Developing Airport Safety Performance Indicators and Index:

The International Civil Aviation Organization (ICAO) established ANNEX19. According to these documents, States and service providers including airports, airlines, etc. are required to comply with Standards And Recommended Practices (SARPS) (ICAO, 2016). This document emphasizes the importance of protecting safety data and information. Those are the fundamentals for the development of indicators and indices. Furthermore, it recommended that the states select quantifiable indicators that take into account their relevance to the service providers, including airports and airlines’ safety objectives. Member countries and service providers refer to these documents as the most influential guidelines. It is also presented in the DOC9859 (Safety Management Manual) examples of indicators that help recognize warnings from fields by setting triggers based on standard deviations (ICAO, 2018).

There has been a guide on Airport Performance Measures (APM) published by ACI (Airport Council International) since 2012 and this guide is available to airports worldwide. The guidebook provides six key performance areas, including core competency, safety, security, and service quality. There are several safety-related indicators, including runway accidents, excursions and incursions, collisions with birds, injuries, and lost work. Based on the type of airport, each indicator is classified according to its applicability (ACI, 2012). The indicator was defined with related parameters and listed the links between the driver and the indicator.

There is also an indicator system that is used by the National Union. The European Aviation Safety Agency (EASA) is composed of members from 31 countries throughout Europe, and it is responsible for certification, standardization, investigation, and monitoring. The European Plan for Aviation Safety (EPAS) includes a safety performance indicator. The major safety indicators include runway excursion, airborne collisions, controlled flight into terrain (CFIT), loss of control during flight, runway incursion, and fire (EASA, 2022).

The development of indicators was not only done by the international organizations described above but also according to the circumstances within the country. Airport Cooperative Research Program (ACRP), an institute specializing in airports supported by the US FAA, presented indicators for the airport sector, such as airside accidents, minor accidents, work-related casualties, runway invasions, and wild animal accidents in the Safety Management Systems guidebook for airports. Another report published, ‘Resource Guide to Airport Performance,’ included 29 key indicators, 132 key indicators, and 679 other indicators. Safety-related indicators include two core, ten major, and 31 other indicators (FAA ACRP, 2011).

The Belgian CAA has, like several other countries, established an aviation safety program in accordance with ICAO DOC9859 and EASA’s safety policy. According to the European Coordination Center for Accident and Incident Reporting Systems (BCAA, 2018), 36 detailed safety performance indicators were developed for accidents and incidents, aerodromes, navigation services, operations, technology, and other prominent fields.

ICAO GASP, Doc 9859, and EU EPAS have been implemented by the Finnish Transport and Communications Agency (TRAFICOM) through its Finnish National Aviation Safety Program (FASP). As stated in TRAFICOM (2018), the EU-level safety objectives must be considered in conjunction with the specific setting of acceptable levels of safety and scope at the national level. Airports, airlines, and ground handlers are provided with safety performance indicators through the FASP program. The safety performance indicators for airport operators are further subdivided into SMS performance, runway incursion/excursion, runway status defects, visibility, air traffic control flight, accidents, a lack of information about obstacles, a ground collision, foreign matter in moving areas, CFIT, as well as neglect of apron area supervision.

In 2008, the Singapore Civil Aviation Authority (CAAS) developed safety indicators for airport operators and airlines as part of the State safety program. Indicators related to airports include aircraft-related accidents, incidents, wildlife collisions, runway intrusions/deviations, and ground collision prevention system recommendations (CAAS, 2018).

In order to measure safety performance, South Korea set up a national aviation safety program through its civil aviation authority, MOLIT (Ministry of Land, Infrastructure and Transport). It is also through this CAA that the state’s aviation safety program was established. Under the theme of air service, each area is divided into sub-parts, such as airport facilities, navigation facilities, air traffic control and air operations, and its indicators are provided. As provided in ICAO Doc9859, the 3-year standard deviation was used as the alert method. There are several indicators related to the airport sector, including ground conflict/accidents, aircraft ground collisions, airport aerodrome facility malfunctions, and navigation safety facility failures. Incheon International Airport Corporation, the primary gate in South Korea, has established 25 safety indicators in support of national indicators since 2013 (IIAC, 2016; Table 1).

ICAO presented representative examples of these indicator systems, such as setting triggers for indicators related to the aviation industry of the state, but these examples were merely indicative. The reason may be the fact that international organizations are responsible for covering the entire aviation industry associated with all states. It is unnecessary to cover the details of service providers such as airports.

In contrast, the ACI did not present any criteria from its guidebook but rather included a number of customized indicators. The ACI and FAA ACRP’s guides present indicators based on their importance as opposed to ICAO’s list of indicators regardless of their relative weight. In addition to their differences, they share the following similarities:

?Various organizations and states have developed airport safety indicators that deal with the same or similar issues. This is due to the similarity between the airport infrastructure’s functional, topographical, and operational characteristics. When safety performance is measured using the same indicators, this provides a positive perspective on comparing safety performance.

?Risk levels, which represent severity and likelihood of occurrence, are commonly used in airport safety indicators. It helps to determine the type and timing of resource allocation according to the risk level. It should be noted, however, that the level of risk is only sometimes constant and reliable due to the ease with which this assessment can be altered. It is important to note that although the severity of a potential occurrence is relatively fixed, the likelihood can vary depending on the location, the time (day/night, weekday/holiday), the season, and the weather. Therefore, it is important to note that variability in likelihood can also distort risk levels and prevent appropriate decision-making.

?Most of the indicators related to airport safety consist of lagging indicators rather than leading indicators. Lagging and Leading indicators can be classified as measured symptoms in advance or events on past results (Lingard et al., 2017). The lagging indicator is outcome-oriented. On the other hand, the leading indicators are process-oriented (ICAO, 2018). Hallowell et al. (2020) studied leading indicators, including pre-task safety meetings, which are able to suppress accidents and have an inverse correlation with injury rates. Given the current indicators system in the airport industry, it cannot be considered the proactive way (Oster Jr et al., 2013). Therefore, leading indicators were extended in this study.

?The overall direction of indicators cannot be measured through the current system due to the mutually independent characteristics of each indicator. Although this fact is generated naturally, it may not be persuasive information for top management because it is difficult to make a comprehensive decision from several individual indicators. The relative weight among indicators is essential in prioritizing decision-making for limited resources. Moreover, there are only a few previous studies considering integrated perspectives.

Given the commonalities described above, this study must include a more innovative and integrated indicator system for sustainable safety management. In order to accomplish this, the study was intended to develop airport workers’ psychology as a component of the indicator. Safety accidents often occur as a result of workers’ minds manifesting in behaviors such as violations and negligence (Reason, 1990). This study attempts to confirm the effects of safety knowledge, attitude, consciousness, climate, and behavioral factors on safety-related behavior and to integrate them with existing indicators.

Safety attitudes have been studied early as an essential predictor of climate (Zohar, 1980). It is also defined as individuals’ belief and confidence about the environment and technology that enable them to continue working safely in their daily work (Wood and Bandura, 1989). Ajzen (2005) divided safety attitudes into favorable and unfavorable ones, saying that favorable attitudes are future-oriented responsibility and constructive beliefs, while unfavorable attitudes are destructive beliefs and characteristics they want to avoid. Fugas (2012) also differentiated between constructive and destructive beliefs regarding safety and argued that positive beliefs should be strengthened while negative beliefs should be reduced. According to Brown et al. (2000), the negative factor affecting worker safety behavior is a cavalier attitude, while the positive factor is safety efficacy. In this study, safety attitudes were divided into two categories: safety efficacy and cavalier attitude.

Safety knowledge shapes safety behavior by ensuring that workers are aware of safety operating procedures and have received appropriate training and instructions (Hofmann et al., 1995). According to Piers et al. (2009), safety knowledge involves understanding the risks involved in the operation and making sure it is distributed and utilized properly. Safety knowledge is divided into explicit and tacit knowledge. Explicit knowledge refers to knowledge that is easy for individuals to convey to organizations because it is official and can be documented. On the other hand, tacit knowledge is generally accumulated by individuals from experience and has characteristics that are variable and difficult to share depending on the situation (Nonaka and Takeuchi, 2007). Safety consciousness refers to “an individual’s awareness of safety issues” (Barling et al., 2002) and operates at both a cognitive and behavioral level (Koster et al., 2011). It is also a factor that significantly impacts safety reporting practices and the incidence of accidents (Barling et al., 2002; Kelloway et al., 2006).

The concept of safety climate is more collective than the concept of safety consciousness. Generally, it refers to the belief that safety should be at the center of organizational policies, procedures, and goals (Griffin and Neal, 2000; Hofmann and Stetzer, 1996; Neal and Griffin, 2002). It is important to note that safety behavior consists of actions taken to protect from danger based on safety knowledge and skills that are necessary for the performance of one’s job (Gressgard, 2014). Several previous studies have identified two types of safety behavior: compliance and participation.

Fig. 1. Overview of the research process

Download Original Figure

Hon et al. (2014) classified safety behavior into safety compliance and active participation in obligations. Jiang and Probst (2015) also said that safety compliance is a behavior required for safety (e.g., safety rules to wear protective gear before work). Neal and Griffin (2000) stated that it includes activities and training to actively participate in safety issues in advance and speak to managers. Borman and Motowidlo (1993) classified safety behavior into safety conformity behavior, a crucial activity to maintain safety, and safety expectation behavior that helps create the surrounding environment but does not affect safety directly.

As part of ISO22301 (2019), an international standard for business continuity planning, safety indicators, and targets should be considered in order to meet the legal requirements of both internal and external stakeholders as well as the government. Indicators of this type apply to all other areas, including this study. Table 2 shows reviewed indicators from international organizations, national guidelines, and previous studies. Academic journals and reports find psychological indicators, including safety awareness and attitude, etc. 25 indicators from Incheon International Airport were also reviewed for candidates.

Based on the above statements, purposive sampling was used to select experienced experts from the airport, airlines, and ground handlers to participate in the survey. The following are details of each survey round. After repeated surveys of experts, the statistical average of the surveys is converged into a group opinion (Rowe and Wright, 2001), and the results are finally summarized after confirming consensus (Baker, Lovell, and Harris, 2006).

3.2.2 Selection of the expert panel (BY Semi-structured interviews)

The first round consisted of experts selected from the same categories as those designated for the Delphi study. The Delphi research panel critically reviewed the findings to gain a deeper understanding of the consensus reached among the researchers and to create a framework for future research efforts in the field (Nicolaidou, et al., 2021).

The criteria for the expert panel were established by applying the Delphi experience criteria of Tran, Trung et al. (2020) in order to select experts who had at least five years of experience as airport safety instructors. The first round of the selection process was conducted on airport experts with an average of six years of experience. An open-ended question type was used to eliminate candidates who were not relevant. As a result of this process, indicators that were not relevant or overlapping were identified and removed.

The second round was conducted with 18 experts from the airport, airlines, and ground handlers with an average working experience of 17 years. A 5-level Likert questionnaire was devised to select from three perspectives: necessity, importance, and urgency. After evaluating the importance of each metric, we set up a comment field for experts to comment on modifying, deleting, or merging metrics.

During the third (last) round, 27 experts from airports, airlines, and ground operators were interviewed, with an average of 11 years of experience. When the survey was conducted, we were experiencing the height of the COVID-19 virus pandemic. Some of our experts became infected with the virus, which resulted in a low return rate. Several experts in airport safety were consulted to reach conclusions about the importance of airport safety indicators.

The 25 indicators were assessed using a questionnaire based on a Likert 5-level scale in order to determine their suitability and expert confidence. The final ten indicators could be selected after calculating the mean and standard deviation of relevance and expert confidence values (3.89 and 4.01, respectively). The result was the confirmation of the existing 7 indicators, as well as 18 new indicators (Fig. 2, Table 3-5).

Fig. 2. Flowchart of the Delphi process for selecting indicators

Download Original Figure

Table 3. Results of the 1^st round Delphi study

No	Item	% of Consensus	C V R	C G D
1	Aircraft accidents	96.7%	0.89	0.00
2	Aircraft airspace intrusion	88.9%	0.89	0.50
3	Vehicle CAA invasion	86.7%	0.89	0.50
4	Aircraft abandonment	88.9%	1.00	0.50
5	Aircraft near miss	86.7%	0.78	0.50
6	Aircraft-aircraft collision	94.4%	0.89	0.00
7	Aircraft-vehicle collision	91.1%	0.89	0.50
8	Aircraft-facility collision	86.7%	0.67	0.50
9	Aircraft-person collision	81.1%	0.44	0.50
10	Vehicle-vehicle collision	73.3%	0.44	0.50
11	Vehicle-person collision	77.8%	0.44	0.00
12	Vehicle-facility collision	68.9%	0.44	0.50
13	Aircraft-aircraft close encounters	84.4%	0.44	0.50
14	Aircraft-vehicle close encounter	82.2%	0.44	0.50
15	Aircraft oil leak	66.7%	0.44	0.50
16	Dangerous goods incident	77.8%	0.22	0.50
17	Bird strike	78.9%	0.44	0.50
18	Reporting mooring control dissatisfaction	65.6%	0.22	0.50
19	FOD collection	75.6%	0.44	0.50
20	Hazardous find reporting	78.9%	0.11	0.50
21	SMS committee	74.4%	0.44	0.50
22	SMS working committee	75.6%	0.44	0.50
23	Safety campaign	63.3%	0.44	0.50
24	Airport operation certification	72.2%	0.22	1.00
25	Navigation safety oversight	71.1%	0.00	1.00

Download Excel Table

Table 4. Results of the 2^nd round Delphi study

No	Item	% of Consensus	C V R	C G D
1	Injuries in transit areas	70.0%	0.110	0.5
2	Number of wake/propeller maintenance/engine aspiration incidents	76.7%	0.440	0.38
3	Wildlife collisions	65.6%	0.220	0.38
4	Wildlife sightings (reports)	95.6%	0.670	0.5
5	Number of safety reports collected	71.1%	0.110	0.5
6	Number of safety reporters	64.4%	0.000	0.5
7	Number of safety training programs	65.6%	0.220	0.38
8	Number of safety audits	63.3%	0.330	0
9	Number of measures implemented according to safety audit points (rate)	68.9%	0.110	0.5
10	Response rate after receiving risk reports	75.6%	0.330	0.5
11	Number of construction works in mobile areas	56.7%	0.560	0.5
12	Number of accidents caused by construction work in mobile areas	65.6%	0.110	0.5
13	Ratio of accidents to the number of construction activities	62.2%	0.330	0.00
14	Number of unauthorized works in airside areas	81.1%	0.440	0.00
15	Employee accidents and injuries	70.0%	0.220	0.50
16	Lost work time	55.6%	0.780	0.00
17	Rescue and firefighting	71.1%	0.220	0.50
18	Number of fatalities in mobile areas	86.7%	0.670	0.38
19	Number of driving violations in the movement zone	72.2%	0.110	0.50
20	Aircraft damage during the aircraft cycle	88.9%	0.780	0.50
21	Aircraft damage while in transit	90.0%	1.000	0.50
22	Equipment damage	65.6%	0.110	0.50
23	Workers’ compensation claims	75.2%	0.440	0.00
24	Airport medical emergency response	70.0%	0.000	0.50
25	Number of environmental violations	64.4%	0.110	0.88
26	Emission limits exceeded	77.8%	0.560	0.38
27	Number of crimes at the airport (violence, theft)	60.0%	0.440	0.00
28	Number of safety seminar participants	60.0%	0.670	0.00
29	Instrumental aircraft damage	81.1%	0.780	0.00
30	Number of injuries/accidents to passengers or other organizations	65.6%	0.000	0.50
31	Fire/smoke incidence rate	68.9%	0.000	0.50
32	Fires in transit areas	92.2%	0.440	0.38
33	Industrial Accident Management	74.4%	0.000	0.50
34	Disaster losses (manpower)	73.3%	0.000	0.50
35	User accidents (passengers, etc.)	64.4%	0.220	0.38
36	Safety complaints	86.7%	0.670	0.38
37	Disaster safety factors	67.8%	0.000	0.50
38	Internal safety culture	88.9%	0.440	0.38
39	External safety culture index	84.4%	0.440	0.38
40	Aviation safety education (SMS) elements	70.0%	0.110	0.50
41	Safety and health education factor	62.2%	0.110	0.50
42	Working days lost due to occupational illness or injury per airport employee	77.8%	0.440	0.50
43	Types of FODs	75.6%	0.440	0.38
44	Number of violations	71.1%	0.000	0.50
45	Failures/damages found	64.4%	0.110	0.50
46	Injuries per workforce size	60.0%	0.560	0.38
47	Safety awareness	83.3%	0.440	0.38
48	Safety attitude	84.4%	0.440	0.38
49	Safety knowledge	82.2%	0.440	0.38
50	Number of flights	68.9%	0.220	0.50
51	Equipment aging	77.4%	0.440	0.00
52	Number of vehicles/equipment	75.3%	0.440	0.38
53	Noise	62.2%	0.330	0.75

Download Excel Table

Table 5. Results of the 3^rd round Delphi study

No	Item	% of Consensus	C V R	C G D
1	Safety awareness	85.4%	0.769	0.50
2	Number of fatalities in airside areas	80.0%	0.670	0.38
3	Number of safety seminar participants	80.8%	0.667	0.00
4	Aircraft damage while in transit	76.3%	0.615	0.50
5	Number of vehicles/equipment	76.2%	0.560	0.50
6	Aircraft damage during the aircraft cycle	75.4%	0.538	0.50
7	Customer safety complaints	85.4%	0.462	0.75
8	Fires in airsideareas	86.9%	0.462	0.88
9	Internal safety culture	76.2%	0.440	0.38
10	External safety culture index	76.5%	0.440	0.38
11	Number of safety reports collected	81.5%	0.385	0.50
12	Types of FOD	80.8%	0.385	0.50
13	Instrumental aircraft damage	75.6%	0.385	1.00
14	Safety knowledge	75.5%	0.385	1.00
15	Safety behavior	76.2%	0.385	1.00
16	Injuries per workforce size	85.4%	0.385	0.38
17	Number of wake/propeller maintenance/engine aspiration incidents	80.0%	0.385	0.38
18	Equipment aging	80.8%	0.385	0.00
19	Working days lost due to occupational illness or injury per airport employee	73.2%	0.110	0.88
20	Lost work time	71.1%	0.110	0.88
21	Number of construction works in airside areas	70.1%	0.110	0.88
22	Emission limits exceeded	65.4%	0.110	0.88
23	Number of unauthorized works in airside areas	60.4%	0.110	0.88
24	Workers’ compensation claims	55.4%	0.110	0.88
25	Number of crimes at the airport (violence, theft)	52.5%	0.110	0.88

Download Excel Table

Lastly, 51 indicators were reclassified and reorganized into four new categories (human factors, accidents/events, air traffic control/airside safety, and air navigation) by adding seven indicators related to air navigation facilities provided by the South Korean State Aviation Safety Program. The following is a classification of indicators from CAT I to CAT IV (Fig. 3).

Fig. 3. Overview of the research process

Download Original Figure

ICAO introduced the approach of applying a standard deviation to the average value of the past three years in selecting alert levels. This study also applied the alert level to set trigger systems. Moreover, the risk level was divided into four levels, and each risk level was expressed as good/moderate/caution/danger, respectively.

There is a need to monitor these indicators, however, the importance of each indicator cannot be rated equally. However, it is difficult to find previous studies that have examined the relative importance of the indicators and integrated them into a composite index. Consequently, an index that sets and integrates the weights of the indicators has been developed. There are two benefits to this approach.

Firstly, an index calculation provides an intuitive understanding of the trend resulting from each indicator incorporated. Using this method, the overall performance of the organization is identified and decisions can be made more effectively.

Secondly, it is possible to compare results over time and with other entities using the same indicators. Comparisons can easily lead to an overall assessment and the setting of short-term (e.g., annual) safety goals.

As a result, resources are used more efficiently in operations. An analytical hierarchy process (AHP) was applied in order to accomplish this. According to Soltanifar and Lofti (2011), AHP uses a weighted voting ranking approach in order to rank criteria and sub-criteria. To perform AHP, it was necessary to classify the 51 indicators for clustering. The safety function is responsible for completing this task in its own area. Category I (CAT I) was classified according to airside safety, air traffic control, and air navigation. It is the same as dividing areas in airport organizations. It is important to note that Category II consists of seven indicators covering areas that constitute Category I - human factors, accidents/events, facilities/ equipment, operational environment, air navigation functions, and communication. A total of 29 indicators are included in the CAT III group, and these indicators can be used to explain CAT II in greater detail. CAT IV follows CAT III and is further separated in the same way as above. As a result of the Delphi survey, 51 final sub-indicators were identified for each area of the CAT I. Some indicators, including the air navigation area, did not have CAT IV indicators in this process. The reason for this is that it has already been sufficiently subdivided and explained in detail.

The survey involved 16 experts from airlines, ground handlers, and the airport in order to determine weights based on AHP. A total of ten answer sheets were evaluated, 16 were used, and six with a low consistency ratio (CR) were removed from the analysis. Each weight of CAT IV indicators was calculated as CAT I weight × CAT II weight × CAT III weight × CAT IV average value (%). The final weighted score of each indicator was calculated as the CAT IV indicator weight × 100. There was no significant effect of some decimal points being corrected in the CAT IV weight results.

Indicators with weights were simulated using data from 2020 to 2022 at Incheon International Airport. There are four warning levels based on the indicator’s 3-year average within the standard deviation range. The details are as follows:

·Green: Good, when the measurement year score is within 1st of the 3-year average

·Yellow: Moderate, when the measurement year score falls within 1sd ~ 2sd of the 3-year average.

·Orange: Caution, when the score in the measurement year falls within 2sd ~ 3sd of the 3-year average.

·Red: Danger, when the measurement year score exceeds 3sd of the 3-year average. Table 3 below shows the indicator, alert level, and index results from 2020 to 2022 (Table 6).