Technical Review

A Research on Assessing and Improving EPTA (English Proficiency Test for Aviation) using Qualitative Research Method

Jin-Kook Choi*, Cynthia Iris Arias Olivares**
Author Information & Copyright
*Professor, Department Aeronautical Science and Flight Operation, Korea National University of Transportation
**Graduate School of Global Convergence, Korea National University of Transportation
Corresponding Author E-mail : irismayh.ia@gmail.com, Corresponding Author Address : 50 Daehak-ro, Chungju-si, Chungbuk 27469, Korea

© Copyright 2022 The Korean Society for Aviation and Aeronautics. This is an Open-Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Received: May 16, 2022; Revised: May 24, 2022; Accepted: May 31, 2022

Published Online: Jun 30, 2022

ABSTRACT

항공교통관제사와 조종사 사이의 의사소통은 항공기 운항 안전에 있어서 매우 중요한 요소이다. 국제민간항공기구(ICAO)는 민간항공운송산업의 발전과 안전 도모를 위해 조종사 영어자격능력 시험을 의무사항으로 규정하고 있다. 왜냐하면 조종사의 우수한 의사소통능력은 비상상황 또는 비정상상황을 즉각적으로 대응할 수 있는 매우 필수적인 능력들 중 하나이기 때문이다. 대한민국 국토교통부는 2006년부터 ICAO EPTA 시험을 민간항공운송에서 조종사 의무자격시험으로 법적으로 규정하고 산하기관인 교통안전공단을 통해 시험을 주관하고 있다. 본 연구는 EPTA 시험에 응시하는 응시자의 시험에 대한 신뢰성을 증진시키고 시험제도의 발전방안을 모색하고자 하였다. 이를 위해 본 연구는 첫째, EPTA 시험제도와 관련한 이론적 배경을 고찰하였다. 둘째, EPTA 시험제도의 국내에서 정착 및 발전과정을 살펴보았다. 셋째, EPTA 시험제도의 신뢰성증진 및 발전방안 모색을 위해 총 15명의 항공전문가들을 대상으로 질적 연구를 수행하였다. 본론의 질적연구를 통해 연구자들은 국외 ICAO 공식 인증 EPTA 시험결과와 국내 교통안전공단이 주관하는 EPTA 시험결과를 바탕으로 교차응시에 따른 EPTA 등급을 비교 ․ 분석할 수 있었으며 시험응시자들에 대한 인터뷰를 통해 국내 EPTA 시험 발전을 위한 개선방안을 도출하였다.

Keywords: EPTA(조종사 영어자격시험); Operational Safety(운항안전); Emergency Situation(비상상황); Communication(의사소통); Air Traffic Controller(항공교통관제사)

I. INTRODUCTION

The ICAO (International Civil Aviation Organization) has endorsed proficiency in international civil aviation since 2003, considering the causes of aircraft accidents and incidents are mainly related to the inadequate English language skills of the crew (ICAO, 2010). The central government of Republic of Korea has implemented EPTA (English Proficiency Test for Aviation), a requirement that pilots and air traffic controllers communicate in English as of 2006, complying with ICAO standards (Choi and Kim, 2020).

First, this paper briefly reviews the theoretical foundations of EPTA. The theoretical review encompassed such topics as test methods, proficiency levels, test ratings, and qualifications for test raters. Furthermore, this paper's qualitative analysis approach involved collecting data primarily through observation and interview. A survey of aviation experts who took overseas tests in aviation English was conducted. Observations were mostly based on comparing overseas test scores with EPTA scores in Korea. The study results and implications based on the results were discussed to propose ways to improve the validity and processes of EPTA exams in Korea.

II. THEORETICAL BACKGROUNDS

2.1 ICAO Standards and Recommendation
2.1.1 Test Methods

Whether direct or semi-direct testing methods are used, it is important that test-takers are evaluated in their use of language related to routine as well as unexpected or complicated situations as evidence of their level of proficiency.

Both direct and semi-direct tests, if well constructed, can elicit speech samples that may be assessed for proficiency in speaking and listening. Each test method has advantages and disadvantages.

Proficient speakers shall communicate effectively in voice-only (telephone or radiotelephone) and in face-to-face situations. The inflexibility arising from the use of standardized, pre- recorded prompts may result in an important limitation in the scope of evaluation available to semi-direct tests. This limitation may be particularly critical in the ability of the test to assess the full range of abilities covered by the “interactions” descriptors of the ICAO Rating Scale. Role-plays and simulations conducted in this mode may be short, unnatural and restricted to the most routine aspects of language use (ICAO Doc. 9835, 6.2.7.7).

2.1.2 Proficiency Level

The language proficiency of aeroplane, airship, helicopter and powered-lift pilots, flight navigators required to use the radiotelephone aboard an aircraft, air traffic controllers and aeronautical station operators who demonstrate proficiency below the Expert Level (Level 6) should be formally evaluated at intervals in accordance with an individual’s demonstrated proficiency level, as follows:

  1. those demonstrating language proficiency at the Operational Level (Level 4) should be evaluated at least once every three years.

  2. those demonstrating language proficiency at the Extended Level (Level 5) should be evaluated at least once every six years.

Formal evaluation is not required for applicants who demonstrate expert language proficiency, for example, native and very proficient non-native speakers with a dialect or accent intelligible to the international aeronautical community.

To fulfill licensing requirements, rating should be carried out by a minimum of two raters. A third expert rater should be consulted in the case of divergent scores. Ideally, an aviation language test will have two primary raters — one language expert and one operational expert — and a third rater who can resolve differences between the two primary raters’ opinions (ICAO Doc. 9835).

2.1.3 Test Rating

Whether rating is conducted “live” during the assessment, or after the test using recordings of the test performance, the rating process should be documented. The test rating process should be documented, and the documentation should include instructions on the extent and nature of evidence that raters should collect (ICAO Doc. 9835). The score reporting process should be documented and scores maintained for the duration of the license. All proficiency tests of speaking ability involving interaction between the test-taker and interlocutor during the test should be recorded on audio or video media.

According to licensing requirements, rating should be carried out by a minimum of two raters. A third expert rater should be consulted in the case of divergent scores. An aviation language test will have two primary raters — one language expert and one operational expert and a third rater who can resolve differences between the two primary raters’ opinions. Initial and recurrent rater training should be documented. The rater training records should be maintained and audits of raters should be conducted and documented periodically (ICAO Doc. 9835).

2.1.4 Test Rater’s Qualification

Raters should demonstrate language proficiency of at least ICAO Extended Level 5 in the language to be tested. If the test is designed to assess ICAO Level 6 proficiency, raters should demonstrate language proficiency at ICAO Expert Level 6. Interlocutors should demonstrate language proficiency of at least ICAO Extended Level 5 in the language to be tested and proficiency at Expert Level 6 if the test is designed to assess ICAO Level 6 proficiency (ICAO Doc 9835).

In the case of semi-direct test prompts (which are pre-scripted and pre-recorded), there should be adequate versions to meet the needs of the population to be tested with respect to its size and diversity. It is not practical to prescribe the number of versions or test prompts required for any specific test situation. The determination of what is adequate in any situation is dependent on specific circumstances.

It is common in large testing initiatives for a testing service to use a version of a test only once before retiring it. In other cases, a testing service develops a number of versions, then recycles them randomly. Test-takers may then generally know the sorts of questions and prompts they will encounter during a test, but will be unable to predict the specific questions and prompts they will encounter during a particular testing interaction.

One security measure that testing organizations may take is to always include at least one completely new prompt or question in every version. A pattern of test-takers achieving high scores on most or all test prompts or questions, but failing the new prompt, may indicate a breach in test security (ICAO Doc 9835).

It is acceptable that a test contains a scripted task in which phraseology is included in a prompt, but the test should not be designed to assess phraseology. An aviation language proficiency test has different aims than a phraseology test. While an aviation language test can include some phraseology as prompts or scene setters, the purpose of the test is to assess plain language proficiency in an operational aviation context.

2.1.5 Standard Phraseology

It is acceptable that a test of plain language in a work-related context could contain a scripted test task or a prompt in which standardized phraseology is included. The test task may be used as a warm-up or as a means of setting a radiotelephony context in which to elicit plain language responses from the test-taker. If phraseology is included in a test prompt, care should be taken that it is used appropriately and that it is consistent with ICAO standardized phraseology.

The compliance with ICAO standardized phraseology is not fully harmonized on a worldwide basis. States publish differences with respect to ICAO Standards. Additionally, users, particularly expert speakers of a language, for all sorts of respectable reasons such as pressure of work, and less respectable reasons such as carelessness and insensitivity, fail to adhere to ICAO standardized phraseology, thereby creating possibilities for misunderstanding in a busy international environment.

2.2 The Progress of EPTA
2.2.1 Major Development of EPTA

As part of the 2006 Aviation Safety Act, the MOLIT (Ministry of Land, Infrastructure, and Transportation) required pilots and air traffic controllers operating between at least two countries among aeronautical communications to obtain a certificates of proficiency in aviation English (MOLIT, 2021). Table 1 displays EPTA's progress. MOLIT began testing pilots and air traffic controllers with EPTA in 2006 using two private testing agencies, G-TELP (General Test of English Language Proficiency) and IAES (International Aviation English Service).

Table 1. The progress of EPTA
Year Changes
2006 - Implementation of EPTA by G-TELP and IAES
2016 - Plan for Improving EPTA with review and analysis
2018 - Preparation Guide for Aviation English Proficiency Test published with sample sets for pilots and air traffic controllers
2019 - Unification of enforcement agency to Korea Transportation Safety Authority (KOTSA)
- Implementation of CBT (computer based test) EPTA for Level 4 and 5
- Interview Teat for Level 6
- Integrated evaluation of listening and speaking skills
2021 - Standard Handbook for EPTA published: Preparation Guide for English Proficiency Test for Aviation
2021 - Research Project of EPTA Improvement by the Ministry of Land, Infrastructure and Transport
Download Excel Table
2.2.2 Revised EPTA

To address the problems that arise in the operation of the aircraft English certification system in 2006, MOLIT reviewed and analyzed the EPTA system. As part of the improvement plan, EPTA will be improved through increased communications with ATC (air traffic control), implementing CBT (computer-based testing) for levels 4 and 5, integrating listening and speaking competencies, and continuing interview procedures for level 6 (MOLIT, 2018).

Following a review of the test system and communications with air traffic controllers, pilots, and consultants, MOLIT has been conducting a CBT EPTA since 2019 (MOLIT, 2021).

2.3 Qualitative Research Method

Qualitative research refers to a method of inquiry in which the researcher, acting as data collection instrument, seeks to answer questions about how or why a particular phenomenon occurs (Rahman, 2016). Questions regarding of what a phenomenon is comprised may also guide qualitative research.

The most fundamental assumption underlying qualitative research is that reality is something socially constructed on an individual basis. Varied methods of qualitative research exist (Leech et al., 2011).

The most fundamental assumption underlying qualitative research is that reality is something socially constructed on an individual basis. Varied methods of qualitative research exist (Leech et al., 2011). Examples of qualitative methods employed in academic research include grounded theory, phenomenology, ethnography, and qualitative description. Each method has its own assumptions and purposes and an appropriate method is chosen based on the research question. For example, a researcher investigating the process involved in the occurrence of a phenomenon would likely choose grounded theory, while a researcher who are interested in the meaning of the phenomenon would utilize phenomenology. Regardless of method, participants are purposefully enrolled based on their familiarity with the phenomenon (Donath et al., 2011).

Data are generally collected via one or a combination of three mechanisms: interviews, observation, or theory review. Data are analyzed inductively via specific, rigorous techniques and then organized in a manner which best answers the research question (Anyan, 2013; Wang, 2006). The objective of qualitative research is not the accumulation of information, but the growth of understanding about phenomena of concern to EPTA exam.

The specialists chosen by authors were the part of the core evaluation team and participated in all aspects of the qualitative research design, implementation, data analysis and dissemination of EPTA results. On the major technical and implementation issues, the specialists were expected to work independently and to engage with a wide-range of stakeholders at ICAO, airlines and a central government agency in the field sites in EPTA cross-sectional project locations. The qualitative research activities would take place in selected research locations where the specialists had taken an EPTA exam.

Before performing qualitative analysis, such 15 experts in Table 2 including a few senior captains were chosen and assigned to take overseas exams. Researchers in this paper analyzed overseas aviation English scores submitted by the experts including the score obtained by EPTA exam in Republic of Korea.

Table 2. The progress of EPTA
Overseas EPTA Certification upon Overseas Aviation English
Experts (Applicants) Location Test result EPTA score Improve Maintain Drop Evaluation
1 COO Instructor U Level 4 Level 3 0 2 1 Easier than EPTA
2 HOO Instructor Level 4 Level 4
3 POO APL Level 4 Level 4
4 LOO Instructor C Level 6 Level 5 0 1 2 Easier than EPTA
5 KOO APL Level 6 Level 4
6 LOO F/O Level 6 Level 4
7 BOO Captain C’ Level 4 Level 4 1 2 More difficult than EPTA
8 ROO Captain Level 4 Level 4
9 YOO Captain Level 4 Level 5
10 COO APL E Level 4 Level 4 1 1 1 Almost the same level
11 AOO APL Level 4 Level 3
12 WOO APL Level 4 Level 5
13 JOO Captain A Level 6 Level 4 0 0 3 Easier than EPTA
14 SOO Instructor Level 6 Level 4
15 FOO Instructor Level 6 Level 4
Download Excel Table

Test results in Table 2 means that test results of the experts who already have taken overseas aviation English tests were converted into EPTA equivalent score level.

2.4 Qualitative Analysis Result
2.4.1 The Advantages of EPTA from the Expert Interviews

From the interviews with subject matter experts, it appears that CBT EPTA has benefits and areas that need improvement. Table 2 shows the advantages and areas for improvement. As a result of the test being operated by a national organization, the test has been improved in many aspects such as fairness and public confidence. There is much less variation depending on the interviewers since recorded voices remain the same.

A set of sample questions containing increased exercises is also helpful for practicing the test. The level of difficulty is adjusted appropriately depending on the job performance. In addition, MOLIT's Preparation Guide for Aviation English Proficiency Test and Standard Handbook for EPTA have proved extremely useful in preparation. As a result of increased communication among ATC members, the situation development is realistic and valid, and standardized terminology is used more often.

2.4.2 The Areas to be Improved on EPTA from the Expert Interviews

From the experts' interviews, it is clear that the areas needing improvement are listed in Table 3. The number of testing facilities needs to be increased, and promotion should be improved, including test guides, web pages, and scoring criteria. It is necessary to improve tests such as adding machine voices, standardizing phraseologies, allowing general English to be used in emergency situations, and allowing level 6 Interviews to be administered to all levels of evaluation. as in Table 4. Improvement in interaction is necessary.

Table 3. The advantages of CBT EPTA
Type Contents
Test improved Fairness and public confidence
Less difference depending on the interviewer
The level of difficulty is appropriately adjusted to fit for the job performance
The increased exercise sample questions
Preparation Guide for Aviation English
Proficiency Test and Standard Handbook for EPTA published
Safety - Safety is improved
- Helpful for flight operation
- ATC Communication is increased
- The use of standardized phraseology increased
- Situation development is realistic and valid
- Satisfied with CBT EPTA in general
Download Excel Table
Table 4. The needs to be improved
Type Contents
Test facilities Need to increase the number of test places
Promotion The need to change test guide and web page
Needs to be actively promoted, such as scoring criteria.
Test Need of improvement in machine voices
The need of improvement on standardized phraseology
Need to improve communication (including general English) in emergency situations
Level 6 Interview should allow all levels of evaluation
The improvement on interaction is needed
Sample questions The increase of sample questions and answer are needed including recorded voices
Education Need educational institution on Aviation English with certification and public trust
Download Excel Table

It is also critical to increase the number of sample questions and answers, with recorded voices, and to establish an educational institution focused on aviation English with certification and public trust for test takers who struggle with tests.

III. CONCLUSION

The purpose of this study is to review improvements for EPTA and the needs of EPTA. In the process of interviewing experts for an effective use of EPTA, we came to the following conclusions regarding EPTA's needs and improvement measures.

Firstly, the subject matter expert agrees that CBT EPTA has been improved and has been helpful in improving aviation safety. The CBT EPTA has been changed to reflect the suggestions of the line and the experts in order to achieve the safety objective. Secondly, we can increase the degree of fairness and public trust because the task is appropriately adjusted for the job performance and the test is administered by a national organization. Thirdly, increasing ATC-related questions will enhance ATC communication. In addition, the participants appreciated the fact that MOLIT has published a Preparation Guide for Aviation English Proficiency Test and a Standard Handbook for EPTA to help with preparation. Lastly, certain improvements need to be implemented, such as increasing the test facilities, improving the test, promoting the web pages, and expanding the sample questions. Education institutions need to be added for Aviation English as well.

EPTA was found to be improved and helpful based on the qualitative analysis. The results of this study are consistent with the results of quantitative studies in previous studies that found EPTA had improved as a result of the survey (Lee and Choi, 2019). If ATC and aviation English capabilities can be integrated, this will improve safety and communication skills as well as continue the improvements on CBT EPTA as safety oriented ICAO member state.

The study is limited in the sense that it is a qualitative study based on interviews with a small group of subject matter experts because there are few aviation specialists on EPTA. It would be more fruitful to increase the number of samples for future research. It is nice to see that the result shows that the progress of EPTA is improved. We hope that the continuous improvement of EPTA will lead to safer air traffic control communication to avoid aviation accidents.

Acknowledgment

This was supported by Korea National University of Transportation in 2021.

References

1.

ICAO, “ICAO Doc 9835. Manual on the Implementation of ICAO Language Proficiency Requirements”, 2010, pp1.1-5.6.

2.

Choi, K. H., and Kim, S. Y., “Face validity analysis of revised English Proficiency Test for Aviation (EPTA)”, Journal of the Korean Society for Aviation and Aeronautics, 28(4), 2020, pp.12-20.

3.

MOLIT, “Article 45 Certification of Proficiency in Aviation English, Ministry of Land, Infrastructure and Transport”, 2021, https://www.law.go.kr/%EB%B2%95%EB%A0%B9/%ED%95%AD%EA%B3%B5%EC%95%88%EC%A0%84%EB%B2%95

4.

Ministry of Land, Infrastructure and Transport, “Preparation Guide for Aviation English Proficiency Test”, 2018, pp.2-8.

5.

Ministry of Land, Infrastructure and Transport, “Preparation Guide for EPTA”, 2021, pp.2-14.

6.

Rahman, S., “The advantages and disadvantages of using qualitative and quantitative approaches and methods in language “Testing and Assessment” research: A literature review”, Journal of Education and Learning, 6(1), 2016, pp.102-104.

7.

Leech, N. L., and Onwuegbuzie, A. J., “Beyond constant comparison qualitative data analysis: Using Nvivo”, School Psychology Quarterly, 26, 2011, pp.70-84.

8.

Donath, C., Winkler, A., Graessel, E., and Luttenberger, K., “Day care for dementia patients from a family caregiver׳s point of view: A questionnaire study on expected quality and predictors of utilisation – Part II”, BMC Health Services Research, 11, 2011, pp.1-7.

9.

Anyan, F., “The influence of power shifts in data collection and analysis stages: A focus on qualitative research interview”, Qualitative Report, 18(36), 2013, pp.1-9.

10.

Wang, J., “Questions and the exercise of power”, Discourse and Society, 17, 2006, pp.529-548.

11.

Lee, J. S., and Choi, J. K, “Satisfaction study of English proficiency test for aviation”, Journal of the Korean Society for Aviation and Aeronautics, 27(4), 2019, pp.114-118.