Pilot performance assessment in simulators: exploring alternative assessment methods

Abstract

Flight crew performance and competency assessment are daunting tasks requiring expertise and training, and still will not be possible without a certain degree of subjectivity. On the other hand, collecting reliable data on flight crew competencies is at the core of Evidence-based Training, a major modernization of training methodology the industry as a whole has embarked on. Data from assessment informs training departments about where pilots seem to be lacking in proficiency, so those issues can be addressed in initial and recurrent training programmes of airlines. The effectiveness of the training hinges on the quality of the data. Accurate interpretation of the data is crucial for the decisions on training to respond to the true needs of commercial pilots. The industry has made a great effort to develop ways to measure crew performance. The role of Human Factors in incidents and accidents has been known for a long time, and the need to assess and train Human Factors has been identified. In recent years, with the introduction of Evidence Based Training (EBT) there has been a shift in focus from task-based assessment to competence based assessment. This study analysed crew performance in 25 videos from simulator sessions in a high fidelity full flight simulator. A checklist of Desired Flight Crew Performance (DFCP) was used to distinguish between high and low performing crews. Then the performance of selected crews was analysed in detail, using Performance Indicators (PI) as developed in EBT. The findings suggest that while the DFCP method was useful for the classification of high and low performing crews, the PI method provided detailed information for the understanding of underlying factors that affected the performance of the crews. The study also considers the value of using PI to understand and emulate well executed flying and problem solving, to change the focus of training from the study of error and accidents, to training best practices and safe operation.