National Academy of Sciences
National Academy of Engineering
Institute of Medicine
National Research Council
Office of News and Public Information
National Academy of Engineering
Back | Home
News from the National Academies

Read Full Report

Date:  Oct. 28, 2009

Contacts:  Rebecca Alvania, Media Relations Officer

Alison Burnette, Media Relations Assistant

Office of News and Public Information

202-334-2138; e-mail <news@nas.edu>

 

FOR IMMEDIATE RELEASE

 

NASA's Pilot Safety Survey Yields Unreliable Data

 

WASHINGTON -- Data from NASA's National Aviation Operations Monitoring Service (NAOMS) project, a survey administered to pilots in order to track safety-related incidents during air travel, should not be used to measure rates or trends in safety in the National Airspace System, says a new report from the National Research Council.  Deficiencies in several aspects of the survey design make the data gathered unreliable.

 

The NAOMS survey -- jointly managed by NASA and the Battelle Memorial Institute and given to more than 29,000 pilots from April 2001 through December 2004 -- was developed as a means to statistically track rates of safety-related incidents and detect trends in those rates over time.  Pilots answered questions regarding numbers of hours and flights flown, and numbers of incidents observed.  Although the survey employed a number of generally accepted practices -- such as computer-assisted telephone interviews and professionally trained interviewers -- several flaws in the design and implementation of the study affected the usefulness of the data that were gathered.

 

According to the committee that wrote the report, many of the survey questions had deficiencies in structure and wording -- including complex structure, multiple-part questions, and vague phrases -- making them potentially difficult for respondents to digest during a telephone interview.  These problems may have led to varying interpretations and judgments on the part of the pilots, thereby impacting the effectiveness of the survey at accurately measuring safety-related incidents. 

 

Substantial fractions of the reported non-zero counts of events and reported flight legs and hours flown had implausibly large values, the committee said.  Furthermore, many respondents appear to have rounded off event counts or reported flight hours to convenient numbers (e.g., numbers ending in 0 or 5), raising serious concerns regarding the accuracy and reliability of the data.  Some of these problems could have been prevented by spending more time ensuring accuracy during the interview and data-entry stages, the report says.

 

Several choices made by the NAOMS project team caused potential biases in the survey's outcome.  The survey was restricted to pilots with specific flight certifications, which led to an over-representation of pilots of wide-body aircrafts, and an under-representation of those using small aircrafts.  Pilots were also able to opt out of the database used to contact potential respondents, and those with commercial licenses opted out at a much higher rate than others.  If those who opted out had witnessed or experienced very different rates of safety events than those who took the survey, the data would be substantially biased.  The NAOMS team should have investigated and identified potential biases early on to develop alternative survey-design strategies.

 

In addition, the survey grouped highly disparate segments of the aviation industry in order to calculate the overall rate of safety incidents.  As a result, there is no way to link the rate of events to aircraft type, or operating environment, hindering any meaningful analysis of the NAOMS data.

 

Issues regarding anonymity and confidentiality of respondents led NASA to release only redacted datasets to the public.  Several of the measures taken to create the redacted datasets further constrained the usefulness of the data within.  For instance, the time of survey response was grouped into years making annual estimates possible, but making it impossible to track changes in incident rates over shorter time frames, such as seasons.  

 

Issues such as these should have been anticipated and addressed in the design stage of the NAOMS project, the report says.  However, the committee found no evidence that the NAOMS team developed or documented their plans for data analysis, nor did they conduct preliminary analyses as data initially became available in order to identify problems early on and refine their survey methodology.

 

The study was sponsored by NASA.  The National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council are private, nonprofit institutions that provide science, technology, and health policy advice under a congressional charter.  The National Research Council is the principal operating agency of the National Academy of Sciences and the National Academy of Engineering.  A committee roster follows.

 

                                                                                                                                                                                                                

Copies of An Assessment of NASA's National Aviation Operations Monitoring Service are available from the National Academies Press; tel. 202-334-3313 or 1-800-624-6242 or on the Internet at http://www.nap.edu.  Reporters may obtain a copy from the Office of News and Public Information (contacts listed above). 

 

#       #       #

 

[ This news release and report are available at http://national-academies.org ]

 

NATIONAL RESEARCH COUNCIL

Division on Engineering and Physical Sciences

Aeronautics and Space Engineering Board

Board on Mathematical Sciences and Their Applications

 

Committee on NASA's National Aviation Operations Monitoring Services (NAOMS)

Project: An Independent Assessment

 

Vijayan N. Nair (co-chair)

Donald A. Darling Professor of Statistics,

Professor of Industrial and Operations Engineering, and

Chair

Department of Statistics

University of Michigan

Ann Arbor     

 

Clinton V. Oster Jr. (co-chair)  

Professor and Associate Dean of Public and Environmental   Affairs

School of Public and Environmental Affairs

Indiana University

Bloomington

 

David L. Banks

Professor

Institute of Statistics and Decision Sciences

Duke University

Durham, N.C.

 

Robert M. Bell

Principal Member, Technical Staff

Statistics Research Department

AT&T Laboratories Research

Florham Park, N.J.

 

Johnny Blair       

Senior Survey Methodologist and Principal Scientist

Abt Associates Inc.

Bethesda, Md.

 

Anthony J. Broderick

Independent Consultant

Catlett, Va.

 

James W. Danaher

Chief

Operational Factors Division

Office of Aviation Safety

National Transportation Safety Board (retired) 

Washington, D.C.

 

Peter Griffiths

Director General of Civil Aviation

United Kingdom Civil Aviation Authority

London

 

Iain M. Johnstone1

Professor

Department of Statistics

Stanford University

Stanford, Calif.

 

Karen Kafadar

Rudy Professor of Statistics and Physics

Department of Statistics

Indiana University

Bloomington

 

Beth Lyall

Founder, President, and Chief Executive Officer

Research Integrations Inc.

Tempe, Ariz.

 

Donald W. Richardson

President and Chief Executive Officer

Donrich Research Inc.

West Palm Beach, Fla.

 

Thomas B. Sheridan2

Ford Professor of Engineering and Applied Physiology Emeritus

Massachusetts Institute of Technology

Cambridge

 

Alfred T. Spain

Senior Vice President of Operations

JetBlue Airways Corp. (retired)

Forest Hills, N.Y.

 

S. Lynne Stokes

Professor of Statistical Science

Southern Methodist University

Dallas

 

RESEARCH COUNCIL STAFF

 

Paul Jackson

Study Director

 

                                                                        

1 Member, National Academy of Sciences

2 Member, National Academy of Engineering