Your airline has implemented a voluntary, nonpunitive safety data reporting program, but how do you know how well it works? Are the data it produces actually improving safety at your carrier, and if so, how can you tell? These are just some of the questions Capt. Leja Noe (Mesa), ALPA Training Council chairman and Central Air Safety chair for her pilot group, posed to the attendees at a Tuesday Air Safety Forum workshop titled “Benchmarking—Good and Bad Practices.”

Noe went around the room, asking the more than two dozen workshop participants, “What is the biggest challenge/success with your voluntary safety program?” Following the ensuing discussion, she facilitated a brainstorming session to determine what constitutes a truly effective program, asking attendees to submit written responses addressing four categories—objective measurement, feedback and education, FOQA, and networking for safety—which the group then reviewed.

Attending pilots like Captains Skip Sampson (PDT) and Greg Downs (UAL) talked about the strengths and challenges of their airlines’ programs, and ways they are working to make improvements. Mitch Serber, a former Comair pilot who works with the Aviation Safety Information Analysis and Sharing (ASIAS) system for MITRE Corp., emphasized the complications inherent in developing effective benchmarking tools. He stressed that programs will often vary depending upon the dynamics of the group.

“Are you receiving reports on all related occurrences?” Serber asked the group, noting that a lack of sufficient data can negatively affect the quality of research findings.

Dr. Doug Farrow, the FAA’s acting manager of the Air Carrier Training Systems & Voluntary Safety Programs Branch Program and manager of the Advanced Qualification Program (AQP), agreed, adding, “You need both the numerator and the denominator to clearly evaluate what’s happening.”

Workshop attendees reviewed a preliminary gap analysis tool that Noe and other ALPA safety representatives are developing for evaluating the effectiveness of safety reporting programs. The tool includes a checklist of objectives and the processes for achieving them.

The Forum’s benchmarking workshop activities complemented exercises and continued discussions initiated at ALPA’s Safety Data Sharing Symposium, which the Association presented last February.