FAA predictive model for accident risk incorporating instrument rating status and total time

[DXB]

I stumbled into this 2015 FAA publication on the interwebs and found it rather fascinating:

https://www.faa.gov/data_research/research/med_humanfacs/oamtechreports/2010s/media/201503.pdf?fbclid=IwAR3QrHgobbc5ZS0w3epndwwdiZzvL13bfU1MUGDq2b1_N8FxCr-0ozqBAxM

My takeaways:

- It certainly highlights a fatal flaw in how R. Craig calculated 50-350 hours pilot total time as the period of highest accident risk in GA in his 2001 book The Killing Zone .  

- It calls into question the premise that instrument-rated GA pilots are less accident-prone overall.

My caveats:

- It is a sophisticated analysis that extrapolates from complex, noisy data.  I suspect the analysis requires numerous potentially false assumptions to generate those plots.

- It appears that all accidents are considered together, without subclassification of ones with fatalities or serious injuries.

- Instrument rated vs. non-instrument rated at the time of accident is treated as a binary variable without consideration of amount of time since the instrument rating. 

- It does not appear to be a peer-reviewed publication. I would be curious to see what independent statistics experts think regarding the quality of the analysis and how predictive the model is really likely to be.

- I am NOT questioning whether getting the IR is a worthwhile activity - it obviously is.

[Flash]

Thanks for posting this, DXB. I agree 100% with your takeaways.

There are problems with the way this study calculates accident rates. If I'm reading this correctly, the study compares number of pilots in the cohort with number of accidents involving such pilots, but it doesn't take into account the possibility (I'd say the probability) that the average pilot with 200-300 flight hours flies a lot less in a year than the average pilot in the 1,900-2,000-hour cohort. This is a major flaw. It's of some use to know how likely a pilot is to have an accident in a given year (if you're an insurer, for example), but it seems misleading to say that a cohort that averages one accident every 1,000 flight hours is safer than a cohort that averages one accident every 10,000 hours if the second cohort flies 20 times as many hours in the typical year. (The numbers in the last sentence are thrown out there for argument's sake; I have no idea what the real numbers are.)

It would be interesting to compare the data for instrument-rated vs. non-instrument-rated pilots for accidents occurring in VMC.

I am glad I got my instrument rating, enjoy using it, and plan to keep on doing so, but this study is a good reminder that flying in IMC adds risk and that unless I'm above average the amount of risk it adds might not be completely counterbalanced by my additional training and experience. That's OK if I'm willing to accept the added risk and am honest with passengers when flying in IMC. The higher accident rate for instrument rated pilots at 

There are a lot of difficulties in this kind of research, especially if you go beyond correlation to trying to infer causal relationships between flight time and accident rate. There are a lot of potential confounding variables. One is that at the lower level of flight hours you're much more likely to find pilots with lower levels of currency; if you don't fly often, you won't build flight time, and if you have a lot of flight time, there's a good chance you fly often. Another is that at the lower level of flight hours you're much more likely to find younger pilots; there aren't going to be a lot of 30-year-old non-professional pilots with 3,000 hours. A third is that GA includes professional as well as amateur pilots, and higher-time pilots are more likely to be professional than lower-time pilots. The paper notes that there is a lot of noise at the high flight hours end of the graph because of low numbers, so it might be impossible to try to control for age, currency, and pro vs. amateur; the cross-tabs would have too small a cohort.

[Flash]

6 minutes ago, Flash said:

There are problems with the way this study calculates accident rates. If I'm reading this correctly, the study compares number of pilots in the cohort with number of accidents involving such pilots, but it doesn't take into account the possibility (I'd say the probability) that the average pilot with 200-300 flight hours flies a lot less in a year than the average pilot in the 1,900-2,000-hour cohort. This is a major flaw. It's of some use to know how likely a pilot is to have an accident in a given year (if you're an insurer, for example), but it seems misleading to say that a cohort that averages one accident every 1,000 flight hours is safer than a cohort that averages one accident every 10,000 hours if the second cohort flies 20 times as many hours in the typical year. (The numbers in the last sentence are thrown out there for argument's sake; I have no idea what the real numbers are.)

This is also likely to be a flaw in the instrument-rated vs. non-instrument-rated comparison. If instrument-rated pilots fly a lot more hours on average than non-instrument-rated ones, then it doesn't tell you a lot to say that on an accidents-per-year basis their accident rate might be slightly higher.

[Flash]

1 minute ago, Flash said:

This is also likely to be a flaw in the instrument-rated vs. non-instrument-rated comparison. If instrument-rated pilots fly a lot more hours on average than non-instrument-rated ones, then it doesn't tell you a lot to say that on an accidents-per-year basis their accident rate might be slightly higher.

It would seem that insurance companies must think that on a claims-in-dollars-per-year basis, instrument-rated pilots have a lower rate than non-instrument-rated (normalized by make and model of airplane), or the insurance companies wouldn't give a discount for getting the rating. 

[DXB]

1 hour ago, Flash said:

It would seem that insurance companies must think that on a claims-in-dollars-per-year basis, instrument-rated pilots have a lower rate than non-instrument-rated (normalized by make and model of airplane), or the insurance companies wouldn't give a discount for getting the rating. 

yeah I wondered about this too - I probably shouldn't be highlighting possible evidence of their having underestimated risk of insuring instrument rated pilots or the flaws in calculating lower accident risk after 350 hours - all our rates are going up enough as it is 

[irishpilot]

Interesting read. It's tough when comparing a non-IR pilot with IR pilot when those two types may fly in very different conditions. It would be interesting to compare the accident rates between those two in VMC conditions and then as a total (which the paper does). Regardless, another factor that greatly affects the data is currency and training. 

[jlunseth]

This is excellent work. Thanks for putting it up. I agree, though, that it would be helpful to compute accident rates in terms of number of accidents per hour flown rather than simply assuming that all pilots, regardless of number of total flight hours, fly the same amount. I believe that is part of the issue with the base numbers, and not an easy number to get. Is the FAA going to start requiring pilots to report the number of hours they fly each year or each month just to get a better database? Probably would be alot of resistance to such an invasion of privacy.

[0TreeLemur]

NTSB reports usually provide the total pilot time, total pilot time in type, and something like total time last 90 days.   With the API to search the NTSB aviation accident database, somebody good with programing could do what you suggest.

[carusoam]

Expect the ADSB out data to provide a lot of data and privacy issues.... 

Best regards,

-a-

[0TreeLemur]

Data suggest that increased probability of having a fatal accident in a M20C is strongly correlated with poor decision making and/or inattentiveness, independent from ratings and/or total time.   The compliment to this suggestion is that good decision making and attentiveness really reduces the probability of having a fatal accident.   I didn't find one summary that show a really good set of decision making that had a fatal outcome.   Nothing close.  I don't raise this to belittle anyone dead or alive, but to point out that the NTSB database clearly shows that bad decision making, including flying outside of comfort, rules, or currency, clearly kills pilots more than any other factor.

Here's a summary that supports this assertion, developed from an analysis of the NTSB database.  It is based on factors that were given by the investigators as causes of all listed fatal accidents in Mooney M20C aircraft (in no particular order) between May, 1993, and February, 2008.  It's a sad list.

1.  Fuel mismanagement

1a.  Pilot did not sump tanks on a Mooney that hadn't flown in months and was parked outdoors.   Rainfall during the period when the plane was parked outdoors: 5 inches.  Fuel starvation on departure due to water ingestion.

1b.  Pilot forgot to switch to the fullest tank before starting a night approach in VMC when the non-fullest tank is almost empty.

1c. Pilot knowingly departed with 10 gallons of fuel for a local flight, then used it all up.  Engine quit 1.3 miles from airport.  Stalled trying to stretch the glide.

1d.  FBO closed.  Pilot did not want to wait for the FBO staff to come to airport to fuel the airplane.  Departed without waiting.  Pilot turned around after realizing insufficient fuel to reach destination, then run out of fuel on night VMC final approach after returning to the very same airport where pilot was unwilling to wait for fuel.

2.   Drugs

2a.  Pilot took cough syrup containing codeine before departing into clouds and becoming spatially disoriented after a vacuum pump failure.

2b.  Pilot took methamphetamine and flew into IMC.  Tried to shoot and NDB approach without an instrument rating, AND ran out of fuel.

3.   Spatial disorientation

3a. Non instrument rated pilot flew into IMC over mountains at 13,900'.

3b. Non instrument rated pilot crashed into trees in low ceilings and fog while trying to find a grass strip.

3c. Non instrument rated pilot flew into IMC while looking for VFR route to airport.   Tried to shoot ILS with radar vectors.   Failed.

3d.  Instrument rated pilot departs night into IMC and on climb out loses control after reporting a "prob".   No problem found with engine, vacuum or gyros.

4.  Failure to fly the airplane

4a.  Pilot flew into the ground while doing low-altitude pipeline reconnaissance flight with a jury-rigged 36 gallon auxiliary fuel tank in the baggage compartment.

4b.  Departure stall with a normally running engine.   4 fatalities, including pilot and three grandchildren.

5.  Lack of training in complex aircraft.  A 100 hour total time pilot with 3 Mooney hours, takes three friends for a ride with no complex training, only to produce a departure stall.  One survivor.

6. Poor decision making (maybe shouldn't be a separate category)

6a.  Two IR pilots warned during wx briefing about light to moderate icing between 6,000 and 18,000 ft and deciding to go anyway.  Flight over mountainous terrain.  Pilots forgot carburetor heat when engine quit at 12,000 ft.  Night forced landing unsuccessful.

6b.  Non instrument rated pilot takes off in mountainous terrain with low ceilings hoping to find a VFR route out.  Doesn't.

7. Pilot Health

7a.  Pilot reports of "chest pains" after departure.  Doesn't make it back to the field.

7b.  69 y.o. solo pilot has a heart attack aloft.

8.  Poor/Incorrect Maintenance

8a.  Clogged fuel strainer in right tank.  Left tank empty.  Resulted in unsuccessful off-field landing.

[exM20K]

@0TreeLemur first off... thanks for using “data” as a plural noun.

at least four of the 18 were at night.  I have approximately 10% of my total logged time as night, and recently, it has been a lot less than that.  Setting aside the stupid pilot tricks (over gross, meth, etc) one take-away here is that night flying raises the stakes significantly if my logbook is typical.

-dan