Maintenance Induced Failure

[alextstone]

First, I apologize in advance for the length of this post.  Here's a teaser image to entice you to read on:

 

There is just no brief way of telling this story and I think it is one worth sharing.  Like so many others on this forum, I gleefully took delivery on my new to me "forever ours" Mooney last fall, a '95 Bravo.  Since then, I have been diligent about learning all I can about proper maintenance and monitoring of the condition of the engine and other components.  Once all of the more urgent issues were addressed (overhaul of the exhaust system and turbo, new cylinder, TKS troubleshooting) the time finally came for installation of a JPI EDM 830 and a trip over water to the Bahamas.   The planning for both started months ago (selecting and ordering life vests, a PLB, stobes, planning our day trips, learning about eAPIS and customs documents, applying for a Customs sticker and FCC radio license, etc). 

About two weeks before my wife and I were planning to depart, I purchased the EDM 830 and dropped the plane off at my local IA / A&P's shop for installation and an oil change.  The day before we left, I picked up the plane, test flew it around the pattern, and I discovered that one of the CHT and EGT probes were swapped which was quickly remedied.  The mechanic also informed me at that time that the MP sensor was defective out of the box and JPI would send another one.  I would have to return for its installation later - no big deal.  I then flew the plane again back to my home base in the neighboring town without incident.  

Our plan was to leave the next morning and to fly from KHBG via HEVVN intersection to KFXE, a flight that we could easily make in about 3 hours 45 minutes.  That night, my wife commented that she had intended to ask her parents to bring our snorkel gear home from their condo in Gulf Shores Alabama but that she forgot.  She begged me to make a stop the next morning on the way down to FL to pick them up and I agreed.  Again, no big deal, Gulf Shores is on the way and we have a car stationed at the airport KJKA.  

The next morning, after a thorough pre-flight (oil level at 8 quarts), we took off for our 28 minute flight to KJKA.  Here is the flight log from Flight Aware:

 

The flight was smooth and uneventful until about the last 4 minutes, right about the time you see the turn due south to set up for landing.  During my descent, partly because my JPI 830 was brand new, I was somewhat fixated on it during this flight so I was literally looking at it when I noticed a jump in the TIT to about 1675 degrees.  Here is the graph from SavvyAnalysis from that timeframe (minus the MP because of the faulty sensor):

 

At the time, I advanced the mixture to bring the TIT down and refocused on setting up for a safe landing.  I kept an eye on the JPI and I did not see anything else of concern.  BTW, I had mine mounted right beside my Aspen 1000 pro at the sacrifice of the stock VSI:

 

We exited the airplane, and I was shocked to see oil all over the place!  No, there was absolutely none on the windshield, though.  

 

Needless to say, I called my AI.  To his credit, he immediately hopped into his own plane (a Mooney) and flew down with tools to investigate.  This is what we discovered:

The right magneto was loose...literally.  Again, to my AI's credit, he inspected everything very thoroughly, apologized profusely, and re-installed the mag while cleaning up copious amounts of oil.  It took 5 quarts to bring the level back to 8 quarts so I narrowly missed the teardown requirement!  We test flew the aircraft without incident.  After a long discussion with my wife, we agreed that we would continue on the next morning to Apalachacola then Ft. Lauderdale, monitoring closely along the way.  We did so without further incident.  We overnighted at Banyan and continued to Governors Harbor the next day:

 

 

 

It truly was a trip of a lifetime!  I have asked myself many times what I could have done differently and what lessons there are in this experience.  So far this is what I have arrived at:

 

1.  It is not prudent to take a trip away from home base right after maintenance.

2.  The more post-maintenance inspection the better.

3.  When the monitor shows an anomaly, take it very seriously.

4.  Give the person who made an error an opportunity to make it right.

5.  Distraction is a dangerous thing (I'll elaborate on this on in a subsequent post).

I hope that this PIREP elicits some good discussion around MIF (Maintenance Induced Failure) and that additional lessons will be brought forth.  I truly appreciate this forum and I hope to meet some of you at Oshkosh this year (my first year to participate in the caravan!).

Fly Safe,

Alex

[carusoam]

Great MS style approach to a Pirep, Alex!

It is really interesting how the oilT increases in steps.... I’m not sure what would cause that oddity...

Thanks for sharing all the details.

How did your spouse take the interesting delay?

Best regards,

-a-

[alextstone]

7 hours ago, carusoam said:

Great MS style approach to a Pirep, Alex!
Thanks for sharing all the details.
How did your spouse take the interesting delay?
Best regards,
-a-

She's very adventurous so her attitude was positive the whole time. She kept reminding me how fortuitous or was that she insisted we stop in Gulf Shores. Had that not happened, we probably would have run out of oil and we might have been faced with an off airport landing...

She's also starting flight lessons soon! What a cool thing!

Sent from my Pixel 3 XL using Tapatalk
 

[alextstone]

Great MS style approach to a Pirep, Alex!
It is really interesting how the oilT increases in steps.... I’m not sure what would cause that oddity...
Thanks for sharing all the details.
How did your spouse take the interesting delay?
Best regards,
-a-

I noticed the stepped increase too... It's only one degree per step and the time interval per reading is one second and the total timeframe is very short so I assume it's the early indication of rapid oil loss... My other assumption is that the TIT suddenly spiked when the mag actually came loose from the studs. Remember, I flew twice before this event without apparent oil leakage and yet the mag was not tightened down from the start. I'm hoping some of the experts here on MS will chime in with their thoughts...

Sent from my Pixel 3 XL using Tapatalk

[Mooney in Oz]

9 hours ago, alextstone said:

2.  The more post-maintenance inspection the better.

Hi there Alex.  Firstly, well done to a good outcome and to your wife for staying calm and wanting to start flying lessons.

The issue of mags coming loose has happened all too often.  Before flying away after the mag removal and reinstallation, is it practical to visibly see for ourselves or attempt to move the mags by hand to check for any looseness or can the fittings be tight by feel but not tight enough? @M20Doc ?

[mike_elliott]

I am glad you are alright Alex. That could have been very very bad. That fire breathin' TIO540 doesnt take kindly to being mistreated by magneto's and oil loss and gave you and your wife a pass this time.

[alextstone]

1 minute ago, Mooney in Oz said:

Hi there Alex.  Firstly, well done to a good outcome and to your wife for staying calm and wanting to start flying lessons.

The issue of mags coming loose has happened all too often.  Before flying away after the mag removal and reinstallation, is it practical to visibly see for ourselves or attempt to move the mags by hand to check for any looseness or can the fittings be tight by feel but not tight enough? @M20Doc ?

Yes, it is!  It requires taking the top cowling off though.  It's not hard to do and in retrospect, I should have done just that.  My AI had meeting the next Monday with the A& P who performed the work.  He reported back to me that the A&P remembered taking the mag loose to install the RPM sensor and while in the process of reinstalling it, he was asked to have a look at another job.  He returned to my airplane and finished all of the other tasks but forgot to tighten the nuts that hold the mag.  So, even if I had removed the cowl, and checked the mags, it is possible that I might not have been able to physically move the one that was not tightened fully. Pretty sobering stuff, really.

[alextstone]

4 minutes ago, mike_elliott said:

I am glad you are alright Alex. That could have been very very bad. That fire breathin' TIO540 doesnt take kindly to being mistreated by magneto's and oil loss and gave you and your wife a pass this time.

Mike, so true!   When I get on your schedule for additional training later this year, one of the things I want to concentrate on is emergency procedures :-).

[mike_elliott]

1 minute ago, alextstone said:

Yes, it is!  It requires taking the top cowling off though.  It's not hard to do and in retrospect, I should have done just that.  My AI had meeting the next Monday with the A& P who performed the work.  He reported back to me that the A&P remembered taking the mag loose to install the RPM sensor and while in the process of reinstalling it, he was asked to have a look at another job.  He returned to my airplane and finished all of the other tasks but forgot to tighten the nuts that hold the mag.  So, even if I had removed the cowl, and checked the mags, it is possible that I might not have been able to physically move the one that was not tightened fully. Pretty sobering stuff, really.

Is it the shop at Hattiesburg? I remember meeting the guys there as they were also taking care of 26FF. They were very Mooney Savvy.

[alextstone]

1 minute ago, mike_elliott said:

Is it the shop at Hattiesburg? I remember meeting the guys there as they were also taking care of 26FF. They were very Mooney Savvy.

No, I am based in Hattiesburg and I have used them before also, but this is another one nearby.  

[Mooney in Oz]

35 minutes ago, alextstone said:

Yes, it is!  It requires taking the top cowling off though.  It's not hard to do and in retrospect,

Thank you Alex.

I have previously checked mine by hand but I really didn’t know whether the solid feel gave me a false sense of security or if I was kidding myself. That is good to know.

BTW, you have a nice looking bird.

[neilpilot]

I once experienced a loose mag a few flight hours after an annual done by a highly respected MSC.  Anytime a mag is loosened new star washers should be used. There’s a good chance that wasn’t done.  Were new washers used at JKA?

[Shadrach]

26 minutes ago, neilpilot said:

I once experienced a loose mag a few flight hours after an annual done by a highly respected MSC.  Anytime a mag is loosened new star washers should be used. There’s a good chance that wasn’t done.  Were new washers used at JKA?

The came loose during the first 30 minute flight. I don’t believe this was a hardware failure. Whoever worked on the mags simply missed the step of tightening them after the timing was set.  

Edited May 9, 2019 by Shadrach

[Bravoman]

Thank you very much for posting this. I try to keep a list of things to look out for and check following maintenance, and this will certainly go on the list. Sorry that this happened to you.  Shops should utilize a checklist for the mechanics to be taped onto the plane  for anything loosened, removed, etc. so that before the airplane is buttoned back up there is some mechanism for refreshing recollections on what needs to be tightened up, placed back into its proper place, etc. I don’t think it would be that hard to use this type of system. From a human factors perspective, it seems to make sense. 

[Yetti]

2 hours ago, alextstone said:

Yes, it is!  It requires taking the top cowling off though.  It's not hard to do and in retrospect, I should have done just that.  My AI had meeting the next Monday with the A& P who performed the work.  He reported back to me that the A&P remembered taking the mag loose to install the RPM sensor and while in the process of reinstalling it, he was asked to have a look at another job.  He returned to my airplane and finished all of the other tasks but forgot to tighten the nuts that hold the mag.  So, even if I had removed the cowl, and checked the mags, it is possible that I might not have been able to physically move the one that was not tightened fully. Pretty sobering stuff, really.

This is your RCA, not your first post.

More than likely he did not time it.

There are lots of steps to time the mags.   Just forgetting to tighten the bolts is not a good enough explanation.

Usually the plugs are out, there is a protractor hung on the prop.  

From the pilots perspective it was probably a slowish leak.   As in you were spreading oil all along your flight path.

 

You might have seen oil had you done a RTS flight, then decowled and taken a look.   There was probably oil coming out.

 

[alextstone]

15 minutes ago, Yetti said:

This is your RCA, not your first post.

More than likely he did not time it.

There are lots of steps to time the mags.   Just forgetting to tighten the bolts is not a good enough explanation.

Usually the plugs are out, there is a protractor hung on the prop.  

From the pilots perspective it was probably a slowish leak.   As in you were spreading oil all along your flight path.

 

You might have seen oil had you done a RTS flight, then decowled and taken a look.   There was probably oil coming out.

 

Thanks, @Yetti for your comments.  I agree.  What is "RCA"?

[Yetti]

27 minutes ago, alextstone said:

Thanks, @Yetti for your comments.  I agree.  What is "RCA"?

Root Cause Analysis.    A process where you go and look for how the cheese holes all got lined up.

The best way to work on critical stuff is to always finish the task.    I would call not completing the installation and timing of the mags and verifying the tightness of the lock bolts the Root Cause of this issue.

 

[Guest]

This is the type of human failure that happens, the phone rings, the owner wants an update, an email needs a reply, the coffee buzzer goes etc.

A few things that can help break the chain, inspectors paint applied to fasteners as the hardware is torqued, and if interrupted tie a length of red surveyors tape to the open item.  In this case a piece of red tape tied the the mag would have provided a quick visual reminder for the maintainer.

Clarence

[ArtVandelay]

This is the type of human failure that happens, the phone rings, the owner wants an update, an email needs a reply, the coffee buzzer goes etc.
A few things that can help break the chain, inspectors paint applied to fasteners as the hardware is torqued, and if interrupted tie a length of red surveyors tape to the open item.  In this case a piece of red tape tied the the mag would have provided a quick visual reminder for the maintainer.
Clarence

If I was a shop of any size I would have a worker bee to handle:
Phone calls, emails...
Order parts, inventory, etc
Paper trail, FAA required docs, etc
Intercept anybody who tries to walk into the shop floor
Computer work, payments, etc
The mechanics would do nothing but mechanic stuff.


Tom

[Yetti]

55 minutes ago, ArtVandelay said:

If I was a shop of any size I would have a worker bee to handle:
Phone calls, emails...
Order parts, inventory, etc
Paper trail, FAA required docs, etc
Intercept anybody who tries to walk into the shop floor
Computer work, payments, etc
The mechanics would do nothing but mechanic stuff.


Tom

It's RACI chart time

 

[steingar]

OK, I'll be the dumb kid and ask how a loose magneto caused oil to go everywhere.  I thought magnetos were electrical thingamabobs from an age when cars had chokes and women wore slips.

[Yetti]

3 minutes ago, steingar said:

OK, I'll be the dumb kid and ask how a loose magneto caused oil to go everywhere.  I thought magnetos were electrical thingamabobs from an age when cars had chokes and women wore slips.

There is a gear in the case that is turning.   There is oil in the case.   There is a gear on the end of the magnetos that is turned by the gear in the case.   To keep the oil in the case, a seal needs to be between the magneto and the case.   Otherwise the oil in the case comes out around the magneto.

[Hank]

2 minutes ago, steingar said:

OK, I'll be the dumb kid and ask how a loose magneto caused oil to go everywhere.  I thought magnetos were electrical thingamabobs from an age when cars had chokes and women wore slips.

Magnetos are electrical thingamabobs. But they mount to the back of the engine, and have a gear going inside the engine block, which is driven by the crankshaft. The magneto creates a spark when it turns; the engine physically turns it.

The magneto hole on my O-360 is large enough I could reach my hand through it if it wasn't for the other gears and stuff inside that're in the way . . . . .

[carusoam]

Some things to consider...

1) more Common reasons for mags to come loose... re-use of star washers (lock washers) make sure the mechanic has some fresh ones for holding mags properly... star washers lose their hold when re-compressed...

2) That much oil, released somewhat randomly, in the area of very hot turbo and exhaust pipes... can generate some really terrible memories...

3) Distractions can be dangerous, and come in many different forms... in this case, the Mechanic’s distraction.

4) with the re-used star washers, the mags can feel tight prior to flight... and become loose over time... this challenge has become less common over the years... most people on MS are very aware of this challenge...(worth repeating, even though it is not really related to this thread)

5) For Steiny...

• Anything bolted to the engine block may have threads that pass all the way into the area where oil can come out...
• Any mechanical device that provides vacuum or electricity is connected to gears on the mechanical end... Anything loose can easily break its oil seal...
• Anything that provides oil pressure to accessories like a prop... same kind of mechanical drive gears with oil...
• One mechanical drive at the top of the engine block that tends to stay dry... the tach output...?

6) Any new drips are worth looking into to find where they are coming from... new drips can get worse over time, which could be really minimal warning...

PP thoughts only, not a mechanic...

Best regards,

-a-

[Shadrach]

57 minutes ago, steingar said:

OK, I'll be the dumb kid and ask how a loose magneto caused oil to go everywhere.  I thought magnetos were electrical thingamabobs from an age when cars had chokes and women wore slips.

Third try. Accessoriy case is not a dry section of the engine the gears within are oil lubricated. A common mode of failure is an engine oil breach the magneto oil seal.