NATS Outage Preliminary Report

[dzeleski]

https://publicapps.caa.co.uk/docs/33/NERL Major Incident Investigation Preliminary Report.pdf
 

The flight plan delivered to FPRSA-R by IFPS had been converted in the usual way into ADEXP. Following the IFPS processing of the flight plan, the ADEXP format of the flight plan contained the original ICAO4444 flight plan plus additional waypoints relevant to its route.
The ADEXP waypoints plan included two waypoints along its route that were geographically distinct but which have the same designator.

.......

 Although there has been work by ICAO and other bodies to eradicate non-unique waypoint names there are duplicates around the world. In order to avoid confusion latest standards state that such identical designators should be geographically widely spaced. In this specific event, both of the waypoints were located outside of the UK, one towards the beginning of the route and one towards the end; approximately 4000 nautical miles apart.

...........

Once the ADEXP file had been received, the FPRSA-R software commenced searching for the UK airspace entry point in the waypoint information per the ADEXP flight plan, commencing at the first line of that waypoint data. FPRSA-R was able to specifically identify the character string as it appeared in the ADEXP flight plan text.

Having correctly identified the entry point, the software moved on to search for the exit point from UK airspace in the waypoint data.

Having completed those steps, FPRSA-R then searches the ICAO4444 section of the ADEXP file. It initially searches from the beginning of that data, to find the identified UK airspace entry point. This was successfully found. Next, it searches backwards, from the end of that section, to find the UK airspace exit point. This did not appear in that section of the flight plan so the search was unsuccessful. As there is no requirement for a flight plan to contain an exit waypoint from a Flight Information Region (FIR) or a country’s airspace, the software is designed to cope with this scenario.

Therefore, where there is no UK exit point explicitly included, the software logic utilises the waypoints as detailed in the ADEXP file to search for the next nearest point beyond the UK exit point. This was also not present. The software therefore moved on to the next waypoint. This search was successful as a duplicate identifier appeared in the flight plan.

Having found an entry and exit point, with the latter being the duplicate and therefore geographically incorrect, the software could not extract a valid UK portion of flight plan between these two points. This is the root cause of the incident.

........

Having raised a critical exception the FPRSA-R primary system wrote a log file into the system log. It then correctly placed itself into maintenance mode and the C&M system identified that the primary system was no longer available. In the event of a failure of a primary system the backup system is designed to take over processing seamlessly. In this instance the backup system took over processing flight plan messages.

As is common in complex real-time systems the backup system software is located on separate hardware with separate power and data feeds. Therefore, on taking over the duties of the primary server, the backup system applied the same logic to the flight plan with the same result. It subsequently raised its own critical exception, writing a log file into the system log and placed itself into maintenance mode.

........

Now that the root cause has been identified further work needs to be undertaken to trace back through the development and testing of the FPRSA-R sub-system to understand whether the combination of events that led to the incident could have been mitigated at some point in the software development cycle. It is our understanding from the manufacturer that the specific area of software related to this investigation is unique to NATS.

[N201MKTurbo]

Wow, thats a crazy acronym word salad!

[N201MKTurbo]

And, why do we care about a software glitch in the UK?

[whiskytango]

We were at LHR waiting for a flight to ATL when this went down.  Our flight was delayed just enough to ensure that our connecting flight from ATL was missed.  It could have been a lot worse.  We heard that a flight from Liverpool to Belfast, just a short hop across the Irish Sea, was delayed 12 hours. 

[0TreeLemur]

I would expect the English to write better English than that to-be verb salad.  It reads as if someone spoke it into a tape recorder....  Just sayin'