A runaway train and the anatomy of a bad alert
The 1988 Gare de Lyon rail accident, and what its instruments, alerts and emergency mechanisms have to say about how we monitor and run software systems.
On the evening of 27 June 1988, a commuter train coming in from the south-eastern suburbs approached Paris Gare de Lyon with around a hundred passengers aboard and almost no working brakes. The driver knew, and had spent the previous two minutes doing everything he could think of to avoid a disaster, including radioing an emergency warning to the control room. The control room knew there was a problem too, to a point - they knew that a train somewhere had no brakes, and that was the full extent of it. Not which train, not which track, not which platform it was heading for. It struck a stationary train waiting to depart for Melun at over 70 km/h, 56 people died, and it remains the deadliest peacetime rail accident France has had.
It's an accident I've been reading about lately, and it got me thinking about how the events of this incident map onto the work I do in software development. Instruments that report the wrong thing, alerts that arrive without the information needed to act on them, an emergency mechanism that wrecked the very recovery it was meant to enable - the problems that contributed to the Gare de Lyon rail accident are problems that can appear in software too, mercifully with less awful consequences.
How you lose the brakes on a train
The chain of events started with a timetable change. Under the new summer schedule, the train from Melun no longer stopped at Le Vert de Maisons, and a passenger who didn't know that pulled the emergency alarm as her platform slid past, got off the train, and left the station. On that class of train the alarm handle didn't warn the driver; it applied the brakes, and left the driver, Daniel Saulin, to walk back through the train and reset the system himself.
The full accounts of what happened next are worth reading - the Wikipedia article is solid, and Max S's long write-up is excellent - and there are numerous YouTube disaster channels on the subject. The investigators' reconstruction went like this. The reset handle was stiff, so Saulin gripped a nearby valve to get leverage, and in hauling on the handle he accidentally knocked the valve closed. This valve was the stopcock on the main brake pipe, the line that feeds compressed air back to every carriage behind the first. With this valve shut, seven of the train's eight cars had no brake supply at all.
The fail-safe did exactly what it was designed to do: with the air supply cut, the brakes on those cars locked on. Saulin and his guard read the locked brakes as a leftover symptom of the emergency stop rather than a new fault, and followed the drill for that instead, walking the train and manually bleeding the air at each wheel to release them. It was a reasonable diagnosis, but of the wrong fault, and by the end of it they had disabled the one mechanism that was correctly reporting the problem.
Back in the cab, meanwhile, the brake pressure gauge read normal. It measured only the first car - the one car in front of the closed valve, and the only car with working brakes. Every instrument Saulin had told him the train was healthy. Nearly half an hour late by now, and with instructions from control to skip the next stop and claw back time, he pulled away without a brake test. The first time the brakes were really needed again was on the descent into Gare de Lyon.
An alert with no name on it
Once he realised, Saulin did what he was supposed to do: he got on the radio and raised the alarm. But in the panic the message went out without his train number, so the controllers now knew there was a runaway train with no brakes somewhere on one of the busiest approaches in Paris, and nothing else. Which train, which line, inbound or outbound - all of it missing from the one message that carried the alarm.
He also pressed the general alarm button, and at this point a mechanical failure became a systems failure. The general alarm halted traffic across a wide area, which seemed like the safest possible move - except it also froze the signalling, wiping the routing the signallers had already set up, which by the published accounts would have diverted an inbound runaway into the empty platform next door. And it lit up the radio, as every driver in the area called in to ask what was going on, jamming the channel the missing train number needed to travel down. So the emergency mechanism undid the one mitigation already in place, and drowned out the follow-up message that would have identified the train. The controllers worked out which train it was less than a minute before impact.
At platform 2, meanwhile, sat a delayed outbound train full of passengers. Station staff had begun evacuating the underground level of the station where these platforms were located, and the outbound train's driver, André Tanguy, stayed in his cab repeating the evacuation order over the intercom until the collision killed him. Fifty-six people died in and around that train.
The changes that would have mattered
Reading it back with the outcome known, you can point at half a dozen places where a different design would have broken the chain, and it's notable how many of them SNCF went on to build.
The first is the alarm handle. After the accident, SNCF changed it so that pulling the handle no longer stops the train; it notifies the driver, who decides where and whether to stop. With that design the emergency stop at Le Vert de Maisons would never have happened, and nor would the reset that closed the valve.
Then there's the gauge. A pressure reading taken at the rear of the brake pipe rather than the front would have read zero, and the train would never have moved. The instrument wasn't broken; it was measuring the wrong scope, and it faithfully reported the one car that was fine.
A brake test after the intervention would have caught it too, and the rules said one should have happened. But a rule that can be skipped under schedule pressure will eventually be skipped under schedule pressure, and the more interesting question is what would have made the check structurally unavoidable - an interlock rather than an instruction.
And there was a brake that would have worked. The Z 5300 class had a rheostatic brake, electric and independent of the failed air system, within reach on the driving desk, and the inquiry found that using it would very probably have slowed the train enough to avoid catastrophe. Drivers of the class were reluctant to use it in normal running, wary of wheel slide when combining it with the air brake, and in the stress of the descent it didn't come to mind.
The radio call is the one modern railways have since answered: from what I've read, today's cab radio systems send the train's identity and position with the call automatically, precisely so a panicking driver doesn't have to remember to. And the general alarm's design - halt everything, freeze everything - assumed the danger was trains moving, when the actual danger that evening was a set of points that could no longer be changed. An emergency mode that preserved the safe routings already set would have put the runaway into an empty platform.
None of these fixes asks anyone involved to have been better, calmer or quicker that evening, which is what makes them the right kind of fix.
What this looks like in software
Each of those failures has a recognisable equivalent in software.
The gauge reading one car is the healthcheck that tests the load balancer and not the application behind it, the dashboard fed from the region that isn't on fire, the synthetic check exercising the happy path while real traffic fails. When a client asks me to review their monitoring, scope is the first thing I want to understand - what exactly each check measures - because a green dashboard measuring the wrong thing is more dangerous than no dashboard at all.
The radio call without a train number is the alert that says something is wrong without saying who, where or what. Every page should carry the service, the environment, a link to the runbook and enough context that the person woken at 3am spends their first minutes acting rather than identifying. Writing that context into the alert takes a few minutes; reconstructing it in the middle of an incident takes far longer, at the worst possible moment.
The deeper fix is the one the railways reached: stop relying on a person to remember the context, and have the system attach it. Defining alerts as code helps here - a CloudWatch alarm written in CloudFormation gets its description, its dimensions and its runbook link reviewed alongside the code it watches, rather than depending on what somebody typed into the console months ago. Structured payloads help too, because an alert that arrives as JSON with named fields for service, environment and severity can be routed, grouped and enriched by tooling before a human ever reads it, where free text has to be parsed by a tired brain. And OpenTelemetry builds the principle in from the start, attaching resource attributes - the service name, the environment, the version - to every trace, metric and log it emits, the software equivalent of the cab radio sending the train's identity with every call.
The jammed radio channel is the alert storm - one root cause, forty downstream alarms, and the notification that matters buried somewhere in the flood while everyone piles into the same channel asking what's going on. Deduplication, alert dependencies and a clear incident-comms structure exist to keep the channel open for the one message with the train number in it.
And the general alarm is the big red button whose blast radius includes the recovery path: the global freeze that also freezes the deploy pipeline you need to ship the fix, the lockdown that revokes the credentials the responders were about to use. Emergency procedures deserve the same failure-mode scrutiny as the systems they protect, and they rarely get it, because nobody enjoys planning for the day the safety mechanism is the problem.
And the rheostatic brake nobody thought to use is the recovery mechanism your team has never exercised: the database failover that has never been pulled in anger, the read-only mode that would shed the load, the manual override documented in a runbook nobody has opened since it was written. People under pressure reach for what they practise, which is the argument for rehearsing emergency mechanisms in calm conditions - game days and failover drills exist for exactly this - rather than trusting that the right control will come to mind on the night it matters.
The system was the failure
The courts initially gave Saulin four years for manslaughter; after SNCF staff struck over what they saw as scapegoating, a retrial cut it back and he served six months. I think the strikers had it right. Nobody in the chain was reckless - a passenger made an understandable mistake, a driver diagnosed the fault the evidence in front of him suggested, controllers pushed to keep a busy timetable moving. The failure lived in the design: in what the instruments measured, what the alerts carried, and what the emergency mechanisms assumed.
It's a worthwhile exercise to read one of your own alerts cold, as if you'd just been woken by it. Does it say who, where and what? Would the person receiving it know which train was running away? If the honest answer is no, that's fixable - and far cheaper to fix before an incident than during one. If you'd like a second pair of eyes on your monitoring or incident response, get in touch.