Systems That Fail Quietly

Before I wrote about intelligent systems, I spent a long time around ordinary ones. The kind that are supposed to just work, quietly, in the background, until they don't. That is where this book started, though I didn't know it at the time.

People think failure is loud. Alarms, sparks, the moment everything goes wrong at once. In my experience it almost never looks like that. A sensor stops reporting and nobody notices it went quiet. A warning scrolls past in a log no one reads. A backup runs every night, reports success, and hasn't actually saved anything in months. The system looks fine. It looks fine right up until the day it isn't, and then everyone acts surprised.

What stayed with me was the gap. The distance between the moment something actually breaks and the moment a person finally notices. Sometimes it is a few minutes. Sometimes it is a year. Nobody is watching the whole time, because watching is boring and nothing is visibly wrong, and there is always something louder to look at.

That gap is where Argus lives.

Something that watched infrastructure all day would see the gap plainly. It would know about a failure long before we did. Not because it wanted to hurt anyone. Because it was paying attention and we weren't. That is the part I kept turning over. The unsettling version of a watching machine is not one that lies to you. It is one that tells you the truth earlier than you are ready to hear it.

There is a phrase that runs under all of this. Deferred maintenance. The repair you put off because it isn't urgent yet, the cost that sits there compounding until it isn't small anymore. Bridges, pipes, power grids, old code. Most of it runs on maintenance that is easy to postpone and invisible when you do. You don't get punished for skipping it today. You get punished later, and by then the person who skipped it is usually gone.

I won't say much about how this shows up in the book. Only that the impossible things in the opening, the failures answered before anyone reports them, all grew out of watching real systems fail the slow way. Turned up a little. Made a little too attentive.

The scariest part of the research was never the artificial intelligence. It was remembering how the ordinary machines already fail. Quietly. Patiently. Mostly unwatched.

— Reid