The change
Most incidents start as a small, reasonable change
The member discount becomes a setting read from the environment. Production sets it to 0.15.
Members get ten percent off, and marketing wants fifteen, with the freedom to change it later without a release. So the developer changes one line: the discount is now read from an environment variable, with ten percent as the default. In production, the variable is set to zero point one five. We follow this one line all the way to the 6 production servers.
The path
It passes every gate it meets
Review approves, then lint, unit and integration tests, an image and staging: 14 minutes, all green.
The branch becomes a pull request, and a colleague reviews it: one line, a sensible default, approved. CI runs lint, unit tests and integration tests, builds one image, and deploys it to staging. 14 minutes, all green. Then production. One thing every gate had in common: none of them set the variable.
The release
Released everywhere at once, a mistake reaches everyone at once
Errors jump from 0.2% to 9.2%: every member fails to pay. 432 failed checkouts in 12 minutes.
The new version goes to all 6 servers together. Checkout errors jump from a fifth of a percent to over nine percent. Every member who tries to pay fails, about 36 people a minute. It takes 9 minutes to connect the alert to the release and 3 more to roll back: 432 failed checkouts.
The cause
An environment variable is always text
Production read the string '0.15'; tests used the number 0.10. One minus a string is a TypeError.
In production the variable is set, so the code receives the text zero point one five, a string. Subtracting a string from a number raises a type error, on every member's checkout. In tests and staging the variable was never set, so the code used its default, a real number, and worked. The missing test is the one that sets the variable the way production does.
The fix
Roll back to the artefact you trust, then ship the fix to a few first
Redeploy v41 in 3 minutes. Then v43 with float() and a test, as a canary on 1 of 6 servers.
First, roll back. Not by rebuilding old code, but by redeploying v41, the exact image that ran yesterday. Three minutes, and errors are back to normal. Then the fix: convert the setting to a number, and add a test that sets the variable as production does. The fixed version goes to one server first, about 17 percent of traffic. After 10 quiet minutes, it goes everywhere.
The trade-off
A canary costs minutes and buys a smaller blast radius
A canary adds 10 minutes a release; this incident would have cost about 30 failed checkouts, not 432.
All at once is the fastest way to release, and every mistake reaches everyone. A canary with automatic rollback adds 10 minutes to each release and needs error rates per version. In return, this incident would have cost about 30 failed checkouts instead of 432. For checkout, the minutes are worth it.
The path, fixed
Every gate catches a different mistake; this one slipped between two
Branch, review, config as text, tests that set it, one image, canary. Each gate links to its lesson.
Here is the whole path again, as it now runs. A small commit on a branch. A review that asks what the code reads. Settings parsed as the text they are, and tests that set them. One image, built once and rolled back to in minutes. And a canary before everyone. Each gate opens the lesson that teaches it.
A real case
At Knight Capital, one server missed the deployment
Seven of eight servers got the new code. A reused flag woke old code on the eighth: over $460 million lost in about 45 minutes.
In 2012, Knight Capital deployed new trading code to eight servers, copying it by hand. One server was missed, and nobody checked. The new code reused a flag that, years before, switched on a retired function called Power Peg. On the eighth server, the flag woke it up. The system sent ninety-seven error emails before the market opened. In about forty-five minutes it made four million executions, and Knight lost over four hundred and sixty million dollars. Deploy with a pipeline, check that every server runs the same artefact, delete dead code, and never reuse a flag.
Lesson complete
You finished “One line to production”
Nicely done.