The change

Most incidents start as a small, reasonable change

One small change# pricing.py- MEMBER_DISCOUNT = 0.10+ MEMBER_DISCOUNT = os.environ.get("MEMBER_DISCOUNT", 0.10)# production environmentMEMBER_DISCOUNT=0.15Marketing wants 15% for members,and to change it later without a release.Main → branch → pull request → review → CI → 6 serversMost incidents start as a small, reasonable change. This one is a single line.Members get 10% off. Marketing wants 15%, and wants to change it again later without waiting for a release. So the developer makesthe discount a setting: the code reads MEMBER_DISCOUNT from the environment and falls back to 0.10. Production sets it to 0.15. The lineis reviewed, passes every test, and is deployed to all 6 servers. Follow it the whole way.

The member discount becomes a setting read from the environment. Production sets it to 0.15.

The path

It passes every gate it meets

BranchPull requestreviewLint1 minUnit tests3 minIntegration5 minBuild image3 minStaging2 minProduction6 serversCI and staging: 14 minutes, all greenReview“Looks good: one line, default kept, tests pass.” Approved.The change passes every gate it meets: review, lint, two kinds of test and a staging deploy, in 14 minutes.Each gate checks something different: review reads the diff, lint the style, unit tests one function, integration tests the parts together,the build makes one image that every later stage uses, and staging runs it like production. All green. Note what every gate has incommon: none of them set MEMBER_DISCOUNT. They all ran with the default.

Review approves, then lint, unit and integration tests, an image and staging: 14 minutes, all green.

The release

Released everywhere at once, a mistake reaches everyone at once

Release v42 to all 6 servers togetherv42v42v42v42v42v42Checkout errorsreleaserollback9.2%0.2%Who failsevery member: 9% of checkouts, 36 a minuteFor how long9 min to notice + 3 min to roll back = 12 min, 432 failed checkoutsReleased everywhere at once, a bad change reaches every customer it can hurt within seconds.The new version goes to all 6 servers together. Checkout errors jump from 0.2% to 9.2%: every member who pays fails, and nobody else.It takes 9 minutes for the error alert and a person to connect it to the release, and 3 more to roll back. 432 checkouts fail in 12minutes. A release strategy decides how many people a mistake can reach before someone sees it.

Errors jump from 0.2% to 9.2%: every member fails to pay. 432 failed checkouts in 12 minutes.

The cause

An environment variable is always text

>>> os.environ.get("MEMBER_DISCOUNT", 0.10)'0.15'>>> 49.90 * (1 - "0.15")TypeError: unsupported operand type(s)for -: 'int' and 'str'What each place passed inWhereValueTypeTests, staging0.10floatProduction'0.15'strThe missing testMEMBER_DISCOUNT=0.15 → priceAn environment variable is always text. Production gave the code a string; every test gave it a number.os.environ only holds strings, so production read '0.15', and 1 - '0.15' is a TypeError on every member's checkout. In tests and staging thevariable was not set, so the code used its default, the number 0.10, and worked. Review read a correct-looking line; CI ran the default.No gate exercised the setting the change existed for. The fix is one call, float(), plus the test that sets the variable.

Production read the string '0.15'; tests used the number 0.10. One minus a string is a TypeError.

The fix

Roll back to the artefact you trust, then ship the fix to a few first

Roll back: redeploy the previous imagev41v41v41v41v41v41Same artefact as yesterday, no rebuild: 3 minThen v43: the fix and its test+ MEMBER_DISCOUNT = float(os.environ.get(...))+ def test_discount_from_env(monkeypatch):+ monkeypatch.setenv("MEMBER_DISCOUNT", "0.15")Released as a canary: 1 of 6 servers, 17% of traffic, watched 10 minv43v41v41v41v41v41Roll back with the artefact you already trust, then ship the fix to a few users first.Rolling back means redeploying v41, the exact image that ran yesterday, not rebuilding old code: 3 minutes and errors are back to 0.2%.The fix converts the setting with float() and adds a test that sets the variable the way production does. v43 then goes to one server first,about 17% of traffic. After 10 minutes with member checkouts at the normal error rate, it goes to the other 5.

Redeploy v41 in 3 minutes. Then v43 with float() and a test, as a canary on 1 of 6 servers.

The trade-off

A canary costs minutes and buys a smaller blast radius

StrategyTime per releaseA bad change reachesThis incidentAll at once14 min + deploy100% of users432 failed checkoutsCanary, 1 of 6, auto rollback+10 min of watchingabout 17%, for 5 minabout 30 failed checkoutsfailed checkouts, to scaleChosen: canary with automatic rollback for checkout; small internal tools still ship all at once.A canary costs minutes on every release and buys a smaller blast radius when one goes wrong.Watching one server for 10 minutes slows every release, and comparing its error rate with the others needs metrics per version. In return, a badchange reaches about 17% of users, and an automatic rollback at 5 minutes limits this incident to about 30 failed checkouts instead of 432.Feature flags go further, but each one is code to remove later. Choose per service: checkout earns the extra minutes.

A canary adds 10 minutes a release; this incident would have cost about 30 failed checkouts, not 432.

The path, fixed

Every gate catches a different mistake; this one slipped between two

1 Commit on a branchsmall, one purposeLesson 5: Git2 Pull request, reviewask: what does it read?Lesson 7: GitHub3 Config is textparse and test settingsLesson 4: data formats4 Tests that set ittest the real inputsLesson 12: tests5 One image, every stageroll back to the last oneLesson 1: artefacts6 Canary, then allwatch, then promoteLesson 11: pipelinesEach gate opens the lesson that teaches it. The two in colour are the ones this change slipped past.Every gate on the path catches a different kind of mistake. This one slipped between two of them.The process was not careless: the change was small, reviewed and tested. What was missing was a test of the input that mattered,a setting read as text. The release strategy decides how much that gap costs. Both are now part of the path, for every change.

Branch, review, config as text, tests that set it, one image, canary. Each gate links to its lesson.

A real case

At Knight Capital, one server missed the deployment

Knight’s order router ran on 8 servers. The new code reached 7.newnewnewnewnewnewnewold codeA reused flagmeant “retail liquidity program” on 7 servers,woke up retired Power Peg code on the 8thBefore the openIn about 45 minutesVolumeLost97 error emails, unread4 million executions, 154 stocks397 million sharesover $460 millionAt Knight Capital in 2012, one server missed a manual deployment, and a reused flag did the rest.Per the SEC, a technician copied new code to seven of the eight servers by hand; no second person reviewed it and no written procedurerequired one. The new code reused a flag that once switched on Power Peg, code that had sat unused for years. On the eighth server itwoke up. Lessons: deploy by pipeline, check every server runs the same artefact, remove dead code, and never reuse a flag.

Seven of eight servers got the new code. A reused flag woke old code on the eighth: over $460 million lost in about 45 minutes.

Lesson complete

You finished “One line to production”

Nicely done.

Open in a new tab