The practices stack

Each practice makes the one before it trustworthy

Version controlwhat changed, whyReviewa second readerTestsstill does its jobCI/CDboring deploysTelemetrywhat is happeningIncidentslearn, write it downwhat production teaches goes into the next commita pipeline without tests is theatretests without telemetry lie about realitytelemetry nobody owns is noiseWriting code costs once. Living in it costs for years, and craft is what keeps that second cost down.Each practice makes the one before it trustworthy. None is added on afterwards: they run through every commit.

Engineering craft is the set of practices that decide what a codebase costs to live in.

The practices stack · Going deeper

The engineering-craft practices and the feedback loops that connect them Version control Code review Testing pyramid CI / CD Observ- ability Incident mgmt Decisions (ADR) Career ladder CI gates merges review feeds commit hygiene SLO miss pages on-call postmortem produces new tests + runbooks ADRs anchor the why for everyone above Feedback loops, not phases a commit triggers CI; CI gates review; review preserves intent; intent feeds ADRs; ADRs survive the people who wrote them

The practices and their feedback loops

None of this is layered on top of the code; it weaves through every commit. Break any link and the work upstream stops being trustworthy.

Git internals

Git names everything by its content, so history cannot be faked

commit a1f3tree 7c2etree 7c2eone folder snapshotblob e69dapp.py, beforeblob 3b18README.mdblob 5d07app.py, after1 byte changedtree f20aone entry differsso a new treeunchanged file: stored once, sharedcommit c94btree f20a · parent a1f3parentso a new commitGit names every object by the hash of its contents, so history cannot change without its name changing.A commit should be one change you can explain in a sentence. Branches and tags are only movable pointers to commits;publish the latest hash and any edit to the past becomes detectable.

A commit points to a snapshot and to its parent. Every object’s name is the hash of its bytes.

Git internals · Going deeper

Git object model — a 2-file repo with one commit, showing blob, tree, commit objects blob a3f1c2… blob 9d4e0b… contents of README.md contents of main.py tree 7c1aa5… 100644 README.md → a3f1c2… 100755 main.py → 9d4e0b… commit c011fd… tree 7c1aa5… parent (none) author Alice 2026-04-01 message: init content-addressed: tamper-evident, deduplicated, garbage-collectable change one byte anywhere and every dependent SHA changes

The object store

Four object kinds: blob, tree, commit and tag, stored under .git/objects by hash (SHA-1 historically, SHA-256 available since Git 2.29). Branches, tags and HEAD are mutable pointers in .git/refs. A lockfile does the same job for dependencies: it pins the exact version of every package so every machine installs the same bytes.

Merging

A merge needs a person only where both sides changed a line

Base: the common ancestordef greet(name): return "Hello, " + nameOurs: added punctuationdef greet(name): return "Hello, " + name + "!"Theirs: added a titledef greet(name, title): return "Hello, " + title + " " + nameline 1: changed on one side onlydef greet(name, title):Git takes the changed side by itselfline 2: changed on both sides<<<<<<< ours "Hello, " + name + "!">>>>>>> theirs "Hello, " + title + " " + namea conflict: Git stops, and a person decides what was meantA merge compares both sides with their common ancestor. Only lines changed on both sides need a person.Merge keeps both histories and joins them; rebase replays your commits on top for a straight line. The files end up the same.Branches that live less than a day keep conflicts small.

A three-way merge compares two versions against their common ancestor.

Merging · Going deeper

Merge versus rebase — same starting graph, two outcomes Merge — preserves topology Rebase — linearises history A B M main merge commit M has two parents · feature SHAs unchanged A' B' main A and B replayed onto the new tip · new SHAs · parallel timeline erased

Merge, rebase and branching

Merge creates a commit with two parents and preserves both histories. Rebase replays commits onto the other tip: same patches, new hashes, a straight line. Trunk-based development keeps branches under a day and hides unfinished work behind flags; GitFlow keeps long-lived branches for versioned releases. Use --force-with-lease, never a plain force push, on shared branches.

Check yourself

baseABCoursA2BCtheirsABC2

The base file has lines A, B, C. Our branch changed line 1; their branch changed line 3. What does Git do when we merge?

Code review

Review works when the change is small and the reply is fast

size of the change, in lines, to scale200 lines: read line by line2,000 lines: approved unreadtime to the first review, to scale4 hours: the author still has the context3 days: the author has moved on, the branch needs rebasing, and the next change waits behind itReview catches what tests cannot: a design that is wrong for this codebase. It works only small and fast.Slow review trains people to batch bigger changes, which are reviewed more slowly still. Block a merge only for what is wrongor unsafe and mark the rest as suggestions. Every review also teaches a second person that part of the code.

Code review: a second engineer reads every change before it merges.

Code review · Going deeper

The pull-request lifecycle and its time budget push branch CI runs review address approve + merge deploy PR opens ≈5–15 min ≈4 h target ≈2 h CI on main re-review on changes Targets that keep the loop fast PR diff under ≈300 lines (review thoroughness drops past this) PR open less than 1 day end-to-end first review response under 4 h during working hours approve / request-changes / comment — three distinct verbs at least one approver other than the author CI must be green before "approve" means anything

The pull-request lifecycle

Targets: a diff under about 300 lines, first response within 4 working hours, open less than a day. Comment, request changes and approve are three distinct verbs: only “request changes” blocks. At least one approver other than the author, and CI green before approval means anything.

Fast test suites

A test suite is only useful if it is fast enough to run

end-to-endintegrationunit tests: thousandsthe whole suite runs in 30 seconds: on every saveseconds tominutes each0.1 to 1 s eachmilliseconds eachend-to-end: hundredsintegrationunit teststhe suite takes 30 minutes: nightly, if at allThe only tests that survive a deadline are the ones fast enough to keep running.Many fast unit tests, fewer integration tests, a handful of end-to-end tests. Invert the shape and the suite turns slow andflaky, and people stop running it. Beware the test that mocks everything: it only tests the mocks.

The test pyramid: many unit tests, fewer integration tests, a few end-to-end tests.

Fast test suites · Going deeper

Testing pyramid (healthy) versus ice-cream cone (anti-pattern) Healthy pyramid Ice-cream cone Unit · ≈70% Integration · ≈20% e2e · ≈10% ≈ms each ≈100ms–1s ≈10s–min Unit · few Integration · some e2e · many slow per test flaky Why the shape matters slow tests at the top get tagged @skip first; flaky tests erode the trust that makes any test useful

Pyramid and ice-cream cone

Unit: one function or class, no I/O. Integration: a component with its real neighbours, such as a service with its database. End-to-end: the whole system through its public interface. Flaky slow tests get skipped under deadline, so the signal collapses exactly when it is needed.

Property-based testing

Give the machine a rule and let it hunt for the breaking input

example test: three inputs you thought of[ ][1][1,2,3]property test: one rule, 100 inputs the machine inventsreverse(reverse(xs)) == xsa bug slips in; the runner finds an input that breaks the rule123456fails123still fails12still fails1passessmallest failurethen it shrinks the input, and reports[1, 2]: the smallest case that still fails,not the long list it tried firstState a rule that must hold for every input, and let the machine hunt for the input that breaks it.Example tests miss the case nobody thought to write: the empty list, the duplicate, the odd character. The cost is decidingwhat must always be true, not only what is true for one case.

Property-based testing checks an invariant against many generated inputs, then shrinks any failure.

Coverage and mutation testing

Coverage shows a line ran, not that anything checked it

your codeif age >= 18: allow()the tool changes one operatorif age > 18: allow()your tests, run against the mutantage 30 is allowedstill passesage 10 is refusedstill passesno test tries age 18, so the mutant survives: a hole in the suiteall 10 mutantsKKKKKKKKSS8 killed, 2 survived: a mutation score of 80%, the healthy floorCoverage says a line ran. Mutation testing says whether any test would notice if the line were wrong.A function with 100% line coverage and no assertions has 0% verification. Each surviving mutant points at the exacttest that is missing.

Mutation testing breaks the code on purpose and counts how many breaks the tests catch.

Coverage and mutation testing · Going deeper

Mutation testing — mutate one operator, rerun the suite, count survivors original code + tests if (a < b) return a; tests: pass coverage: 100% mutate mutant A — killed if (a <= b) return a; tests: 1 fail → test caught it mutant B — survived if (a > b) return a; tests: pass → coverage gap Mutation score = killed / (killed + survived) under 60% — coverage is theatre 80–95% — solid suite 95%+ high-assurance cost: 10–100× the suite's wall-clock; runs nightly, not on every PR

Killed and surviving mutants

Typical mutations: < to <=, + to -, a condition negated. Mutation score is killed over total. It is slow, so teams run it on changed files or nightly. Coverage is a presence test; mutation testing is an absence test.

Check yourself

total: adds with +mutant: adds with −test: calls total([1,2,3])no assertion

A test calls total([1, 2, 3]) and asserts nothing. Coverage reports 100%. A mutation tool changes + to − inside total. What happens?

Continuous delivery and canaries

Ship small and often, and release to 1% before everyone

CommitLint20 sUnit tests3 minBuild2 minIntegration5 minArtefactready to deployon a clean machine, cheapest checks first, the whole run under about 15 minutesCanaryreal traffic, on a dial1% of users5%25%everyoneerror rate or slowest requests cross a limit: traffic snaps back by itselfMake deploys small, frequent and reversible, and release to 1% of users before everyone.Ten deploys a day means ten small changes, each revertible by name. A feature flag separates deployed from released: codecan sit in production switched off. Blue-green is the simpler switch: two full environments and one load balancer.

CI/CD builds and tests every commit on a clean machine. A canary exposes a release gradually.

Continuous delivery and canaries · Going deeper · 1 of 3

A typical CI/CD pipeline — sequential gates with parallel fan-out commit lint unit build integration staging e2e prod ≈20s ≈3 min ≈2 min ≈5 min deploy ≈10 min canary Properties of a good pipeline fail fast — cheap stages first (lint, unit) gate the expensive ones (e2e, deploy) deterministic — same commit produces the same artefact, every time, on a clean runner total wall-clock under ≈15 min — past that, devs context-switch deploy is reversible — rollback is a button, not a rebuild feature flag = deploy without release code shipped flag = off flag flipped feature live

A typical pipeline

Fail fast: cheap stages gate expensive ones. Deterministic: the same commit gives the same artefact on a clean runner. Speed comes from running suites in parallel and caching unchanged dependencies. Continuous delivery produces a deployable artefact on every green build; continuous deployment also rolls it out without a human.

Continuous delivery and canaries · Going deeper · 2 of 3

Blue-green and canary deployment patterns compared Blue-green Canary load balancer blue · serving v1.0 · 100% green · idle v1.1 · 0% 1. deploy v1.1 to green (idle) 2. smoke-test green from inside 3. flip LB: green serves, blue idles 4. rollback = flip LB back (seconds) load balancer stable · 95% v1.0 canary · 5% v1.1 1. send 5% of traffic to canary 2. watch error rate, p99, saturation 3. ramp: 5% → 25% → 50% → 100% 4. auto-rollback on metric breach Trade-offs blue-green — instant cutover; double infra cost; one bad version exposed to 100% canary — partial blast radius; needs metric-driven gates; harder for stateful changes both presuppose backwards-compatible schema (expand-migrate-contract)

Blue-green and canary

Blue-green: two full environments, flip the load balancer, roll back in seconds, at double the infrastructure and with everyone exposed at once. Canary: a small share of traffic, metric gates, automatic rollback; harder for stateful changes. Both assume a backwards-compatible schema.

Continuous delivery and canaries · Going deeper · 3 of 3

SLI versus SLO over a 28-day window, with error budget burn-down 100% 99.95 99.9 99.8 day 0 day 28 SLO: 99.9% SLI (measured) budget exhausted · freeze Error budget = (1 − SLO) × total events in window spent on: risky deploys, migrations, chaos drills, experimental rollouts when burnt: stop new features, harden the existing system, restore the budget for next window

Telemetry and the error budget

Metrics alert, logs explain one event, traces follow one request across services. A 99.9% objective over 28 days allows 40.3 minutes of failure; a 15-minute bad deploy leaves 25.3. At 99.99% the budget is 4 minutes. Lesson 12, Operations, covers this in full.

Safe schema migrations

You cannot un-drop a column: expand, migrate, then contract

1 Expandadd the new column, keep the oldemailprimary_emailapp v1 writes email2 Migratecopy the data, write to bothemailprimary_emailapp v2 writes both, reads new3 Contractweeks later, drop the old columndroppedprimary_emailapp v2 writes primary_emailrolling the app back to v1 is safe: the old column is still there and still filledno way back from hereA dropped column cannot be rolled back, so change a schema in three steps and delete last.During the window the database speaks both versions, so the previous release still runs. A pipeline that can deploy butcannot roll back is optimistic, not reliable: keep the previous artefact ready to load in seconds.

Expand, migrate, contract keeps rollback possible while a database schema changes.

Check yourself

table nowprimary_emailemail was droppedapp v1reads email

In one release, a team ships app v2 and drops the old email column. v2 turns out to have a bug, so they roll the app back to v1. What happens?

Postmortems

After an outage, keep asking why until the answer is the system

symptomcheckout p99 jumped from 80 ms to 4 s for 18 minuteswhy?the orders table held a long row lock during a migrationcodewhy?the migration backfilled a hot table in one transactiondesignwhy?the engineer copied the pattern from a small table’s migrationteamwhy?the migration template does not tell hot tables from coldtoolingwhy?no review step exists for schema changes on busy tablesprocessfix here: revert itprevents this incidentfix here: add the gateprevents the next twentyKeep asking why until the answer is about the system. That is where one fix prevents the most.Blameless: assume people acted reasonably on what they knew, so the outage is a system bug. Punish the person and the nextmistake gets hidden. Every action item needs an owner and a date, or the write-up changed nothing.

A blameless postmortem descends from the symptom to the process gap that allowed it.

Postmortems · Going deeper · 1 of 2

Five whys — descending from a surface symptom to a systemic cause Symptom why? why? why? why? why? checkout p99 jumped from 80 ms to 4 s for 18 minutes the orders table held a long row-level lock during a migration the migration ran a backfill in one transaction over a hot table the engineer copied a pattern from a smaller table's migration the migration template doesn't distinguish hot from cold tables no review step exists for schema changes against high-QPS tables code fact code fact design fact team fact tooling fact process fact

Five whys

The first why usually surfaces a code fact; the last, a process fact. The number five is approximate; the discipline is to go past the first plausible stop. Postmortems have a fixed form: summary, minute-by-minute timeline written first, contributing factors, and action items each with an owner.

Postmortems · Going deeper · 2 of 2

The four phases of an incident, with the MTTR budget marked detect respond mitigate resolve alert fires on-call paged user impact ends root cause fixed MTTR (target: hours, not days) postmortem follows resolve · written within 5 working days · timeline first, analysis second An automated alert beats a customer email every time. Mitigation comes before resolution: roll back or flag off first, find the cause later.

The phases of an incident

Mitigation stops the harm (roll back); resolution finds why the bad change passed. Teams optimise time to recovery. On-call means one primary and one backup, weekly handoffs, and every alert linked to a runbook. Lesson 12 covers incident response.

Design docs and decisions

Write down why, because the people who know will leave

ADR-0007 Use Postgres for the orders serviceStatusacceptedContextorders need transactions across tablesDecisionPostgres; Redis only as a cacheConsequencesslower reads; one database to operateone page, under 500 words, kept in the repository“Why Postgres and not Redis?”asked two years later, by someone newNo recordthree people, threeanswers, two wrongADR-0007one dated answer,with its reasonsADR-0007 acceptedADR-0023 replaces itnever edited: a new record supersedes the old oneCode records what the system does. Only writing records why, and the people who knew will leave.An ADR captures a decision after it is made; an RFC is the longer proposal argued over before code is written. Link themfrom the code and from pull requests, or nobody reads them.

An ADR, architecture decision record: one page of context, decision, status and consequences.

Design docs and decisions · Going deeper

An ADR card: title, status, context, decision, consequences ADR-0007 · Use PostgreSQL for the transactional store status: accepted · 2026-03-14 Context Decision Consequences We need a primary store for billing and account data. Requirements: strong consistency, transactional joins across 5–10 tables, < 50 ms p99 read latency, ≈2k QPS sustained, point-in-time recovery. Team has prior PostgreSQL experience; prior MySQL outage in 2024. Adopt PostgreSQL 16 in primary-replica configuration. Use logical replication for the analytics warehouse. No multi-master. + proven ACID semantics; rich query planner; team familiarity. + extensions (pgvector, PostGIS) cover known future needs. − vertical scaling ceiling at ≈16 TB; sharding will require ADR-NNNN. − logical replication adds operational surface vs. physical-only. − commits to the SQL dialect and admin tooling for years. one page · immutable once accepted · supersede with a new ADR, never edit

ADRs and RFCs

Michael Nygard’s template (2011): context, decision, status, consequences. An RFC is negotiated before the work; an ADR is recorded after the decision. Use MUST, SHOULD and MAY precisely. An ADR never cited within six months is a candidate for archiving.

Seniority

Seniority is scope and judgement, not years

Juniorimplements solutionsa commitMiddesigns solutionsa featureSeniordefines problemsa subsystemStaffpicks the right problemsa systemPrincipalredefines the problemthe organisationup to senior the question is:how good is my own work?from staff on the question is: how muchdo my decisions let other people get done?more of the value comes from judgement, less from execution; the label under each step is the scope of impactThe ladder measures the scope of your impact and how much of it is judgement. Not years served.Junior to mid is becoming reliable; mid to senior, autonomous; senior to staff, deciding what that autonomy should aim at.

The career ladder: implement, design, define the problem, choose the right problems, redefine them.

Seniority · Going deeper

The career ladder plotted as scope of impact against share of work that is judgement org-wide single PR scope of impact execution judgement share of value created by judgement → Junior Mid Senior Staff Principal implements solutions designs solutions defines problems defines the right problems redefines what "problem" means artefacts: commit · feature · subsystem · system · org-wide convention audience: code reviewer · tech lead · adjacent team · org leadership · industry

Scope against judgement

Each step is a change in what the role is. A healthy staff role crosses team boundaries, has a sponsor at director level and a written charter; a staff engineer spending 80% of the week in one team’s meetings is a tech lead with a bigger title.

Summary

Craft is feedback at every timescale, seconds to years

Secondsunit testson every savewrong logicMinutesthe pipelineunder 15a broken buildHoursreviewreply within 4a wrong designDayscanary ramp1% to everyonewrong at scaleWeekserror budget28-day windowslow decayYearsdecision recordsoutlive the teamlost reasonsthe row beneath: what each loop catches that the faster one before it could not seeCraft is feedback at every timescale, from the test that runs on save to the record read years later.That completes how systems are built and kept running. The last lesson looks ahead, at what is changing underneath them.

Each slower loop catches what the faster one cannot. Next: the frontier, what is changing underneath.

Summary · Going deeper

The engineering-craft practices and the feedback loops that connect them Version control Code review Testing pyramid CI / CD Observ- ability Incident mgmt Decisions (ADR) Career ladder CI gates merges review feeds commit hygiene SLO miss pages on-call postmortem produces new tests + runbooks ADRs anchor the why for everyone above Feedback loops, not phases a commit triggers CI; CI gates review; review preserves intent; intent feeds ADRs; ADRs survive the people who wrote them

The practices on one map

Conventions worth knowing: Semantic Versioning, Conventional Commits, OpenTelemetry for telemetry, and the Google SRE books for error budgets and postmortems. Pipeline credentials live in a secret manager: a leaked CI token is a leaked production credential.

Lesson complete

You finished “Engineering Craft”

Answer the questions along the way to see your score here.

Open in a new tab