Engineering craft is the set of practices that decide what a codebase costs to live in.
Every system in this course was built by a team, and then lived in for years. Writing the code is the small cost. Living in it is the large one. Craft is a chain of practices. Version control records what changed and why. Review puts a second reader on every change. Tests say it still does its job. A pipeline makes deploying boring. Telemetry shows what is really happening. And incidents get learned from and written down. What production teaches flows back into the next commit. And every link depends on its neighbours. A pipeline with no tests is theatre. Tests with no telemetry lie about reality. Telemetry that nobody owns is noise. This lesson walks the chain.
The practices stack · Going deeper
The practices and their feedback loops
None of this is layered on top of the code; it weaves through every commit. Break any link and the work upstream stops being trustworthy.
Git internals
Git names everything by its content, so history cannot be faked
A commit points to a snapshot and to its parent. Every object’s name is the hash of its bytes.
Version control first, and the picture underneath the commands. A commit points to a tree, which is a snapshot of a folder. The tree points to blobs, which are file contents. Each of these is named by a hash of its own bytes. Now change one byte in one file. Its contents are different, so its hash is different: a new blob. The tree lists that hash, so the tree changes too: a new tree. The file that did not change keeps its hash, and is simply shared. Nothing is copied. And the commit names the tree, so there is a new commit, which also records its parent. That is a history. Because every name is derived from content, nobody can alter an old commit without every later name changing. The history proves itself.
Git internals · Going deeper
The object store
Four object kinds: blob, tree, commit and tag, stored under .git/objects by hash (SHA-1 historically, SHA-256 available since Git 2.29). Branches, tags and HEAD are mutable pointers in .git/refs. A lockfile does the same job for dependencies: it pins the exact version of every package so every machine installs the same bytes.
Merging
A merge needs a person only where both sides changed a line
A three-way merge compares two versions against their common ancestor.
Two people edit the same file on two branches. How does Git combine them? It uses three versions, not two. The base: the last version both sides agreed on. Ours: someone added an exclamation mark to the greeting. Theirs: someone added a title argument. Take the first line. Compared with the base, only their side changed it. So Git takes their version without asking. The second line was changed by both sides, differently. Git cannot know whether the exclamation mark should survive the new title. It stops, marks both versions in the file, and a person decides. Without the common ancestor, every differing line would need a person. With it, only the true collisions do.
Merging · Going deeper
Merge, rebase and branching
Merge creates a commit with two parents and preserves both histories. Rebase replays commits onto the other tip: same patches, new hashes, a straight line. Trunk-based development keeps branches under a day and hides unfinished work behind flags; GitFlow keeps long-lived branches for versioned releases. Use --force-with-lease, never a plain force push, on shared branches.
Check yourself
The base file has lines A, B, C. Our branch changed line 1; their branch changed line 3. What does Git do when we merge?
Each line was changed on at most one side relative to the base, so both changes apply. A conflict needs the same region changed on both sides.
Code review
Review works when the change is small and the reply is fast
Code review: a second engineer reads every change before it merges.
Tests can tell you a change is correct. They cannot tell you it is right for this codebase: the wrong shape, the misleading name, the abstraction that will leak. That needs a second reader. And whether review works depends on two numbers. Size. A change of two hundred lines gets read line by line. A change of two thousand lines gets approved unread, because nobody can hold it in their head. And time. A review that comes back in four hours reaches an author who still has the context. One that comes back in three days reaches someone who has moved on, whose branch has gone stale, with the next change stuck behind it. Worse, slow review teaches people to send bigger changes. Keep both numbers small.
Code review · Going deeper
The pull-request lifecycle
Targets: a diff under about 300 lines, first response within 4 working hours, open less than a day. Comment, request changes and approve are three distinct verbs: only “request changes” blocks. At least one approver other than the author, and CI green before approval means anything.
Fast test suites
A test suite is only useful if it is fast enough to run
The test pyramid: many unit tests, fewer integration tests, a few end-to-end tests.
A test is code that runs your code and checks the result. With tests, every change comes with proof that yesterday’s behaviour still holds. But the shape of the suite matters more than its size. The healthy shape is a pyramid. Thousands of unit tests at the base, each checking one small piece with no network and no disk. Fewer integration tests. A handful that drive the whole system end to end. Look at the speeds. Milliseconds each at the bottom. Seconds to minutes each at the top. So this whole suite runs in thirty seconds, on every save. Turn it upside down: hundreds of slow end-to-end tests and few unit tests. Now the suite takes thirty minutes. It runs at night, if at all. And a test nobody runs protects nothing.
Fast test suites · Going deeper
Pyramid and ice-cream cone
Unit: one function or class, no I/O. Integration: a component with its real neighbours, such as a service with its database. End-to-end: the whole system through its public interface. Flaky slow tests get skipped under deadline, so the signal collapses exactly when it is needed.
Property-based testing
Give the machine a rule and let it hunt for the breaking input
Property-based testing checks an invariant against many generated inputs, then shrinks any failure.
There is a limit to ordinary tests. You write three examples: the empty list, one element, three elements. They pass. That proves the code works for the three inputs you thought of. A property test states a rule instead. Reverse a list twice and you get the original back. For every list. The machine then invents a hundred inputs and checks the rule on each. Suppose a bug slips in. The runner stumbles on a six-element list that breaks the rule. Now the clever part. It shrinks the input. Three elements: still fails. Two: still fails. One: passes. So the report shows you the two-element list, the smallest case that fails. Not a haystack. The needle.
Coverage and mutation testing
Coverage shows a line ran, not that anything checked it
Mutation testing breaks the code on purpose and counts how many breaks the tests catch.
How do you know your tests are any good? The usual number, coverage, only says which lines were executed. Here is a sharper question. Take a line of your code: if age is at least eighteen, allow. A tool changes one operator. At least becomes greater than. The code is now wrong for exactly one age. Run the tests against this broken version. Age thirty is allowed: still passes. Age ten is refused: still passes. No test tries eighteen. The broken version survives, and you have found a hole that coverage reported as fully covered. Do this for every operator. Here, eight of ten mutants were killed by some test and two survived: a score of eighty percent, which is about where a healthy suite begins.
Coverage and mutation testing · Going deeper
Killed and surviving mutants
Typical mutations: < to <=, + to -, a condition negated. Mutation score is killed over total. It is slow, so teams run it on changed files or nightly. Coverage is a presence test; mutation testing is an absence test.
Check yourself
A test calls total([1, 2, 3]) and asserts nothing. Coverage reports 100%. A mutation tool changes + to − inside total. What happens?
With no assertion, no result can make the test fail. The line is executed, which is all coverage measures, and never checked.
Continuous delivery and canaries
Ship small and often, and release to 1% before everyone
CI/CD builds and tests every commit on a clean machine. A canary exposes a release gradually.
It works on your laptop. That says almost nothing, because your laptop is not production. So every commit is built and tested automatically on a clean machine. Cheap checks first: the style check in seconds, unit tests in a few minutes, then the build, then integration tests. Out comes an artefact that is ready to deploy. Keep the whole run under about fifteen minutes, or people stop waiting for it. Then release it carefully. A canary sends one percent of real users to the new version. If the numbers hold, five percent. Then a quarter. Then everyone. And if the error rate or the slowest requests cross a limit at any step, traffic snaps back to the old version by itself. The goal of all this is that deploying becomes boring.
Continuous delivery and canaries · Going deeper · 1 of 3
A typical pipeline
Fail fast: cheap stages gate expensive ones. Deterministic: the same commit gives the same artefact on a clean runner. Speed comes from running suites in parallel and caching unchanged dependencies. Continuous delivery produces a deployable artefact on every green build; continuous deployment also rolls it out without a human.
Continuous delivery and canaries · Going deeper · 2 of 3
Blue-green and canary
Blue-green: two full environments, flip the load balancer, roll back in seconds, at double the infrastructure and with everyone exposed at once. Canary: a small share of traffic, metric gates, automatic rollback; harder for stateful changes. Both assume a backwards-compatible schema.
Continuous delivery and canaries · Going deeper · 3 of 3
Telemetry and the error budget
Metrics alert, logs explain one event, traces follow one request across services. A 99.9% objective over 28 days allows 40.3 minutes of failure; a 15-minute bad deploy leaves 25.3. At 99.99% the budget is 4 minutes. Lesson 12, Operations, covers this in full.
Safe schema migrations
You cannot un-drop a column: expand, migrate, then contract
Expand, migrate, contract keeps rollback possible while a database schema changes.
Rolling back code is easy: run the previous version. Rolling back a database is not. A column you dropped is gone. So schema changes are done in three steps. Say we are renaming a column. Expand. Add the new column beside the old one. The current version of the app does not even notice. Migrate. Copy the data across, and ship a version of the app that writes to both columns and reads the new one. Contract. Weeks later, when nothing reads the old column, drop it. Look at what this buys. All through the first two steps, the old column is still there and still being filled. If the new version misbehaves, you roll back and the old version simply works. Only the last step is irreversible, and you take it when the risk has passed.
Check yourself
In one release, a team ships app v2 and drops the old email column. v2 turns out to have a bug, so they roll the app back to v1. What happens?
Rolling back code does not roll back the schema. Keeping the old column until v1 is no longer a rollback target is the whole point of expanding first and contracting last.
Postmortems
After an outage, keep asking why until the answer is the system
A blameless postmortem descends from the symptom to the process gap that allowed it.
Things will break. What matters is whether the same thing breaks again. Here is a real-shaped example. The symptom: checkout slowed from eighty milliseconds to four seconds, for eighteen minutes. Why? A migration held a long lock on the orders table. You could stop here, revert the migration, and prevent this incident. Keep going. Why did it hold the lock? It updated a busy table in one big transaction. Why? The engineer copied the pattern from a small table, where it was fine. Why? The template does not distinguish busy tables from quiet ones. Why? Because there is no review step for schema changes on busy tables. That is a fact about the process, not about a person. Fix it there and you prevent the next twenty incidents.
Postmortems · Going deeper · 1 of 2
Five whys
The first why usually surfaces a code fact; the last, a process fact. The number five is approximate; the discipline is to go past the first plausible stop. Postmortems have a fixed form: summary, minute-by-minute timeline written first, contributing factors, and action items each with an owner.
Postmortems · Going deeper · 2 of 2
The phases of an incident
Mitigation stops the harm (roll back); resolution finds why the bad change passed. Teams optimise time to recovery. On-call means one primary and one backup, weekly handoffs, and every alert linked to a runbook. Lesson 12 covers incident response.
Design docs and decisions
Write down why, because the people who know will leave
An ADR, architecture decision record: one page of context, decision, status and consequences.
Code tells you what a system does. It never tells you why. So when a real decision is made, write one page. What was true that forced a choice. What we decided. What we are giving up. Give it a number and keep it next to the code. Two years later someone new asks: why this database and not that one? Without the record, three engineers give three answers, two of them wrong, and nobody can tell which. The people who decided have left. With the record, there is one answer, with a date and its reasons. And when the decision changes, you do not edit the old record. You write a new one that replaces it, so the history of the reasoning survives as well.
Design docs and decisions · Going deeper
ADRs and RFCs
Michael Nygard’s template (2011): context, decision, status, consequences. An RFC is negotiated before the work; an ADR is recorded after the decision. Use MUST, SHOULD and MAY precisely. An ADR never cited within six months is a candidate for archiving.
Seniority
Seniority is scope and judgement, not years
The career ladder: implement, design, define the problem, choose the right problems, redefine them.
Last, the people. What does it mean to grow as an engineer? A junior engineer takes a well-defined problem and implements a solution. A mid-level engineer takes a fuzzy problem and designs the solution. A senior engineer looks at an area, sees what should exist, and defines the problem. Notice the scope growing underneath: a commit, a feature, a subsystem. Up to here, the question is: how good is my own work? Then the role changes. A staff engineer chooses, from the dozen things the organisation could do, the two it should. A principal engineer changes what the organisation thinks its problems are. From staff on, the question is: how much do my decisions let other people get done?
Seniority · Going deeper
Scope against judgement
Each step is a change in what the role is. A healthy staff role crosses team boundaries, has a sponsor at director level and a written charter; a staff engineer spending 80% of the week in one team’s meetings is a tech lead with a bigger title.
Summary
Craft is feedback at every timescale, seconds to years
Each slower loop catches what the faster one cannot. Next: the frontier, what is changing underneath.
The whole lesson on one line of time. In seconds, unit tests catch wrong logic. In minutes, the pipeline catches a broken build. In hours, review catches a wrong design. Over days, a canary catches what only shows at scale. Over weeks, the error budget catches slow decay. And over years, decision records keep the reasons from being lost. Each loop catches what the faster one before it could not see. That completes how systems are built and kept. One lesson remains: the frontier, and what is changing underneath all of it.
Summary · Going deeper
The practices on one map
Conventions worth knowing: Semantic Versioning, Conventional Commits, OpenTelemetry for telemetry, and the Google SRE books for error budgets and postmortems. Pipeline credentials live in a secret manager: a leaked CI token is a leaked production credential.
Lesson complete
You finished “Engineering Craft”
Answer the questions along the way to see your score here.