Kinds of test

Many fast tests at the bottom, few slow ones at the top

Kind of testTestsEachAllEnd-to-endthe whole system, as a user meets it560 s5 minIntegrationseveral parts together, with a real database502 s100 sUnitone function, on its own50010 ms5 sMany fast tests at the bottom, few slow ones at the top.Here 500 unit tests finish in 5 s, while 5 end-to-end tests take 5 min. Fast tests run on everysave and every commit, so they catch mistakes while those are cheap. Slow tests prove that the partswork together, and are kept for the journeys that matter. A suite the other way up is soon ignored.

Unit tests are many and fast. End-to-end tests are few and slow.

Test doubles · Going deeper

A unit test replaces what is slow or outside your control with a double

health.pydef fetch_cpu(host): ... # asks the server, over the networkdef report(host): cpu = fetch_cpu(host) return f'{host}: {"warning" if cpu > 75 else "ok"}'test_health.pydef test_report_warns_above_75(monkeypatch): monkeypatch.setattr(health, 'fetch_cpu', lambda host: 87) assert health.report('web-01') == \ 'web-01: warning'✓1 passed in 0.01s, with no server and no networkDoubleWhat it isExampleStubreturns a fixed answerfetch_cpu always gives 87, as aboveMocka stub that also records how it was calledfetch_cpu.assert_called_once_with('web-01')Fakea small working replacementan in-memory database, a local file storeA double stands in for whatever is slow, costly or outside your control, so a unit test checks your logic alone.monkeypatch undoes the swap when the test ends. Replace at the edge of your code: the function that calls the network,the clock, the cloud. A double proves nothing about the real service: if its API changes, these tests stay green.Closing that gap is the job of the integration tests one level up the pyramid.

A test double keeps a unit test fast, and proves nothing about the real service.

Unit tests

A test is an executable statement of what the code must do

def state(cpu, limit=75): return 'warning' if cpu > limit else 'ok'def test_state_at_the_limit(): assert state(75) == 'ok' assert state(76) == 'warning'the code under testa test: its name says what must holdexactly at the limit: still okone above it: a warning✓pytest: 1 passed in 0.01sLater, someone changes > to >=✗FAILED assert state(75) == 'ok'caught in the pipeline, before the mergeA test is an executable statement of what the code must do.It runs the code on a known input and asserts on the result. Good tests probe the edges, wheremistakes live: the limit itself, zero, an empty list, a missing file. Once written, a test guardsthat behaviour for good. Whoever breaks it, including you in a year, finds out within minutes.

Call the code with a known input and assert on the result.

Parametrised tests · Going deeper

One test function, many cases: each passes or fails under its own name

test_health.py@pytest.mark.parametrize('cpu, expected', [ (0, 'ok'), (75, 'ok'), (76, 'warning'), (100, 'warning'),])def test_state(cpu, expected): assert state(cpu) == expectedThen someone changes > to >= in state():only the case at the limit, 75, fails.pytest 9, terminal 58 columns wide$ python3 -m pytest -v...test_health.py::test_state[0-ok] PASSED [ 25%]test_health.py::test_state[75-ok] PASSED [ 50%]test_health.py::test_state[76-warning] PASSED [ 75%]test_health.py::test_state[100-warning] PASSED [100%]=================== 4 passed in 0.01s ====================$ python3 -m pytest -q...FAILED test_health.py::test_state[75-ok] - AssertionErr...1 failed, 3 passed in 0.01sOne test function, four cases. Each case runs, passes or fails, and is named by its own inputs.The two cases that matter sit side by side: 75 and 76, either side of the limit. Adding a case is one line, so edgecases get written. Without parametrize, four asserts in one test stop at the first failure and hide the rest.pytest -k "76 or 100" runs only the cases whose name matches (2 passed, 2 deselected); -x stops at the first failure.

parametrize runs one test once per case and reports each separately.

Quality gates

Machines check what they can, so reviewers can think

GateFor exampleCatchesTakesFormatterblack, prettierinconsistent layout, which it simply fixes1 sLinterruff, eslintunused names, risky patterns, likely bugs2 sType checkermypy, tsctext passed where a number is expected10 sTestspytest, jestbehaviour that is wrongminutesSecurity scanof dependencies and secretsvulnerable packages, leaked keysminutesThen a person reviews: is this the right change, is it clear, and what is missing?Machines check what machines can check, so that reviewers can think.The first three are static analysis: they read the code without running it. Run them in the editorand again in the pipeline. The earlier a mistake is caught, the less it costs: seconds in the editor,minutes in the pipeline, an incident in production.

Format, lint, types, tests, scan. Then a person.

Quality gates · Going deeper · 1 of 2

What test coverage does and does not tell you

def ship(order): if order.total > 1000: require_approval(order) pack(order) send(order)never run by any test: the large-order path is uncheckedCoverage: 4 of 5 lines, 80%Coverage shows which lines the tests ran. It cannot show that they were checked.The red line has never been executed by a test, which is worth knowing. But a line can run withoutanything being asserted about it, so high coverage is no proof of quality. Use it to find untestedcode, not as a number to hit.

Coverage finds untested lines. It does not prove the tested ones are right.

Code review · Going deeper · 2 of 2

What a reviewer looks for, once the machines are done

Review design and behaviour first, style last: the linter has already had its say.

The gates have checked layout, types and tests. A person is there for what a machine cannot judge. Google's engineering practices list what to look at, roughly in this order.

Look for, in this order

  1. Design. Does the change belong here, and does it fit the rest of the system?
  2. Functionality. Does it do what the author intended, including at the edges and under concurrency?
  3. Complexity. Could a reader understand it quickly? Is it solving a problem that does not exist yet?
  4. Tests. Are they there, and would they fail if the code were wrong?
  5. Naming and comments. Names say what a thing is. Comments say why, not what.
  6. Style and documentation. Left to the formatter and the style guide. Docs updated where behaviour changed.
1 business daythe longest a review request should wait for a first response
Nit:prefix for a comment the author is free to ignore
Better, not perfectapprove once the change clearly improves the code overall

As the author

  • Keep it small. One purpose per pull request. A small change gets read; a large one gets skimmed.
  • Say what and why. The description should let a reviewer start without asking.
  • Answer every comment. With a change or a reason. If the code needed explaining, improve the code.
Tone: Review the code, not the person. "This loop reads the file twice" is a comment; "you are careless" is not.

Lesson complete

You finished “Testing and code quality”

Nicely done.

Open in a new tab