The path to production

One change passes six stations on its way to users

Editorchange the sourceVS Codelessons 2 to 4Repositorycommit and pushGit, GitHublessons 5 to 7Buildtest and packagea pipelinelessons 11 and 12Artefactone versioned fileimage, zip, packageDeployto an environmentCLI, Terraformlessons 9 and 10Serviceserving usersreached by its APIlesson 8One change passes six stations on its way to users. This course visits each.Source code is text. A build turns it into an artefact: one file that can be copied anywhere. Deployingputs the artefact where it can run, and a running copy is a service. Automating the path from commit toservice is called CI/CD: continuous integration and continuous delivery.

Source is built into an artefact, which is deployed and runs as a service.

Languages

Something must turn source code into instructions

CompiledC, C++, Go, RustSourceCompilerbefore runningMachine codeone file per platformProcessorBytecodeJava, C#SourceCompilerbefore runningBytecodethe same file anywhereVirtual machineJVM, .NET runtimeInterpretedPython, JavaScript, BashSourceInterpreterreads the source while the program runsProcessorSource code is written for people. Something must turn it into instructions for the machine.Compiled languages translate once, ahead of time: fast to run, and many errors show up at build time.Interpreted languages translate while running: there is nothing to build, but the machine that runs theprogram needs the interpreter installed, in the right version. The purple box is what you ship.

Compiled languages ship machine code. Interpreted languages ship the source.

Dependencies

Most of what you ship was written by someone else

Your servicethe code you wroterequestsurllib3 · certifi · idnacharset-normalizerflaskwerkzeug · jinja2 · itsdangerousclick · blinker · markupsafeboto3botocore · s3transfer · jmespathpython-dateutil · six · urllib3You chose 3 packages. 15 more came with them.Each dependency has dependencies of its own. A manifest (requirements.txt, package.json) lists what youchose. A lock file records the exact version of everything, so every build gets the same code. A flaw inany of them is a flaw in your service, which is why builds scan their dependencies.

Three packages you chose bring fifteen more. A lock file pins them all.

Dependencies · Going deeper · 1 of 2

Reading a version number

2MAJORbreaking changesread the notes before upgrading31MINORnew featuresexisting code keeps working0PATCHfixes onlybugs and security holesWritten in a manifestAcceptsThat is==2.31.0exactly this version2.31.0~=2.31.0patch updates2.31.0 up to, not including, 2.32>=2.31,<3minor and patch updates2.31.0 up to, not including, 3.0A version number is a promise about what changed.Semantic versioning has three parts. Take patch updates promptly, minor ones routinely and major onesdeliberately. With a lock file, each upgrade is a visible change that can be reviewed and undone.

Major may break things. Minor adds. Patch fixes.

Lock files · Going deeper · 2 of 2

Every ecosystem has a manifest, a lock file and a strict install

EcosystemManifest: your choicesLock file: exact versionsInstall exactly what the lock saysPython, pippyproject.tomlrequirements.txt (pinned)pip install --require-hashes -r requirements.txtPython, uvpyproject.tomluv.lockuv sync --lockedNode.js, npmpackage.jsonpackage-lock.jsonnpm ciGogo.modgo.sum (checksums)go build (never edits go.mod)Rust, CargoCargo.tomlCargo.lockcargo build --lockedTerraformrequired_providers.terraform.lock.hclterraform init -lockfile=readonlymanifest: flask>=3.0resolvelock: flask==3.1.0 and the 6 packages it needs, each with a hashThe manifest states a wish. The lock file records the fact. Commit both, and build from the lock.A range such as >=3.0 or ^1.4.2 gives a different answer next month. The lock pins every package, including those younever chose, usually with a hash, so a file swapped upstream under the same version is refused. npm install rewritesthe lock when it disagrees with the manifest; npm ci stops with an error instead. Pipelines use the strict form.

The manifest is a wish; the lock file is the fact. Build from the lock.

The artefact

Build once, and promote the same artefact

shop-api:1.4.2from commit 9f3c2e1Developmentruns shop-api:1.4.2DATABASE_URL=dev-dbLOG_LEVEL=debugsameTestruns shop-api:1.4.2DATABASE_URL=test-dbLOG_LEVEL=infosameProductionruns shop-api:1.4.2DATABASE_URL=prod-dbLOG_LEVEL=warnThe artefact is identical.Only the configuration differs:Build once. Promote the same artefact through every environment.Then what was tested is exactly what ships. Only configuration changes, supplied from outside asenvironment variables or a settings file, and never baked into the build. Secrets such as passwordscome from a secret store, not from the repository.

The same artefact runs everywhere. Only configuration changes.

Tags and digests · Going deeper

A tag is a label that can move; a digest names the exact bytes

$ sha256sum app.pyef91da4140d3a12d714e2fb94df72bfe3551873bb9a797aebf7397317d45ce31 app.py$ sha256sum app2.pya7d8387b803ea3c5cd951417e4f360203b99d431ffd7bc8455f64e4949e2382f app2.pyapp2.py has one more space than app.py.All 64 hex digits change.An image digest is the same idea:SHA-256 over the image manifest.ReferenceExampleCan it move?Use it forVersion tagshop-api:1.4.2yes: a push can reuse the namepeople, release noteslatestshop-api:latestyes: it is only the default tag namenothing in productionDigestshop-api@sha256:ef91da41...no: it is the hash of the contentdeploys and rollbacksA tag is a label that someone can move. A digest is the SHA-256 of the artefact, so it cannot.To promote "the same artefact", record the digest that passed the tests and deploy that digest to each environment.docker pull shop-api@sha256:... fetches exactly those bytes, whatever the tags say today. latest does not mean newest:it is whatever was last pushed without a tag. Registries can also be set to refuse overwriting a tag (immutable tags).

Deploy by digest: then every environment provably runs the same artefact.

Environments

Each environment answers one question

EnvironmentThe question it answersIts dataWho uses itDevelopmentdoes it work at all?inventedthe developerTestdoes it pass every automated test?generated for each runthe pipelineStagingdoes it behave under real conditions?like production, anonymisedtesters and the teamProductionare users getting what they need?realcustomersEach environment answers one question before the change moves on.A bug caught in development costs minutes; in production it costs customers. The gates betweenenvironments are automated tests, a review by a colleague and, before production, an approval. Stagingis only useful while it matches production closely.

Development, test, staging, production: each one a gate.

Runtime

"It works on my machine" is a runtime mismatch

Developer's laptopthe applicationrequests 2.31Python 3.12macOS✓ worksServerthe applicationrequests 2.25Python 3.9Linux✗ fails: versions differContainer image, on any hostthe applicationrequests 2.31Python 3.12any Linux host✓ the same everywhere"It works on my machine" is a runtime mismatch.A program needs more than its own code: the right language version, the right libraries and an operatingsystem that provides them. A container image packs all of that into the artefact, so the runtime travelswith the application.

A program needs its language version and libraries. A container carries them along.

CPU architecture · Going deeper

An image carries its libraries, not a kernel, and is built for one CPU

In the imagethe applicationlibraries: requests 2.31Python 3.12OS files: Debian user spaceSupplied by the hosta Linux kernelthe CPU: amd64 or arm64Image built forHost CPUResultlinux/amd64amd64 (x86-64)✓runslinux/arm64amd64 (x86-64)✗exec format errorlinux/arm64arm64✓runsboth, under one tageither✓the registry serves the matchBuild for both, under one tag$ docker buildx build --platform linux/amd64,linux/arm64 \ -t registry.example.com/shop-api:1.4.2 --push .An image carries its libraries and runtime, but borrows the kernel, and is compiled for one CPU type.A laptop with Apple silicon builds linux/arm64 images by default. Most cloud servers are amd64, where that image stopsat start with "exec format error" (typical output). Emulation hides the problem on the laptop, at several times therun time. The kernel is always the host's: on macOS and Windows, Docker runs a small Linux virtual machine to have one.

An arm64 image on an amd64 server fails with exec format error. Build for both.

Lesson complete

You finished “How software is built and shipped”

Nicely done.

Open in a new tab