Workflows

A workflow is a YAML file: events start jobs of steps

.github/workflows/ci.ymlon: pull_request: push: branches: [main]jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 - run: pip install -r requirements.txt - run: pyteston: the events that start the workflowany pull requestor a push to mainjobs: independent units, in parallel unless told otherwiseruns-on: a runner, a fresh virtual machine for this jobsteps: run in order, on that one machineuses: an action, a packaged step someone publishedrun: any shell commanda non-zero exit code fails the step and the jobA workflow is a YAML file in the repository: events start jobs, and jobs run steps.It is versioned and reviewed like the code it tests. This one is continuous integration: every pullrequest is installed and tested on a clean machine before anyone merges it.

on says when. jobs say where. steps say what.

Triggers · Going deeper

Five triggers cover most workflows, and each has its filters

on: push: branches: [main] paths: ['src/**'] pull_request: types: [opened, synchronize, reopened] schedule: - cron: '30 5 * * 1-5' workflow_dispatch: workflow_call:1pushfilter by branch, tag or path; with no filter, any push starts it2pull_requestthese three types are the default; a fork gets a read-only token, no secrets3schedulecron in UTC: 05:30, Monday to Friday. Every 5 minutes at most; may start late4workflow_dispatcha Run button, the API and gh workflow run; can take typed inputs5workflow_callmakes this file a reusable workflow that others call with uses:on: lists the events. The filters under each event decide which of them count.schedule and workflow_dispatch read the workflow file on the default branch only. Scheduled runs can be delayed whenGitHub is busy, so avoid the top of the hour. pull_request_target runs with secrets and a write token even for a fork:never check out and run the fork's code in it. A pull request that changes only docs/ does not match paths: src/**.

push, pull_request, schedule, workflow_dispatch and workflow_call start most workflows.

CI and CD

CI proves every change; CD ships the artefact that passed

lint20 stest3 minin parallelbuildthe artefact, 2 mindeployto stagingapprovalby a persondeployto productionContinuous integration: a verdict in 5 minContinuous delivery: the same artefact, onwardsCI proves every change. CD ships the artefact that passed.Jobs with no dependency run in parallel, and "needs" puts the others in order. When a job fails,those after it do not run, so a broken change cannot reach production. With a manual approvalbefore production this is continuous delivery; without one it is continuous deployment.

Lint and test, then build once, then deploy the same artefact.

Outputs and conditions · Going deeper

A value crosses from step to job through GITHUB_OUTPUT and needs

jobs: build: runs-on: ubuntu-latest outputs: version: ${{ steps.v.outputs.version }} steps: - id: v run: echo "version=1.4.2" >> "$GITHUB_OUTPUT" deploy: needs: build if: github.ref == 'refs/heads/main' runs-on: ubuntu-latest steps: - run: ./deploy.sh ${{ needs.build.outputs.version }}1A step writes name=valueit appends a line to the file named by $GITHUB_OUTPUT2The job publishes itoutputs: maps a job output to a step output, by step id3needs: orders the jobsand opens needs.build.outputs to the later job4if: decides whether to runa skipped or failed job also skips those that need itJobs share nothing: a step writes a value, its job publishes it, and a later job reads it through needs.For later steps of the same job, write to $GITHUB_ENV instead. Outputs are short strings; files travel as artefacts.To run a job even after a failure, for a clean-up or a notification, give it if: always() or if: failure().

Write name=value to $GITHUB_OUTPUT, publish it as a job output, read it through needs.

Secrets

A pipeline holds the keys to production: store none in it

In the workflow fileenv: KEY: "AbC123xyz..."✗neverreadable by everyone with therepository, and in its historyfor goodA stored secretenv: KEY: ${{ secrets.CLOUD_KEY }}✓acceptableencrypted in the settings,injected at run time andmasked in the logsA short-lived tokenpermissions: id-token: write✓bestthe cloud trusts the pipeline'sidentity and issues a tokenfor this run onlyA pipeline holds the keys to production. Store none of them in the file.Secrets live encrypted in the repository, organisation or environment settings, and are not handedto pull requests from forks. Better still is a token issued per run, because nothing long-livedexists to steal. Give each workflow only the permissions it needs.

Never in the file. A stored secret is acceptable. A per-run token is best.

Deployment

A canary reaches 1% of users before it reaches everyone

Share of traffic:version 2and version 1Stage 11%1,000 usersStage 210%10,000 usersStage 3100%100,000 usersA canary sends a small share of real traffic to the new version first.If version 2 is broken, 1,000 of 100,000 users notice instead of all of them, and sending them backtakes seconds. Each stage is held while the pipeline compares the error rate and response time ofthe two versions, and then promotes or rolls back by itself.

1%, then 10%, then everyone, with a check at every stage.

Deployment · Going deeper

Blue-green and rolling deployment

Blue-greenRouterBlue: version 1, idleGreen: version 2, livetwo complete environmentsall traffic switches at once, and back as fastcosts double while both existRollingv2v2v1v1servers are replaced a few at a timeno spare capacity needed; both versions serve togetherThree strategies, one aim: change production without an outage and with a way back.Blue-green switches everything at once between two environments. Rolling replaces servers inbatches. Canary shifts a growing share of traffic. All need the new and old versions to work sideby side for a while, which mostly constrains how the database may change.

Blue-green switches all at once. Rolling replaces in batches.

Artefacts

Each job starts clean, so artefacts carry files between

Job: buildrunner 1, a fresh machinedist/app.zipwiped when the job endsuploadArtefact storagekept with the rundownloadJob: deployrunner 2, another fresh machinedist/app.zipEvery job starts on a clean machine, so files must be carried from job to job.The clean start is what makes a pipeline reproducible: nothing is left over from the last run. Anartefact carries build output to later jobs and can be downloaded afterwards. A cache is different:it only saves time by keeping downloaded dependencies, and is never the only copy of anything.

A fresh runner per job. An artefact passes files on.

Artefacts · Going deeper · 1 of 2

Matrices and runners

strategy: matrix: python: ["3.10", "3.11", "3.12"] os: [ubuntu-latest, windows-latest]Python 3.10Python 3.11Python 3.12ubuntuone jobone jobone jobwindowsone jobone jobone job3 × 2 = 6 jobs, in parallelA matrix runs the same job once for every combination.Each gets its own runner. Note the quotes: unquoted, 3.10 would be read as the number 3.1. Hostedrunners are provided and wiped by GitHub. A self-hosted runner is your own machine, for specialhardware or a private network, and it is yours to patch and to secure.

A matrix multiplies one job across versions and systems.

Limits and defaults · Going deeper · 2 of 2

The limits that end a run, expire a file or evict a cache

Limit onValueWhat it meansOne job6 hours (5 days self-hosted)the job is cancelled; set timeout-minutes far lowerOne workflow run35 daysincludes waiting for runners and for approvalsA matrix256 jobs per runthe most that one run may generateArtefacts and logs90 days by defaultthen deleted; set retention-days on an uploadCache10 GB per repository by defaultunused for 7 days: removed. Over the limit: oldest go firstGITHUB_TOKENthe job, at most 24 hoursit stops working when the job endsscheduleevery 5 minutes at mostpublic repositories: paused after 60 days without activityThe defaults are generous. A stuck job burns six hours of runner time unless you set timeout-minutes.timeout-minutes goes on a job or on a step: ten minutes for a test job that normally takes three is a sensible ceiling.An artefact is output you need to keep with the run; a cache is only a speed-up and may vanish at any time.Figures are for GitHub-hosted runners on github.com at the time of writing; plans and self-hosted runners differ.

A job may run for 6 hours by default. Set timeout-minutes.

Lesson complete

You finished “CI/CD pipelines”

Nicely done.

Open in a new tab