CI/CD

Every commit becomes a tested image, then a release

Commitpush to GitBuildmake the imageTestunit tests, scansContinuous integration: every commit is built and testedPublishimage to a registryDeployroll out to the clusterContinuous delivery: always ready to releaseA pipeline turns every commit into a tested image, and then into a running release.Continuous integration (CI) builds and tests every small change, so problems surface in minutes.Continuous delivery (CD) keeps a tested image ready to release, with a person approving.Continuous deployment removes that approval: every change that passes goes live.

Continuous integration builds and tests. Continuous delivery keeps it ready to release.

CI/CD · Going deeper

The three terms and the tools

The three termsContinuous integrationEvery change is merged, built and tested automaticallyContinuous deliveryEvery passing change can be released; a person presses the buttonContinuous deploymentEvery passing change is released, with no buttonTools the exam may nameJenkins, GitHub Actions, GitLab CIGeneral CI/CD systems that run outside or beside the clusterTektonPipelines defined as Kubernetes objects, run as podsArgo WorkflowsA workflow engine that runs each step as a podArgo CD, FluxGitOps delivery: the cluster follows a Git repositoryArgo Rollouts, FlaggerProgressive delivery: canary and blue-green releasesCI produces the image. CD, increasingly through GitOps, puts it in the cluster.

Delivery keeps a manual approval. Deployment does not.

GitOps

The cluster pulls its desired state from Git

PUSH: THE PIPELINE REACHES INTO THE CLUSTERCI pipeline,holds the cluster keysappliesclustera change made by hand goes unnoticedPULL: AN AGENT IN THE CLUSTER FOLLOWS GITGitthe desired statepullsAgentArgo CD or Flux,inside the clusterdrift is detected and put backGitOps: Git holds the desired state, and an agent in the cluster keeps pulling it in.To change production you change Git, through a reviewed pull request. The agent notices andapplies it. Nothing outside the cluster needs the cluster's keys, every change has an authorand a history, and rolling back is reverting a commit.

Git holds the desired state. An agent in the cluster pulls it and corrects drift.

GitOps · Going deeper

The GitOps principles, Argo CD and Flux

The four OpenGitOps principles1. DeclarativeThe system's desired state is expressed declaratively2. Versioned and immutableIt is stored with a complete version history3. Pulled automaticallySoftware agents pull the desired state from the source4. Continuously reconciledAgents keep comparing actual with desired, and correct itThe two CNCF graduated GitOps toolsArgo CDAn "Application" object points at a repository; has a web UI showing sync statusFluxA set of controllers (source, kustomize, helm); driven by objects and the CLIIn sync, out of syncWhether the cluster matches Git; "drift" is the differenceInfrastructure as codeThe same idea for cloud resources: Terraform, OpenTofu, CrossplaneGitOps is the Kubernetes reconciliation loop, extended to start from Git.

Declarative, versioned, pulled, reconciled.

Check yourself

Gitreplicas: 3GitOps agent comparesClusterreplicas: 5someone scaled the Deployment by hand

What does the agent do next?

Packaging

Helm fills in templates. Kustomize patches manifests

Charttemplates with blanks+Valuesreplicas: 3, tag: 1.4helm installfills in the blanksReleasecan be rolled backBaseplain manifests+Overlaypatches for productionkubectl apply -kmerges the twoManifestsno templates usedHelm fills in templates. Kustomize patches plain manifests. Both give one app many settings.Helm is the package manager: a chart is a package you can share, install and upgrade. Kustomizeis built into kubectl: a base, with a small overlay of changes on top for each environment.

Helm: chart plus values gives a release. Kustomize: base plus overlay gives manifests.

Packaging · Going deeper

Helm and Kustomize terms

HelmChartA package: templates, default values and metadataReleaseOne installation of a chart in a cluster; each upgrade is a new revisionRepositoryWhere charts are published; Artifact Hub is the public search for themhelm upgrade, helm rollbackMove a release to a new revision, or back to an earlier oneKustomizekustomization.yamlLists the manifests to use and the changes to makeBaseThe shared manifestsOverlayPer-environment patches: names, labels, replicas, imageskubectl apply -kBuilds and applies; no extra tool to installBoth are graduated or built in, and both work under Argo CD and Flux.

Chart, release, repository; base and overlay.

Release strategies

Rolling, blue-green and canary move traffic differently

Share of traffic on the new version, as time passesRolling updatepods replaced a few at a time0%33%67%100%Blue-greentwo full copies, one switch0%0%100%100%Canarya small share first, then more0%5%25%100%Three ways to release: gradually, all at once with a spare, or to a few users first.A rolling update is the Deployment default, and both versions serve for a while. Blue-greenswitches all traffic in one step, so going back is instant, at double the cost. A canary watchesa small share on the new version before sending more.

Rolling replaces step by step. Blue-green switches at once. Canary starts small.

Release strategies · Going deeper

Strategies compared

StrategyHow it worksExtra capacityGoing backRecreatestop all the old, start the newnone; there is downtimeredeploy the oldRolling updatereplace pods step by stepa little (maxSurge)kubectl rollout undoBlue-greenrun both, switch the Servicea full second copyswitch back: instantCanarya small share of traffic firsta few extra podssend the share backA/B testingsplit users to compare versionsa few extra podsa product decisionSettings and toolsmaxSurge, maxUnavailableHow many pods a rolling update may add, or take away, at one timeArgo Rollouts, FlaggerAutomate canary and blue-green, and roll back when metrics turn badA Deployment knows Recreate and RollingUpdate. The others need a mesh, a gateway or a tool.

Capacity and rollback differ. Argo Rollouts and Flagger automate them.

Check yourself

0%5%25%100%share of traffic on the new version, as time passes

Which release strategy is this?

Debugging a rollout

A bad release stalls, and one command undoes it

DeploymentwebReplicaSet, revision 1image web:1.4podpodpodReplicaSet, revision 2image web:1.5ImagePullBackOffthe rollout stalls, andthe old pods still servekubectl rollout undo deployment/webback to revision 1, and revision 2 is scaled to zeroA Deployment keeps its old ReplicaSets, so a bad release can be undone with one command.Each change to the pod template creates a new ReplicaSet and a new revision. A rolling update onlyremoves old pods as new ones become ready, so a broken image stalls the rollout without an outage.kubectl rollout status shows the stall, history lists the revisions, and undo goes back.

Each revision is a ReplicaSet. kubectl rollout undo returns to the last one.

Debugging a rollout · Going deeper · 1 of 2

Rollout and debugging commands

CommandWhat it is forkubectl rollout status deploy/webWatch a rollout until it finishes or stallskubectl rollout history deploy/webList the revisionskubectl rollout undo deploy/webReturn to the previous revisionkubectl rollout restart deploy/webReplace every pod, for example to pick up a new Secretkubectl debug -it web --image=busyboxAdd an ephemeral container with tools to a running podkubectl port-forward svc/web 8080:80Reach a Service from your own machinekubectl top podsCurrent CPU and memory use (needs metrics-server)kubectl get events --sort-by=.lastTimestampWhat happened in the namespace, in orderkubectl debug exists because good images contain no shell and no tools.An ephemeral container joins a running pod only for the investigation, and is never restarted.

rollout status, history, undo, and kubectl debug.

Debugging a rollout · Going deeper · 2 of 2

Clue words for delivery and debugging

When the question saysThe answer isRemember"build and test every commit"Continuous integrationCI"released with no manual approval"Continuous deploymentdelivery keeps the approval"Git is the single source of truth"GitOpspull, then reconcile"agent in the cluster syncs from Git"Argo CD or Fluxboth CNCF graduated"package manager for Kubernetes"Helmcharts and releases"template-free customisation"Kustomizebases and overlays"small share of users first"Canaryprogressive delivery"two environments, switch traffic"Blue-greeninstant rollback"default Deployment strategy"RollingUpdatemaxSurge, maxUnavailable"return to the previous revision"kubectl rollout undoold ReplicaSet kept"debug a pod with no shell"kubectl debugephemeral container"pipelines as Kubernetes objects"Tektonsteps run as podsDomain 3 is 16% of the exam: GitOps, Helm and the release strategies carry most of it.

GitOps, Helm, canary and rollout undo.

Check yourself

Installedthe application from a chart, with HelmUpgradedto a new chart version, and errors beganNeedsthe previous revision of the release back

Which command fits what the team needs?

Lesson complete

You finished “Application delivery and debugging”

Answer the questions along the way to see your score here.

Open in a new tab