The 4Cs

Security is four layers, each inside the next

CLOUDaccounts, networks,the machinesCLUSTERAPI access, RBAC,network policyCONTAINERtrusted images,no root userCodeyour application and its librariesSecurity is four layers, one inside the next: cloud, cluster, container, code.Each layer can only be as safe as the layer around it. Perfect code is no help when the clusterlets anyone in, and a locked-down cluster is no help when the cloud account is open. Defendevery layer, and do not rely on one: defence in depth.

Cloud, cluster, container, code: each is only as safe as the layer around it.

The 4Cs · Going deeper

What protects each layer

LayerWhat can go wrongWhat protects itCloudA leaked cloud key; an open networkIAM, private networks, encrypted disksClusterAn API server open to the internetAuthentication, RBAC, NetworkPolicy, audit logsContainerAn image with known vulnerabilitiesImage scanning and signing, non-root, read-onlyCodeA vulnerable library; a secret in the sourceDependency scanning, TLS, code reviewTools the exam may nameTrivyScans images and manifests for known vulnerabilitiesFalcoWatches running containers for suspicious behaviour (runtime security)OPA, KyvernoPolicy engines that accept or reject objects at admissionShift left: find the problem in the image and the manifest, before it reaches the cluster.

Each layer has its own threats and controls. Trivy scans, Falco watches at runtime.

RBAC

Who, bound to what they may do

Roleget, list, watchon pods, in namespace devwhat may be doneSubjecta user, a group or aServiceAccount: here "ci"whoRoleBindingjoins the twothe grant✓ ci lists the pods in dev✗ ci deletes a pod in dev✗ ci lists the pods in prodRBAC only ever adds permission. Nothing is allowed until a role grants it.A Role lists verbs on resources. A RoleBinding gives that Role to a subject. There are no denyrules, so the way to restrict someone is to grant less: least privilege. Pods have an identitytoo, a ServiceAccount, and it should be given only the access the application needs.

A Role lists permissions. A RoleBinding gives it to a subject. Nothing else is allowed.

RBAC · Going deeper

The request path and the RBAC objects

The request path1. AuthenticationWho are you? Certificates, tokens, or an identity provider (OIDC)2. AuthorizationMay you do this? RBAC answers yes or no3. Admission controlIs the object acceptable? It can be changed (mutating) or rejected (validating)RBAC objectsRolePermissions inside one namespaceClusterRolePermissions across the cluster, or on cluster-wide things such as nodesRoleBindingGrants a Role or ClusterRole inside one namespaceClusterRoleBindingGrants a ClusterRole everywhereServiceAccountThe identity of a pod; every namespace has one named "default"Every request to the API server passes all three gates, in this order.

Authentication, authorization, then admission control.

Check yourself

Roleget and list on pods, in the namespace devBindinggives that Role to the ServiceAccount "ci"Request"ci" asks to delete a pod in dev

The request in the last row arrives. What happens?

Pod security

Three levels of what a pod may ask for

privileged containerhost networkhostPath volumerun as rootgain privilegesPrivilegedno limits: for system add-ons✓✓✓✓✓Baselineblocks the known escapes✗✗✗✓✓Restrictedcurrent hardening practice✗✗✗✗✗Pod Security Standards are three levels of what a pod is allowed to ask for.Each level allows less than the one before. You apply a level to a namespace with a label, and thebuilt-in Pod Security Admission controller can enforce it (reject the pod), audit it or only warn.A privileged pod on the host network is, in effect, root on the node.

Privileged, Baseline, Restricted: each allows less than the one before.

Pod security · Going deeper

Security contexts and Secrets

Security context: settings on a pod or containerrunAsNonRoot: trueRefuse to start when the image would run as rootallowPrivilegeEscalation: falseThe process can never gain more rights than it began withreadOnlyRootFilesystem: trueThe container cannot change its own filescapabilities: drop ALLRemove the pieces of root power that Linux hands outSecretsWhat a Secret isA small object for passwords, tokens and keys, kept apart from the imagebase64 is not encryptionAnyone who can read the Secret can decode itProtect it withRBAC on who may read it, and encryption at rest in etcdUsed by a pod asEnvironment variables, or files in a mounted volumeA Secret is only as private as the RBAC rules and etcd encryption around it.

A security context hardens one pod. A Secret is encoded, not encrypted.

Troubleshooting

The status says where the pod is stuck

Schedulepick a nodePendingno node fits itsrequests or taintsPullfetch the imageImagePullBackOffwrong name or tag, orno access to the registryStartrun the containerCrashLoopBackOffthe process keeps exitingand being restartedReadypass the readiness probeRunning, 0/1 readythe probe fails, so theService sends it nothingA pod starts in four stages, and its status tells you which stage it is stuck at.Read the STATUS column first. Pending is a scheduling problem. ImagePullBackOff is an imageproblem. CrashLoopBackOff is an application problem: the container starts, then dies. Runningbut not ready is a probe problem. "BackOff" means Kubernetes is retrying, with a growing wait.

Pending, ImagePullBackOff, CrashLoopBackOff, or running but not ready.

Troubleshooting · Going deeper

Statuses, meanings and where to look

StatusIt meansLook atPendingnot scheduled yetdescribe pod: read the EventsErrImagePull, ImagePullBackOffthe image cannot be pulledcheck the name, tag and pull secretCreateContainerConfigErrora ConfigMap or Secret is missingcreate it, or fix the nameCrashLoopBackOffthe container exits again and againlogs --previousOOMKilled (exit code 137)it went over its memory limitraise the limit, or fix the leakRunning, but 0/1 readythe readiness probe is failingdescribe pod: the probe eventsEvictedthe node ran short of memory or diskcheck the node and the requestsCompletedthe process finished with exit code 0normal for a JobNode is NotReadythe kubelet stopped reportingdescribe node, check the kubeletExit code 0 is success. Anything else is a failure, and 137 means the kernel killed it.

Each status points to one kind of cause. OOMKilled is exit code 137.

Check yourself

NAME READY STATUS RESTARTSapi-7d9f4 0/1 CrashLoopBackOff 6

What is the most likely cause?

Debugging commands

Three questions, three commands

Why did it not start?kubectl describe pod webEvents: scheduling, imagepulls, probes, killsWhat did the application say?kubectl logs webits output; add --previousfor the run that crashedWhat is true inside it?kubectl exec -it web -- sha shell in the container:files, DNS, connectionsThree questions, three commands: describe for the cluster's view, logs for the application's.Start with kubectl get pods to see the status, then describe the pod and read the Events at thebottom. When the container did start, read its logs. When it is running but wrong, go insidewith exec. Kubernetes keeps events for about an hour, so look soon.

describe for events, logs for the application's output, exec to look inside.

Debugging commands · Going deeper

Clue words for security and troubleshooting

When the question saysThe answer isRemember"cloud, cluster, container, code"The 4Cslayers of security"who may do what on which resource"RBACRole and RoleBinding"permissions across all namespaces"ClusterRolewith a ClusterRoleBinding"identity for a pod"ServiceAccountnot a user account"privileged, baseline, restricted"Pod Security Standardsset by namespace label"store a password or token"Secretbase64, so guard it"validate or change objects on the way in"Admission controllerafter authorization"detect threats at runtime"Falcowatches system calls"no node has enough CPU"Pendinga scheduling failure"container keeps restarting"CrashLoopBackOffread logs --previous"see the events for one pod"kubectl describeevents are at the bottom"exceeded its memory limit"OOMKilledexit code 137Security and troubleshooting are the other half of domain 2.

RBAC, ServiceAccount, Pending and CrashLoopBackOff.

Check yourself

StatusPending, for the last ten minutesEvents0/3 nodes are available: 3 Insufficient cpuLogsnone: no container has started

Which change gets this pod running?

Lesson complete

You finished “Security and troubleshooting”

Answer the questions along the way to see your score here.

Open in a new tab