kubernetes · from first principles

the database that
runs your containers

Everything in kubernetes reduces to two ingredients: a declarative store of desired state and a bunch of control loops that drag reality toward it. Once you see that, every component stops being magic — it's either reading the database, writing to it, or reconciling the difference.

written for engineers who already know what a pod is and want the internals, the terminology, and the failure modes — mechanism first, jargon second.

01 the idea: two hard problems, one design

Before kubernetes there were two problems every infrastructure team solved by hand, badly:

Placement. I have 40 machines and 300 services. Which service goes on which machine, how do I avoid stacking CPU hogs on one box, and what happens when a machine dies at 3am?
State convergence. I said "5 replicas behind a load balancer" — but then one crashed, an operator fat-fingered a config, and a deploy went out. Who makes sure reality matches what I said, forever, without me babysitting it?

Kubernetes' answer to both: stop issuing commands, start declaring state. You don't say "start container X on node 7" — you say "a deployment named web should have 3 replicas of this image". The system then owns the job of making that true, continuously. This is the declarative model, and it's the single idea the whole system is built around.

The trick that makes it work is that the desired state lives in a highly available database (etcd), and every piece of machinery reads from and writes to that database. Kubernetes is, at heart, a database with side effects: when the data says "3 replicas" and reality says "2", some component notices and makes the side effect happen (starts a container), then records what it did back into the database.

┌─────────────────────────┐ ┌──────────────────────────────┐ │ desired state (etcd) │ │ reality (nodes) │ │ "3 replicas of web" │ │ only 2 web pods running │ └────────────┬─────────────┘ └──────────────┬───────────────┘ │ │ │ ┌──────────────┐ │ └───────►│ control loop │◄────────────┘ │ observe → │ │ diff → act → │ │ record │ └──────────────┘

This is why kubernetes feels different from bash scripts or config management: there is no "end of the run". The system never finishes, it only converges — and if something drifts, the loops pull it back. It's the difference between edge-triggered automation (act once, on an event) and level-triggered control (act until the state matches).

key idea

Every object in kubernetes has a spec (what you want) and a status (what actually is). The status is written back by the system, not by you. Spec vs status is the entire contract: you write specs, components write statuses, controllers read both and close the gap.

see it yourself

kubectl get pod web-xyz -o yaml — the spec block is yours, status (containerStatuses, conditions, podIP) is the kubelet's report. Watch the status field update as the pod boots: kubectl get pod -w.

02 the reconciliation loop — the core primitive

The atomic unit of kubernetes is the reconcile loop. Every controller — the deployment controller, the scheduler, the kubelet, your own operator — is some flavor of this:

// pseudocode, but this is genuinely the whole pattern
func reconcile(object) {
    observed  := read(object.status)      // what actually is
    desired   := read(object.spec)        // what was asked for
    if observed == desired {
        return                         // converged, nothing to do
    }
    act(desired, observed)           // start/stop/update things
    return                             // loop will run again
}

Three properties make this pattern absurdly robust, and they're worth understanding deeply because they explain most of kubernetes' behavior:

Idempotency. The act must be safe to repeat. If the controller crashes after starting a container but before recording the status, it restarts, re-reads state, sees the container already exists, and does nothing. "Apply until true" means partial failures self-heal without checkpointing.
Level-triggered, not edge-triggered. The loop reacts to the current state, not to "a thing happened". Miss an event? Doesn't matter — the next pass sees current reality. This is why k8s survives missed heartbeats, dropped watches, and controller restarts.
Convergence over correctness. The system doesn't guarantee no wrong intermediate states — it guarantees the system heads toward the desired state and gets there eventually. During a deploy, for a moment, you have mixed versions. That's accepted and bounded by higher-level machinery (rolling updates), not prevented.

Contrast this with imperative tooling: a bash script says "stop A, then start B". If it dies between the two steps, you're in an undefined state forever unless you wrote recovery code. Kubernetes bakes the recovery code into the architecture — every object is perpetually being compared against its spec, so undefined states are temporary by construction.

key idea

Kubernetes is best understood as thousands of independent reconcile loops, each owning one small slice of the world (a deployment, a node, a pod), each reading specs and writing statuses, none of them trusting each other. There is no orchestrator-with-a-plan — the "plan" is the current state of the database, and the intelligence is distributed.

failure mode

A reconcile loop only fixes drift it can observe. If the desired state is wrong (typo in an env var, bad image tag), the loop faithfully converges reality toward your mistake. Kubernetes is excellent at making wrong things consistently true.

03 cluster anatomy: who reads, who writes, who acts

A cluster is two planes. The control plane is the brain — it holds state and makes decisions but never runs your workload. The data plane is the muscle — nodes that run containers and route traffic. The clean split is the reason control-plane failure and workload failure are different events (section 12).

CONTROL PLANE DATA PLANE (nodes) ┌─────────────────────────────────┐ ┌──────────────────────────────┐ │ etcd (3–5 nodes, raft) │ │ kubelet │ │ ▲ │ only writer is: │ │ └─► CRI ─► containerd │ │ │ ▼ │ │ └─► OCI runtime ─► c │ │ kube-apiserver ◄───────────────┼──────┼── watch (spec.nodeName==me) │ │ ▲ │ │ │ kube-proxy (svc rules) │ │ │ ▼ watches │ │ CNI plugin (pod network) │ │ scheduler controller-mgr │ │ pods ── your containers │ └─────────────────────────────────┘ └──────────────────────────────┘ ▲ kubectl / humans / CI

etcd — the one and only store of cluster state. Nothing else holds state.
kube-apiserver — the single gateway. Every read and write, from kubectl to the kubelet, goes through it. It's the only component that talks to etcd.
kube-scheduler — watches for pods with no node assigned, picks a node, writes the answer back (to the apiserver, which persists it).
kube-controller-manager — actually dozens of controllers bundled into one binary: deployments, replicasets, node lifecycle, endpoints, serviceaccounts, garbage collection…
cloud-controller-manager — same idea, for cloud-specific things (creating load balancers, attaching volumes).
kubelet — on every node; the only component that actually starts containers. Watches the apiserver for pods assigned to its node.
kube-proxy — on every node; implements the Service abstraction by programming routing rules (iptables / IPVS / eBPF).
container runtime — containerd or CRI-O, spoken to via CRI (the Container Runtime Interface). Docker itself is long gone from the runtime path (dockershim removed in 1.24).

Notice the one-directional pattern: all writes go kubectl → apiserver → etcd; all components act by watching the apiserver. No component talks directly to another (scheduler never calls the kubelet). Communication via shared state is what lets every component be stateless-ish, restartable, and horizontally scalable.

key idea

If you want to know what any k8s component does, ask one question: what does it watch, and what does it write back? Scheduler watches unscheduled pods → writes nodeName. Kubelet watches its node's pods → writes container statuses. kube-proxy watches Services+Endpoints → writes iptables rules (locally, not to etcd).

04 etcd — the database everything stands on

etcd is a distributed key-value store built on the Raft consensus protocol. Every cluster object — pods, services, configmaps, secrets, RBAC rules, events — is a key in etcd. There is no other source of truth. This has deep consequences:

raft, quorum, and why clusters are 3 or 5

Raft keeps the replicas consistent by requiring a majority (n/2 + 1) to agree on every write. With 3 nodes, you can lose 1 and keep working; with 5, you can lose 2. An even number of nodes adds nothing — 4 nodes also tolerate only 1 failure — which is why you'll never see a 4-node etcd recommended. Losing the majority (quorum loss) means no writes can commit: the apiserver can't persist anything, and while the cluster reads fine, it's effectively frozen. The current leader steps down, and the remaining members can't elect a new one.

Because Raft makes a minority unable to commit writes, split-brain is impossible by construction — the worst case is frozen, not divergent.

watches and leases — the two features k8s is built on

Watch: a client subscribes to a key or prefix and receives a stream of changes. The entire controller architecture is watches all the way down — "when a pod appears, do X" is literally a watch event triggering a reconcile. This is why k8s reacts in milliseconds to kubectl apply.
Lease: a key with a TTL, renewable by its holder. Used for node heartbeats (kubelet renews a lease every ~10s; if it stops, the node goes NotReady), leader elections between controller replicas, and distributed locking.

mvcc, compaction, defrag — the operational reality

etcd keeps every historical version of every key (MVCC) so watches can replay. Those old versions accumulate until compaction deletes them; space isn't actually returned to the filesystem until a defrag. An etcd whose compaction is misconfigured grows unbounded, and a bloated etcd means slow commits, which means a slow apiserver, which means a slow everything. This is one of the most common production incidents in k8s — usually caught by etcd's built-in "database space exceeded" alarms.

failure modes

Lose 1 of 3: fine, still quorum. Lose 2 of 3: no writes; control plane frozen; existing workloads keep running (see section 12). Restart all 3 members at once: usually fine — they re-elect. But if you restore from a bad snapshot or clone a member's data dir, you can violate quorum invariants. Never "fix" a broken etcd by copying data directories around; there's a documented disaster-recovery procedure for a reason.

see it yourself

kubectl get events --sort-by=.lastTimestamp — events are just k8s objects stored in etcd, and they're usually what bloats it. kubectl get lease -n kube-system shows leader-election and heartbeat leases live.

05 the API server — the gatekeeper

The apiserver is the only door into the system. kubectl, the kubelet, the scheduler, humans, CI — all speak to it over the same REST+watch API, and it is the sole writer to etcd. Everything you've ever done with kubectl was an HTTP request to this one component.

It's stateless (all state in etcd), so it scales horizontally behind a load balancer — 3 replicas is the typical HA setup, and they're interchangeable. This matters for failure reasoning: an apiserver restart loses nothing, and taking one out for maintenance is invisible.

the request path: three checkpoints before a write commits

kubectl apply
  → TLS handshake + authentication   (certs, tokens, OIDC → who are you)
  → authorization             (RBAC → are you allowed)
  → admission                 (mutating, then validating → does it violate policy)
  → etcd commit
  → response

Admission is where interesting engineering lives: mutating admission can rewrite your request before it lands (inject sidecars — how Istio's mesh sidecar gets into every pod; add default fields), and validating admission can reject it (enforce that all images come from your registry, forbid hostPath volumes, run policy engines like OPA/Gatekeeper). Admission webhooks are just your own HTTPS services the apiserver calls during this phase — this is the extension point for org-wide policy.

resourceVersion and optimistic concurrency

Every object carries a resourceVersion — a monotonically increasing version derived from etcd's revision counter. When a controller updates an object, it sends the resourceVersion it read. If the object changed in between, the write is rejected with a 409 conflict, and the controller re-reads and retries. No locks, no deadlocks — just "if it moved, start over". This is the concurrency mechanism behind the entire system, and why two controllers can safely poke at the same object.

key idea

Why force everything through one API instead of letting components read etcd directly? Because the apiserver is where schema, validation, authn/authz, and admission policy live. It's the compatibility and security boundary. Components get a clean, versioned, watched abstraction — and you get a system where every action is audited and gated in one place.

failure mode

Apiserver down = no kubectl, no scheduling, no new pods, no edits. But: pods already running keep running, kubelets keep managing them locally, probes and restarts keep working. It's an operations outage, not an availability outage — for a while.

06 the scheduler — bin packing as an optimization

The scheduler's entire job: find the best node for each pod that has no nodeName yet. "Best" is the key word — it's a classic filter-then-score pipeline over a candidate list of all nodes.

filter (hard constraints — any fail = node rejected)

Resources fit: does the node have enough CPU/memory (counting only requests, not limits — see the kubelet section)?
Taints/tolerations: taints repel pods; only pods with a matching toleration may land. (This is how you mark a node "GPU only" or "cordoned".)
nodeSelector / node affinity: "only nodes with disk=ssd".
Ports: no conflict on hostPort.
Volume topology: the node is in a zone where the pod's PVC's storage exists.

score (soft preferences — rank the survivors)

Plugins assign points: least requested spreads load, balanced allocation avoids one-resource-skewed nodes, image locality prefers nodes that already have the image cached, topology spread prefers spreading replicas across zones/hosts. The winner gets bound: the scheduler writes spec.nodeName to the pod via the apiserver. That write is the scheduler's only output — and it happens through the same optimistic-concurrency path as everything else.

what the scheduler is not

It is stateless — it keeps a cached snapshot of the cluster (via informers) but holds no persistent queue. Unscheduled pods just sit in etcd with empty nodeName; a scheduler restart simply re-watches and picks up where it left off. It also never moves running pods — if a node dies, the pods on it are marked dead (not rescheduled); new pods get created by the controller that owns them (section 12).

preemption and priority

When nothing fits, a pod with a high priorityClass can trigger preemption: the scheduler picks victims of lower priority on the best-scoring node and evicts them (gracefully, via the apiserver), then binds the preemptor. It's polite but ruthless — this is how you guarantee "this job gets a node even if we have to throw someone off".

failure mode

Pods stuck Pending is almost always a filter failure — the scheduler can't say why out loud, but kubectl describe pod will (unfulfilled requests, unbound PVC, no toleration). Also: a scheduler that can't reach the apiserver stops scheduling silently; the cluster looks fine while the pending queue grows. Scheduler health is a watch-it metric, not a "someone would notice" metric.

see it yourself

kubectl get events -w while deploying shows Scheduled → Pulling → Started as a live transcript of scheduler and kubelet reconciling.

07 controllers & the informer pattern

kube-controller-manager is not one thing — it's dozens of controllers sharing a process: deployment, replicaset, endpoints, node-lifecycle, serviceaccount, garbage-collector, and more. Each one is the reconcile loop from section 02, wired up with three standard pieces:

apiserver ──watch──► informer ──► local cache ──► workqueue ──► reconcile()
                     (list, then        (events go in,    (rate-limited,
                      watch forever;     deduped,          retried w/
                      events feed        keyed by object)  backoff)
                      a delta queue)

Informer: does a list once to build a local cache, then watches forever for changes. It's why controllers don't hammer the apiserver — they read their own cached copy and only get woken by change events. Every serious k8s client library implements this pattern (client-go's informers).
Workqueue: events land as object keys, deduplicated (100 events for the same pod = 1 queue item), with rate limiting and exponential retry. If reconcile fails, the key goes back in the queue later. This is where "eventually consistent" lives.
Reconcile: the controller's brain. Reads current state from its cache, compares with desired, makes changes via the apiserver, and (critically) expects the informer to tell it when reality settles.

the controller chain: deployment → replicaset → pod

Controllers compose. The deployment controller watches Deployments and creates/updates a ReplicaSet whose spec says "3 pods of image v2". The replicaset controller watches ReplicaSets, counts the pods matching its selector, and creates/deletes pods through the apiserver until the count matches. The scheduler assigns those pods to nodes. The kubelet runs them. Each layer is ignorant of the layers above and below — a deployment never talks to a pod, it just declares a ReplicaSet and trusts the chain.

Ownership is tracked with ownerReferences on each object. The garbage collector (another controller) watches these: delete a Deployment and its ReplicaSet and pods cascade-delete, because every orphan whose owner vanished is garbage. This is how kubectl delete deployment web tears down a whole tree with one object deletion.

key idea

The single most powerful realization in k8s: everything above a pod is just controllers watching specs and creating more specs. A Deployment is a controller that makes ReplicaSets; a ReplicaSet is a controller that makes Pods; Pods are the last spec that has a side effect in the real world (a container). Once you see the chain, "what does an operator do" is obvious — it's a deployment-style controller with domain knowledge.

failure mode

Controller-manager down = the cluster stops converging. Deployments freeze mid-rollout, dead pods aren't replaced, node health stops being evaluated. Everything running keeps running — the system just stops self-healing. This is the "zombie cluster" state.

08 the kubelet — where spec becomes process

The kubelet is the only component that talks to the container runtime, and it runs on every node. It watches the apiserver for pods with nodeName == its node, and for each one it's the local project manager: pull images, create namespaces, start containers, run probes, report status back.

CRI: one interface, many runtimes

The kubelet speaks the Container Runtime Interface (gRPC) to containerd or CRI-O; the runtime then drives an OCI-compliant low-level runtime (runc, kata, gVisor) that actually clone()s the process. This interface layering (CRI → runtime → OCI) is why kubernetes supports containerd, CRI-O, and exotic runtimes interchangeably — and why Docker-the-daemon became unnecessary: it was always just another CRI shim (removed in 1.24).

the pause container: a pod is a group, not a thing

A pod is a set of containers sharing namespaces. To make that concrete, the kubelet first starts a tiny pause/sandbox container that does nothing but sleep — it owns the pod's network and IPC namespaces. Every real container then joins those namespaces. Consequences:

All containers in a pod share localhost (same netns) and an IP — talk to each other on loopback, no service discovery needed.
Restarting your app container doesn't change the pod IP, because the pause container's netns survives.
Sidecars work: a logging or proxy container in the same pod sees the same network and lifecycle.

namespaces + cgroups: isolation and fairness

Isolation comes from Linux namespaces (pid, net, ipc, mnt, uts, user): each container gets its own process table, filesystem view, and network stack. Fairness and limits come from cgroups (CPU shares, memory limits, pids limit, IO): this is where requests and limits become kernel policy.

requests = reserved/guaranteed minimum. The scheduler only ever sums requests when deciding if a node has room.
limits = hard ceiling. CPU over the limit gets throttled; memory over the limit gets OOM-killed (the container restarts).
Set no limits and one leaky pod can eat the node — a classic incident. The kubelet does have node-level pressure eviction as a backstop (see QoS below).

probes: three questions, three different consequences

probe	question	failure consequence
startup	has it finished booting?	restart the container (gates the others; for slow-starting apps)
liveness	is it alive (not deadlocked)?	restart the container — the app itself is broken
readiness	can it serve traffic right now?	remove from service endpoints — app is alive but shouldn't get traffic

The distinction is the classic interview question and the classic production bug: a liveness probe that fails under transient load restarts healthy-but-slow apps in a death spiral; a readiness probe that fails gracefully drains traffic. Liveness kills, readiness detaches.

QoS classes: who dies first under pressure

From requests/limits, the kubelet derives a pod's QoS class, which sets OOM-kill priority when the node runs out of memory: BestEffort (no requests/limits) dies first, then Burstable (requests < limits), and Guaranteed (requests == limits on every container) dies last. This is why "set proper requests/limits" isn't style advice — it's literally the node's eviction policy, and BestEffort pods are your sacrificial lambs.

the kubelet as reporter

Everything the kubelet does, it reports back: container statuses, pod conditions (Ready, PodScheduled), node conditions, heartbeats (a lease renewed ~every 10s). The control plane has no telemetry other than what kubelets write into status fields. When kubectl describe node shows conditions, you're reading kubelet reports.

failure mode

If the kubelet stops heartbeating, the node-lifecycle controller marks the node NotReady (after ~40s of silence) and eventually taints it NoExecute, evicting its pods so they reschedule elsewhere. But a partitioned kubelet (node fine, network to apiserver broken) keeps running its pods while the control plane "reschedules" them elsewhere — congratulations, you now have two copies of every pod. Section 12.

09 networking — one hard rule, then everything is layered on top

Kubernetes imposes exactly one requirement on the network: every pod gets its own IP, and every pod can reach every other pod directly, without NAT — across nodes, across the whole cluster. How that's implemented is deliberately unspecified, and that's what a CNI plugin (Container Network Interface) does: flannel (overlay via VXLAN), Calico (BGP or overlay), Cilium (eBPF).

Mechanically: the CNI plugin allocates a per-node subnet, creates a network namespace for the pod (the pause container's — see section 08), plugs in a veth pair (one end in the pod, one on the host), assigns the IP, and wires routes so traffic to any pod IP finds the right node. Overlay networks wrap packets so they survive routing between nodes; Calico-style L3 routes pod subnets directly when the underlying network allows it.

Services: a stable name for an unstable set of pods

Pod IPs die with their pods. A Service gives you a stable ClusterIP (virtual IP) + DNS name (web.default.svc.cluster.local) that load-balances to whatever pods currently match a label selector. The important mental model: a ClusterIP is not a listener — nothing is bound to it; ss -tlnp on a node will never show it. It exists as routing rules, programmed by kube-proxy:

iptables mode (classic default): kube-proxy watches Services and EndpointSlices, and rewrites node-local iptables to DNAT the ClusterIP to a randomly chosen backend pod IP. A config change = rewrite thousands of rules (why huge clusters saw slow service updates).
IPVS mode: uses the kernel's IPVS load balancer — faster, real scheduling algorithms, scales better.
eBPF (Cilium): replaces kube-proxy entirely — service translation happens in the kernel at the socket level. No kube-proxy daemon at all.

Endpoints / EndpointSlices are the objects holding "which pod IPs back this service" — maintained by the endpoints controller, which watches pods and readiness. Readiness probe fails → pod IP removed from EndpointSlices → kube-proxy rewrites rules → traffic stops. That's the whole chain of "readiness detaches".

Headless Services (clusterIP: None) skip the VIP: DNS returns the pod IPs directly. This is how stateful apps find individual pods (see StatefulSets). NodePort opens the port on every node; LoadBalancer makes the cloud provision an LB in front (via cloud-controller-manager). Ingress is L7 routing (host/path → service) — an API object that needs an ingress controller (nginx, Contour…) running as a pod to actually implement it. People conflate "ingress" with "nginx pod" constantly; they're spec vs implementation.

DNS: CoreDNS runs in-cluster, watches Services/Pods, and answers service.namespace.svc.cluster.local queries. Every pod's /etc/resolv.conf points at it. Kube-DNS/CoreDNS going down is a full service-discovery outage — a genuinely nasty failure mode.

key idea

Three decouplings to keep straight: Service decouples name from IP (stable VIP over ephemeral pods). EndpointSlice decouples the service definition from its current backends. CNI decouples the k8s network model from any particular network implementation. Every layer is a contract, and each contract is what lets vendors/plugins slot in.

failure modes

kube-proxy lag: between a pod dying and rules being rewritten, connections hit a dead IP (brief 5xxs — why endpoints-based LB is eventually consistent). Conntrack table exhaustion on busy nodes. Pods that can't reach the ClusterIP of their own service (hairpin). And the big one: NetworkPolicies are enforced by the CNI, and plain flannel doesn't enforce them — by default, no pod-to-pod firewall exists at all. Your cluster is open by default.

10 storage — decoupling providers from consumers

Storage follows the same shape as everything else in k8s: a contract (interface) between two parties that shouldn't know about each other.

PV (PersistentVolume) — a cluster-scoped object representing actual storage: "there exists a 100Gi disk at NFS server X" or "an AWS EBS volume vol-abc123". Think of it as the storage itself, registered with the cluster. Provisioned by an admin, or dynamically by a provisioner.
PVC (PersistentVolumeClaim) — a namespaced request: "I need 10Gi, read-write-once, fast class". An app pod references a PVC; the cluster matches it to a PV.

The decoupling is the point: the developer asks for capacity with a PVC; ops/cloud answers with PVs. The PV/PVC pair is the malloc of kubernetes — the claim is the malloc(), the volume is the heap block, and the binding is a one-to-one lease until the claim is deleted.

dynamic provisioning and storage classes

Nobody wants to hand-create PVs. A StorageClass names a provisioner plus parameters ("the cloud disk plugin, SSD tier, in eu-west-1a"), and a PVC that names the class causes a PV to be created on demand. The provisioner speaks CSI (Container Storage Interface) — the storage world's version of CRI/CNI: a standard gRPC interface that any vendor's driver implements, so k8s core never needs to know about EBS vs Ceph vs NetApp. Provision → attach → mount is the lifecycle; attachdetach controller + kubelet split the work, with volume topology keeping the pod on a node in the volume's zone.

Reclaim policies close the loop: PVC deleted → PV either Retains (waits for an admin), Deletes (cloud volume destroyed), or Recycles (wipe and reuse — legacy).

failure modes

PVC stuck Pending forever = no PV matches (wrong storage class, no provisioner, quota). A network volume that hangs can wedge pods in Terminating and block node drains. Deleting a PV with Retain and assuming the cloud disk is gone = the classic orphaned-EBS-cost incident. And ReadWriteOnce means one node, not one pod — two pods on different nodes sharing a RWO volume will both "mount" it and corrupt data.

11 higher-level controllers — deployment, statefulset, and the operator insight

Everything in this section is, per section 07, just a controller generating specs. The differences are the policies they encode.

deployment: rolling updates as a two-replicaset trick

A Deployment manages a ReplicaSet per revision. On update: create new ReplicaSet with the new pod template, scale it up, scale the old one down — the overlap between the two curves is governed by maxSurge (how many extra pods may exist) and maxUnavailable (how many may be down). Rollback is just reversing the curve with the previous ReplicaSet (kept per revisionHistoryLimit). No magic: the "rolling update" you see is literally two ReplicaSets scaling against each other, which kubectl rollout status renders as a progress bar.

statefulset: identity for pods

Plain pods are cattle; a StatefulSet gives them name-shaped identities: pods get ordinal names (db-0, db-1, …), stable DNS (db-0.db.default.svc — hence headless services), stable per-pod PVCs, and ordered create/scale/update (one at a time, in order). This is what stateful systems need: a quorum member that comes back must come back as the same member, with the same disk, addressable by its peers.

But know the boundary: a StatefulSet gives identity, not immortality. The pod can still die and be rescheduled; the app still has to handle restart, resync, and re-election. Running etcd or Kafka on StatefulSets without understanding their consensus protocols is how you get db-2 forever-CrashLoopBackOff-ing.

daemonset, job, cronjob

DaemonSet: exactly one pod per node, always. That's how kube-proxy, CNI agents, and log collectors deploy themselves. New node joins → pod appears there automatically.
Job: run to completion, N completions, N parallel. restartPolicy: OnFailure/Never — a different failure model than services (retry vs replace).
CronJob: a Job factory on a schedule — with its own timezone and concurrency-policy footguns.

HPA: scaling as another control loop

The HorizontalPodAutoscaler is a controller that watches metrics (CPU via metrics-server, custom via prometheus-adapter) and sets replicas on your Deployment every ~15s. It's reconcile loops all the way up: your scale target is just another spec field, and the HPA is just another controller. (VPA does the same vertically — requests/limits — by restarting pods, which is why it's scarier.)

CRDs and operators: the whole system in miniature

A CRD (CustomResourceDefinition) extends the apiserver with your own resource type — a PostgresCluster object that kubectl can get/apply/watch like anything built-in. That alone does nothing; an operator is what makes it real: a controller (informer + reconcile, section 07) watching your CRD, encoding domain knowledge — provision storage, bootstrap, replication, failover, backup, upgrade. The prometheus-operator, etcd-operator, all of it: Deployment-plus-StatefulSet-plus-backup-logic, written as a reconcile function.

The full-circle insight: a Deployment is a built-in operator for stateless apps. Operators aren't exotic machinery, they're the same machinery with your expertise in the loop.

failure mode

An operator is only as good as its reconcile logic — and a buggy reconcile loop is a persistent bug: it will faithfully re-break your system every few seconds. CRDs are also global and irreversible-ish (deleting a CRD deletes all its objects). Careful what your operator considers "desired".

12 failure modes — what actually breaks, and what doesn't

Kubernetes is designed around one architectural bet: the data plane must keep serving even when the brain dies. Here's the failure model, component by component.

a node dies

kubelet heartbeats stop → lease expires → node-lifecycle controller marks the node NotReady (~40s) → taints it NoExecute → pods are evicted and recreated elsewhere by their controllers (not "moved" — new pods, new IPs). DaemonSet pods tolerate the taint and wait. Meanwhile the dead node's workloads are simply gone: k8s doesn't do migration, it does replacement. If you have 1 replica of something, you have downtime — the controller can't resurrect what it doesn't know about.

apiserver dies

No kubectl, no scheduling, no edits. But kubelets have their assigned pods and keep running them, probes keep working, restarts happen locally. The cluster serves traffic but can't change. Hours-long apiserver outage with stable workloads = users never notice.

etcd loses quorum

Control plane frozen — worse than apiserver death because it's the persistence layer: no state can be written, and no new apiserver instance can help. Workloads still run. Recovery is the documented etcd disaster-recovery path (snapshot restore), not vibes. This is why etcd gets its own dedicated nodes, its own disks, and its own alerts.

control plane ↔ node network partition (the nasty one)

Node can't reach the apiserver, apiserver can't reach the node. Two things happen at once: the kubelet keeps its pods running (it can't hear the eviction order), and the control plane marks the node dead and schedules replacements of those same pods on healthy nodes. Result: duplicate pods, split state, two writers to the same database-backed service. K8s has no global "is this pod still alive" oracle beyond heartbeats. This is why multi-AZ designs pin stateful pods to zones (topology spread, node affinity) and why you run at least 3 replicas across failure domains.

kube-proxy / endpoints lag

A pod dies; until kube-proxy reprograms rules, connections hit a dead backend. Brief, self-healing, but it's why readiness + fast probes matter: a pod that's slow to be removed from endpoints is a pod still receiving traffic it can't serve.

memory pressure and QoS

Node OOM → kubelet evicts in QoS order: BestEffort first, then Burstable, Guaranteed last. Un-bounded pods (no limits) are the ones that take the node down and get everyone evicted. This is the concrete reason "set requests/limits" is infrastructure policy, not pedantry.

voluntary disruption: drains and PDBs

Draining a node (maintenance) evicts pods — a PodDisruptionBudget is your contract for "at least 2 of these must survive". Without PDBs, a rolling node drain can happily take your entire quorum down one member at a time. PDBs protect against voluntary disruption only; node death is involuntary and ignores them — that's what replicas are for.

priority and preemption

When the cluster is full, high-priority pods evict low-priority ones (section 06). Default priority is 0 — which means everyone is evictable. Production policy: explicit priorityClasses, and never run something critical at the default.

quota and limits

ResourceQuota caps total resources per namespace, LimitRange forces defaults on pods that don't set requests/limits. Namespaces without them are the wild west: one team's unbounded pod consumes the node and evicts your BestEffort pods. Multi-tenant clusters without quotas are single-tenant clusters with extra steps.

key idea

The throughline: k8s failures come in two families — control-plane failures (things stop changing; traffic keeps flowing) and data-plane failures (things keep changing; traffic stops). Diagnosing which family you're in — "can I still edit things, but traffic is broken?" vs "traffic is fine, but kubectl is frozen?" — cuts the search space in half before you look at a single log.

13 life of a pod — the whole system in one trace

Every component from this page, exercised by one kubectl apply. Read this top to bottom once and you've seen the whole architecture fire.

kubectl apply -f pod.yamlHTTP POST to the apiserver. Authn (who), authz (may they), admission (mutating webhooks inject sidecars, validating webhooks check policy) — then the pod object is committed to etcd with status Pending.
scheduler noticesIts informer sees a pod with no nodeName. Filters: resources fit (summing requests), taints tolerated, affinity matches. Scores: least-requested wins. Writes spec.nodeName back through the apiserver (optimistic concurrency: 409 → re-read → retry).
kubelet on the chosen node wakes upIts informer sees a pod assigned to it. Calls the runtime via CRI: create the pause/sandbox container — the pod's netns and IPC ns now exist.
CNI wires the networkThe CNI plugin allocates an IP from the node's subnet, creates the veth pair into the pod's netns, programs routes. The pod has an IP before any app container exists.
images pulled, containers startedcontainerd pulls (respecting imagePullPolicy, imagePullSecrets), creates namespaces, applies cgroup limits, execs the app. Init containers run first, in order, to completion; then app containers; then postStart hooks.
probes beginReadiness probe starts passing → kubelet writes containerStatuses.ready=true into pod status, sets the Ready condition. Until this moment, the pod is invisible to Services.
the endpoints controller reactsIt watches pods: a ready pod matching a Service's selector → its IP lands in an EndpointSlice. kube-proxy's informer sees that → rewrites iptables/IPVS so the ClusterIP can DNAT to this pod. Traffic flows.
steady statekubelet keeps probing, heartbeats every ~10s (node lease), status updates flow to etcd. All of this is just watches and status writes — the cluster's "running" state is a continuously refreshed database.
deletionkubectl delete → apiserver sets deletionTimestamp (finalizers, if any, must clear first — that's how operators run cleanup) → kubelet sends SIGTERM → preStop hook runs → grace period (terminationGracePeriodSeconds, default 30s) expires → SIGKILL → sandbox torn down → IP released.

see it yourself

Run kubectl get pod -w in one terminal and kubectl apply -f pod.yaml in another: watch Pending → ContainerCreating → Running → Ready in real time, then kubectl describe pod for the event log the components left behind. Every status flip is a write from some component's reconcile loop.

14 terminology glossary

Every term from the page plus the ones you'll hear in interviews, one line each. Type to filter.

admission controller: Plugin chain in the apiserver that can mutate or reject requests before they hit etcd — the policy checkpoint (see webhook).
affinity / anti-affinity: Scheduling preferences/constraints: "prefer nodes with disk=ssd" (node affinity), "don't co-locate with pods matching X" (pod anti-affinity — how you spread replicas across hosts/zones).
annotation: Non-identifying metadata on objects (like labels, but not used for selection). Where tools stash state: kubectl.kubernetes.io/restartedAt, ingress annotations, etc.
API server (kube-apiserver): The single gateway for all reads/writes and the only component that talks to etcd. Serves REST + watch; fronted by authn/authz/admission.

CNI: Container Network Interface — the plugin standard for pod networking (flannel, Calico, Cilium). CNI plugin = whatever actually implements "every pod has an IP".
CRI: Container Runtime Interface — the gRPC API the kubelet uses to talk to containerd/CRI-O. Decouples k8s from any specific runtime.
CSI: Container Storage Interface — the equivalent plugin standard for storage: vendors ship a driver, k8s provisions/attaches/mounts volumes through it.
ClusterIP: The default Service type: a virtual IP for the service, implemented as routing rules (not a listener). Stable entry point to an unstable set of pods.
ConfigMap: Non-secret key/value config, exposed to pods as env vars or files. Note: not encrypted, not secret storage.
container runtime: The software that actually runs containers on a node (containerd, CRI-O), driven by the kubelet through CRI.
controller: A reconcile loop: watches some object, compares spec vs status, acts to converge them. Deployment controller, node-lifecycle controller, your operator — all controllers.
control plane: The brain: etcd + apiserver + scheduler + controller-manager(s). Decides and stores; never runs your workloads.
cordon: Mark a node unschedulable (new pods won't land; existing ones stay). First step of a drain.
CRD: CustomResourceDefinition — extends the apiserver with your own resource type. The raw material operators are built from.
CronJob: Controller that creates Jobs on a schedule (cron syntax, with timezone + concurrency-policy settings).

DaemonSet: Controller guaranteeing exactly one pod per node — how kube-proxy, CNI agents, and node exporters run everywhere automatically.
Deployment: Controller managing a ReplicaSet per revision, giving you rolling updates, rollbacks, and scale. The default way to run stateless apps.
drain: Gracefully evict all pods from a node (respecting PDBs) before maintenance. kubectl drain node.
data plane: The nodes: kubelets, runtimes, kube-proxy, pods. Runs the workloads and routes the traffic.

EndpointSlice: Object holding the ready pod IPs backing a Service; maintained by the endpoints controller. kube-proxy watches it to program rules. (Supersedes the older flat Endpoints object.)
etcd: Distributed KV store (Raft consensus) holding all cluster state. Quorum of n/2+1; watches + leases are the features k8s is built on.
eviction: Forcibly removing a pod from a node — by the kubelet (node pressure), by the scheduler (preemption), by a drain (voluntary), or by the API (node NotReady).

F–G

finalizer: A marker on an object that blocks deletion until a controller clears it — the hook operators use to run cleanup before an object disappears.
garbage collector: Controller that deletes objects whose ownerReferences owner is gone. Makes kubectl delete deployment cascade through ReplicaSets to pods.

H–I

headless service: A Service with clusterIP: None: DNS resolves to the individual pod IPs instead of a VIP. Required for StatefulSet pod discovery.
HPA: HorizontalPodAutoscaler — controller that adjusts replicas based on metrics (~every 15s). Scaling is just another reconcile loop.
imagePullPolicy: When to pull the image: Always / IfNotPresent / Never. Classic pitfall: latest + IfNotPresent = stale code that "worked yesterday".
informer: Client-side list-then-watch cache of objects with an event stream. The pattern every controller and client library uses to avoid hammering the apiserver.
ingress: API object declaring L7 routing (host/path → service). Requires an ingress controller (nginx, Contour…) running as pods to actually take effect.
init container: Container that runs to completion before app containers start — migrations, config prep, waiting for dependencies. Separate image, shared pod resources.

J–L

Job: Controller that runs pods to completion (N completions, N in parallel) with retry semantics — for tasks, not services.
kubelet: The node agent. Watches pods assigned to its node, drives the runtime via CRI, runs probes, reports status. The only component that starts containers.
kube-proxy: Node agent implementing Services by programming iptables/IPVS rules (or absent entirely under eBPF CNIs). Watches Services + EndpointSlices.
label / selector: Key/value metadata on objects, and the query language matching it. The glue between pods and Services, Deployments, and almost everything else. Selectors are the only link between controller and pod.
lease: etcd key with TTL, renewable by holder — used for node heartbeats, leader election, and locks.
LimitRange: Namespace-level policy forcing defaults/min/max on requests and limits. Without it, pods without limits are legal and dangerous.
liveness probe: Health check that restarts the container on failure. For detecting deadlocked/broken processes — not for transient slowness (that's a death spiral).
LoadBalancer: Service type that provisions a cloud load balancer in front of NodePorts, via the cloud-controller-manager.

M–N

manifest: A YAML/JSON file describing desired objects. "Applying a manifest" = declaring state to the apiserver.
namespace: Virtual cluster: a scoping boundary for objects, names, quotas, and RBAC. Not a security boundary by itself (network isolation needs NetworkPolicy).
NetworkPolicy: Declarative pod-to-pod firewall. Only enforced if the CNI supports it — flannel alone enforces nothing; the default posture is open.
node: A worker machine (VM or bare metal) running kubelet + runtime + kube-proxy. Represented as a cluster object with capacity, conditions, and heartbeats.
NodePort: Service type that opens a port (30000–32767) on every node, forwarding to the service. Building block under LoadBalancer.

O–P

OCI: Open Container Initiative — the image format + runtime spec. runc/kata/gVisor are OCI runtimes; containerd/CRI-O manage them.
operator: Custom controller + CRD encoding domain knowledge (deploy, failover, backup, upgrade). A Deployment is a built-in operator for stateless apps.
ownerReference: Pointer from an object to its parent. Powers cascade deletion via the garbage collector.
pause container: The tiny sandbox container each pod starts with; it owns the pod's net/IPC namespaces so real containers can share them and the pod IP survives restarts.
PDB: PodDisruptionBudget — "at least N of these must survive" contract, honored by voluntary disruptions (drains). Node death ignores it.
pod: The smallest schedulable unit: one or more containers sharing network/IPC namespaces, IP, volumes, and lifecycle. The atom of the system.
preemption: High-priority pod evicting lower-priority pods to win a node when the cluster is full. Governed by PriorityClass.
PriorityClass: Named priority level for pods; drives preemption order. Default is 0 — and everyone's evictable at 0.
probe: Health check the kubelet runs against a container (exec/HTTP/TCP). Startup gates the others; liveness restarts; readiness detaches from services.
PV / PVC: PersistentVolume = registered storage (the resource); PersistentVolumeClaim = a request for storage (the malloc). Binding links them 1:1.

Q–R

QoS class: Guaranteed / Burstable / BestEffort, derived from requests/limits — the OOM kill order under node memory pressure. BestEffort dies first.
quorum: Majority of etcd members required to commit writes (n/2+1). Lose it and the cluster stops persisting state; split-brain is impossible by design.
RBAC: Role-Based Access Control: Roles/ClusterRoles (permissions) bound to users/serviceaccounts via (Cluster)RoleBindings. The authorization layer.
readiness probe: Health check that removes the pod from service endpoints on failure. Alive but not serving — drain traffic, don't restart.
replica: One copy of a pod, managed by a controller. Replicas are fungible by default — that's the entire premise of stateless apps on k8s.
ReplicaSet: Controller ensuring N pods matching a selector exist. Deployments drive ReplicaSets; ReplicaSets drive pods.
requests / limits: requests = guaranteed minimum (what the scheduler sums); limits = hard ceiling (CPU throttled, memory OOM-killed past it). The basis of QoS and scheduling.
ResourceQuota: Namespace-level cap on total resource consumption (CPU, memory, object counts). Multi-tenancy's seatbelt.
resourceVersion: Object version stamp (from etcd's revision counter). Sent back on updates for optimistic concurrency — changed underneath you → 409 → re-read, retry.

scheduler: Component that assigns pods to nodes: filter (hard constraints) then score (preferences), then binds via spec.nodeName. Stateless; pods wait in etcd, not in its memory.
secret: Object for sensitive data (tokens, keys). Base64, not encrypted by default — enable encryption-at-rest, and prefer mounted files over env vars.
service: Stable name + VIP over a set of pods (label selector), load-balanced by kube-proxy. The decoupling of "who am I" from "where do I run".
ServiceAccount: Identity for pods; every pod gets one automatically, with a token the apiserver authenticates. What RBAC roles are bound to.
sidecar: Auxiliary container in the same pod, sharing network/volumes: log shippers, service-mesh proxies, config refreshers. Same lifecycle as the main app.
spec / status: The k8s contract: you write spec (desired), the system writes status (actual). Controllers reconcile the two.
StatefulSet: Controller giving pods stable identity: ordinal names, stable DNS, per-pod PVCs, ordered operations. Identity, not immortality — the app still handles restarts.
static pod: Pod the kubelet runs from a local manifest directory, not from the apiserver — how kubeadm runs the control plane itself.
StorageClass: Named class of storage + provisioner params; a PVC referencing it triggers dynamic PV creation via CSI. The "instance type" of volumes.

T–W

taint / toleration: Taints repel pods from a node ("GPU only", "draining"); tolerations let specific pods land anyway. Scheduling's No-Entry signs.
terminationGracePeriodSeconds: Time between SIGTERM and SIGKILL on pod deletion (default 30s). Your app's window to drain connections (with preStop hooks).
watch: The streaming API: subscribe to object changes instead of polling. The event backbone of every controller.
workqueue: Per-controller queue: dedupes events by key, rate-limits, retries with backoff. Where "eventually consistent" is implemented.
webhook (admission): Your own HTTPS endpoint called during admission to mutate/validate requests — the extension point for org policy (sidecar injection, image allowlists).