the database that
runs your containers

01 the idea: two hard problems, one design

Before kubernetes there were two problems every infrastructure team solved by hand, badly:

1. Placement. I have 40 machines and 300 services. Which service goes on which machine, how do I avoid stacking CPU hogs on one box, and what happens when a machine dies at 3am?
2. State convergence. I said "5 replicas behind a load balancer" — but then one crashed, an operator fat-fingered a config, and a deploy went out. Who makes sure reality matches what I said, forever, without me babysitting it?

Kubernetes' answer to both: stop issuing commands, start declaring state. You don't say "start container X on node 7" — you say "a deployment named web should have 3 replicas of this image". The system then owns the job of making that true, continuously. This is the declarative model, and it's the single idea the whole system is built around.

The trick that makes it work is that the desired state lives in a highly available database (etcd), and every piece of machinery reads from and writes to that database. Kubernetes is, at heart, a database with side effects: when the data says "3 replicas" and reality says "2", some component notices and makes the side effect happen (starts a container), then records what it did back into the database.

This is why kubernetes feels different from bash scripts or config management: there is no "end of the run". The system never finishes, it only converges — and if something drifts, the loops pull it back. It's the difference between edge-triggered automation (act once, on an event) and level-triggered control (act until the state matches).

02 the reconciliation loop — the core primitive

The atomic unit of kubernetes is the reconcile loop. Every controller — the deployment controller, the scheduler, the kubelet, your own operator — is some flavor of this:

// pseudocode, but this is genuinely the whole pattern
func reconcile(object) {
    observed  := read(object.status)      // what actually is
    desired   := read(object.spec)        // what was asked for
    if observed == desired {
        return                         // converged, nothing to do
    }
    act(desired, observed)           // start/stop/update things
    return                             // loop will run again
}

Three properties make this pattern absurdly robust, and they're worth understanding deeply because they explain most of kubernetes' behavior:

1. Idempotency. The act must be safe to repeat. If the controller crashes after starting a container but before recording the status, it restarts, re-reads state, sees the container already exists, and does nothing. "Apply until true" means partial failures self-heal without checkpointing.
2. Level-triggered, not edge-triggered. The loop reacts to the current state, not to "a thing happened". Miss an event? Doesn't matter — the next pass sees current reality. This is why k8s survives missed heartbeats, dropped watches, and controller restarts.
3. Convergence over correctness. The system doesn't guarantee no wrong intermediate states — it guarantees the system heads toward the desired state and gets there eventually. During a deploy, for a moment, you have mixed versions. That's accepted and bounded by higher-level machinery (rolling updates), not prevented.

Contrast this with imperative tooling: a bash script says "stop A, then start B". If it dies between the two steps, you're in an undefined state forever unless you wrote recovery code. Kubernetes bakes the recovery code into the architecture — every object is perpetually being compared against its spec, so undefined states are temporary by construction.

03 cluster anatomy: who reads, who writes, who acts

A cluster is two planes. The control plane is the brain — it holds state and makes decisions but never runs your workload. The data plane is the muscle — nodes that run containers and route traffic. The clean split is the reason control-plane failure and workload failure are different events (section 12).

• etcd — the one and only store of cluster state. Nothing else holds state.
• kube-apiserver — the single gateway. Every read and write, from kubectl to the kubelet, goes through it. It's the only component that talks to etcd.
• kube-scheduler — watches for pods with no node assigned, picks a node, writes the answer back (to the apiserver, which persists it).
• kube-controller-manager — actually dozens of controllers bundled into one binary: deployments, replicasets, node lifecycle, endpoints, serviceaccounts, garbage collection…
• cloud-controller-manager — same idea, for cloud-specific things (creating load balancers, attaching volumes).
• kubelet — on every node; the only component that actually starts containers. Watches the apiserver for pods assigned to its node.
• kube-proxy — on every node; implements the Service abstraction by programming routing rules (iptables / IPVS / eBPF).
• container runtime — containerd or CRI-O, spoken to via CRI (the Container Runtime Interface). Docker itself is long gone from the runtime path (dockershim removed in 1.24).

Notice the one-directional pattern: all writes go kubectl → apiserver → etcd; all components act by watching the apiserver. No component talks directly to another (scheduler never calls the kubelet). Communication via shared state is what lets every component be stateless-ish, restartable, and horizontally scalable.

04 etcd — the database everything stands on

etcd is a distributed key-value store built on the Raft consensus protocol. Every cluster object — pods, services, configmaps, secrets, RBAC rules, events — is a key in etcd. There is no other source of truth. This has deep consequences:

raft, quorum, and why clusters are 3 or 5

Raft keeps the replicas consistent by requiring a majority (n/2 + 1) to agree on every write. With 3 nodes, you can lose 1 and keep working; with 5, you can lose 2. An even number of nodes adds nothing — 4 nodes also tolerate only 1 failure — which is why you'll never see a 4-node etcd recommended. Losing the majority (quorum loss) means no writes can commit: the apiserver can't persist anything, and while the cluster reads fine, it's effectively frozen. The current leader steps down, and the remaining members can't elect a new one.

Because Raft makes a minority unable to commit writes, split-brain is impossible by construction — the worst case is frozen, not divergent.

watches and leases — the two features k8s is built on

• Watch: a client subscribes to a key or prefix and receives a stream of changes. The entire controller architecture is watches all the way down — "when a pod appears, do X" is literally a watch event triggering a reconcile. This is why k8s reacts in milliseconds to kubectl apply.
• Lease: a key with a TTL, renewable by its holder. Used for node heartbeats (kubelet renews a lease every ~10s; if it stops, the node goes NotReady), leader elections between controller replicas, and distributed locking.

mvcc, compaction, defrag — the operational reality

etcd keeps every historical version of every key (MVCC) so watches can replay. Those old versions accumulate until compaction deletes them; space isn't actually returned to the filesystem until a defrag. An etcd whose compaction is misconfigured grows unbounded, and a bloated etcd means slow commits, which means a slow apiserver, which means a slow everything. This is one of the most common production incidents in k8s — usually caught by etcd's built-in "database space exceeded" alarms.

05 the API server — the gatekeeper

The apiserver is the only door into the system. kubectl, the kubelet, the scheduler, humans, CI — all speak to it over the same REST+watch API, and it is the sole writer to etcd. Everything you've ever done with kubectl was an HTTP request to this one component.

It's stateless (all state in etcd), so it scales horizontally behind a load balancer — 3 replicas is the typical HA setup, and they're interchangeable. This matters for failure reasoning: an apiserver restart loses nothing, and taking one out for maintenance is invisible.

the request path: three checkpoints before a write commits

kubectl apply
  → TLS handshake + authentication   (certs, tokens, OIDC → who are you)
  → authorization             (RBAC → are you allowed)
  → admission                 (mutating, then validating → does it violate policy)
  → etcd commit
  → response

Admission is where interesting engineering lives: mutating admission can rewrite your request before it lands (inject sidecars — how Istio's mesh sidecar gets into every pod; add default fields), and validating admission can reject it (enforce that all images come from your registry, forbid hostPath volumes, run policy engines like OPA/Gatekeeper). Admission webhooks are just your own HTTPS services the apiserver calls during this phase — this is the extension point for org-wide policy.

resourceVersion and optimistic concurrency

Every object carries a resourceVersion — a monotonically increasing version derived from etcd's revision counter. When a controller updates an object, it sends the resourceVersion it read. If the object changed in between, the write is rejected with a 409 conflict, and the controller re-reads and retries. No locks, no deadlocks — just "if it moved, start over". This is the concurrency mechanism behind the entire system, and why two controllers can safely poke at the same object.

06 the scheduler — bin packing as an optimization

The scheduler's entire job: find the best node for each pod that has no nodeName yet. "Best" is the key word — it's a classic filter-then-score pipeline over a candidate list of all nodes.

filter (hard constraints — any fail = node rejected)

• Resources fit: does the node have enough CPU/memory (counting only requests, not limits — see the kubelet section)?
• Taints/tolerations: taints repel pods; only pods with a matching toleration may land. (This is how you mark a node "GPU only" or "cordoned".)
• nodeSelector / node affinity: "only nodes with disk=ssd".
• Ports: no conflict on hostPort.
• Volume topology: the node is in a zone where the pod's PVC's storage exists.

score (soft preferences — rank the survivors)

Plugins assign points: least requested spreads load, balanced allocation avoids one-resource-skewed nodes, image locality prefers nodes that already have the image cached, topology spread prefers spreading replicas across zones/hosts. The winner gets bound: the scheduler writes spec.nodeName to the pod via the apiserver. That write is the scheduler's only output — and it happens through the same optimistic-concurrency path as everything else.

what the scheduler is not

It is stateless — it keeps a cached snapshot of the cluster (via informers) but holds no persistent queue. Unscheduled pods just sit in etcd with empty nodeName; a scheduler restart simply re-watches and picks up where it left off. It also never moves running pods — if a node dies, the pods on it are marked dead (not rescheduled); new pods get created by the controller that owns them (section 12).

preemption and priority

When nothing fits, a pod with a high priorityClass can trigger preemption: the scheduler picks victims of lower priority on the best-scoring node and evicts them (gracefully, via the apiserver), then binds the preemptor. It's polite but ruthless — this is how you guarantee "this job gets a node even if we have to throw someone off".

07 controllers & the informer pattern

kube-controller-manager is not one thing — it's dozens of controllers sharing a process: deployment, replicaset, endpoints, node-lifecycle, serviceaccount, garbage-collector, and more. Each one is the reconcile loop from section 02, wired up with three standard pieces:

apiserver ──watch──► informer ──► local cache ──► workqueue ──► reconcile()
                     (list, then        (events go in,    (rate-limited,
                      watch forever;     deduped,          retried w/
                      events feed        keyed by object)  backoff)
                      a delta queue)

• Informer: does a list once to build a local cache, then watches forever for changes. It's why controllers don't hammer the apiserver — they read their own cached copy and only get woken by change events. Every serious k8s client library implements this pattern (client-go's informers).
• Workqueue: events land as object keys, deduplicated (100 events for the same pod = 1 queue item), with rate limiting and exponential retry. If reconcile fails, the key goes back in the queue later. This is where "eventually consistent" lives.
• Reconcile: the controller's brain. Reads current state from its cache, compares with desired, makes changes via the apiserver, and (critically) expects the informer to tell it when reality settles.

the controller chain: deployment → replicaset → pod

Controllers compose. The deployment controller watches Deployments and creates/updates a ReplicaSet whose spec says "3 pods of image v2". The replicaset controller watches ReplicaSets, counts the pods matching its selector, and creates/deletes pods through the apiserver until the count matches. The scheduler assigns those pods to nodes. The kubelet runs them. Each layer is ignorant of the layers above and below — a deployment never talks to a pod, it just declares a ReplicaSet and trusts the chain.

Ownership is tracked with ownerReferences on each object. The garbage collector (another controller) watches these: delete a Deployment and its ReplicaSet and pods cascade-delete, because every orphan whose owner vanished is garbage. This is how kubectl delete deployment web tears down a whole tree with one object deletion.

08 the kubelet — where spec becomes process

The kubelet is the only component that talks to the container runtime, and it runs on every node. It watches the apiserver for pods with nodeName == its node, and for each one it's the local project manager: pull images, create namespaces, start containers, run probes, report status back.

CRI: one interface, many runtimes

The kubelet speaks the Container Runtime Interface (gRPC) to containerd or CRI-O; the runtime then drives an OCI-compliant low-level runtime (runc, kata, gVisor) that actually clone()s the process. This interface layering (CRI → runtime → OCI) is why kubernetes supports containerd, CRI-O, and exotic runtimes interchangeably — and why Docker-the-daemon became unnecessary: it was always just another CRI shim (removed in 1.24).

the pause container: a pod is a group, not a thing

A pod is a set of containers sharing namespaces. To make that concrete, the kubelet first starts a tiny pause/sandbox container that does nothing but sleep — it owns the pod's network and IPC namespaces. Every real container then joins those namespaces. Consequences:

• All containers in a pod share localhost (same netns) and an IP — talk to each other on loopback, no service discovery needed.
• Restarting your app container doesn't change the pod IP, because the pause container's netns survives.
• Sidecars work: a logging or proxy container in the same pod sees the same network and lifecycle.

namespaces + cgroups: isolation and fairness

Isolation comes from Linux namespaces (pid, net, ipc, mnt, uts, user): each container gets its own process table, filesystem view, and network stack. Fairness and limits come from cgroups (CPU shares, memory limits, pids limit, IO): this is where requests and limits become kernel policy.

• requests = reserved/guaranteed minimum. The scheduler only ever sums requests when deciding if a node has room.
• limits = hard ceiling. CPU over the limit gets throttled; memory over the limit gets OOM-killed (the container restarts).
• Set no limits and one leaky pod can eat the node — a classic incident. The kubelet does have node-level pressure eviction as a backstop (see QoS below).

probes: three questions, three different consequences

The distinction is the classic interview question and the classic production bug: a liveness probe that fails under transient load restarts healthy-but-slow apps in a death spiral; a readiness probe that fails gracefully drains traffic. Liveness kills, readiness detaches.

QoS classes: who dies first under pressure

From requests/limits, the kubelet derives a pod's QoS class, which sets OOM-kill priority when the node runs out of memory: BestEffort (no requests/limits) dies first, then Burstable (requests < limits), and Guaranteed (requests == limits on every container) dies last. This is why "set proper requests/limits" isn't style advice — it's literally the node's eviction policy, and BestEffort pods are your sacrificial lambs.

the kubelet as reporter

Everything the kubelet does, it reports back: container statuses, pod conditions (Ready, PodScheduled), node conditions, heartbeats (a lease renewed ~every 10s). The control plane has no telemetry other than what kubelets write into status fields. When kubectl describe node shows conditions, you're reading kubelet reports.

09 networking — one hard rule, then everything is layered on top

Kubernetes imposes exactly one requirement on the network: every pod gets its own IP, and every pod can reach every other pod directly, without NAT — across nodes, across the whole cluster. How that's implemented is deliberately unspecified, and that's what a CNI plugin (Container Network Interface) does: flannel (overlay via VXLAN), Calico (BGP or overlay), Cilium (eBPF).

Mechanically: the CNI plugin allocates a per-node subnet, creates a network namespace for the pod (the pause container's — see section 08), plugs in a veth pair (one end in the pod, one on the host), assigns the IP, and wires routes so traffic to any pod IP finds the right node. Overlay networks wrap packets so they survive routing between nodes; Calico-style L3 routes pod subnets directly when the underlying network allows it.

Services: a stable name for an unstable set of pods

Pod IPs die with their pods. A Service gives you a stable ClusterIP (virtual IP) + DNS name (web.default.svc.cluster.local) that load-balances to whatever pods currently match a label selector. The important mental model: a ClusterIP is not a listener — nothing is bound to it; ss -tlnp on a node will never show it. It exists as routing rules, programmed by kube-proxy:

• iptables mode (classic default): kube-proxy watches Services and EndpointSlices, and rewrites node-local iptables to DNAT the ClusterIP to a randomly chosen backend pod IP. A config change = rewrite thousands of rules (why huge clusters saw slow service updates).
• IPVS mode: uses the kernel's IPVS load balancer — faster, real scheduling algorithms, scales better.
• eBPF (Cilium): replaces kube-proxy entirely — service translation happens in the kernel at the socket level. No kube-proxy daemon at all.

Endpoints / EndpointSlices are the objects holding "which pod IPs back this service" — maintained by the endpoints controller, which watches pods and readiness. Readiness probe fails → pod IP removed from EndpointSlices → kube-proxy rewrites rules → traffic stops. That's the whole chain of "readiness detaches".

Headless Services (clusterIP: None) skip the VIP: DNS returns the pod IPs directly. This is how stateful apps find individual pods (see StatefulSets). NodePort opens the port on every node; LoadBalancer makes the cloud provision an LB in front (via cloud-controller-manager). Ingress is L7 routing (host/path → service) — an API object that needs an ingress controller (nginx, Contour…) running as a pod to actually implement it. People conflate "ingress" with "nginx pod" constantly; they're spec vs implementation.

DNS: CoreDNS runs in-cluster, watches Services/Pods, and answers service.namespace.svc.cluster.local queries. Every pod's /etc/resolv.conf points at it. Kube-DNS/CoreDNS going down is a full service-discovery outage — a genuinely nasty failure mode.

10 storage — decoupling providers from consumers

Storage follows the same shape as everything else in k8s: a contract (interface) between two parties that shouldn't know about each other.

• PV (PersistentVolume) — a cluster-scoped object representing actual storage: "there exists a 100Gi disk at NFS server X" or "an AWS EBS volume vol-abc123". Think of it as the storage itself, registered with the cluster. Provisioned by an admin, or dynamically by a provisioner.
• PVC (PersistentVolumeClaim) — a namespaced request: "I need 10Gi, read-write-once, fast class". An app pod references a PVC; the cluster matches it to a PV.

The decoupling is the point: the developer asks for capacity with a PVC; ops/cloud answers with PVs. The PV/PVC pair is the malloc of kubernetes — the claim is the malloc(), the volume is the heap block, and the binding is a one-to-one lease until the claim is deleted.

dynamic provisioning and storage classes

Nobody wants to hand-create PVs. A StorageClass names a provisioner plus parameters ("the cloud disk plugin, SSD tier, in eu-west-1a"), and a PVC that names the class causes a PV to be created on demand. The provisioner speaks CSI (Container Storage Interface) — the storage world's version of CRI/CNI: a standard gRPC interface that any vendor's driver implements, so k8s core never needs to know about EBS vs Ceph vs NetApp. Provision → attach → mount is the lifecycle; attachdetach controller + kubelet split the work, with volume topology keeping the pod on a node in the volume's zone.

Reclaim policies close the loop: PVC deleted → PV either Retains (waits for an admin), Deletes (cloud volume destroyed), or Recycles (wipe and reuse — legacy).

11 higher-level controllers — deployment, statefulset, and the operator insight

Everything in this section is, per section 07, just a controller generating specs. The differences are the policies they encode.

deployment: rolling updates as a two-replicaset trick

A Deployment manages a ReplicaSet per revision. On update: create new ReplicaSet with the new pod template, scale it up, scale the old one down — the overlap between the two curves is governed by maxSurge (how many extra pods may exist) and maxUnavailable (how many may be down). Rollback is just reversing the curve with the previous ReplicaSet (kept per revisionHistoryLimit). No magic: the "rolling update" you see is literally two ReplicaSets scaling against each other, which kubectl rollout status renders as a progress bar.

statefulset: identity for pods

Plain pods are cattle; a StatefulSet gives them name-shaped identities: pods get ordinal names (db-0, db-1, …), stable DNS (db-0.db.default.svc — hence headless services), stable per-pod PVCs, and ordered create/scale/update (one at a time, in order). This is what stateful systems need: a quorum member that comes back must come back as the same member, with the same disk, addressable by its peers.

But know the boundary: a StatefulSet gives identity, not immortality. The pod can still die and be rescheduled; the app still has to handle restart, resync, and re-election. Running etcd or Kafka on StatefulSets without understanding their consensus protocols is how you get db-2 forever-CrashLoopBackOff-ing.

daemonset, job, cronjob

• DaemonSet: exactly one pod per node, always. That's how kube-proxy, CNI agents, and log collectors deploy themselves. New node joins → pod appears there automatically.
• Job: run to completion, N completions, N parallel. restartPolicy: OnFailure/Never — a different failure model than services (retry vs replace).
• CronJob: a Job factory on a schedule — with its own timezone and concurrency-policy footguns.

HPA: scaling as another control loop

The HorizontalPodAutoscaler is a controller that watches metrics (CPU via metrics-server, custom via prometheus-adapter) and sets replicas on your Deployment every ~15s. It's reconcile loops all the way up: your scale target is just another spec field, and the HPA is just another controller. (VPA does the same vertically — requests/limits — by restarting pods, which is why it's scarier.)

CRDs and operators: the whole system in miniature

A CRD (CustomResourceDefinition) extends the apiserver with your own resource type — a PostgresCluster object that kubectl can get/apply/watch like anything built-in. That alone does nothing; an operator is what makes it real: a controller (informer + reconcile, section 07) watching your CRD, encoding domain knowledge — provision storage, bootstrap, replication, failover, backup, upgrade. The prometheus-operator, etcd-operator, all of it: Deployment-plus-StatefulSet-plus-backup-logic, written as a reconcile function.

The full-circle insight: a Deployment is a built-in operator for stateless apps. Operators aren't exotic machinery, they're the same machinery with your expertise in the loop.

12 failure modes — what actually breaks, and what doesn't

Kubernetes is designed around one architectural bet: the data plane must keep serving even when the brain dies. Here's the failure model, component by component.

a node dies

kubelet heartbeats stop → lease expires → node-lifecycle controller marks the node NotReady (~40s) → taints it NoExecute → pods are evicted and recreated elsewhere by their controllers (not "moved" — new pods, new IPs). DaemonSet pods tolerate the taint and wait. Meanwhile the dead node's workloads are simply gone: k8s doesn't do migration, it does replacement. If you have 1 replica of something, you have downtime — the controller can't resurrect what it doesn't know about.

apiserver dies

No kubectl, no scheduling, no edits. But kubelets have their assigned pods and keep running them, probes keep working, restarts happen locally. The cluster serves traffic but can't change. Hours-long apiserver outage with stable workloads = users never notice.

etcd loses quorum

Control plane frozen — worse than apiserver death because it's the persistence layer: no state can be written, and no new apiserver instance can help. Workloads still run. Recovery is the documented etcd disaster-recovery path (snapshot restore), not vibes. This is why etcd gets its own dedicated nodes, its own disks, and its own alerts.

control plane ↔ node network partition (the nasty one)

Node can't reach the apiserver, apiserver can't reach the node. Two things happen at once: the kubelet keeps its pods running (it can't hear the eviction order), and the control plane marks the node dead and schedules replacements of those same pods on healthy nodes. Result: duplicate pods, split state, two writers to the same database-backed service. K8s has no global "is this pod still alive" oracle beyond heartbeats. This is why multi-AZ designs pin stateful pods to zones (topology spread, node affinity) and why you run at least 3 replicas across failure domains.

kube-proxy / endpoints lag

A pod dies; until kube-proxy reprograms rules, connections hit a dead backend. Brief, self-healing, but it's why readiness + fast probes matter: a pod that's slow to be removed from endpoints is a pod still receiving traffic it can't serve.

memory pressure and QoS

Node OOM → kubelet evicts in QoS order: BestEffort first, then Burstable, Guaranteed last. Un-bounded pods (no limits) are the ones that take the node down and get everyone evicted. This is the concrete reason "set requests/limits" is infrastructure policy, not pedantry.

voluntary disruption: drains and PDBs

Draining a node (maintenance) evicts pods — a PodDisruptionBudget is your contract for "at least 2 of these must survive". Without PDBs, a rolling node drain can happily take your entire quorum down one member at a time. PDBs protect against voluntary disruption only; node death is involuntary and ignores them — that's what replicas are for.

priority and preemption

When the cluster is full, high-priority pods evict low-priority ones (section 06). Default priority is 0 — which means everyone is evictable. Production policy: explicit priorityClasses, and never run something critical at the default.

quota and limits

ResourceQuota caps total resources per namespace, LimitRange forces defaults on pods that don't set requests/limits. Namespaces without them are the wild west: one team's unbounded pod consumes the node and evicts your BestEffort pods. Multi-tenant clusters without quotas are single-tenant clusters with extra steps.

13 life of a pod — the whole system in one trace

Every component from this page, exercised by one kubectl apply. Read this top to bottom once and you've seen the whole architecture fire.

1. kubectl apply -f pod.yamlHTTP POST to the apiserver. Authn (who), authz (may they), admission (mutating webhooks inject sidecars, validating webhooks check policy) — then the pod object is committed to etcd with status Pending.
2. scheduler noticesIts informer sees a pod with no nodeName. Filters: resources fit (summing requests), taints tolerated, affinity matches. Scores: least-requested wins. Writes spec.nodeName back through the apiserver (optimistic concurrency: 409 → re-read → retry).
3. kubelet on the chosen node wakes upIts informer sees a pod assigned to it. Calls the runtime via CRI: create the pause/sandbox container — the pod's netns and IPC ns now exist.
4. CNI wires the networkThe CNI plugin allocates an IP from the node's subnet, creates the veth pair into the pod's netns, programs routes. The pod has an IP before any app container exists.
5. images pulled, containers startedcontainerd pulls (respecting imagePullPolicy, imagePullSecrets), creates namespaces, applies cgroup limits, execs the app. Init containers run first, in order, to completion; then app containers; then postStart hooks.
6. probes beginReadiness probe starts passing → kubelet writes containerStatuses.ready=true into pod status, sets the Ready condition. Until this moment, the pod is invisible to Services.
7. the endpoints controller reactsIt watches pods: a ready pod matching a Service's selector → its IP lands in an EndpointSlice. kube-proxy's informer sees that → rewrites iptables/IPVS so the ClusterIP can DNAT to this pod. Traffic flows.
8. steady statekubelet keeps probing, heartbeats every ~10s (node lease), status updates flow to etcd. All of this is just watches and status writes — the cluster's "running" state is a continuously refreshed database.
9. deletionkubectl delete → apiserver sets deletionTimestamp (finalizers, if any, must clear first — that's how operators run cleanup) → kubelet sends SIGTERM → preStop hook runs → grace period (terminationGracePeriodSeconds, default 30s) expires → SIGKILL → sandbox torn down → IP released.

14 interview questions — and the mechanisms behind the answers

These are the questions that separate "used kubernetes" from "understands kubernetes". Every answer is one paragraph from the sections above — the point of this page is that after reading it, you can give these cold.

How to use this section: cover the answer, say yours out loud, then check. If your answer starts with "well, in my last company we…" you've dodged the mechanism — and interviewers listen for exactly that dodge.

1. what actually happens when you kubectl apply a deployment?

Walk the chain, not the commands. The apiserver authenticates, authorizes, runs admission, and writes the Deployment object to etcd. The deployment controller (via its informer) sees it and creates a ReplicaSet. The ReplicaSet controller creates Pods. The scheduler binds each pod to a node (writing nodeName back to etcd). The kubelet sees the assignment, creates the sandbox, wires the network via CNI, starts containers. When ready, the endpoints controller adds the pod IP to the EndpointSlice, and kube-proxy programs the Service. Name five of those stages and you're ahead of most candidates.

2. liveness probe vs readiness probe — what's the actual difference?

Liveness failure restarts the container — "the process is broken, kill it". Readiness failure removes the pod from Service endpoints — "the process is fine, not ready to serve". The trap is the death spiral: a liveness probe that fails under load kills and restarts healthy-but-slow apps in a loop, while a readiness probe failing under the same load just drains traffic until things recover. Liveness kills, readiness detaches.

3. the apiserver just went down. are your applications down?

No — and that's by design. Pods already scheduled keep running: the kubelet doesn't need the apiserver to keep containers alive, run probes, or do local restarts. What stops: kubectl, scheduling, edits, deploys — the cluster can no longer change. It's an operations outage, not an availability outage, and the correct follow-up question is "for how long, and do I need to change anything in the meantime?"

4. what's the difference between requests and limits?

requests are the guaranteed minimum and the number the scheduler sums when bin-packing. limits are the hard ceiling: CPU past limit gets throttled, memory past limit gets OOM-killed. From the pair, the kubelet derives the pod's QoS class — BestEffort (neither set) dies first under node memory pressure, Guaranteed last. The one-liner: requests drive scheduling, limits drive the blast radius.

5. is a ClusterIP a real IP? where does it actually live?

It's virtual — nothing binds it; ip addr on any node will never show it. It exists as routing rules programmed by kube-proxy (or an eBPF CNI): a DNAT from the ClusterIP to one of the ready pod IPs in the EndpointSlice. DNS resolves the name to the ClusterIP; the ClusterIP is a symbol the data plane translates to a real pod. That's why readiness matters — an unready pod is simply absent from the EndpointSlice, so the rules never point at it.

6. a node just went NotReady. walk me through your investigation.

Read the node object first: conditions (MemoryPressure, DiskPressure, PIDPressure), last heartbeat time, events. NotReady ~40s after heartbeats stop means the kubelet stopped renewing its lease — so the question becomes "is the node up, is the kubelet up, what is it saying". Meanwhile the node-lifecycle controller will taint and evict its pods so they reschedule elsewhere. The order is always: is the machine alive → is the agent alive → what does the agent report → let the controllers converge.

7. a pod is stuck Pending. enumerate the reasons.

Pending means the scheduler hasn't bound it, which means a filter failed: no node with enough free requests, taints no one tolerates, nodeSelector/affinity matching nothing, an unbound PVC, hostPort conflicts, or simply no schedulable node. kubectl describe pod names it. The mechanism to recite: filters reject hard, scoring only ranks survivors — so it's never "the node was busy", it's "no node passed".

8. CrashLoopBackOff — how do you debug it?

Exit code first, logs second, config third. kubectl describe pod shows the previous container's exit code: 137 = OOM-killed (raise the limit), 1 = the app crashed (read the logs), 0 with a restarted pod = a probe killed it. kubectl logs --previous gets the dead container's last words. And distinguish it from ImagePullBackOff — same looping state, completely different problem (image name/registry/auth).

9. how does a rolling update actually work?

Two ReplicaSets scaling against each other: the deployment controller creates a new RS with the new template, scales it up while scaling the old one down, bounded by maxSurge (how many extra pods may exist) and maxUnavailable (how many may be missing). Rollback reverses the curves using the previous RS, which is kept per revisionHistoryLimit — that's all kubectl rollout undo is. No orchestration engine, just two counters moving.

10. a node is partitioned from the control plane, but the node itself is healthy. what happens?

The nastiest scenario in the system: the kubelet keeps running its pods (it can't hear the eviction order), while the control plane sees heartbeats stop and reschedules those same pods on healthy nodes. Now you have duplicates — potentially two writers to the same database. There's no global "is this pod still alive" oracle, which is why stateful apps get pinned to zones and quorum apps get 3+ replicas across failure domains. This answer, given unprompted, wins interviews.

11. when do you use a StatefulSet instead of a Deployment?

When pods need identity, not just count: stable ordinal names (db-0), stable DNS (hence headless services), stable per-pod PVCs, ordered operations. That's what quorum members need to recognize each other across restarts. But a StatefulSet gives identity, not immortality — the app must still survive kill/restart, and you must still understand its consensus protocol. Running etcd on StatefulSets without understanding raft is how you get db-2 in CrashLoopBackOff forever.

12. what is an operator, really?

A CRD plus a custom controller whose reconcile function encodes domain knowledge: provision, bootstrap, replicate, fail over, back up, upgrade. Mechanically it's the informer + workqueue + reconcile pattern, with your expertise in the loop. The punchline interviewers want: a Deployment is a built-in operator for stateless apps — operators aren't a different species, they're the same mechanism with more taste.

13. two pods share one PVC with ReadWriteOnce. can both use it?

ReadWriteOnce means one node, not one pod. Two pods on the same node can mount and share it. Two pods on different nodes both mounting the same RWO network volume is the classic silent data-corruption setup — filesystems don't handle two writers. Multi-node sharing means ReadWriteMany (usually NFS-like storage) or a redesigned data path (and block devices with multiple mounters is how you corrupt disks).

14. ingress vs ingress controller — what's the difference?

Ingress is an API object: a spec declaring host/path → service routing. It does nothing by itself. An ingress controller (nginx, Contour…) is software running as pods that watches Ingress objects and implements the routing. It's the same spec/implementation split the entire system runs on — and conflating the two is the single most common mid-level k8s mistake.

15. you're getting 409 Conflict responses on writes. what's happening?

Optimistic concurrency: every object carries a resourceVersion, and a write must send the version it read. Someone wrote in between — the write is rejected, and the correct response is re-read and retry. No locks, no deadlocks, just "if it moved, start over". Controllers doing this in a loop is the system working as designed, not an error.

16. etcd loses quorum. what stops working?

Writes. With fewer than n/2+1 members reachable, raft can't commit, so the apiserver can't persist anything: no kubectl edits, no scheduling, no status updates. Everything already running keeps running. It never corrupts or splits — a minority can't elect itself leader — it freezes. And you never run 4 nodes "for extra safety": 4 tolerates the same single failure as 3. Recovery is the documented snapshot-restore procedure, not improvisation.

15 terminology glossary

Every term from the page plus the ones you'll hear in interviews, one line each. Type to filter.

admission controller

Plugin chain in the apiserver that can mutate or reject requests before they hit etcd — the policy checkpoint (see webhook).

affinity / anti-affinity

Scheduling preferences/constraints: "prefer nodes with disk=ssd" (node affinity), "don't co-locate with pods matching X" (pod anti-affinity — how you spread replicas across hosts/zones).

annotation

Non-identifying metadata on objects (like labels, but not used for selection). Where tools stash state: kubectl.kubernetes.io/restartedAt, ingress annotations, etc.

API server (kube-apiserver)

The single gateway for all reads/writes and the only component that talks to etcd. Serves REST + watch; fronted by authn/authz/admission.

CNI

Container Network Interface — the plugin standard for pod networking (flannel, Calico, Cilium). CNI plugin = whatever actually implements "every pod has an IP".

CRI

Container Runtime Interface — the gRPC API the kubelet uses to talk to containerd/CRI-O. Decouples k8s from any specific runtime.

CSI

Container Storage Interface — the equivalent plugin standard for storage: vendors ship a driver, k8s provisions/attaches/mounts volumes through it.

ClusterIP

The default Service type: a virtual IP for the service, implemented as routing rules (not a listener). Stable entry point to an unstable set of pods.

ConfigMap

Non-secret key/value config, exposed to pods as env vars or files. Note: not encrypted, not secret storage.

container runtime

The software that actually runs containers on a node (containerd, CRI-O), driven by the kubelet through CRI.

controller

A reconcile loop: watches some object, compares spec vs status, acts to converge them. Deployment controller, node-lifecycle controller, your operator — all controllers.

control plane

The brain: etcd + apiserver + scheduler + controller-manager(s). Decides and stores; never runs your workloads.

cordon

Mark a node unschedulable (new pods won't land; existing ones stay). First step of a drain.

CRD

CustomResourceDefinition — extends the apiserver with your own resource type. The raw material operators are built from.

CronJob

Controller that creates Jobs on a schedule (cron syntax, with timezone + concurrency-policy settings).

DaemonSet

Controller guaranteeing exactly one pod per node — how kube-proxy, CNI agents, and node exporters run everywhere automatically.

Deployment

Controller managing a ReplicaSet per revision, giving you rolling updates, rollbacks, and scale. The default way to run stateless apps.

drain

Gracefully evict all pods from a node (respecting PDBs) before maintenance. kubectl drain node.

data plane

The nodes: kubelets, runtimes, kube-proxy, pods. Runs the workloads and routes the traffic.

EndpointSlice

Object holding the ready pod IPs backing a Service; maintained by the endpoints controller. kube-proxy watches it to program rules. (Supersedes the older flat Endpoints object.)

etcd

Distributed KV store (Raft consensus) holding all cluster state. Quorum of n/2+1; watches + leases are the features k8s is built on.

eviction

Forcibly removing a pod from a node — by the kubelet (node pressure), by the scheduler (preemption), by a drain (voluntary), or by the API (node NotReady).

finalizer

A marker on an object that blocks deletion until a controller clears it — the hook operators use to run cleanup before an object disappears.

garbage collector

Controller that deletes objects whose ownerReferences owner is gone. Makes kubectl delete deployment cascade through ReplicaSets to pods.

headless service

A Service with clusterIP: None: DNS resolves to the individual pod IPs instead of a VIP. Required for StatefulSet pod discovery.

HPA

HorizontalPodAutoscaler — controller that adjusts replicas based on metrics (~every 15s). Scaling is just another reconcile loop.

imagePullPolicy

When to pull the image: Always / IfNotPresent / Never. Classic pitfall: latest + IfNotPresent = stale code that "worked yesterday".

informer

Client-side list-then-watch cache of objects with an event stream. The pattern every controller and client library uses to avoid hammering the apiserver.

ingress

API object declaring L7 routing (host/path → service). Requires an ingress controller (nginx, Contour…) running as pods to actually take effect.

init container

Container that runs to completion before app containers start — migrations, config prep, waiting for dependencies. Separate image, shared pod resources.

Job

Controller that runs pods to completion (N completions, N in parallel) with retry semantics — for tasks, not services.

kubelet

The node agent. Watches pods assigned to its node, drives the runtime via CRI, runs probes, reports status. The only component that starts containers.

kube-proxy

Node agent implementing Services by programming iptables/IPVS rules (or absent entirely under eBPF CNIs). Watches Services + EndpointSlices.

label / selector

Key/value metadata on objects, and the query language matching it. The glue between pods and Services, Deployments, and almost everything else. Selectors are the only link between controller and pod.

lease

etcd key with TTL, renewable by holder — used for node heartbeats, leader election, and locks.

LimitRange

Namespace-level policy forcing defaults/min/max on requests and limits. Without it, pods without limits are legal and dangerous.

liveness probe

Health check that restarts the container on failure. For detecting deadlocked/broken processes — not for transient slowness (that's a death spiral).

LoadBalancer

Service type that provisions a cloud load balancer in front of NodePorts, via the cloud-controller-manager.

manifest

A YAML/JSON file describing desired objects. "Applying a manifest" = declaring state to the apiserver.

namespace

Virtual cluster: a scoping boundary for objects, names, quotas, and RBAC. Not a security boundary by itself (network isolation needs NetworkPolicy).

NetworkPolicy

Declarative pod-to-pod firewall. Only enforced if the CNI supports it — flannel alone enforces nothing; the default posture is open.

node

A worker machine (VM or bare metal) running kubelet + runtime + kube-proxy. Represented as a cluster object with capacity, conditions, and heartbeats.

NodePort

Service type that opens a port (30000–32767) on every node, forwarding to the service. Building block under LoadBalancer.

OCI

Open Container Initiative — the image format + runtime spec. runc/kata/gVisor are OCI runtimes; containerd/CRI-O manage them.

operator

Custom controller + CRD encoding domain knowledge (deploy, failover, backup, upgrade). A Deployment is a built-in operator for stateless apps.

ownerReference

Pointer from an object to its parent. Powers cascade deletion via the garbage collector.

pause container

The tiny sandbox container each pod starts with; it owns the pod's net/IPC namespaces so real containers can share them and the pod IP survives restarts.

PDB

PodDisruptionBudget — "at least N of these must survive" contract, honored by voluntary disruptions (drains). Node death ignores it.

pod

The smallest schedulable unit: one or more containers sharing network/IPC namespaces, IP, volumes, and lifecycle. The atom of the system.

preemption

High-priority pod evicting lower-priority pods to win a node when the cluster is full. Governed by PriorityClass.

PriorityClass

Named priority level for pods; drives preemption order. Default is 0 — and everyone's evictable at 0.

probe

Health check the kubelet runs against a container (exec/HTTP/TCP). Startup gates the others; liveness restarts; readiness detaches from services.

PV / PVC

PersistentVolume = registered storage (the resource); PersistentVolumeClaim = a request for storage (the malloc). Binding links them 1:1.

QoS class

Guaranteed / Burstable / BestEffort, derived from requests/limits — the OOM kill order under node memory pressure. BestEffort dies first.

quorum

Majority of etcd members required to commit writes (n/2+1). Lose it and the cluster stops persisting state; split-brain is impossible by design.

RBAC

Role-Based Access Control: Roles/ClusterRoles (permissions) bound to users/serviceaccounts via (Cluster)RoleBindings. The authorization layer.

readiness probe

Health check that removes the pod from service endpoints on failure. Alive but not serving — drain traffic, don't restart.

replica

One copy of a pod, managed by a controller. Replicas are fungible by default — that's the entire premise of stateless apps on k8s.

ReplicaSet

Controller ensuring N pods matching a selector exist. Deployments drive ReplicaSets; ReplicaSets drive pods.

requests / limits

requests = guaranteed minimum (what the scheduler sums); limits = hard ceiling (CPU throttled, memory OOM-killed past it). The basis of QoS and scheduling.

ResourceQuota

Namespace-level cap on total resource consumption (CPU, memory, object counts). Multi-tenancy's seatbelt.

resourceVersion

Object version stamp (from etcd's revision counter). Sent back on updates for optimistic concurrency — changed underneath you → 409 → re-read, retry.

scheduler

Component that assigns pods to nodes: filter (hard constraints) then score (preferences), then binds via spec.nodeName. Stateless; pods wait in etcd, not in its memory.

secret

Object for sensitive data (tokens, keys). Base64, not encrypted by default — enable encryption-at-rest, and prefer mounted files over env vars.

service

Stable name + VIP over a set of pods (label selector), load-balanced by kube-proxy. The decoupling of "who am I" from "where do I run".

ServiceAccount

Identity for pods; every pod gets one automatically, with a token the apiserver authenticates. What RBAC roles are bound to.

sidecar

Auxiliary container in the same pod, sharing network/volumes: log shippers, service-mesh proxies, config refreshers. Same lifecycle as the main app.

spec / status

The k8s contract: you write spec (desired), the system writes status (actual). Controllers reconcile the two.

StatefulSet

Controller giving pods stable identity: ordinal names, stable DNS, per-pod PVCs, ordered operations. Identity, not immortality — the app still handles restarts.

static pod

Pod the kubelet runs from a local manifest directory, not from the apiserver — how kubeadm runs the control plane itself.

StorageClass

Named class of storage + provisioner params; a PVC referencing it triggers dynamic PV creation via CSI. The "instance type" of volumes.

taint / toleration

Taints repel pods from a node ("GPU only", "draining"); tolerations let specific pods land anyway. Scheduling's No-Entry signs.

terminationGracePeriodSeconds

Time between SIGTERM and SIGKILL on pod deletion (default 30s). Your app's window to drain connections (with preStop hooks).

watch

The streaming API: subscribe to object changes instead of polling. The event backbone of every controller.

workqueue

Per-controller queue: dedupes events by key, rate-limits, retries with backoff. Where "eventually consistent" is implemented.

webhook (admission)

Your own HTTPS endpoint called during admission to mutate/validate requests — the extension point for org policy (sidecar injection, image allowlists).