| 1 | # Kubernetes Cluster Architecture |
| 2 | |
| 3 | > Research document for the Blueprint / Isometric capability-showcase deck. Source material gathered from `kubernetes.io` official documentation (2026-05). Concrete facts only; the Strategist composes the final slide copy. |
| 4 | |
| 5 | ## 1. Cluster topology overview |
| 6 | |
| 7 | A Kubernetes cluster is a two-plane system: |
| 8 | |
| 9 | - **Control plane** — global decision-making. Schedules workloads, reacts to events, holds the desired-state of every API object. |
| 10 | - **Data plane (worker nodes)** — runs the actual application Pods. One or more nodes; production HA clusters spread the control plane and etcd across ≥3 nodes and across availability zones. |
| 11 | |
| 12 | Minimum cluster = 1 control-plane host + 1 worker node. Production = ≥3 control-plane hosts, ≥3 etcd members (odd numbers for quorum), worker pool sized by workload. |
| 13 | |
| 14 | All components communicate through **kube-apiserver** — the only component that talks to etcd. Every read/write of cluster state goes through the API server. This is the architectural spine. |
| 15 | |
| 16 | ## 2. Control plane components |
| 17 | |
| 18 | Five components run the control plane: |
| 19 | |
| 20 | ### 2.1 kube-apiserver |
| 21 | - Front-end of the cluster; exposes the Kubernetes HTTP API |
| 22 | - The **only** component that talks to etcd directly |
| 23 | - Horizontally scalable — multiple `kube-apiserver` instances behind a load balancer |
| 24 | - Validates and admits requests, runs admission controllers, then persists to etcd |
| 25 | |
| 26 | ### 2.2 etcd |
| 27 | - Distributed, consistent, highly-available key-value store |
| 28 | - Holds **all** cluster state (API objects, config, secrets) |
| 29 | - Raft consensus protocol; needs an odd number of members (typically 3 or 5) |
| 30 | - Backup is operationally critical — losing etcd loses the cluster |
| 31 | - HA topologies: stacked (etcd co-located with control-plane host) vs. external etcd |
| 32 | |
| 33 | ### 2.3 kube-scheduler |
| 34 | - Watches the API server for Pods with no `nodeName` assigned |
| 35 | - Picks a node for each unscheduled Pod based on: |
| 36 | - Resource requests (CPU / memory / GPU) |
| 37 | - Hardware / software / policy constraints |
| 38 | - Affinity & anti-affinity rules |
| 39 | - Taints & tolerations |
| 40 | - Topology spread, data locality, inter-workload interference |
| 41 | - Two-phase pipeline: **filtering** (which nodes are feasible?) → **scoring** (which feasible node is best?) |
| 42 | - Writes the binding decision back to the API server — does not directly talk to the node |
| 43 | |
| 44 | ### 2.4 kube-controller-manager |
| 45 | - One binary, many logically separate controllers; each runs a `watch → diff desired-vs-actual → act` reconcile loop |
| 46 | - Core controllers: |
| 47 | - **Node controller** — detects and responds to node failures |
| 48 | - **Job controller** — creates Pods to drive Job objects to completion |
| 49 | - **EndpointSlice controller** — maps Services to Pod IPs |
| 50 | - **ServiceAccount controller** — creates default ServiceAccounts in new namespaces |
| 51 | - **ReplicaSet / Deployment / StatefulSet / DaemonSet controllers** — workload primitives |
| 52 | - Leader-elected for HA: only one replica is active at a time |
| 53 | |
| 54 | ### 2.5 cloud-controller-manager (optional) |
| 55 | - Only on cloud-hosted clusters; absent in on-premises / bare-metal / local environments |
| 56 | - Embeds cloud-vendor-specific control logic so the core Kubernetes binary stays vendor-neutral |
| 57 | - Sub-controllers: |
| 58 | - **Node controller** — confirms whether deleted nodes really were deleted at the cloud provider |
| 59 | - **Route controller** — sets up cloud routes between nodes |
| 60 | - **Service controller** — creates / updates / deletes cloud load balancers for Services of type `LoadBalancer` |
| 61 | |
| 62 | ## 3. Worker node components |
| 63 | |
| 64 | Three components run on every worker node: |
| 65 | |
| 66 | ### 3.1 kubelet |
| 67 | - The node-level agent |
| 68 | - Watches the API server for Pods scheduled to its node |
| 69 | - Takes PodSpecs, instructs the container runtime to start the containers, supervises health via probes |
| 70 | - Reports node and Pod status back through the API server |
| 71 | - Reconciles continuously — if a container dies, kubelet restarts per `restartPolicy` (`Always` / `OnFailure` / `Never`) |
| 72 | - Heartbeats via Lease objects in the `kube-node-lease` namespace |
| 73 | |
| 74 | ### 3.2 kube-proxy (optional) |
| 75 | - Implements the Kubernetes **Service** abstraction at the node level |
| 76 | - Maintains network rules that translate the Service virtual IP into a backing Pod IP |
| 77 | - Implementation backends: |
| 78 | - `iptables` (default) — netfilter rules, lower CPU overhead |
| 79 | - `IPVS` — kernel L4 load balancer, better at very large Service counts |
| 80 | - `nftables` — modern netfilter framework (newer clusters) |
| 81 | - `userspace` — legacy, slower, deprecated |
| 82 | - Optional: a CNI plugin (Cilium, Calico-eBPF) can replace kube-proxy entirely |
| 83 | |
| 84 | ### 3.3 Container runtime |
| 85 | - Actually runs the containers |
| 86 | - CRI (Container Runtime Interface) standardizes the contract between kubelet and the runtime |
| 87 | - Supported runtimes: **containerd**, **CRI-O**, any CRI-conformant implementation |
| 88 | - Docker Engine removed as a built-in runtime (since v1.24) |
| 89 | |
| 90 | ## 4. Pod lifecycle |
| 91 | |
| 92 | A Pod is the smallest deployable unit — one or more containers sharing a network namespace and storage volumes. |
| 93 | |
| 94 | **Phases** (set by the control plane): |
| 95 | |
| 96 | | Phase | Meaning | |
| 97 | |---|---| |
| 98 | | `Pending` | Accepted; not all containers running yet (waiting on scheduling, image pull, etc.) | |
| 99 | | `Running` | Bound to a node; ≥1 container is starting / running / restarting | |
| 100 | | `Succeeded` | All containers exited 0; will not restart | |
| 101 | | `Failed` | All containers exited; ≥1 with non-zero status | |
| 102 | | `Unknown` | API server cannot reach the node holding the Pod | |
| 103 | |
| 104 | **Container states**: `Waiting` / `Running` / `Terminated` (each container in the Pod is tracked independently) |
| 105 | |
| 106 | **Probes** (kubelet runs them): |
| 107 | - `livenessProbe` — fail ⇒ kubelet restarts the container |
| 108 | - `readinessProbe` — fail ⇒ kubelet removes Pod from Service endpoints (no restart) |
| 109 | - `startupProbe` — gates the other two until the application has finished initializing |
| 110 | - Probe methods: `exec` (run command) / `httpGet` / `tcpSocket` / `grpc` |
| 111 | |
| 112 | **Termination flow** (graceful, default 30s): |
| 113 | 1. API server marks the Pod for deletion |
| 114 | 2. `preStop` hook (if defined) runs in the container |
| 115 | 3. `SIGTERM` to the container main process |
| 116 | 4. Up to `terminationGracePeriodSeconds` to exit cleanly |
| 117 | 5. `SIGKILL` if still alive |
| 118 | |
| 119 | A Pod is scheduled **once** in its lifetime. Failed Pods are replaced (by their controller — Deployment / StatefulSet / Job — with a new UID), not rescheduled. |
| 120 | |
| 121 | ## 5. Service networking |
| 122 | |
| 123 | A Service gives a stable virtual IP and DNS name in front of an ever-changing set of Pods. |
| 124 | |
| 125 | **Service types**: |
| 126 | |
| 127 | | Type | Where reachable | How | |
| 128 | |---|---|---| |
| 129 | | `ClusterIP` (default) | Inside the cluster only | Virtual IP in the Service CIDR; kube-proxy programs the iptables/IPVS rules | |
| 130 | | `NodePort` | Any node's IP at a static port (30000–32767) | NodePort opens on every node; routes to the ClusterIP | |
| 131 | | `LoadBalancer` | External cloud LB | cloud-controller-manager provisions the LB; targets the NodePort | |
| 132 | | `ExternalName` | DNS CNAME | API server returns a CNAME record; no proxying | |
| 133 | |
| 134 | **EndpointSlices** (replaced the older `Endpoints` resource in v1.21): track the Pod IPs backing a Service. Controller continuously rebuilds the slice as Pods come and go. |
| 135 | |
| 136 | **DNS**: every Service is resolvable at `<svc>.<namespace>.svc.cluster.local`. Cluster DNS (CoreDNS) is a mandatory addon. |
| 137 | |
| 138 | **Headless Service** (`clusterIP: None`): returns Pod IPs directly via DNS A records, no virtual IP — used by StatefulSets so each replica gets its own resolvable name. |
| 139 | |
| 140 | ## 6. Storage |
| 141 | |
| 142 | Persistent storage in Kubernetes is split across three resources: |
| 143 | |
| 144 | - **PersistentVolume (PV)** — a piece of provisioned storage; cluster-scoped resource with its own lifecycle |
| 145 | - **PersistentVolumeClaim (PVC)** — a Pod's request for storage; namespaced |
| 146 | - **StorageClass** — an admin-defined recipe for dynamically provisioning PVs on demand |
| 147 | |
| 148 | **Access modes**: |
| 149 | |
| 150 | | Mode | Meaning | |
| 151 | |---|---| |
| 152 | | `ReadWriteOnce` (RWO) | Mountable read-write on a single node | |
| 153 | | `ReadOnlyMany` (ROX) | Mountable read-only on many nodes | |
| 154 | | `ReadWriteMany` (RWX) | Mountable read-write on many nodes | |
| 155 | | `ReadWriteOncePod` (RWOP) | Mountable read-write by a single Pod | |
| 156 | |
| 157 | **Reclaim policies** (what happens when the PVC is deleted): `Retain` / `Delete` / `Recycle` (deprecated). |
| 158 | |
| 159 | **CSI (Container Storage Interface)** is the modern plugin contract — every storage backend (cloud block storage, NFS, Ceph, etc.) ships a CSI driver, the in-tree plugins are deprecated. |
| 160 | |
| 161 | **Volume modes**: `Filesystem` (default — mounted as a directory) vs. `Block` (raw block device). |
| 162 | |
| 163 | ## 7. High availability and topology |
| 164 | |
| 165 | For production: |
| 166 | |
| 167 | - **Control plane** — ≥3 control-plane hosts, kube-apiserver load-balanced, kube-scheduler and kube-controller-manager leader-elected |
| 168 | - **etcd** — ≥3 members across hosts; **odd number** for quorum (3 or 5) |
| 169 | - **Multi-zone** — spread control-plane hosts and worker nodes across availability zones; lose one zone, the cluster keeps going |
| 170 | - **Two topologies** for control plane: |
| 171 | - **Stacked** — each control-plane host runs an etcd member alongside kube-apiserver (simpler, less HW) |
| 172 | - **External etcd** — etcd cluster on dedicated hosts (better failure isolation, more HW) |
| 173 | |
| 174 | ## 8. Self-healing and observability |
| 175 | |
| 176 | What Kubernetes recovers from automatically: |
| 177 | |
| 178 | - Container crash → kubelet restarts per `restartPolicy` with exponential backoff (100ms → 5min cap, resets after 10min healthy) |
| 179 | - Pod failure → controller (Deployment / ReplicaSet / StatefulSet) creates a new Pod |
| 180 | - Node failure → node controller marks the node `NotReady`; after `pod-eviction-timeout` (default 5min) Pods on it are marked for deletion, controllers replace them on other nodes |
| 181 | - Workload imbalance → kube-scheduler places the new Pods on under-loaded nodes |
| 182 | |
| 183 | What it does **not** recover from: |
| 184 | |
| 185 | - Lost etcd quorum — cluster state is gone, manual restore from backup required |
| 186 | - Lost API server — no new decisions possible, existing workloads keep running on their nodes until kubelet hits its watch timeout |
| 187 | |
| 188 | **Observability addons** (universal in production): |
| 189 | |
| 190 | - **DNS** (CoreDNS) — required; cluster DNS resolution |
| 191 | - **Metrics** — metrics-server + Prometheus |
| 192 | - **Logs** — Fluent Bit / Fluentd → central store (Loki, Elasticsearch, cloud log services) |
| 193 | - **Dashboard** — optional web UI |
| 194 | |
| 195 | ## 9. Component communication summary |
| 196 | |
| 197 | The traffic pattern is simple: |
| 198 | |
| 199 | ``` |
| 200 | kubectl / clients ──HTTPS──► kube-apiserver ◄──gRPC──► etcd |
| 201 | ▲ |
| 202 | │ watch / report |
| 203 | ┌───────────────────────────┼───────────────────────────┐ |
| 204 | │ │ │ |
| 205 | kube-scheduler kube-controller-manager cloud-controller-manager |
| 206 | │ |
| 207 | │ watch |
| 208 | ┌───────────────────────────┼───────────────────────────┐ |
| 209 | │ │ │ |
| 210 | kubelet (node 1) kubelet (node 2) kubelet (node N) |
| 211 | │ │ │ |
| 212 | ▼ ▼ ▼ |
| 213 | container runtime container runtime container runtime |
| 214 | + kube-proxy + kube-proxy + kube-proxy |
| 215 | ``` |
| 216 | |
| 217 | Every line is API-server-mediated. No component talks directly to another except via the API server (and only the API server talks to etcd). This is what makes Kubernetes pluggable and observable — every state transition is an API event. |
| 218 | |
| 219 | ## Sources |
| 220 | |
| 221 | - [Cluster Architecture | Kubernetes](https://kubernetes.io/docs/concepts/architecture/) |
| 222 | - [Kubernetes Components | Kubernetes](https://kubernetes.io/docs/concepts/overview/components/) |
| 223 | - [Service | Kubernetes](https://kubernetes.io/docs/concepts/services-networking/service/) |
| 224 | - [Pod Lifecycle | Kubernetes](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/) |
| 225 | - [Persistent Volumes | Kubernetes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) |
| 226 |