Containers

A pod can look healthy in Kubernetes while the model behind it is still not ready to answer a request. That is the gap I wanted to measure. Kubernetes Ready means the pod passed the readiness condition you configured. It does not automatically mean the model is loaded, resident in memory, or able to complete inference. For a normal web service, an HTTP check is often good enough. With an LLM serving pod, it can be too shallow. The process may be running. The API may respond. The model file may even be on disk. The first real request can still spend several seconds loading the model before it completes. I ran controlled Ollama recovery experiments on Kubernetes to see how big that window was. Ready Is Only One Point in the Recovery Path I measured five timestamps: Plain Text T0 - pod replacement requested T1 - Kubernetes reports Ready T2 - inference runtime responds to HTTP T3 - first post-recovery inference request begins T4 - inference request completes successfully Figure 1: Kubernetes Ready vs. Functional Recovery That gave me four useful timings: Plain Text Kubernetes recovery = T1 - T0 Runtime recovery = T2 - T0 Functional recovery = T4 - T0 Ready -> inference gap = T4 - T1 The last one is where the problem becomes visible. The Results I ran 10 pod-replacement tests for each configuration: local Minikube on Mac, CPU-onlyAzure Standard_D16s_v5 Linux VM running Minikube, CPU-onlyOllamallama3.2:1bllama3.2:3bsame 2 CPU / 4 GiB container limit for the 1B and 3B comparison Figure 2: LLM Recovery Experiment Architecture Mean results: MetricLocal 1BLocal 3BAzure 1BAzure 3BKubernetes Ready1.66 s1.96 s1.61 s1.69 sRuntime reachable2.43 s2.44 s2.19 s2.17 sFunctional recovery11.11 s16.27 s5.43 s7.73 sReady -> inference9.45 s14.31 s3.83 s6.05 sModel load5.51 s8.60 s2.16 s3.96 s Kubernetes reported the pod Ready in about two seconds or less in all four configurations. Successful inference came later. The mean Ready-to-inference gap ranged from about 3.8 seconds to 14.3 seconds. The Azure environment was faster than the local environment for the inference-dependent part of recovery, but the gap was still there. I did not try to explain the cross-platform difference with one cause. CPU, storage, virtualization, architecture, and cache behavior can all affect the result. The point was simpler: Kubernetes recovery and inference recovery were not the same event. There Is More Than One Kind of "Ready" The experiments also exposed a few other states that are easy to mix together. The Runtime Can Be Up While the Model Is Gone One early version used emptyDir for Ollama model storage. After pod replacement, Ollama started normally. But: Shell ollama list returned no model. The runtime had recovered. The model artifact had not. Moving the model data to a PVC fixed the persistence problem. The Model Can Be on Disk Without Being Loaded A larger llama3.1:8b test made this very clear. Before inference, ollama list showed the model artifact, but ollama ps showed nothing resident. Cgroup memory usage was only around 14 MiB. After the first request, the model became resident and memory rose to roughly 5.27 GiB. So "model exists" and "model is ready to serve" are different checks. A Warm Node Can Make Recovery Look Better I also ran 10 warm-cache and 10 cold-cache tests for the 3B model on the same Azure node. For the cold condition: Shell sync echo 3 > /proc/sys/vm/drop_caches This clears the Linux page cache, dentries, and inode caches. It is a host filesystem/page-cache test, not an Ollama-specific model cache. metricwarmcoldFunctional recovery7.58 s8.09 sReady -> inference5.70 s6.26 sModel load3.95 s4.61 sRequest wall time5.11 s5.70 s Model load increased by about 16.6% under the cold condition. Kubernetes recovery barely moved. That is a useful warning for repeated recovery tests on the same node: the host may be helping more than you realize. Model Residency Can Overlap During memory testing, loading the 3B model under a 4 GiB limit once failed with: Shell signal: killed It looked like the 3B model did not fit. That was not the actual problem. A 1B model from an earlier request was still resident. When I tested the 3B model alone under the same limit, it worked, and the cgroup showed no OOM kill. The failure came from overlapping residency, not the 3B model by itself. A simple runtime health check would not have told me that. So What Should Readiness Check? A normal readiness probe usually asks something like: Plain Text Is the HTTP endpoint responding? That proves the runtime is reachable. For an LLM workload, I care about a stronger question: Plain Text Can this pod actually complete inference with the model it is supposed to serve? One way to test that is with a minimal inference request: YAML readinessProbe: exec: command: - sh - -c - | curl -sf -X POST http://localhost:11434/api/generate \ -H 'Content-Type: application/json' \ -d '{"model":"llama3.2:1b","prompt":"ping","stream":false}' \ | grep -q '"done":true' periodSeconds: 2 failureThreshold: 1 The exact command will depend on the serving image. The point is not curl. The point is that readiness now checks the model-serving path, not just the process. What Happened During Rollouts? For the 3B readiness test, I sampled Kubernetes EndpointSlice state at roughly 0.5-second intervals during 10 local rollouts and 10 Azure rollouts. metriclocal 3Bazure 3BMean new-endpoint non-serving duration47.6 s11.0 sSampled intervals with zero ready + serving endpoints00Rollouts observed1010 Across those 20 rollouts, I did not observe a sampled interval with zero ready-and-serving endpoints. That is not the same as proving packet-level availability between every sample. What it does show is that the replacement endpoint stayed out of Service eligibility until the inference-aware readiness condition succeeded. That is much closer to what I wanted Ready to mean. Readiness Is a Contract This was the main lesson for me. Readiness is not a universal definition of application health. It is a contract between the workload and Kubernetes. For a normal API, the contract might be: Plain Text My process is initialized and can accept requests. For an LLM workload, it may need to be closer to: Plain Text The runtime is running. The model exists. The model can be loaded. Inference can complete. If the probe only checks the first line but the team reads Ready as all four, the problem is not Kubernetes. The signal is just weaker than the expectation. What This Does Not Prove These tests were CPU-only. They used Ollama. They measured same-node pod replacement. And they used 10 repetitions per condition. So the numbers here should not be treated as universal timings or production SLAs. Cold-node relocation is also a separate problem. Moving an LLM workload to another node brings node-local cache state and possibly image or model acquisition into the recovery path. I am measuring that separately rather than mixing it into these same-node results. Takeaway In these experiments, Kubernetes readiness came back quickly. Inference recovery followed a different timeline. The mean Ready-to-inference gap ranged from about 3.8 seconds to 14.3 seconds, depending on the model and environment. The fix is not to distrust Kubernetes. It is to make the readiness condition represent the state you actually care about. A pod can be healthy. The runtime can answer HTTP. The model can exist on disk. And inference can still not be ready. Those are different states. For the full experiment setup, raw results, methodology, environment captures, and ongoing cold-node work, see the project write-up and repository: https://github.com/opscart/k8s-llm-recovery-lab. For the full experiment setup, methodology, raw results, environment captures, and ongoing cold-node work, see the complete OpsCart write-up and project repository.

By Shamsher Khan CORE

Every Java developer who runs services on Kubernetes has watched this scene play out. Traffic spikes, the autoscaler adds a pod, and then everyone waits. The container is running in two seconds. The application is not ready for another twelve seconds. During those ten seconds, your existing pods absorb the extra load, latency climbs, and if things are bad enough, the autoscaler panics and adds even more pods that are also not ready. I spent years treating Spring Boot startup time as a fact of life, the way you treat weather. Then I found out the JVM has had a fix for a big chunk of it since Java 12; it works beautifully inside Docker, and almost nobody bakes it into their images. It is called Class Data Sharing, CDS for short, and this article shows you how to make your Docker build do the work Where Those Twelve Seconds Actually Go When a Spring Boot application starts, the JVM is not mostly running your code. It is loading classes. A plain REST service with Spring Web, Spring Data, and a driver or two loads somewhere between ten and twenty thousand classes before it serves its first request. For every single one of those classes, the JVM does the same ritual. Find the class file inside a jar, read the bytes, parse them, verify the bytecode is legal, and build the internal metadata structures it needs at runtime. Thousands of times. Every startup. In every pod. Here is the part that should bother you. Your container image never changes after you build it. The same jar, the same classes, the same parsing work, repeated identically in every pod that ever starts from that image. The JVM is solving the same puzzle again and again and throwing away the answer each time. CDS is the JVM saying: let me solve it once, write the answer to a file, and just memory map that file next time. What a CDS Archive Is A CDS archive is a file, usually ending in .jsa, that contains classes already parsed and verified, stored in the exact internal format the JVM uses in memory. On startup, the JVM maps this file straight into memory. No finding, no parsing, no verifying. The work was done ahead of time. You have been using CDS without knowing it. Modern JDKs ship with a default archive covering the core JDK classes, which is why java -version is fast. The step almost everyone skips is creating an archive for your application classes, all fifteen thousand of them. That is where the real win lives. The mechanism has one rule that matters for us. The archive must be created with the same JVM and the same classpath that will use it. That rule sounds annoying until you realize a Docker image is the one place in your entire infrastructure where JVM and classpath are frozen forever. Docker is not just compatible with CDS. It is the perfect home for it. The Training Run Creating the archive takes two steps. First you do a training run, where the JVM starts your application, watches which classes get loaded, and writes the list down. Then you exit, and the JVM turns that list into the archive. Since Java 13, this is pleasantly simple: Shell java -XX:ArchiveClassesAtExit=app.jsa -jar app.jar Run the app, let it come up, stop it, and app.jsa appears. From then on you start the app like this: Shell java -XX:SharedArchiveFile=app.jsa -jar app.jar There is an obvious question here. The training run wants to actually start the application, and inside docker build there is no database, no message broker, nothing to connect to. A Spring Boot app that cannot reach Postgres will crash during training. Spring Boot 3.3 solved this neatly. Setting one property makes the application run through its entire startup sequence, create all bean definitions, and then exit just before touching the outside world: Shell java -Dspring.context.exit=onRefresh -XX:ArchiveClassesAtExit=app.jsa -jar app.jar The application loads nearly everything it will ever load, writes the archive, and exits cleanly with no infrastructure needed. This is exactly what a Docker build stage can do. The Dockerfile Here is the complete picture: a multi-stage build where the image trains itself: Shell FROM eclipse-temurin:21-jdk-alpine AS build WORKDIR /build COPY . . RUN ./mvnw -B package -DskipTests # Explode the jar so the classpath is stable RUN java -Djarmode=tools -jar target/app.jar extract --destination /app FROM eclipse-temurin:21-jre-alpine AS runtime WORKDIR /app COPY --from=build /app /app # Training run: start the context, record classes, exit RUN java -Dspring.context.exit=onRefresh \ -XX:ArchiveClassesAtExit=/app/app.jsa \ -jar /app/app.jar ENV JAVA_TOOL_OPTIONS="-XX:SharedArchiveFile=/app/app.jsa" ENTRYPOINT ["java", "-jar", "/app/app.jar"] Two details in there deserve a closer look. The extract step unpacks the fat jar into a folder with the dependencies laid out as plain files. CDS is picky about the classpath being identical between training and real runs, and a fat jar with nested jars inside it makes that fragile. The exploded layout keeps the classpath boring and stable, which is exactly what CDS wants. On Spring Boot 3.2 and older, the same idea works through the layertools jarmode instead. The training run happens as a RUN instruction, which means it executes once at build time on your CI server. Every container that ever starts from this image inherits the archive for free. You did the class loading homework once, in the build, and ten thousand pod starts copy the answer. What You Get Numbers vary with how heavy your application is, but the pattern is consistent. A typical Spring Boot 3 web service that started in 10 to 12 seconds lands somewhere between 5 and 7. The JVM portion of startup shrinks dramatically, and as a bonus, the archive is memory-mapped and shared, so if you run several JVMs on one node, they share those pages and total memory drops too. You can verify the archive is actually being used, which I recommend, because CDS fails silently by design. If something mismatches, it just quietly falls back to normal class loading: Shell docker run --rm my-service -Xlog:class+load=info | head -5 Classes loaded from the archive say source: shared objects file. If you see jar paths instead, the archive is being ignored, and the log will usually tell you why. The usual culprit is a classpath that differs from training, even by one entry. One honest caveat. The training run exercises startup, not your traffic. Classes that only load when a specific endpoint gets hit for the first time are not in the archive, so those first requests still do normal loading. The archive covers the framework and wiring, which is most of the cost, but it is not a magic warm-up for everything. Why This Beats the Alternatives You Have Heard Of Whenever container startup time comes up, someone mentions GraalVM native images, and native images are impressive. Millisecond startup is real. But they come with a price list: long build times, a closed-world assumption that fights with reflection, some libraries that simply do not work, and a different runtime profile you have to learn to debug. CDS costs you five lines of Dockerfile. Your application is still a completely normal JVM application. Same debugging, same profilers, same libraries, same behavior, just faster out of the gate. For most teams, that trade-off is not even close. It also stacks with what is coming. Project Leyden's AOT cache in Java 24 and beyond is essentially this same idea grown up, caching not just parsed classes but resolved linkage and compiled code. The Dockerfile pattern you build today, a training run at build time producing a cache file shipped in the image, is exactly the shape Leyden uses. Learning it now means the future is a flag change. The Takeaway Your Docker image is immutable. Your JVM does expensive, perfectly repeatable work on every startup. Those two facts fit together like puzzle pieces, and a training run inside docker build is where they connect. One extra build step, and every pod your autoscaler ever creates comes up in half the time. The next time you watch a rollout crawl because pods take forever to go ready, remember that the answer was hiding inside the build all along.

What if you could multiplex roughly 250 stateful agent sessions across eight Kubernetes worker Pods, then reactivate any one without losing its in-memory or filesystem state? The repository's demo reports 30x+ actor-to-worker oversubscription for that sample workload, with sub-second activation. It is a demonstration, not a production capacity guarantee. Agent Substrate is interesting not because it makes Kubernetes faster, but because it challenges a common deployment pattern around Kubernetes: coupling a workload's logical lifecycle to the compute allocated to run it. For AI agents that spend most of their time idle while retaining valuable state, separating those lifecycles could become an important building block for operating agents at significantly higher density. Kubernetes gave us an exceptionally durable abstraction for running workloads: the Pod. But emerging agent workloads expose places where that abstraction may become inefficient. They can be stateful, sandboxed, and overwhelmingly idle. A one-agent-per-Pod deployment model, while simple, couples each session's logical lifecycle to a Pod's runtime lifecycle. When sessions spend much of their time idle, that coupling can leave compute capacity allocated to workloads that are not actively executing. This is not a claim that Kubernetes is obsolete. It is an exploration of where Kubernetes remains the right substrate--provisioning capacity, managing worker Pods, and enforcing infrastructure policy--and where an agent-specific control plane may need a faster path for high-frequency lifecycle operations. The Thesis: Make Running Optional The central idea is surprisingly simple: an agent does not need to occupy compute merely because it exists. Agent Substrate calls an instance of a managed workload an actor. The deliberately broader term matters: an actor does not have to be an AI agent; it can be any OCI workload that benefits from being bursty, checkpointable, and independently suspendable. The system provides an agent-oriented workload runtime and control plane; it is not an agent framework or SDK. An actor can be suspended into a snapshot containing process memory, filesystem state, or both. The worker Pod is then freed. When another request arrives, the system restores the actor to a ready worker and routes the request to it. In this model, the actor becomes the logical workload, while the Pod becomes temporary compute capacity. That gives the architecture three defining properties: Warm capacity replaces per-session capacity. A smaller pool of ready workers serves a much larger population of actors over time.State survives worker reassignment. The next activation need not use the worker that ran the actor previously.Requests can initiate activation. The router can hold a request while the control plane brings a suspended actor back. The project's architecture document defines north-star targets including 100 ms p95 activation, one billion active and idle actors per cluster, and 1,000 wakeup events per second. These are architectural targets, not production benchmarks or guarantees. That distinction is important because the repository itself is candid that large parts of the architecture are still evolving. Architecture at a Glance The control flow becomes easier to follow once the logical workload is separated from the physical capacity: Why the Pod Becomes an Awkward Unit for Agents A conventional one-Pod-per-session deployment model can become inefficient when sessions spend most of their lifetime idle but retain valuable state. Agent-like workloads have a different shape: They wait much more than they compute.They may execute untrusted code, so multi-tenancy often means one sandbox per session.They keep useful state in memory and local filesystem changes.Their active periods can be short enough that creating and initializing a dedicated Pod for each session becomes noticeable user-facing latency. In a one-Pod-per-session design, the logical unit a user cares about — a coding session, sandboxed tool, or stateful agent — does not align cleanly with the physical unit Kubernetes schedules: a Pod. One straightforward approach is to keep each session's Pod alive. Agent Substrate asks whether that physical allocation can be temporary instead. Its answer is a pool of pre-provisioned worker Pods plus a separate actor record that tracks identity, lifecycle state, placement, and snapshots. This is an important architectural departure from Kubernetes' conventional control-plane model. Kubernetes intentionally optimizes for declarative desired state and asynchronous reconciliation. Agent Substrate moves high-churn actor state out of the Kubernetes API machinery so that wakeup, placement, and snapshot transitions can occur without making each actor a Kubernetes object. The point is not that Valkey or Redis is universally "faster than etcd." The interesting boundary is low-frequency desired state versus high-frequency runtime state: the two have different frequency and latency profiles, so the system gives them different control paths. Three Planes of State The resource model separates declarative configuration from dynamic runtime records. Operationally, snapshot contents form a useful third plane: STATE TYPEWHERE IT LIVESWHYActorTemplate, WorkerPool, and SandboxConfigKubernetes CRDsLow-frequency infrastructure configuration benefits from Kubernetes RBAC, auditability, and reconciliation.Actors, workers, assignments, lifecycle state, and snapshot referencesControl-plane store (Redis/Valkey by default; experimental PostgreSQL support is also available)These records change on lifecycle transitions and need low-latency reads and writes.Snapshot contentsNode-local storage for Pause; object storage for snapshots committed during SuspendSnapshot scopes trade off locality, durability, and transfer cost. An ActorTemplate defines an actor class: its container image, snapshot behavior, and compatible worker selection. A WorkerPool declares warm Pods. An Actor is a specific instance that moves between workers through its lifetime. Here is a trimmed ActorTemplate from the repository's multi-template demo: YAML apiVersion: ate.dev/v1alpha1 kind: ActorTemplate metadata: name: counter spec: containers: - name: counter image: ko://github.com/agent-substrate/substrate/demos/counter workerSelector: matchLabels: workload: multi-template-shared The important omission is a dedicated Pod. The template describes the workload and selects compatible reusable capacity; a separate WorkerPool provides the warm Pods, while the actor's identity and lifecycle remain independent of whichever worker hosts it. The deeper architectural pattern is a separation of three related lifecycles: infrastructure, workload, and execution. Kubernetes manages infrastructure capacity; the actor control plane manages logical workload identity, placement, and lifecycle; snapshot and sandbox machinery preserve and reconstitute execution state. Once those lifecycles are separated, a worker Pod becomes a reusable execution slot rather than the identity of the workload itself. An actor is addressed by (atespace, name), not by name alone. That is more than a naming detail: the glossary defines an atespace as a logical actor isolation boundary, not a replacement for Kubernetes namespaces or a sandbox security boundary. The same actor name can exist in different atespaces. The atespace also appears in the actor's routable DNS name: Plain Text ..actors.resources.substrate.ate.dev This is the first place the project begins to look less like a set of Kubernetes objects and more like a runtime: stable logical identity remains while the physical worker assignment changes. The Request Path: Routing Becomes Placement The most consequential design choice is that ingress is part of activation. The networking architecture provides the actor DNS model and an Envoy-based router. The router's ext_proc handler reads the actor reference from the request authority, calls the control plane to ensure that actor is running, and then selects the assigned worker as the upstream. Plain Text Error: Parse error on line 22: ...atunnel Ateom->>Actor: Forward over ----------------------^ Expecting '+', '-', '()', 'ACTOR', got 'participant_actor' The ingress detail matters. The router does not forward directly to the actor's application endpoint. It opens an mTLS connection to the worker's atunnel listener. atunnel validates the router and forwards only to the actor currently assigned to that worker. That makes routing part of the security boundary, not merely service discovery. There is also a practical admission-control insight here. A saturated worker pool should not turn a burst of requests into an unbounded queue. The router can park a bounded number of requests while it retries transient capacity and control-plane conditions; once the parking limit is reached, it sheds new work. Activation latency is therefore not just a restore-time problem. It is also a backpressure problem. What Happens During Suspend and Resume The control plane coordinates a distributed workflow rather than pretending this is a single atomic operation. For a resume, it locks the actor, reads its state and template, selects an eligible idle worker, asks the node-level supervisor to restore a snapshot or cold boot, and marks the actor running only after the worker is ready. For a suspend, it checkpoints state, persists the requested snapshot scope, clears the worker assignment, and returns the worker to the pool. The details reveal deliberate distributed-systems trade-offs. In the default Redis backend, actor and worker records are separate keys that may occupy different cluster slots, so they cannot be updated in one cross-slot action. The implementation uses per-record version checks, actor locking, ordering, retries, and idempotent workflow steps to coordinate these transitions. The architectural implication is that lifecycle operations are treated as recoverable workflows rather than atomic infrastructure mutations. A repeated lifecycle call can discover completed steps and move forward instead of blindly redoing them. The Worker Is a Reusable Sandbox, Not the Actor Below the control plane, atelet runs as a DaemonSet and manages the node-side work: image preparation, OCI bundle assembly, snapshot transfer, and communication with the worker. ateom runs inside the worker Pod and drives the sandbox runtime. The repository currently defines gVisor and microVM sandbox classes. In the gVisor path, ateom drives runsc checkpoint and restore. Depending on the configured scope, snapshots can preserve process and filesystem state, allowing an actor to resume later on another worker. This is why the demo can show an in-memory counter continuing after a suspend/resume cycle: the application was restored, rather than restarted from scratch. The security model should be described with care. Sandboxing and mTLS are real implementation elements, and the project has a detailed threat model. But that document explicitly says security hardening remains early. A fair reading is that Agent Substrate is making the right boundaries visible--sandbox, worker reuse, actor identity, snapshot access, and router-to-worker authentication--rather than claiming those boundaries are already production complete. What the Demos Prove--and What They Do Not The most accessible proof is the counter demo. A tiny HTTP service increments an in-memory counter. Create an actor in an atespace, send requests through atenet-router, suspend it, and resume it. The counter continues. The demo makes the abstract claim concrete: memory and filesystem state can outlive a worker assignment. The README's published density demonstration goes further: about 250 stateful actors multiplexed across eight physical worker Pods. The repository also includes examples for Claude Code multiplexing, request parking, autoscaled worker pools, and different templates sharing a worker pool. These examples validate the model and its developer experience. They do not prove the project's one-billion-actor target, production reliability, or a universal cost model. Treating that distinction honestly makes the architecture more interesting, not less: the open questions are precisely where the difficult engineering begins. From Traffic Locality to Compute Locality My previous DZone article, "Zone-Aware Routing in Kubernetes", examined a related infrastructure question: how should a platform place traffic so requests stay local when that improves latency, resilience, or cost? That work led me to a broader question: if locality matters for packets, what happens when locality also matters for stateful compute? Zone-aware routing asks where traffic should go. Agent Substrate raises a harder question: where should the compute state itself live when a workload can disappear from one worker and reappear on another? This turns locality from a networking concern into a workload-lifecycle concern. That change has consequences: Scheduling cannot be evaluated only by where free CPU exists; snapshot location and resume cost matter too.Routing cannot be evaluated only by endpoint availability; it can trigger a state transition.Security cannot stop at the Pod boundary; worker reuse and snapshot access become first-class concerns.Autoscaling cannot only count replica demand; it must account for how long actors remain active, parked, or suspended.Storage cannot be treated as an afterthought; snapshot placement, transfer time, durability, and locality become part of the activation path. This suggests a broader infrastructure question for agent workloads. The challenge is not simply running more containers; it is hosting large populations of mostly-idle, stateful, potentially untrusted processes without allocating dedicated compute to each one. Where the Hard Work Remains The project is unusually direct about its unfinished work: control-plane performance and reliability, worker autoscaling, identity and policy, actor network isolation, storage design, observability, and support for different sandbox runtimes all remain active areas of development. Those concerns are not peripheral; they determine whether the architecture can operate reliably at the scale it targets. A system that makes activation fast must still decide how to shard state, restore safely, apply policy before execution, isolate one actor from the state left by another, and reason about locality without turning every wakeup into a storage bottleneck. Four questions are especially important: Snapshot locality. If an actor's state is remote, resume latency becomes partly a storage and network-transfer problem.Snapshot correctness. Checkpointing a live process is not equivalent to serializing application state. Open connections, timers, external leases, credentials, and dependencies can make a restored process semantically different from a freshly initialized one.Activation bursts. Multiplexing improves average utilization, but a correlated wake-up event can turn many inexpensive idle actors into a sudden demand spike. The system therefore needs admission control and worker autoscaling that respond to activation pressure, not only steady-state utilization.Fairness. A small number of highly active actors can monopolize workers unless scheduling and admission control account for competing demand. Agent Substrate is therefore more compelling as an emerging architectural pattern than as a product claim. Its value is in making these trade-offs explicit and providing a runnable implementation that exposes where the abstractions are strong and where they remain unfinished. Conclusion The Pod is unlikely to disappear. But it may stop being the only unit we think about when we build infrastructure for agents. Kubernetes remains a powerful system for provisioning and operating compute. Agent Substrate is exploring what happens when the logical lifecycle of an agent is separated from the lifecycle of the Pod that temporarily runs it. If agents become ubiquitous--long-lived, intermittently active, stateful, and capable of executing untrusted code--the infrastructure challenge will not simply be running more Pods. It will be deciding where an agent should exist when it is inactive, how quickly it can become active, and how efficiently thousands or millions of them can share the same underlying compute. That is the problem Agent Substrate is attempting to solve. At a billion actors, the central question is no longer how to run more agents. It is how to make running optional. Further Reading Agent Substrate repositoryArchitectureThreat modelRequest parkingCounter demo Agent Substrate is Apache-2.0 licensed, explicitly not an officially supported Google product, and in active early development. The architectural analysis and opinions in this article are my own.

Most Docker content targets web developers shipping stateless services. However, data engineers, who represent a huge and growing population of Dockers users, are mostly left to figure things out alone, and it shows. The get pipelines that pass locally, but explode on clusters. They pit notebook-only development against expensive cloud workspaces, and more. This article applies six years of production data platform experience in financial services and healthcare to a question nobody answers well: How to you make a laptop behave like a lakehouse? A Familiar Routine If you build data pipelines for a living, you've lived this story. Your PySpark job runs perfectly in a cloud notebook. You productionize it, push it through CI, deploy it to the cluster, and it fails. A dependency mismatch. A different Spark minor version. A Delta Lake protocol feature your local wheel doesn't know about. A timezone default nobody set. Web developers solved "works on my machine" a decade ago with containers. Data engineers, somehow, are still developing against shared cloud workspaces, paying per-minute cluster costs to debug a GROUP BY, and discovering environment drift in production. This article is the workflow I wish someone had handed me years ago: a fully containerized lakehouse development environment — Spark, Delta Lake, object storage, a catalog, and orchestration — that runs on a laptop, mirrors production closely enough to trust, and plugs into CI without mocks. The Real Problem: Data Pipelines Have Four Environments, Not One A typical stateless web service has one environment to reproduce: the app runtime. A data pipeline has at least four, and they drift independently: The compute runtime — Spark version, Scala version, JVM, Python, native libs (Arrow, Parquet, libhdfs).The table format layer — Delta Lake / Iceberg versions and protocol versions, which are not the same thing.The storage layer — S3/ADLS semantics: multipart uploads, eventual consistency quirks, path-style vs virtual-hosted access.The orchestration layer — the scheduler's Python environment, which is famously not your job's environment. Mocking any one of these in tests means you aren't testing the thing that breaks. The goal of containerizing a lakehouse is to pin all four layers in code and version them together. Step 1: A Reproducible Spark Image You Actually Control Don't develop against latest. Build a base image that pins every layer of the compute runtime and treat it like an artifact: Dockerfile # syntax=docker/dockerfile:1.7 FROM eclipse-temurin:17-jre-jammy AS base ARG SPARK_VERSION=3.5.4 ARG DELTA_VERSION=3.3.0 ARG HADOOP_AWS_VERSION=3.3.6 RUN apt-get update && apt-get install -y --no-install-recommends \ python3.11 python3-pip tini && \ rm -rf /var/lib/apt/lists/* # Pin Spark itself, not just PySpark RUN curl -fsSL https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop3.tgz \ | tar -xz -C /opt && mv /opt/spark-${SPARK_VERSION}-bin-hadoop3 /opt/spark ENV SPARK_HOME=/opt/spark PATH=$PATH:/opt/spark/bin PYTHONHASHSEED=0 TZ=UTC # Delta + S3 connectors resolved at build time, never at job submit time RUN /opt/spark/bin/spark-shell --packages \ io.delta:delta-spark_2.12:${DELTA_VERSION},org.apache.hadoop:hadoop-aws:${HADOOP_AWS_VERSION} \ -e "println(\"deps cached\")" && \ cp /root/.ivy2/jars/*.jar /opt/spark/jars/ COPY requirements.lock /tmp/ RUN pip install --no-cache-dir -r /tmp/requirements.lock # Never run Spark as root RUN useradd -m -u 1001 spark USER 1001 ENTRYPOINT ["/usr/bin/tini", "--"] Three details that matter more than they look: --packages at build time, not submit time. Resolving connector JARs at spark-submit is the #1 source of "it worked yesterday" failures — Maven Central is a runtime dependency you didn't mean to have.PYTHONHASHSEED=0 and TZ=UTC kill two classes of "non-deterministic only in prod" bugs.A lockfile, not requirements.txt. Compile with pip-compile or uv pip compile so transitive dependencies (looking at you, pandas/pyarrow) can't drift. Step 2: The Lakehouse-In-A-Box With Docker Compose Here's the part most teams never build: the rest of the lakehouse, locally. MinIO stands in for S3 (it speaks the same API), and a real Spark master/worker pair stands in for the cluster, because local[*] mode hides every serialization and shuffle bug you'll meet in production. Dockerfile # compose.yaml services: spark-master: build: . command: /opt/spark/sbin/start-master.sh environment: [SPARK_NO_DAEMONIZE=true] ports: ["7077:7077", "8080:8080"] spark-worker: build: . command: /opt/spark/sbin/start-worker.sh spark://spark-master:7077 environment: - SPARK_NO_DAEMONIZE=true - SPARK_WORKER_MEMORY=4g - SPARK_WORKER_CORES=2 depends_on: [spark-master] deploy: replicas: 2 # >1 worker = real shuffles, real serialization minio: image: minio/minio:RELEASE.2025-09-07T16-13-09Z command: server /data --console-address ":9001" environment: MINIO_ROOT_USER: localdev MINIO_ROOT_PASSWORD: localdev-secret ports: ["9000:9000", "9001:9001"] volumes: [lake-data:/data] healthcheck: test: ["CMD", "mc", "ready", "local"] interval: 5s mc-init: # create the bronze/silver/gold buckets on boot image: minio/mc:latest depends_on: { minio: { condition: service_healthy } } entrypoint: > /bin/sh -c "mc alias set local http://minio:9000 localdev localdev-secret && mc mb -p local/lakehouse/bronze local/lakehouse/silver local/lakehouse/gold" volumes: lake-data: Point Spark at MinIO with three config lines and your medallion pipeline reads and writes s3a://lakehouse/... paths exactly like production: Python spark = (SparkSession.builder .config("spark.hadoop.fs.s3a.endpoint", "http://minio:9000") .config("spark.hadoop.fs.s3a.path.style.access", "true") .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") .getOrCreate()) docker compose up and you have bronze → silver → gold on your laptop. Total cloud cost of a debugging session: $0. Step 3: Integration Tests That Run Real Spark — Testcontainers The payoff of all this is CI you can trust. With Testcontainers, your pipeline tests spin up the same images your developers use: Python import pytest from testcontainers.minio import MinioContainer from pyspark.sql import SparkSession @pytest.fixture(scope="session") def lake(request): with MinioContainer("minio/minio:RELEASE.2025-09-07T16-13-09Z") as minio: yield minio def test_silver_dedup_keeps_latest_record(lake, spark): # write duplicate customer events to bronze bronze_path = f"s3a://test/bronze/customers" write_fixture_events(spark, bronze_path, duplicates=True) run_silver_dedup(spark, bronze_path, "s3a://test/silver/customers") result = spark.read.format("delta").load("s3a://test/silver/customers") assert result.count() == EXPECTED_UNIQUE assert latest_record_wins(result) No mocked DataFrames. No unittest.mock.patch("boto3..."). The test exercises Delta's actual transaction log against actual object storage. When this suite is green, deployments stop being scary. A pattern I use in regulated environments: keep a fixtures/ directory of small, synthetic Parquet files that mirror production schemas (never production data), and version them with the code. Schema drift then fails a unit test instead of a 2 a.m. pipeline run. Step 4: One Image From Laptop → CI → Production The final principle: the image you test is the artifact you ship. Multi-stage builds let one Dockerfile serve dev (with Jupyter, debuggers) and prod (minimal, non-root): Dockerfile FROM base AS dev USER root RUN pip install --no-cache-dir jupyterlab pytest debugpy USER 1001 FROM base AS prod COPY --chown=1001:1001 src/ /app/src/ COPY --chown=1001:1001 jobs/ /app/jobs/ # nothing else — no notebooks, no test deps, no shell tools you don't need In CI: build once, tag with the git SHA, run the Testcontainers suite against prod, scan it (Docker Scout, or your registry's scanner), sign it, and promote that exact digest through staging to the scheduler. Whether the scheduler is Airflow's DockerOperator/KubernetesPodExecutor or a managed Spark platform pulling custom containers, the principle holds: environments are immutable, versioned, and identical by construction. Lessons Learned From Production Run ≥2 workers locally.local[*] mode never serializes between JVMs. The day you switch to a real cluster, every closure-capture and UDF-pickling bug appears at once. Two 2-core workers in Compose surfaces them on day one.Pin the table format protocol, not just the library. Delta and Iceberg both evolve table protocol versions. A newer writer can produce tables an older reader can't open. Encode the protocol version in your image build args and test reads with the oldest reader you support.MinIO is a stand-in, not a clone. It won't reproduce S3 request throttling or cross-region latency. Keep a small smoke-test suite that runs against real object storage nightly; do everything else locally.Resource-limit your local Spark. Without SPARK_WORKER_MEMORY caps, a skewed join will cheerfully eat your laptop. Limits also force you to think about partitioning early — which is the point.Treat the orchestrator's image as layer four. Airflow DAG-parse environments drift too. Containerize the scheduler with the same lockfile discipline as the jobs. Production Considerations Before you take this pattern to a real platform team, three things to plan for: secrets (local Compose uses throwaway creds; production should inject via your cloud's secret manager or Docker secrets — never baked into images), image provenance (sign images and generate SBOMs in CI; regulated industries will ask, and in 2026 the tooling is mature enough that "we didn't get to it" no longer flies), and base image hygiene (start from minimal, hardened bases and rebuild on a schedule, not just on code change — CVEs don't wait for your sprint). Conclusion Containers gave application developers reproducibility ten years ago. Data engineering is finally having the same moment — and the teams that containerize their lakehouse development loop ship faster, test honestly, and stop paying cloud bills to find typos. Try it: clone the Compose stack above, point your gnarliest pipeline at it, and see what breaks locally that used to break in prod. Then tell me about it — I'd genuinely like to hear which layer drifted on you. If this was useful, follow me here and on LinkedIn. Next up in this series: load-testing Delta merge performance locally, and contract testing between pipeline stages.

It stopped being just a packaging tool the day our onboarding doc got shorter instead of longer. Three weeks into a new ML platform job, I asked a coworker why the 'getting started' doc had a section called 'If conda breaks, try the alternative.' He laughed in a way that told me it wasn't a joke. Every new hire spent their first two days fighting Python versions, CUDA driver mismatches, and a vector database that someone had installed locally in 2022 and nobody dared touch. We had four individuals on the team, each with distinct working setups, and "it works on my machine" was no longer a mere punchline; it had become a regular agenda item during our daily standup meetings. That's the environment I inherited, and it's the reason I ended up rebuilding our entire local AI dev loop around Docker Compose instead of the notebook-and-prayer setup we'd been running. Why This Isn't Just a Packaging Problem The instinct on most teams is to treat Docker as something you reach for at deploy time. You write the model, get it working in a notebook, and only think about containers once it's time to ship. That instinct falls apart with AI workloads specifically because the dev-time dependencies are just as fragile as the prod ones. A GPU-backed embedding model, a local vector store, a retrieval service, and an orchestration layer all need to talk to each other during development, not just in production. If your local loop doesn't mirror that, you spend your debugging time chasing environment drift instead of chasing actual bugs. That was our exact situation, and it cost us roughly a day of onboarding per person plus a steady trickle of 'works for me' bug reports that turned out to be dependency version mismatches. The Setup We Rejected First Our initial response was to improve the Conda environment file and create a more detailed README. In hindsight, that was doomed from the start. Conda solved the Python dependency problem reasonably well but said nothing about the GPU driver version, the vector database binary, or the fact that two people were running Ollama locally with completely different default models pulled. We also floated the idea of just giving everyone a cloud dev environment with GPU access baked in. It solved the consistency problem, but the latency for interactive debugging was miserable, and the monthly bill for keeping GPU instances warm for a six-person team was not something I wanted to defend in a budget review. Neither approach addressed the real issue: we needed one definition of the environment that was runnable identically on a Mac laptop and a Linux workstation. What We Actually Built We moved the whole local AI stack into a single Compose file: an inference service running a small local model, a vector store, and the application layer, all networked together the same way they'd be networked in staging. Here's a trimmed version of what that looked like: YAML services: llm: image: ollama/ollama:latest volumes: ["ollama-data:/root/.ollama"] deploy: resources: reservations: devices: - driver: nvidia count: 1 capabilities: [gpu] vectordb: image: pgvector/pgvector:pg16 environment: POSTGRES_PASSWORD: devpass volumes: ["pgdata:/var/lib/postgresql/data"] app: build: ./app depends_on: [llm, vectordb] environment: OLLAMA_HOST: http://llm:11434 That file, plus a one-line 'docker compose up,' replaced two days of onboarding pain with about fifteen minutes. New hires no longer needed tribal knowledge about which conda channel had the right cuDNN build. It also resolved unforeseen bugs by ensuring everyone used the same version of the embedding model, eliminating reports of differing search results caused by dependency drift. The GPU Passthrough Headache Here's where things got tricky. GPU passthrough on Linux with the NVIDIA Container Toolkit is straightforward once it's configured, but it's not portable to Apple Silicon, and half our team was on M-series MacBooks. We ended up maintaining two Compose override files: one that requests GPU reservations for Linux workstations and one for Mac that falls back to CPU inference with a smaller quantized model, accepting slower generation for the sake of a working local loop. It's not elegant, and I still dislike maintaining two code paths for something as basic as "run the model," but the alternative was blocking half the team from working locally at all, which is worse. Where I'd Push Back on the Hype There's a growing narrative that Docker is quietly turning into a full AI platform with model registries, one-command local model pulls, and built-in GPU scheduling for dev. Some of that is genuinely useful, and I would rather not undersell it. But I'd push back on treating Docker as a replacement for a real experiment-tracking or model-serving platform in production. What it's good at is collapsing the dev-time chaos into something reproducible; it is not a substitute for proper GPU orchestration at scale, and teams that try to run Compose-style setups in production tend to relearn the lessons Kubernetes already solved, just slower and with worse observability. The platform shift is real at the development layer. I'm far more skeptical that it fully extends to production serving without a lot of additional tooling wrapped around it. Key Takeaways Treat local AI dev environments with the same seriousness as production ones. Dependency drift in embedding models and vector stores causes real, challenging-to-trace bugs.Conda and README discipline don't solve GPU driver and binary-level mismatches; a single Compose definition does.Plan for hardware heterogeneity early: GPU passthrough doesn't travel to Apple Silicon, so budget for a CPU fallback path.Don't overextend this pattern into production serving; Compose is a dev-loop win, not a Kubernetes replacement. Conclusion What changed for our team wasn't really about Docker getting new AI-specific features, though some of that helped. Realizing that the development environment for an AI application is as complex and failure-prone as production and treating it as an afterthought cost us real engineering hours each week. Whether Docker keeps expanding into model management and becomes a genuine AI platform, or whether that space gets carved up by more specialized tools, I think the underlying lesson holds either way: if your local AI loop isn't reproducible, nothing built on top of it will be either. I'm curious how far other teams have pushed this before Compose starts creaking. Is there a scale at which this pattern breaks down, or a project where you gave up and rebuilt around something heavier?

The first time I containerized a fine-tuned Llama model for a client's internal search tool, the build finished at 38 gigabytes. I remember staring at the terminal thinking there was no way that was right. It was right. The image included a CUDA base, PyTorch with every backend compiled in, model weights baked directly into the layer, and a pip cache that had not been cleaned. Pushing that to our registry took eleven minutes on a good connection. Pulling it onto a fresh node during an autoscale event took even longer, and by the time the pod was ready, the traffic spike it was supposed to handle had already passed. That's the moment I stopped treating LLM containers like regular application containers, because they are not the same animal at all. Why This Problem Actually Matters Most Docker advice out there is written for stateless web services, small images, fast cold starts, and horizontal scaling on demand. LLM workloads break almost every assumption baked into that advice. The artifact is huge, the runtime is GPU-bound, startup involves loading gigabytes into VRAM, and half your "application code" is actually a C++/CUDA binary blob you didn't write and can't easily trim. If you treat an inference container like a Flask app with a bigger base image, you end up with slow deploys, wasted GPU spend, and autoscaling that technically works but arrives too late to matter. The First Wrong Turn: One Image to Rule Them All Our early approach was a single monolithic image model with weights, tokenizer, inference server, and dependencies all baked together, rebuilt on every model version bump. It felt simple. It wasn't. Every retrain meant rebuilding a 30+ GB image even when the code hadn't changed a single line. Registry storage costs gradually increased until someone in finance questioned why our container registry bill resembled that of a second AWS account. Worse, rollbacks were painful because reverting to a previous model meant pulling an entire previous image rather than swapping a much smaller artifact. The solution that actually worked was separating the model weights from the serving image entirely. The image contains the runtime, the inference server (we used vLLM for most of our transformer workloads), and pinned dependencies. Weights live in object storage and are pulled at container start via an init container or a lazy loading entry point. The approach felt counterintuitive at first. Are we effectively transitioning the slower process to startup instead of build time? — but it turned out to be the right trade. Startup pulls are parallelizable, cacheable on the node, and don't bloat the registry. Build time dropped from twenty-plus minutes to under four. A Smaller Base Image Than You'd Expect This is where the challenges began. Everyone defaults to using nvidia/cuda:*-devel images because the framework documentation recommends them, but these devel images include the entire CUDA toolkit, which contains compilers that you will never use at runtime. Switching to the runtime variant and only installing the exact CUDA and cuDNN versions your framework's wheel actually needs cuts roughly 4GB off the base alone. A minimal multi-stage build looks something like this: Dockerfile FROM nvidia/cuda:12.1.0-devel-ubuntu22.04 AS builder RUN pip install --no-cache-dir vllm==0.4.2 FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04 COPY --from=builder /usr/local/lib/python3.10 /usr/local/lib/python3.10 COPY --from=builder /usr/local/bin/python3.10 /usr/local/bin/ ENV MODEL_PATH=/mnt/models ENTRYPOINT ["python3", "-m", "vllm.entrypoints.api_server"] The build stage compiles anything that needs the full toolkit; the runtime stage only carries what's needed to execute. It's a basic Docker pattern, but I've seen it skipped constantly on ML teams because the assumption is always, "the model is the heavy part; the image doesn't matter." The model is heavy, sure, but a bloated base image adds real minutes to every autoscale event, and in production that's the difference between absorbing a traffic spike and dropping requests. The OOM Kill: Nobody Explained Well This is the war story I bring up most often. We had a container that ran fine locally and in staging, then got silently killed in production under load — no crash log, no stack trace, just a pod restart and a confused on-call engineer at 2 AM. It turned out to be the kernel OOM killer, not an application-level exception, because our memory limit accounted for the model weights in VRAM but excluded the growing KV cache for long-context requests plus the CPU-side tokenizer buffers. GPU memory and container memory limits are two completely separate accounting systems, and Kubernetes will happily kill your pod over host RAM even if your GPU has headroom to spare. The fix was unglamorous: we set explicit memory requests and limits with a real margin above peak KV cache usage, moved batch size and max sequence length into environment-configurable values instead of hardcoding them, and added a lightweight health assessment that reported GPU memory utilization alongside the standard liveness probe. None of that is exotic. All of it was missing because we'd copy-pasted a manifest template built for a stateless API and never revisited the resource math for a model that holds state in memory for the duration of a request. Where I'd Push Back on Common Advice A lot of guidance recommends one model per container for isolation, and for many teams that's right. But if you're serving several small fine-tunes of the same base model, that pattern wastes GPU memory by duplicating base weights across containers. We transitioned to a multi-adapter setup, where one base model is loaded once, and LoRA adapters are swapped for each request; this approach is more complex operationally but reduces the GPU footprint by nearly half. I wouldn't consider it a default; it represents a level of complexity that is justified only after demonstrating that plain per-model containers are indeed the bottleneck. I'd also push back on containerizing every workload the same way. Batch inference and real-time serving have almost opposite goals: one wants throughput and tolerates slow cold starts; the other needs rapid readiness and predictable latency. We split these into separate images with separate resource profiles, even though it meant more Dockerfiles. Fewer surprises beat fewer files. Key Takeaways Separate model weights from the serving image; bake them in the runtime and pull weights at startup from object storage.Use CUDA runtime images, not devel images, unless you genuinely compile something at container start.Account for GPU memory and host memory as two separate budgets; KV cache growth is the usual silent killer.Split batch and real-time serving into different images; their optimization goals are conflicting.Don't reach for multi-adapter serving or other density tricks until you've measured that plain per-model containers are actually the bottleneck. Closing Thought None of this required exotic tooling, no custom orchestrator, and no proprietary platform. It required treating the container as part of the model's runtime behavior rather than a packaging afterthought bolted on after the research work was done. The teams that struggle most with this approach usually aren't lacking Docker knowledge; they're applying web-service intuition to a workload that behaves nothing like a web service. If you're mid-migration on something similar, I'd genuinely ask: are you optimizing your image for build convenience or for what actually happens the moment traffic hits a cold node? Those answers are rarely the same, and figuring out which one you've been solving for is usually the first real fix.

If different Docker Engine versions are running simultaneously in a Docker Swarm cluster, this may lead not to an obvious service outage but to a more subtle scenario: partial traffic degradation on individual nodes. In this case, the issue appeared on one of the manager nodes, Traefik started reporting an unavailable status (health=0) for the router-app service, and the cause, according to the working hypothesis, was related to differences in iptables rules and overlay networking between Docker 28.1.1 and 28.2.2. On June 22, 2025, this exact scenario occurred in the production cluster of the backend infrastructure for a socially significant public transportation mobile application. The system serves about 2 million users, several tens of thousands of daily active users, and a total load of around 1000–1600 RPS, so even partial degradation at a single entry point affected a high-load segment of traffic and could have had a noticeable impact on SLA metrics if it had not been localized in time. Context At the time of the incident, the Docker Swarm cluster consisted of 5 manager nodes, several dozen worker nodes, and approximately 40–50 services. External HTTP traffic passed through Traefik, deployed on each of the five manager nodes, and was then routed by Traefik to the backend application containers. One of the key services was router-app, responsible for building public transportation routes on the frontend. It was one of the critical entry-point services with the highest SLA, and any disruption to its availability could have led to severe penalties from the customer, so any deviation in its availability required an immediate response. Grafana dashboards showed application availability through Traefik health-check statuses. Those statuses were generated based on HTTP health-check endpoints implemented by the developers for most services, primarily the most critical ones. This was enough to quickly localize the problem at the ingress traffic level. The cluster had one important characteristic. Some nodes were running Ubuntu 20.04 (focal), while others were running Ubuntu 22.04 (jammy), and different APT repositories were pulling different Docker Engine versions. As a result, after another scheduled Docker update on the nodes, the production environment ended up with a mix of nodes running 28.1.1 and 28.2.2 at the same time. The docker node ls screenshot additionally confirmed that mixed versions were present not only on worker nodes but also on manager nodes, including swarm4 and swarm5. How the Incident Manifested The incident was detected not through user complaints and not through a general service outage, but through Traefik monitoring. The triggered alerts showed partial unavailability of one of the router-app containers, after which the health-check dashboard confirmed that the issue affected not the entire service but one of the manager nodes. This is important because the red blocks on the dashboard did not indicate a complete outage of router-app. It meant that Traefik on one of the manager nodes started receiving health=0 when checking that service’s backend endpoint, while the other manager nodes continued to see the backend as healthy. In practice, it looked like this: traffic through one of the manager nodes stopped reaching the router-app containers correctly, but Traefik, running on all five manager nodes, automatically excluded requests to the unhealthy entry point. As a result, from the outside the incident appeared as partial degradation rather than full unavailability. That is exactly what made the situation tricky. Fault tolerance limited the impact of the incident, but the underlying cause remained inside the cluster and continued to affect one of the entry points. What Docker Showed After localizing the issue to one of the manager nodes, it became clear that the cause should be sought not in router-app itself but in the network path between Traefik and the backend containers. At the same time, docker service ps did not show a widespread service failure, and the containers still appeared as running. The next useful signal came from the dockerd logs on swarm5. Repeated messages appeared there, including Peer delete operation failed, neighbor entry not found, and errors related to deleting FDB and neighbor entries for the VXLAN interface vx-001001-5lk08. For example: Plain Text Jun 22 16:38:09 swarm5 dockerd: time="2025-06-22T16:38:09.366807202Z" level=warning msg="Peer delete operation failed" error="could not delete fdb entry for nid:5lk08r7jjvtq5idqggzeygmlv eid:4e7d63d00fffaa6be7ce6362f47acd7912f7c11e5ac6e018393722decc16c210 into the sandbox:neighbor entry not found for IP 10.170.0.37, mac 02:42:0a:1b:14:2c, link vx-001001-5lk08" Jun 22 16:38:09 swarm5 dockerd: time="2025-06-22T16:38:09.765092537Z" level=warning msg="error deleting neighbor entry" error="no such file or directory" ifc=vx-001001-5lk08 ip=10.170.0.138 mac="02:42:0a:1b:14:65" Such messages were highly consistent with problems in Docker Swarm’s overlay network. In essence, Docker was trying to delete network records that were no longer present in the tables, which usually points to desynchronization of network state at the VXLAN, FDB, or neighbor-table level. By themselves, these messages still did not provide a complete explanation, but they pushed the investigation in the right direction. It became clear that the problem was not in the application’s business logic but in the network layer on one of the nodes. Additionally, docker node inspect self --pretty on swarm5 showed that from Swarm’s point of view the node looked normal: State: Ready, Availability: Active, Raft Status: Reachable, Leader: No, while Engine Version was already 28.2.2. This was an important point: the control plane still considered the node healthy, even though at the traffic-flow and network-state level it was already behaving differently. Diagnostics The investigation was carried out at the node level. The tools used included docker node ls, docker version, apt-cache policy docker-ce, as well as ip link show, bridge fdb show, ip neigh show, and comparisons of iptables chains across different nodes. The key fact became visible after docker node ls. The cluster was not homogeneous in terms of Docker Engine version: some manager nodes and some worker nodes were already running 28.2.2, while the others remained on 28.1.1. This led to the assumption that the issue might be at the iptables rules level. After that, iptables had to be compared separately on healthy and problematic nodes. On swarm5, a full rules dump was collected using the combination of iptables -S, iptables -t nat -S, and iptables -t mangle -S. Those rules showed the DOCKER, DOCKER-FORWARD, DOCKER-INGRESS, and DOCKER-USER chains, as well as ACCEPT, DROP, and DNAT rules for traffic through docker_gwbridge, published ports, and ingress routing.... To test the hypothesis, not only the problematic swarm5 but also the first manager node, swarm1, was compared, where a stable stack with Docker 28.1.1 had long been running. On swarm1, the output of iptables -S and iptables -t nat -S showed the expected picture: the DOCKER and DOCKER-INGRESS chains contained a full set of ACCEPT and DNAT rules for all published ports (80, 8080–8082, and dozens of internal service ports) with symmetric dport/sport pairs, while DOCKER-USER effectively boiled down to a clean RETURN. Taken together with the dump from swarm5, this reinforced the conclusion that on nodes running 28.1.1, the iptables configuration for ingress and routing was consistent, and the differences seen on 28.2.2 were related not to manual changes but to the behavior of Docker Engine itself. After that, iptables had to be compared separately on other nodes running different Docker versions. On nodes with 28.1.1, the DOCKER and DOCKER-USER chains and the associated rules were in the expected state, whereas on nodes with 28.2.2 some of the required rules were missing or the chains were reduced to a minimal RETURN. This explained the observed behavior well. The services remained running, Swarm did not appear broken, but external traffic and part of the overlay routing through a specific manager node were working incorrectly, causing Traefik on that node to report health=0 for router-app. It is worth noting separately that journalctl -u docker and docker service ps did not provide a simple direct cause for the incident. They did not show a picture of a general failure, so the conclusion had to be assembled from several sources: Traefik monitoring, dockerd logs, Docker versions, and the state of iptables on different nodes. Fix Once the main hypothesis had narrowed down to mismatched Docker Engine versions, the solution was fairly straightforward: return the cluster to a homogeneous configuration by rolling back to version 28.1.1 as the fastest solution. The rollback was performed for swarm4, swarm5, and all worker nodes where version 28.2.2 had already been installed. To do this, a specific package version was pinned via apt, then Docker was restarted, and the installed version was verified. One version of the commands looked like this: Shell apt-cache madison docker-ce | grep 28.1.1 apt-get install docker-ce=5:28.1.1-1~ubuntu.22.04~jammy \ docker-ce-cli=5:28.1.1-1~ubuntu.22.04~jammy \ containerd.io && systemctl restart docker && docker --version Additionally, it made sense to check the package sources and, if necessary, remove conflicting APT entries so that the nodes would no longer receive an unsuitable Docker version from another repository. In practice, it looked like this: Shell sudo rm /etc/apt/sources.list.d/download_docker_com_linux_ubuntu.list sudo apt update After the rollback, docker version was checked again, as well as the DOCKER and DOCKER-USER chains. After Docker Engine had been unified to 28.1.1 on both manager and worker nodes, the issue disappeared. From the perspective of external behavior, this was confirmed immediately. Health checks in Traefik returned to the green zone, and the partial unavailability of router-app on one of the manager nodes could no longer be reproduced. Root Cause Based on the available data, the most well-founded working version is this: in this environment, Docker Engine 28.2.2 formed or applied iptables rules related to DOCKER, DOCKER-USER, FORWARD, ingress, and overlay networking differently. In a mixed cluster, this led to one of the manager nodes no longer forwarding traffic correctly to the router-app backend containers, even though from the perspective of the control plane and service state this did not look like a direct failure. It is important here not to overstate what the data allows. This case does not prove a universal upstream bug in Docker 28.2.2 for all Swarm installations, but it does show that even closely related Docker Engine versions can affect the cluster’s network plane differently, especially when different Ubuntu distributions and different package sources are present in production at the same time. What Follows From This The first conclusion is simple: Docker Swarm is sensitive to Docker Engine version mismatches. If some manager or worker nodes have been updated while others have not, this can lead not only to version drift as an organizational problem, but also to practical issues with traffic, published ports, and overlay routing. The second conclusion is that after updating Docker, it is necessary to check not only docker version but also the node’s network behavior. The minimum set includes iptables -L DOCKER -v -n, iptables -L DOCKER-USER -v -n, checking published ports, ingress/overlay state, and health checks from the edge proxy. The third conclusion is that it is useful to maintain a single baseline stack across all nodes. One Ubuntu LTS distribution, unified repositories, and the same update order reduce the chance that cluster state will remain formally healthy while part of the network traffic is already being handled incorrectly. The fourth conclusion concerns update order. In our case, that was exactly how it happened, but it is worth noting separately. When Docker is updated in a Swarm cluster, it is better to update worker nodes first, then manager nodes, and the leader last, while after each stage separately checking the node’s behavior under real traffic conditions (service availability, correct routing, and published ports). In our case, enhanced monitoring was in place, so no additional manual checks of node behavior in traffic were required: if any part of the infrastructure became unavailable, we would promptly receive an alert. The final conclusion relates to monitoring. In this case, Traefik not only helped limit the impact of the incident by routing around the unhealthy node, but also provided the first precise signal that the problem was localized to a specific entry point rather than existing at the level of the entire service or the entire cluster.

Introduction The meeting that changed the platform team's direction was not a technical one. It was a conversation with a product engineer who had been at the company for eight months and had never successfully deployed to production without help from someone on the platform team. Not because she lacked skill. She was smart, experienced, and had successfully launched production systems at two previous jobs, but getting a working service into production meant dealing with fifteen different configuration files across four repositories, figuring out how Helm values files and Kustomize overlays worked together, and knowing which of the three CI pipeline templates to use based on whether the service needed a sidecar, a job scheduler, or neither. She had read the documentation. It was accurate. It just didn't tell her what to do when the documented path didn't match the state of her specific service in her environment. The platform team had built powerful infrastructure. They had not built a usable platform. That distinction between infrastructure and platform is where most Kubernetes-based internal developer platform efforts go wrong, and it's worth being precise about what it means. Infrastructure vs Platform: A Practical Distinction Infrastructure is the machinery: the Kubernetes clusters, the networking layer, the CI pipelines, the secrets management system, and the monitoring stack. A platform is the interface that makes that machinery accessible to developers who aren't Kubernetes experts without requiring them to become ones. The confusion between the two produces a situation that's extremely common in engineering organizations: a technically sophisticated infrastructure that's effectively only usable by the people who built it. The test for whether you have a platform or just infrastructure is simple: can a developer who joined three months ago deploy a new service to production without asking anyone for help? Not by following a tutorial someone wrote last year that may or may not still be accurate, but through tooling that guides them through a current, correct process. If the answer is no, you have infrastructure. The platform is the missing layer. This statement is not an argument against complexity in the underlying system. Kubernetes is complex, and that complexity exists for beneficial reasons: flexibility, programmability, and a rich ecosystem. The platform layer should absorb the complexity, rather than exposing it to every developer who needs to ship a service. What the First Attempt Got Wrong The infrastructure team built the first version of the internal platform in their spare time, juggling it with other priorities. It consisted of a set of Helm chart templates, a GitHub Actions workflow library, and a wiki with deployment instructions. This approach is how most internal platforms start, and it has a predictable failure mode: the templates encode the assumptions of the people who wrote them, the wiki goes stale within weeks, and the gap between the documented process and the actual state of the infrastructure grows invisibly until it becomes a significant tax on every developer who hits it. The fundamental mistake was treating platform work as documentation work rather than product work. A wiki is not a platform. A set of templates that require understanding to use correctly is not a platform. A platform is software that makes the correct path the easy path, that validates inputs before they cause problems downstream, and that fails loudly and helpfully rather than silently and mysteriously. The second attempt started from a different premise: the platform is a product, developers are its users, and the measure of success is whether they can do their jobs without needing the platform team. The Abstraction Layer: Custom Resources and Admission Webhooks The technical decision that made the most difference was introducing a custom resource definition (CRD) that represented a service in the platform's domain model, not a Kubernetes Deployment or Service, but a higher-level construct that encoded the platform's opinionated defaults and generated the underlying Kubernetes objects from a much simpler specification. YAML # Platform-level CRD: what developers actually write apiVersion: platform.company.com/v1 kind: AppService metadata: name: payment-api namespace: production spec: image: payment-api:v1.4.2 tier: backend # drives resource limits, network policy replicas: 3 port: 8080 healthCheck: /healthz env: DATABASE_URL: secretRef: payment-db-credentials This twelve-line manifest replaced the hundred-plus lines of Kubernetes YAML that developers had previously been required to write and maintain. The controller running in the cluster, a standard Kubernetes operator built with controller-runtime, read the AppService resource and generated the Deployment, Service, HorizontalPodAutoscaler, PodDisruptionBudget, and NetworkPolicy that the platform's standards required, with defaults applied consistently across every service. The key design decision was what to expose in the CRD and what to hide. The tier field is a prime example: rather than exposing resource requests and limits directly, which requires understanding what values are appropriate for the cluster, the CRD accepts a tier label (frontend, backend, worker, batch) that maps to a predefined resource profile. A backend tier service receives a specific CPU and memory allocation appropriate for the cluster's node types. A batch tier service receives a different profile with different eviction priorities. The developer specifies intent; the platform enforces the appropriate configuration. Go # Controller logic: tier maps to resource profile (Go pseudocode) func resourceProfileForTier(tier string) corev1.ResourceRequirements { profiles := map[string]corev1.ResourceRequirements{ "frontend": { Requests: corev1.ResourceList{ corev1.ResourceCPU: resource.MustParse("100m"), corev1.ResourceMemory: resource.MustParse("128Mi"), }, Limits: corev1.ResourceList{ corev1.ResourceCPU: resource.MustParse("500m"), corev1.ResourceMemory: resource.MustParse("256Mi"), }, }, "backend": { Requests: corev1.ResourceList{ corev1.ResourceCPU: resource.MustParse("250m"), corev1.ResourceMemory: resource.MustParse("256Mi"), }, Limits: corev1.ResourceList{ corev1.ResourceCPU: resource.MustParse("1000m"), corev1.ResourceMemory: resource.MustParse("512Mi"), }, }, // batch, worker profiles follow same pattern } return profiles[tier] } Admission webhooks complemented the CRD by catching misconfiguration before it reached the cluster. A validating webhook checked every AppService manifest against a set of rules: the image tag must not be 'latest,' the health verification path must respond within the cluster, and secret references must exist in the target namespace and return a clear error message describing exactly what was wrong and how to resolve it. This shifted error detection from 'runtime, after deployment' to 'submission time, before anything breaks,' which dramatically reduced the debugging load on both developers and the platform team. The Golden Path and Its Limits The CRD and controller approach works well when services fit the platform's model. Here's where things became challenging: not all services fit the platform's model. A service that needed custom init containers, a service that required a specific affinity rule because of a hardware dependency, and a batch job with a complex retry policy that didn't map cleanly to the tier abstraction. Each of these required either extending the CRD or breaking the abstraction and falling back to raw Kubernetes YAML. The temptation is to keep extending the CRD to cover every case. Resist it. A CRD that tries to expose every Kubernetes feature is just a more complicated way to write Kubernetes YAML, and it loses the simplicity that made the abstraction valuable. The better model is a golden path, the CRD for the 80% of services that fit the standard model, and a documented escape hatch for the 20% that don't. The escape hatch is raw Kubernetes resources, maintained by the teams that need them, with the platform team providing support rather than ownership. The key is being honest with developers about which path they're on. A service using the AppService CRD gets platform-managed defaults, automatic updates when the platform evolves, and first-class support. A service using raw Kubernetes resources owns its own configuration and gets best-effort support. That distinction in the support model is what makes the trade-off legible rather than arbitrary. What I'd Do Differently In hindsight, the most important investment was the admission webhook, and I'd build it earlier. The CRD and controller took significant time to design and implement. The webhook could have been built in a few days and would have immediately improved the developer experience by catching misconfiguration at submission time rather than deployment time. Validation before generation is higher-leverage than generation that might produce something invalid. I'd also measure platform adoption from day one. Which teams are using the AppService CRD? Which teams are on raw Kubernetes? What's the conversion rate of new services to the platform abstraction? Without that data, the platform team relies on intuition instead of evidence to guide their investment. The teams that adopt slowly are often the ones with the most valuable feedback about where the abstraction doesn't fit, and they're also the teams most likely to be quietly maintaining fragile custom configurations that will become incidents later. When should you not build a CRD-based platform abstraction? If you have fewer than fifteen to twenty engineers deploying services, the overhead of designing, building, and maintaining a CRD-based platform abstraction almost certainly exceeds the value. Helm charts and excellent templates get you most of the way there with a fraction of the complexity. The operator pattern earns its cost when you have enough services that inconsistency becomes a real operational problem when the differences between how services are configured start causing incidents and nobody can tell you why a particular service is configured the way it is. Key Takeaways Infrastructure and platform are different things. Infrastructure is the machinery; a platform is the interface that makes machinery accessible without requiring expertise in its internals. Most Kubernetes-based IDPs stop at infrastructure. Custom resource definitions let you define a domain model that encodes your platform's opinions. Developers specify intent (tier, replicas, port); the controller generates the correct Kubernetes objects with consistent defaults applied. Admission webhooks shift error detection from runtime to submission time. A clear error message at kubectl apply is worth more than a mysterious pod crash two minutes later. Maintain a golden path for the majority of services and a documented escape hatch for the rest. A CRD that tries to cover every Kubernetes feature loses the simplicity that justified building it. Conclusion The platform engineer's job is to make complexity disappear, not by eliminating it, but by absorbing it into tooling so that the people building products don't have to carry it. That's a harder problem than building the infrastructure itself, and it requires a fundamentally different mindset: less systems engineering, more product thinking. Who are the users? What tasks do they need to accomplish? Where does the current experience fail them? The teams building internal developer platforms who get the process right tend to look, from the outside, like they have unusually productive engineering organizations. Individual contributions ship faster, incidents caused by misconfiguration drop, and the platform team spends less time on support and more time on improvements. The causal chain runs directly from platform quality to engineering output, even though it's usually measured differently. The open question is whether the CRD-based abstraction model scales to genuinely heterogeneous service fleets, the kinds of organizations where services span multiple languages, multiple deployment patterns, and multiple infrastructure dependencies. The golden path works when most services look similar enough that a shared abstraction is useful. What occurs to the platform model when 40% of services utilize the escape hatch? At that point, is the abstraction still earning its cost, or is it adding complexity without delivering the simplicity it promised?

Sponsored By: NutanixThe following is sponsored content. It may not reflect the views of our editorial staff. The Kubernetes scaling problem nobody talks about Enterprise platform teams encounter the same pattern repeatedly: a Kubernetes platform works well enough that nobody wants to change it. This happens gradually as teams make reasonable technology choices: selecting different ingress controllers, secrets management tools, CD platforms, or observability software. Individually, none of these decisions is a problem. Months later, however, they’ve created a Kubernetes environment that only a handful of people understand. As soon as that one person gets sick or leaves the company, maintaining or improving the platform becomes much more difficult. Mark Dastmalchi-Round, a Solutions Architect at Nutanix with decades of experience in platform engineering, describes the pattern in blunt terms: “Configuration drift, exacerbated by the fact that multicloud is increasingly becoming the new reality.” Over time, that drift compounds. Companies get acquired, technology merges, and silos form. Suddenly, organizations are managing clusters that look nothing alike and are often held together by institutional knowledge. As a solution, proprietary overlays have sought to address these issues, with mixed results. They tend to reduce overall surface area (fewer choices lead to fewer points of divergence), but often at a cost to portability and extensibility, which is what made Kubernetes so attractive in the first place. A more durable approach is to build on Kubernetes-native primitives, adding governance and operational consistency without replacing the workflows teams already use. The remainder of this article will demonstrate what that looks like in practice. What an open platform actually means in enterprise Kubernetes “Open platform” is a common phrase in the Kubernetes ecosystem, but it’s worth defining what that term actually means in practice. Dastmalchi-Round defines an open platform as one that “exposes industry-standard APIs and, where possible, uses pure upstream open-source projects.” The distinction isn't whether the platform is open source. It's whether it relies on Kubernetes-native APIs and tooling or introduces proprietary CRDs, workflows, and CLIs that make migration difficult. As he notes, "You can still get lock-in with open source, because if it is only one vendor's solution and they layer all of their stuff on top of standard tooling, you are now dependent on their abstractions." The difference is easier to see when comparing an open platform with a proprietary overlay. Comparing Open Kubernetes Platforms and Proprietary Overlays Dimension Open Platform (NKP) Proprietary Overlay Core CRDs Standard upstream (Cluster API, FluxCD, Helm) Vendor-specific, migration cost is high GitOps engine FluxCD (CNCF project) Proprietary sync engine App packaging Helm + OCI (industry standard) Custom catalog format Monitoring stack Pure upstream CNCF (Prometheus, Grafana) Wrapped / vendor-branded Exit cost Clusters survive platform removal Manifests tied to platform APIs Third-party tooling Works if it runs on Kubernetes Requires certified integration Nutanix Kubernetes Platform (NKP) applies these principles by building on upstream Kubernetes components rather than replacing them. As Dastmalchi-Round puts it, the real test is what survives if you remove the platform. "With NKP, the clusters are pure upstream Kubernetes,” says Dastmalchi-Round. “The monitoring stack is pure upstream CNCF projects. GitOps is provided by FluxCD. Your manifests and charts are standard Helm." In other words, the operational tooling may change, but the underlying applications and deployment artifacts remain portable. Raw manifests to managed artifacts: Helm and OCI packaging in NKP Most enterprise teams start with a collection of Kubernetes YAML manifests that work for a single application or environment. While those manifests are typically stored in version control, they aren't easily reusable across environments, self-service for other teams, or packaged in a way that supports consistent versioning and rollback. Helm addresses those limitations by packaging manifests into versioned, parameterized charts. For existing applications, the process typically starts by converting Kubernetes manifests into a standard Helm chart, either manually or with tools such as Helmify. The result is a familiar Helm project structure built around Chart.yaml, parameterized templates, and a values.yaml file, giving teams a reusable deployment artifact instead of a collection of static manifests. Deployment-specific settings, such as image tags, replica counts, and resource limits, move into a values.yaml file, while the underlying templates remain unchanged. Those deployment-specific settings are defined in the chart's values.yaml file. For example: # values.yaml — the self-service interface for application teams replicaCount: 2 image: repository: registry.example.com/myapp tag: "2.1.0" pullPolicy: IfNotPresent resources: limits: cpu: 500m memory: 256Mi requests: cpu: 250m memory: 128Mi ingress: enabled: true host: myapp.internal.example.com annotations: kubernetes.io/ingress.class: "traefik" serviceAccount: create: true name: "myapp-sa" Versioning makes deployments reproducible across environments while providing a clear history of releases. Teams can promote the same chart through development, staging, and production with confidence, then roll back to a previous version if needed. OCI registries address the next challenge: distributing and versioning those charts. Instead of relying on a separate chart repository, teams can store Helm charts alongside container images as immutable, versioned artifacts. Because chart versions can't be overwritten, deployments are reproducible and easier to audit. The approach also fits existing registry workflows. Organizations using Harbor, Amazon ECR, or similar registries can manage container images and Helm charts in the same place, using the same authentication, access controls, and security policies. For example: # Package the chart locally helm package ./myapp --version 2.3.0 # Authenticate to the OCI registry (same registry as your container images) helm registry login registry.example.com \ --username $REGISTRY_USER \ --password $REGISTRY_PASSWORD # Push is stored as an OCI artifact alongside container images helm push myapp-2.3.0.tgz oci://registry.example.com/charts # Any team can pull without touching the source repo helm pull oci://registry.example.com/charts/myapp --version 2.1.0 # Inspect the chart before deploying helm show values oci://registry.example.com/charts/myapp --version 2.1.0 The goal of packaging is to create a self-service deployment model. Once packaged, Helm charts are registered with the NKP catalog, where they appear alongside built-in platform applications as versioned deployment artifacts. Application teams can deploy them by configuring only the settings that vary between environments, while platform teams focus on maintaining reusable application catalogs instead of manually managing deployments. FluxCD deployments, overrides, and upgrades Once Helm charts are stored in an OCI registry, FluxCD keeps deployed clusters aligned with the desired state defined in Git. It continuously reconciles each cluster against that source of truth, automatically correcting configuration drift. In multi-cluster environments, each cluster follows the same reconciliation process using its own configuration. NKP's FluxCD implementation centers on two resources: HelmRepository, which points to the OCI registry, and HelmRelease, which specifies the chart version, configuration values, and target namespace. # Source: points FluxCD at your OCI chart registry apiVersion: source.toolkit.fluxcd.io/v1beta3 kind: HelmRepository metadata: name: internal-charts namespace: flux-system spec: type: oci url: oci://registry.example.com/charts interval: 5m # poll for new chart versions every 5 minutes # Release: declares desired state for a specific deployment apiVersion: helm.toolkit.fluxcd.io/v2beta3 kind: HelmRelease metadata: name: myapp-production namespace: production spec: interval: 10m chart: spec: chart: myapp version: "2.3.0" sourceRef: kind: HelmRepository name: internal-charts namespace: flux-system values: replicaCount: 3 resources: limits: cpu: 1000m memory: 512Mi ingress: host: myapp.prod.example.com Although teams interact with NKP through its web interface, those actions are ultimately represented as standard Kubernetes resources. Configuration changes become declarative objects that FluxCD reconciles like any other GitOps workflow, making the deployment model transparent and compatible with standard Kubernetes tooling without relying on proprietary deployment workflows. Teams typically promote the same chart version from development to staging and production while applying environment-specific overrides through HelmRelease values rather than modifying the chart itself. Promotion becomes a Git commit instead of a manual deployment, with FluxCD automatically reconciling and applying the change. FluxCD also provides continuous drift detection. If someone manually changes a resource in the cluster, FluxCD restores it to the state defined in Git during the next reconciliation cycle. Rolling back a deployment is simply a Git revert, with Git history providing a complete audit trail of configuration changes. How to integrate third-party tools without losing openness Enterprise platform teams are often asked to integrate tools such as vulnerability scanners, cost management dashboards, and application performance monitoring (APM) platforms. The tools themselves aren't the problem. The problem is managing each one through a separate deployment and maintenance process, increasing operational complexity over time. NKP addresses this by treating third-party software like any other platform application. Whether it's an upstream open-source project or a commercial product distributed as a Helm chart, it follows the same Helm-over-OCI packaging model and is deployed and managed through FluxCD. The outcome is a consistent deployment and lifecycle workflow across both first- and third-party applications. For example, an upstream Helm chart such as Redis can be published to the NKP catalog and managed through the same deployment workflow as a first-party application, avoiding the need for a separate integration process. Because this approach relies on standard Kubernetes resources, Helm charts, Git, and Kubernetes RBAC, those workloads remain portable across platforms. As Dastmalchi-Round summarizes, "If it works on Kubernetes, it will work on NKP." Dastmalchi-Round notes that the biggest integration challenges typically come from tools that rely on rigid deployment models, particularly older operator-based packages that expose little configuration. "A few years ago, there was a trend of people overusing the operator pattern for packaging applications," he says. "Operators have their uses, but when they became the distribution artifact, they often resulted in big, opaque blobs running in your cluster. If they didn't do exactly what you needed, you were out of luck." As more vendors have adopted Helm-based packaging, those limitations have become less common. Examples of Third-Party Tool Integrations in NKP Integration Type Packaging Model Configuration Upgrade Path NKP Catalog Security scanner (e.g., Trivy) Helm chart via OCI values.yaml in Git FluxCD HelmRelease bump Yes Custom Grafana dashboard Helm chart + ConfigMap Dashboard JSON in Git Chart version update Yes Cost management (e.g., OpenCost) Helm chart via OCI values.yaml in Git FluxCD HelmRelease bump Yes Service mesh (e.g. Istio) Helm chart via OCI IstioOperator CRDs in Git Controlled chart upgrade Yes Legacy operator-only tool Operator bundle Operator-managed CRDs Operator version update Requires evaluation In practice, the less a tool depends on proprietary deployment mechanisms, the easier it is to integrate, manage, and move between Kubernetes platforms. Conclusion: the platform that gets out of the way NKP doesn't replace Kubernetes workflows—it builds on them. Helm packages applications, OCI registries distribute them, Git defines the desired state, and FluxCD keeps deployments in sync. Instead of introducing proprietary workflows, NKP brings these familiar tools together with the governance, lifecycle management, and self-service capabilities required for enterprise-scale operations. It standardizes these workflows across any environment, including public clouds, on-premises, and edge locations. For enterprise teams, the value lies in achieving consistency without sacrificing portability. As Dastmalchi-Round notes, the question isn't whether lock-in exists, but how costly it is to leave. By relying on upstream Kubernetes components, Helm charts, and GitOps workflows, organizations retain portable applications and deployment artifacts even if they choose a different platform in the future. In the end, an open platform shouldn’t be defined by its licensing model. It should be defined by how much of your platform remains yours if you decide to move on.

This guide explains zone-aware routing from a Kubernetes-first point of view. It covers: why zones matter in cloud platformswhich topology labels Kubernetes places on nodeshow Kubernetes first tried to solve locality through Servicewhat gaps remained after those Service-based featureshow Gateway API implementations such as Envoy Gateway and kgateway built on top of that foundation Why Zones Matter In cloud platforms, a zone is a logical failure domain inside a region. Zones usually have low-latency networking within the zone, but crossing zones can increase both latency and cost. That cost is not theoretical. AWS documents that traffic within the same Availability Zone is free, while traffic that crosses Availability Zones typically incurs data transfer charges, and cross-zone transfer is generally billed in both directions, so a single round trip can be charged twice. See: AWS Architecture Blog: Overview of Data Transfer Costs for Common ArchitecturesAmazon EC2 pricing: Data Transfer This is one reason distributed systems try to keep traffic local when they can, while still preserving failover to other zones. The Topology Information Kubernetes Already Has Kubernetes did not start by inventing zone-aware traffic policies. It started by carrying topology information on nodes. The two most important well-known labels are: topology.kubernetes.io/regiontopology.kubernetes.io/zone According to the Kubernetes reference, these labels are populated on Node objects by the kubelet or the external cloud-controller-manager when the cluster is integrated with a cloud provider. In non-cloud environments, operators can set them manually if the topology model still makes sense. Reference: Kubernetes well-known labels: topology.kubernetes.io/zone In managed clusters, these labels are commonly present by default. Here is the kind of node data Kubernetes typically exposes: YAML apiVersion: v1 kind: Node metadata: name: ip-10-0-12-34.ec2.internal labels: kubernetes.io/hostname: ip-10-0-12-34.ec2.internal topology.kubernetes.io/region: us-east-1 topology.kubernetes.io/zone: us-east-1a That topology data is useful for scheduling, spreading replicas, volume placement, and eventually traffic routing. The Original Service Model The original Kubernetes Service abstraction solved a different problem first: stable discovery and virtual IPs for ephemeral Pods. At the beginning, the model was simple: a Service selected a set of Podskube-proxy programmed forwarding rulestraffic could be sent to any healthy endpoint behind the Service That was excellent for reachability and abstraction, but it had no built-in notion of zone locality. The gap was straightforward: the Service abstraction knew which endpoints existed, but not that a client in zone-a should usually prefer endpoints in zone-a. Kubernetes' First Attempts to Improve Locality Through Services Kubernetes gradually added locality-aware behavior on top of Service, mostly by improving how endpoint selection works. Internal Traffic Policy One early mechanism was internalTrafficPolicy: Local. This tells kube-proxy to use only node-local endpoints for cluster-internal traffic. Example: YAML apiVersion: v1 kind: Service metadata: name: my-service spec: selector: app: my-app ports: - port: 80 targetPort: 8080 internalTrafficPolicy: Local Reference: Kubernetes Service Internal Traffic Policy This helps with node locality, but it is not zone-aware routing. Its limitations are important: it is node-local, not zone-localif a node has no local endpoint, the Service behaves as if it has zero endpoints from that node's perspectiveit is too strict for many multi-zone workloads that want zonal preference, not node affinity So this was useful, but it did not really solve multi-zone locality. Topology Aware Routing With Services Kubernetes next introduced Topology Aware Hints, now called Topology Aware Routing. This works through two components: The EndpointSlice controller looks at endpoint and node topology.kube-proxy consumes hints from EndpointSlices and prefers endpoints closer to the client zone. Historically, the Service-side configuration was commonly exposed through the service.kubernetes.io/topology-mode: Auto annotation: YAML apiVersion: v1 kind: Service metadata: name: zone-aware-backend annotations: service.kubernetes.io/topology-mode: Auto spec: selector: app: backend ports: - port: 80 targetPort: 8080 Conceptually, the flow looks like this: This was Kubernetes' first real zone-aware answer at the Service layer. It is useful historical context, but it is no longer the clearest Service-level API to emphasize for new users. Traffic Distribution Preferences Kubernetes later added trafficDistribution as a clearer way to express routing preferences. In current Kubernetes documentation, the relevant zone-level preference is: PreferSameZone The older PreferClose name is documented as deprecated in favor of PreferSameZone, though you may still see PreferClose in some provider and implementation docs that have not yet caught up. Example: YAML apiVersion: v1 kind: Service metadata: name: zone-aware-backend spec: selector: app: backend ports: - port: 80 targetPort: 8080 trafficDistribution: PreferSameZone Reference: Kubernetes Service trafficDistribution This is a better API shape than older annotations because it is explicit in the Service spec and described as a preference rather than a strict guarantee. In practice, that means current Kubernetes guidance emphasizes trafficDistribution: PreferSameZone, while the older topology-mode: Auto path is best understood as part of the feature's evolution. What Gap Remained After Service-Based Locality Kubernetes Services improved a lot, but they still left several gaps. The Behavior Is Best Effort Topology-aware routing is not a hard guarantee. Kubernetes documents multiple safeguard cases where the system falls back to cluster-wide routing. Examples include: too few endpointsimpossible balanced allocationmissing topology labels on one or more nodesmissing hints for one or more endpointsno hinted endpoint for the local zone That is correct for safety, but it means the behavior is heuristic and conditional. It Assumes a Certain Traffic Shape Kubernetes explicitly documents that Topology Aware Routing works best when traffic is roughly evenly distributed and when there are enough endpoints per zone. If most traffic originates from one zone, local subsets can overload while the global service still looks healthy. It Is Scoped to the Service Datapath This is the most important architectural gap. Service-level topology features influence how kube-proxy chooses endpoints for Service traffic. They do not automatically solve every higher-level data plane. In particular, they do not by themselves define: how an L7 gateway proxy should understand its own zonehow an Envoy-based gateway should configure locality-aware upstream load balancinghow a gateway controller should express stricter local preference versus simple best-effort localityhow policy should attach to particular routes, gateways, or backends That left room for Gateway API implementations to expose richer locality controls. Why Gateway API Implementations Stepped In Gateway API is intentionally expressive and extensible. It standardizes core routing objects, but implementations often add policy CRDs to expose features that are specific to their data plane. That distinction matters here: Gateway API itself does not define one universal, cross-implementation zone-aware policy. Instead, it gives implementations room to expose locality behavior in a way that matches their proxy and control-plane design. Reference: Gateway API overview This is where zone-aware routing became more explicit at the gateway layer. Instead of relying only on kube-proxy's Service behavior, gateway implementations can: understand the proxy's own localityread backend endpoint localityconfigure the underlying proxy's load balancer directlyexpose locality policies as route or backend-attached configuration Example of How Envoy Gateway Addresses the Gap Envoy Gateway supports two paths: Reusing Kubernetes Service-level locality such as Topology Aware Routing or trafficDistributionConfiguring zone awareness directly through BackendTrafficPolicy Reference: Envoy Gateway zone-aware routingEnvoy zone-aware routing Example BackendTrafficPolicy: YAML apiVersion: gateway.envoyproxy.io/v1alpha1 kind: BackendTrafficPolicy metadata: name: zone-aware-routing spec: targetRefs: - group: gateway.networking.k8s.io kind: HTTPRoute name: zone-aware-routing loadBalancer: type: RoundRobin zoneAware: preferLocal: minEndpointsThreshold: 1 force: minEndpointsInZoneThreshold: 1 That is a meaningful step beyond plain Service because the gateway layer is now explicitly participating in locality-aware upstream balancing. Example of How kgateway Addresses the Gap kgateway takes a similar approach in spirit: proxy locality is made explicit, and backend load-balancing behavior is configured through policy rather than relying only on Service heuristics. At a high level, kgateway combines: Gateway proxy locality configurationBackend-attached load-balancing policyNative Envoy locality-aware upstream load balancingEndpoint locality metadata that Envoy can use directly Architectural Summary The progression looks like this: Kubernetes Service solved stable discovery and reachability.internalTrafficPolicy improved node-local routing, but not zonal routing.Topology Aware Routing and trafficDistribution added zone-aware preferences to the Service datapath.Gateway API implementations extended the model so L7 gateways and proxies could make explicit locality-aware decisions themselves. Practical Takeaways Kubernetes already provides the topology metadata needed for zone-aware decisions.Service-native locality is useful, but it is heuristic and scoped to the Service datapath.Zone-aware traffic for gateways usually needs the gateway implementation to understand locality too.Modern Gateway API implementations fill that gap by attaching locality-aware load-balancing policy closer to the L7 data plane. Where Zone-Aware Routing Matters in Practice Zone-aware routing usually becomes worth the added operational attention when one or both of these are true: The workload has a tight latency budget, especially at p95 or p99The system moves enough east-west traffic that even a small per-GB cross-zone charge becomes material Common examples include: Gaming platforms, where matchmaking, player session state, inventory, and real-time coordination are sensitive to a few extra milliseconds of network delayFinancial services, where payment, quote, fraud, or checkout paths care more about predictable tail latency than average latencyLarge SaaS and enterprise control planes, where a gateway fans out to many internal APIs and the aggregate cross-zone traffic becomes a real monthly costAI inference, media delivery, logging, and telemetry pipelines, where payload sizes are large enough that bandwidth cost matters even when latency is less critical Worked Example: Multiplayer Gaming Backend Suppose a regional game API runs gateway proxies and backend pods in three zones. Players connect to a gateway in zone-a, and that gateway calls a player-state service that is also deployed in zone-a, zone-b, and zone-c. Assume the following: 25,000 requests per second reach the player-state service from zone-athe combined request and response payload is about 40 KiB per callcross-zone traffic is billed at a representative $0.01 per GBwithout zone awareness, only about one third of those calls stay in zone-a, while the other two thirds go to zone-b or zone-c Actual billing varies by provider, region, and direction of transfer, but the point of the example is that a seemingly small per-GB rate compounds quickly on hot service paths. That means the traffic volume from zone-a to the player-state service is about: 25,000 x 40 KiB per second, or roughly 1 GB/s totalif two thirds of that traffic crosses zones, that is about 0.67 GB/s of cross-zone trafficover a 30-day month, that is about 1.7 million GBat $0.01 per GB, that is about $17,000 per month in cross-zone transfer for just that one service path That is the cost side. The latency side can matter even more for the player experience. If each cross-zone hop adds only 1-3 ms, a request path that fans out to several internal services can add multiple milliseconds of extra tail latency. For a gaming workload, that can affect: matchmaking responsivenesssession join timethe smoothness of player state or presence updateshow stable the system feels during traffic spikes and retries This is why zone-aware routing is not only a cost optimization. In some industries, it is a user-experience and SLO control. Worked Example: Large SaaS Control Plane The same logic applies outside gaming. Consider a large enterprise SaaS platform where each incoming API request hits a gateway and then fans out to an auth service, tenant metadata service, feature-flag service, and audit pipeline. Even if each individual backend call is small, the gateway can generate a large amount of aggregate east-west traffic. In that kind of system, zone-aware routing helps in two ways: it removes avoidable cross-zone traffic from the steady-state hot pathit reduces the chance that a multi-hop request burns several extra milliseconds just on internal network distance For that kind of platform, the business case is usually a combination of lower regional data-transfer cost, tighter latency distributions, and better failure-domain alignment. Conclusion Zone-aware routing is the story of a single idea moving down the stack. Kubernetes started with topology labels on nodes, then taught the Service datapath to prefer local endpoints through internalTrafficPolicy, Topology Aware Routing, and trafficDistribution. Those features are valuable, but they are best-effort and they stop at the Service boundary, which leaves L7 gateways unable to reason about their own locality. Gateway API implementations such as Envoy Gateway and kgateway pick the idea up from there, making proxy locality explicit and pushing locality-aware load balancing into Envoy where it can act on real endpoint metadata. The practical guidance is short. Start with the Service-native controls, because they are simple and often enough. Reach for gateway-level locality policy when you have a tight tail-latency budget, or enough east-west traffic that cross-zone transfer becomes a line item you can see. In both cases, the goal is the same: keep traffic local when you safely can, and fail across zones when you must. Further Reading Kubernetes ServiceKubernetes Topology Aware RoutingKubernetes Service Internal Traffic PolicyKubernetes well-known topology labelsGateway API overviewAWS Architecture Blog: Data transfer costs