Kubernetes on bare metal: the 4 cloud integrations you must build

TL;DR

Kubernetes on bare metal fails in four specific places a cloud used to cover: machine provisioning, node identity, LoadBalancer services, and block storage. Oxide’s public integration work is the clearest inventory of those seams anyone has published — three shipped, and the fourth is blocked by a hardware constraint rather than by engineering time. Plan them as four separate projects, because the seam that stalls a cluster fastest is the one that leaves every LoadBalancer service sitting at <pending> forever.

What is a bare-metal Kubernetes cluster?

A bare-metal Kubernetes cluster is one where four integration seams are yours to own: provisioning, node identity, service load balancing, and volume attachment. Everything else about the cluster is identical to a managed one. These four are not.

For most of Kubernetes’ life, “the cloud provider” was a box in the architecture diagram that nobody opened. It is now an explicitly empty box. KEP-2395 removed cloud-provider code from the Kubernetes tree in phases — the feature gates were promoted to GA and locked in the v1.31 cycle, and the last remaining in-tree provider was deleted on 6 May 2024. Every provider today is an out-of-tree component that someone deploys as pods in your cluster.

Seam	What a cloud supplied	What you supply	Failure mode if missing
Provisioning	Machines on API request	Cluster API provider, node driver, or image pipeline	No self-service capacity; clusters grow by hand
Node identity	cloud-controller-manager	An out-of-tree CCM for your infrastructure	No provider ID, wrong addresses, dead nodes linger
Load balancing	Managed LB per Service	Service controller or MetalLB-style allocator	EXTERNAL-IP stuck at <pending>
Storage	Block volumes on demand	CSI driver	PersistentVolumeClaim never binds

Oxide is a useful case study because they built all four against their own rack and wrote down what each one took. They are the rare vendor publishing the seam list rather than the marketing summary — including the seam they have not finished.

Seam 1 — Provisioning: something has to make the machines

The first seam is the least glamorous and the most immediately obvious: a cluster that cannot create machines is a cluster that grows by ticket.

Oxide filled it three times, for three different customer shapes. A Rancher node driver (rancher-machine-driver-oxide) is an executable plugin that translates Rancher’s provisioning calls into Oxide API requests — the cheapest possible integration, and one that had a customer in production shortly after release. An Omni infrastructure provider (omni-infra-provider-oxide) creates Talos Linux instances and registers them with Omni. And CAPOx (cluster-api-provider-oxide) is the Cluster API infrastructure provider: clusters described as custom resources and reconciled by a controller, which is the declarative, Kubernetes-native option.

The detail worth stealing from that work is not the architecture, it is the failure. Building the Omni provider took roughly seven weeks — 24 September to 12 November 2025, timed to land at KubeCon North America — and a chunk of that went into a bug that has nothing to do with Kubernetes: Talos’ filesystem probe only attempted ISO 9660 superblock reads, so it could not see Oxide’s FAT12-formatted user-data volume. The shipped workaround was padding the user data with comments until an ISO superblock got created. Two upstream issues came out of it.

That is what these projects actually look like. The integration logic is tractable; the schedule risk lives in the boot path, the image format, and the probe order of a distro you did not write.

Seam 2 — Node identity: the boring seam everyone skips

The cloud-controller-manager is the component that tells Kubernetes what a machine is. Per the Kubernetes architecture docs, its node controller updates each Node object with the server’s unique identifier from the provider API, annotates and labels it with region and resource information, obtains the node’s hostname and network addresses, and — the part people forget — verifies node health by asking the provider whether an unresponsive server has actually been deleted, removing the Node object if so.

Oxide’s CCM (oxide-cloud-controller-manager) ships a node controller and a service controller, and their framing of it is the right one: it “provides a durable extension point inside each cluster.” It is the thing already running with credentials and a reconcile loop, so every later seam hangs off it.

This is the seam that gets skipped, because a cluster comes up fine without it. Nodes register, pods schedule, everything looks healthy. What you have actually accepted is that no Node carries a provider ID, addresses are whatever the kubelet inferred from its own interfaces, and a machine destroyed out from under the cluster remains a Node object until a human deletes it. Then you go to build seam 3 or seam 4, both of which want to map a Kubernetes node to an infrastructure instance, and the mapping does not exist.

Build the CCM first, even though it fixes nothing you can see today.

Seam 3 — LoadBalancer: the seam that fails loudest

Apply a Service of type: LoadBalancer on a cluster with no service controller and you get the single most common bare-metal Kubernetes support question: EXTERNAL-IP shows <pending>, and stays there. It is not a slow allocation. Nothing in the cluster is listening for that Service at all, so the address is never requested and the field is never written.

Oxide’s implementation shows the mechanics a substitute has to reproduce. A floating IP delivers traffic to a single instance, and Oxide “translates the destination address of inbound traffic to the instance’s internal IP before sending the traffic to the instance” — so the packet arrives at a node bearing that node’s own address, and the ordinary Service dataplane spreads it across pods from there. The Service status carries two entries: the floating IP in Proxy mode, and the node’s internal IP in VIP mode.

The stated limitation is the interesting part: only externalTrafficPolicy: Cluster is supported today. That is a real trade, not a footnote. Cluster means a packet can take a second hop from the receiving node to a pod elsewhere, and the client source IP is lost to SNAT along the way — so your access logs see node addresses, and anything doing IP-based rate limiting or geo rules has to read X-Forwarded-For instead. Local would avoid both, but it requires the load balancer to health-check each node individually and stop sending traffic to nodes with no local endpoint, which is a materially larger amount of infrastructure.

Seam 4 — Storage: the seam hardware can block

The other three seams are engineering time. This one is not, and it is the most instructive item in the whole story.

Oxide’s CSI plugin is designed — it has an RFD, number 595 — and it is not shipped, because of a lifecycle mismatch that no driver code can paper over. Oxide requires an instance to be stopped before a disk is attached or detached. Kubernetes requires the opposite: when a pod moves to another node, the volume must detach from a running machine and attach to another running machine, live. Until disk hot-plug support lands across the hypervisor and the API layers, a conformant CSI driver is not implementable.

So the interim answer is Longhorn on Oxide local disks. And the reason given for that specific shape is a good general rule: it avoids stacking two replication layers on top of each other. Oxide’s storage already replicates, so running a distributed storage system that also replicates over the top means paying for the same durability twice — in write amplification and in capacity — for no additional safety.

The generalisable lesson: when you audit a platform, check the volume attach lifecycle before you check anything else. It is the one seam where the answer can be “not this quarter,” and it is invisible until you try to reschedule a stateful pod.

Four mistakes this inventory prevents

1. Treating “bare metal” as one project. It is four, with different owners, different failure modes, and one of them potentially gated on firmware. A single line item in a migration plan will be wrong by an order of magnitude.
2. Skipping the CCM because the cluster works without it. Node identity is invisible debt until seams 3 and 4 need it, and then it is on the critical path.
3. Assuming a LoadBalancer allocator is the whole job. MetalLB genuinely solves seam 3, which is why it feels like the answer — it is the seam whose absence you notice on day one. It does not touch provisioning, node identity, or storage.
4. Stacking replication. Running a replicating distributed filesystem over already-replicated storage is the most common way bare-metal clusters end up slow and expensive at the same time.

Should you run Kubernetes on bare metal at all?

Only if you are getting something back that a managed platform cannot give you. That is a genuine calculation, not a rhetorical question — I ran a version of it in Docker Swarm vs Kubernetes: the $166/mo reality check, and the honest answer for a lot of workloads is that the orchestration bill was never the expensive part. If your platform work can sit on someone else’s control plane, running CI/CD on Cloudflare Workflows is the shape of that alternative: zero seams, because you own nothing underneath.

The case that does justify it is usually hardware you cannot rent on acceptable terms — GPUs at a utilisation level where tuning vLLM throughput flags beats renting more of them, or a workload whose performance is dominated by the hypervisor rather than the code, the way slow LLM inference in macOS VMs turns out to be a virtualisation problem rather than a model problem.

When that is your situation, own the seams deliberately, in this order: CCM first because everything else keys off node identity, load balancing second because it is what unblocks users, provisioning third, and storage scoped honestly against what your hardware can currently do. The alternative — discovering the four seams one production incident at a time — is the same project with worse sequencing.

Frequently asked questions

What does a cloud provider actually do for Kubernetes?

It fills four integration seams that core Kubernetes deliberately leaves empty: it provisions the machines, it tells the cluster what each machine is via a cloud-controller-manager, it turns a Service of type LoadBalancer into a real external address, and it attaches block volumes through a CSI driver. None of that code lives in Kubernetes itself anymore. KEP-2395 reached GA and locked its feature gates in the v1.31 cycle, and the last in-tree provider was deleted in May 2024, so every provider today is an out-of-tree component someone has to supply.

Why does my LoadBalancer service stay stuck at pending?

Because nothing in the cluster is listening for that Service and allocating an address. The service controller inside a cloud-controller-manager is the component that calls an infrastructure API, gets an external IP, and writes it back into the Service status. On bare metal with no cloud-controller-manager and no MetalLB-style substitute, that reconcile loop simply never runs, so EXTERNAL-IP sits at <pending> indefinitely. It is not a timeout, and waiting does not resolve it.

Do I need a cloud-controller-manager if my cluster already starts fine?

A cluster will come up without one, which is exactly why the seam gets skipped. What you lose is node identity: Node objects carry no provider ID, addresses and region labels are whatever the kubelet guessed, and a machine that dies stays in the cluster as a Node object because nothing asks the infrastructure API whether it still exists. That debt compounds, because both the volume attach path and the load balancer path key off the provider ID the node controller was supposed to write.

Is MetalLB enough to run Kubernetes on bare metal?

MetalLB is a good answer to one of the four seams. It gives a Service of type LoadBalancer a real address on a network you control, which unblocks the failure that stops clusters fastest. It does not provision machines, does not maintain node identity, and does not attach block volumes, so treat it as one component of a bare-metal platform rather than the platform itself.

Why is a CSI driver harder to build than the other integrations?

Because it can be blocked by the hardware rather than by engineering time. Kubernetes assumes a volume can be attached to a running node on demand, since that is what happens when a pod is rescheduled. Oxide’s CSI work, tracked in RFD 595, is gated on disk hot-plug support landing across the hypervisor and the API, because today an instance has to be stopped before a disk is attached or detached. No amount of driver code works around a stop-to-attach lifecycle.

How long does it take to build one of these integrations?

Oxide’s Omni infrastructure provider went from start to a working demo at KubeCon North America in about seven weeks, from 24 September to 12 November 2025. That is one seam, built by a team with full access to its own API and its own hardware to test on. Treat it as a floor rather than an estimate, and note that a meaningful share of that time went into an unrelated filesystem-probe bug in Talos rather than into the integration logic.

Sources

• Matthew Sanabria, Oxide Computer — Kubernetes on Oxide: How customer needs shaped our integrations (the Rancher node driver, Omni infrastructure provider, CAPOx, the CCM, the floating-IP LoadBalancer mechanics, and RFD 595)
• Kubernetes documentation — Cloud Controller Manager (node controller, route controller and service controller responsibilities)
• Kubernetes Enhancements — KEP-2395: Removing In-Tree Cloud Provider Code (the phased removal, GA in the v1.31 cycle, last in-tree provider deleted 6 May 2024)