kube-proxy vs Cilium: What Replacing the Kubernetes Datapath Actually Changes

Every Service in your cluster is a lie that kube-proxy tells on every node — and at a few thousand Services, the cost of telling that lie with iptables becomes measurable in packet latency and CPU. Cilium's kube-proxy replacement doesn't tell the lie faster; it moves it somewhere cheaper. For east-west traffic, the Service VIP never even makes it onto the wire.

That distinction — where the Service-to-endpoint translation happens — is the entire kube-proxy vs Cilium debate. Get it, and everything else (the benchmarks, the caveats, the migration steps) falls into place.

What kube-proxy actually does

A ClusterIP is a virtual address. No pod owns it, no interface holds it, and nothing answers ARP for it. When a pod sends a packet to 10.96.0.10:80, something on the node must rewrite the destination to a real pod IP before the packet goes anywhere. That something is kube-proxy.

kube-proxy runs as a DaemonSet, watches the API server for Service and EndpointSlice changes, and programs the node's Linux networking to do the VIP → endpoint DNAT. It is not a proxy in the data plane sense — no packet flows through the kube-proxy process. It is a control-plane agent that writes rules and lets the kernel do the work.

How it writes those rules is where the two classic modes diverge.

iptables mode: linear rules, linear pain

In iptables mode, kube-proxy renders every Service into a chain of netfilter rules: one chain per Service, one rule per endpoint, with probability-weighted jumps to fake random load balancing:

# What one 3-endpoint Service looks like in iptables
-A KUBE-SVC-XYZ -m statistic --mode random --probability 0.333 -j KUBE-SEP-A
-A KUBE-SVC-XYZ -m statistic --mode random --probability 0.500 -j KUBE-SEP-B
-A KUBE-SVC-XYZ -j KUBE-SEP-C

Two problems compound at scale:

1. Lookup is O(n). iptables chains are evaluated sequentially. The first packet of every new connection walks the rule list until it hits a match. At 1,000 Services with 10 endpoints each, that is tens of thousands of rules, and p99 first-packet latency starts to show it.
2. Updates are expensive. iptables replaces rules by rewriting tables under a lock. Historically, one pod becoming ready in one Deployment triggered a full iptables-restore of every Service rule on every node; kube-proxy now restores only the chains that changed (MinimizeIPTablesRestore — on by default since Kubernetes 1.27, GA in 1.28), which blunts the worst of it. But in churny clusters — autoscaling, spot instances, frequent deploys — the watch → sync → reprogram loop still costs CPU, and sync latency still means traffic keeps hitting terminated pods until the next sync lands.

Add conntrack on top: every connection through the DNAT path allocates a conntrack entry, the table has a finite size, and when it fills you get silently dropped SYNs that manifest as random timeouts nobody can reproduce.

IPVS mode: better lookups, same architecture

IPVS mode replaces the per-Service chains with the kernel's IP Virtual Server — a hash-table-based L4 load balancer. Lookups become O(1), you get real scheduling algorithms (round-robin, least-connection), and 10,000 Services stops being scary.

But IPVS fixes the lookup, not the model. kube-proxy still shells out to iptables for SNAT, masquerade, and NodePort edge cases, so you run both subsystems. Endpoint updates are cheaper than full-table rewrites but still go through the same watch → sync → program loop. Conntrack is still in the path. And packets still traverse the full netfilter hook chain — PREROUTING, FORWARD, POSTROUTING — for every hop.

IPVS raised kube-proxy's ceiling. It did not change what kube-proxy is.

How Cilium replaces it

Cilium's kube-proxy replacement (kubeProxyReplacement: true) implements the Service abstraction entirely in eBPF programs attached at three places in the kernel, chosen by traffic direction. If you want the broader Cilium picture first — CNI, network policy, identity model — start with Cilium and eBPF networking; this section is just the Service datapath.

East-west: socket-level load balancing

For pod-to-Service traffic inside the cluster, Cilium attaches eBPF programs to the cgroup socket hooks (connect(2), sendmsg(2), recvmsg(2)). When an application calls connect() to a ClusterIP, the eBPF program rewrites the destination address to a backend pod IP before the first packet is ever built.

Read that again, because it is the headline: there is no per-packet NAT for east-west traffic. The translation happens once, at socket creation, in the syscall path. The packet leaves the pod already addressed to the real backend. No DNAT, no reverse translation on the reply, no conntrack entry for the VIP, no netfilter traversal for the Service logic at all.

This is why Cilium's east-west numbers do not just beat iptables — they beat IPVS. IPVS still translates packets; Cilium stops the translation from being a packet-path operation entirely.

North-south: tc and XDP

Traffic arriving from outside — NodePort, LoadBalancer, ExternalIP — cannot be caught at a socket hook because there is no local socket making the call. Here Cilium attaches eBPF at the tc (traffic control) ingress hook on the NIC, or, for the fastest option, at XDP — which runs before the kernel even allocates an sk_buff for the packet. XDP-based NodePort handling can process and forward packets at a rate iptables cannot approach, because most of the kernel network stack never runs.

Backend selection, session affinity, and NAT for these paths live in eBPF hash maps — O(1) lookups, and updates that touch one map entry per endpoint change rather than rewriting a global table.

What measurably changes

Numbers vary by kernel, NIC, and workload, so treat these as directions rather than gospel — but the directions are consistent across Cilium's published benchmarks and independent tests:

The practical translation: below roughly a thousand Services with modest churn, you will struggle to measure a difference your users feel. Past that — and especially in clusters with aggressive autoscaling or thousands of endpoints behind single Services — the iptables curve bends upward while the eBPF curve stays flat.

What you gain beyond speed

The kube-proxy replacement is not only a performance play. It unlocks features kube-proxy structurally cannot offer:

DSR (Direct Server Return). With kube-proxy, external traffic hitting a NodePort on the "wrong" node gets SNAT'd before forwarding, so the backend's reply can route back through the same node — which destroys the client source IP unless you use externalTrafficPolicy: Local and accept uneven load spreading. Cilium's DSR mode lets the backend pod reply directly to the client, preserving source IP and cutting a hop, without the Local-policy trade-off. The catch: most DSR dispatch modes require native routing — if you run tunnel encapsulation, Geneve dispatch is the one that works.

Maglev consistent hashing. Cilium can use Google's Maglev algorithm for backend selection on north-south traffic. When a backend is added or removed, only a minimal fraction of flows are re-hashed — existing connections mostly stay pinned to their backends. For anything stateful behind a NodePort, this is the difference between a rolling deploy being invisible and it resetting a chunk of live connections.

Hubble observability. Because every Service translation flows through Cilium's datapath, Hubble can show you flow-level answers to "which pod talked to which Service, and was it dropped, and by which policy?" — with zero sidecars and zero packet capture. When you later layer on zero-trust network policies, the same datapath enforces them, and Hubble tells you what they did.

Plus smaller items — graceful backend draining, termination-aware load balancing — that kube-proxy either cannot do or does coarsely.

What you give up, and the caveats

This is the section vendors skip. Do not.

Kernel version is a hard dependency. The full replacement wants a reasonably modern kernel — current Cilium releases document 5.10 (or a vendor-backported equivalent like RHEL 8's 4.18) as the baseline, and newer features (BIG TCP, netkit, some DSR dispatch modes) gate on newer kernels still. On a distro with an ancient kernel, Cilium silently falls back or refuses features. cilium-dbg status inside the agent pod tells you what you actually got; believe it, not the Helm values.

NodePort edge cases exist. Socket-level LB means a pod connecting to its own node's NodePort behaves differently than with kube-proxy. Applications that bind to specific interfaces, rely on seeing the VIP as the destination address (some older service meshes and iptables-based tooling do), or do odd things with SO_REUSEPORT can surprise you. Anything that itself injects iptables rules expecting kube-proxy's chains — legacy Istio iptables interception being the classic example — needs checking. If you run a mesh, read what a service mesh actually does alongside this before you migrate, because two things rewriting connection destinations is a debugging session waiting to happen.

Your debugging muscle memory is now wrong. iptables -L -t nat returns nothing useful. The Service table lives in eBPF maps, and the tools are cilium-dbg service list, cilium-dbg bpf lb list, and Hubble. These are better tools once learned — but at 3 a.m. during an incident, "better but unfamiliar" loses to "worse but memorized." Budget real time for the team to relearn the datapath.

Managed cluster support is uneven.

• EKS is the friendliest: replace the AWS VPC CNI with Cilium (or run it in chaining mode, which does not give you the full kube-proxy replacement) and it works; EKS Anywhere ships Cilium as its default CNI. You own the CNI swap, though — see the VPC CNI guide for what you are replacing.
• AKS offers Azure CNI powered by Cilium as a first-party option — but Microsoft's managed flavor does not expose every OSS feature, and kube-proxy replacement behavior is theirs to configure, not yours. The trade-offs mirror the broader Cilium vs Calico on AKS decision.
• GKE uses Cilium under the hood as GKE Dataplane V2 — you get eBPF Service handling and no kube-proxy on Dataplane V2 clusters, but it is Google's build with Google's feature gates; you cannot Helm-upgrade your way to OSS features.

If you are choosing a provider partly on networking flexibility, that difference is worth weight in the EKS vs GKE vs AKS comparison.

Migration: the actual steps

On a self-managed cluster (kubeadm, Cluster API, bare metal), the migration is short but ordered:

# 1. Install/upgrade Cilium with the replacement enabled
helm upgrade --install cilium cilium/cilium \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=<API_SERVER_IP> \
  --set k8sServicePort=6443

k8sServiceHost/k8sServicePort matter: with kube-proxy gone, Cilium cannot reach the API server through the kubernetes ClusterIP that it is itself responsible for implementing. Point it at the real endpoint.

# 2. Remove kube-proxy and its state
kubectl -n kube-system delete ds kube-proxy
kubectl -n kube-system delete cm kube-proxy

# 3. Flush kube-proxy's leftover iptables rules on each node
iptables-save | grep -v KUBE | iptables-restore

Step 3 is the one people skip. Stale KUBE-* chains do not just sit there harmlessly — they can keep intercepting traffic and mask whether the eBPF path is actually handling it. On kubeadm clusters, also set skipPhases: [addon/kube-proxy] (or kubeadm init --skip-phases=addon/kube-proxy) so upgrades do not reinstall it behind your back.

1# 4. Validate — do not assume
2cilium status --wait
3# Agent, operator, and Hubble health — everything should report OK
4
5kubectl -n kube-system exec ds/cilium -- cilium-dbg status --verbose | grep -A5 KubeProxyReplacement
6# Look for: KubeProxyReplacement: True — and exactly which features are active on this kernel
7
8cilium connectivity test
9# Full functional test: ClusterIP, NodePort, pod-to-pod, policy

cilium connectivity test deploys real workloads and exercises every Service path. It takes several minutes. Run it anyway — it is the difference between "the DaemonSet is Running" and "Services actually resolve to endpoints."

On managed clusters, the sequence differs: GKE Dataplane V2 and AKS's Cilium mode handle kube-proxy removal for you at cluster or nodepool creation; on EKS you follow roughly the self-managed path but must also decide the VPC CNI question first.

When to not bother

Honest answer: most clusters should keep kube-proxy.

• Small and medium clusters. Under ~500 Services with normal churn, iptables mode's overhead is noise. You would be trading a well-understood, boring component for operational novelty with no measurable win.
• Teams without Linux depth. If nobody on-call can read cilium-dbg output or reason about kernel versions, the failure modes cost more than the latency saves.
• Old kernels you cannot change. RHEL-lineage nodes on stock old kernels will get a degraded feature set. Half a replacement is worse than none.
• Heavy dependence on iptables-integrating tooling you cannot upgrade or validate.

The inverse profile — thousands of Services, high endpoint churn, conntrack exhaustion incidents in your postmortems, NodePort throughput as a real bottleneck, or you already want Cilium for network policy and Hubble anyway — is where the replacement pays for itself quickly. If Cilium is already your CNI, running it without the kube-proxy replacement means maintaining two Service datapaths; finishing the job is usually the simpler steady state.

Frequently Asked Questions

Does Cilium fully replace kube-proxy?

Yes, when kubeProxyReplacement: true and the kernel supports it — ClusterIP, NodePort, LoadBalancer, ExternalIP, session affinity, and HostPort are all handled in eBPF, and the kube-proxy DaemonSet can be deleted. Run cilium-dbg status inside the agent pod and confirm KubeProxyReplacement: True; a partial/fallback mode means some paths still expect kube-proxy.

Is IPVS mode good enough instead?

Often, yes. IPVS fixes the O(n) lookup problem and comfortably handles very large Service counts. What it does not fix: per-packet NAT and conntrack for east-west traffic, the dual iptables+IPVS maintenance surface, or any of the feature gaps (DSR, Maglev, socket-level LB, Hubble). If lookup latency was your only problem, IPVS is the smaller change.

Can I run kube-proxy and Cilium's replacement together?

Cilium as a CNI alongside kube-proxy (kubeProxyReplacement: false) is a common starting point. Running the full replacement with kube-proxy still installed leaves both programming the datapath — it mostly works because the eBPF hooks run first, but it is not a supported steady state. Migrate cleanly: enable the replacement, delete kube-proxy, flush the rules.

What kernel do I need?

Current Cilium releases document 5.10 (or a vendor-backported equivalent like RHEL 8's 4.18) as the baseline — older releases went as low as v4.19.57 for the base replacement — and some features (IPv6 BIG TCP, netkit, newer DSR dispatch modes) want considerably newer kernels. Modern Ubuntu LTS, Amazon Linux 2023, Bottlerocket, and COS are all fine. Check cilium-dbg status --verbose in the agent pod for the per-feature verdict on your actual nodes.

Does this apply on EKS, GKE, and AKS?

GKE Dataplane V2 is Cilium with kube-proxy replacement, managed by Google. AKS offers Azure CNI powered by Cilium with similar semantics. On EKS you do it yourself by replacing the VPC CNI with Cilium. In all three cases you get the datapath benefits, but only self-managed Cilium gives you the full OSS feature surface and upgrade control.

Will migrating break existing connections?

Expect a brief disruption window during the switchover as established flows tracked by conntrack/iptables hand over to the eBPF path — plan a maintenance window or roll node pools rather than flipping a live fleet. New connections pick up the eBPF path immediately.

The bottom line

kube-proxy is not broken — it is a 2015 answer running on 2015 kernel primitives, and it still serves the majority of clusters perfectly well. Cilium's replacement is a categorically different design: translate Services at the socket for east-west, at XDP/tc for north-south, and keep state in O(1) maps instead of rule chains. If your cluster is big enough or churny enough to feel iptables' ceilings — or you want DSR, Maglev, and Hubble regardless — the migration is four commands and a validation run. If it is not, the most senior move is the boring one: leave kube-proxy alone and revisit when the Service count says otherwise.

See also

• Cilium and eBPF Kubernetes networking — the full Cilium picture beyond the Service datapath
• Kubernetes network policies for zero trust — the policy layer the same datapath enforces
• eBPF observability with Tetragon, Hubble and Pixie — what Hubble gives you once Cilium owns the datapath
• Cilium vs Calico on AKS — the CNI-level comparison on Azure
• Managed Kubernetes compared: EKS vs GKE vs AKS — where each provider stands on Cilium support

Official References

• Service — the Service types and how traffic reaches pods
• Virtual IPs and Service Proxies — how kube-proxy's iptables, IPVS and nftables modes differ
• Cilium introduction — the eBPF dataplane and what replaces kube-proxy
• Cilium network policy — L3/L4/L7 policy and identity-based enforcement