Custom Pod Networking on EKS with VPC CNI

By default, every pod on an EKS node gets its IP address from the same subnet the node itself lives in — the VPC CNI attaches secondary ENIs to the node and pulls pod IPs from that node's primary subnet CIDR. That works fine until the subnet is small and the cluster isn't: once the subnet's free addresses run out, new pods stay Pending with an IP allocation error, and the fix isn't more nodes, it's more address space that the current architecture doesn't have anywhere to put.

Custom networking decouples the two: pods draw IPs from secondary CIDR blocks and subnets you carve out specifically for them, while nodes keep their original primary subnet. This tutorial is the hands-on configuration path — for the conceptual background on how VPC CNI allocates addresses in the first place, see EKS Networking and VPC CNI Guide.

What You'll Build

• A secondary VPC CIDR block with dedicated pod subnets, one per availability zone
• ENIConfig custom resources mapping each AZ to its pod subnet and security group
• aws-node DaemonSet configuration that enables custom networking and auto-labels nodes by AZ
• Prefix delegation layered on top to multiply usable IPs per ENI
• A quick look at security groups for pods, the related feature you'll likely need next

Step 1: Confirm You Actually Have an IP Problem

Do the arithmetic before touching anything. VPC CNI's max-pods-per-node ceiling is driven by ENI slots, not CPU or memory. For an m5.large:

• Maximum ENIs per instance: 3
• IPv4 addresses per ENI: 10
• Max pods = ENIs × (IPs per ENI − 1) + 2 = 3 × 9 + 2 = 29

Every one of those IPs comes from the node's primary subnet. A /24 subnet has 251 usable addresses after AWS reserves five — shared across every node's primary IP, every pod IP, and anything else placed in that subnet (internal load balancers, control plane ENIs). That's room for roughly eight m5.large nodes' worth of pods before the subnet is exhausted, regardless of how much room is left on the instances themselves or in your account's EC2 limits. Custom networking doesn't change the ENI math — it changes which subnet those IPs are drawn from, so you can size that subnet independently of whatever CIDR the node subnets were planned with (see AWS VPC Design for EKS if that original planning is what's cornering you now).

Step 2: Enable Custom Networking on the CNI

Set the env var that tells aws-node to stop assigning pod IPs from the primary ENI and start looking for ENIConfig resources instead:

kubectl set env daemonset aws-node -n kube-system AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true

Also set ENI_CONFIG_LABEL_DEF now, before any ENIConfig objects exist — it controls which node label the CNI reads to decide which ENIConfig applies to a given node:

kubectl set env daemonset aws-node -n kube-system ENI_CONFIG_LABEL_DEF=topology.kubernetes.io/zone

topology.kubernetes.io/zone is populated automatically by the AWS cloud provider on every node at join time, so this single setting is what lets Step 5 skip manual per-node labeling entirely.

If aws-node is an EKS managed addon — which it is on most clusters — set these through the addon instead. The managed addon reconciles the DaemonSet back to its own spec, so kubectl set env works until the next addon update or reconcile and then silently reverts, taking custom networking down with it. Put the values in the addon's configurationValues:

aws eks update-addon --cluster-name my-cluster --addon-name vpc-cni \
  --configuration-values '{"env":{"AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG":"true","ENI_CONFIG_LABEL_DEF":"topology.kubernetes.io/zone"}}' \
  --resolve-conflicts PRESERVE

The equivalent Helm values, if you manage the CNI yourself rather than as an addon:

env:
  AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG: "true"
  ENI_CONFIG_LABEL_DEF: "topology.kubernetes.io/zone"

With custom networking on, the primary ENI (eth0) on every node stops handing out pod IPs — it's used for node-to-node and control plane traffic only. That's a small reduction in max-pods-per-node (one ENI's worth of slots is no longer available for pods), traded for pod subnets you can size and grow independently.

Step 3: Carve Out Secondary CIDR Blocks and Pod Subnets

Associate a secondary CIDR with the VPC — the 100.64.0.0/16 CGNAT range is the common choice here specifically because it won't collide with typical 10.0.0.0/8 or 172.16.0.0/12 VPC allocations:

aws ec2 associate-vpc-cidr-block \
  --vpc-id vpc-0123456789abcdef0 \
  --cidr-block 100.64.0.0/16

Create one subnet per AZ inside that new CIDR, sized generously — this is the address space you were short on, so don't recreate the same problem at a bigger scale:

1aws ec2 create-subnet \
2  --vpc-id vpc-0123456789abcdef0 \
3  --cidr-block 100.64.0.0/19 \
4  --availability-zone us-east-1a \
5  --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=pod-subnet-us-east-1a}]'
6
7aws ec2 create-subnet \
8  --vpc-id vpc-0123456789abcdef0 \
9  --cidr-block 100.64.32.0/19 \
10  --availability-zone us-east-1b \
11  --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=pod-subnet-us-east-1b}]'
12
13aws ec2 create-subnet \
14  --vpc-id vpc-0123456789abcdef0 \
15  --cidr-block 100.64.64.0/19 \
16  --availability-zone us-east-1c \
17  --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=pod-subnet-us-east-1c}]'

Associate each new subnet with a route table that already has the routing you need (NAT gateway for egress, VPC peering, etc.) — reuse the existing private route table rather than building routing from scratch:

aws ec2 associate-route-table \
  --subnet-id subnet-0aaaa1111bbbb2222 \
  --route-table-id rtb-0123456789abcdef0

Repeat the association for the other two pod subnets against the route table appropriate to their AZ.

Step 4: Create an ENIConfig per Availability Zone

ENIConfig is the CRD the VPC CNI reads to find out which subnet and security group secondary ENIs should use in a given AZ. Naming the object after the AZ itself is what makes the ENI_CONFIG_LABEL_DEF=topology.kubernetes.io/zone setting from Step 2 work with no further wiring:

1# eniconfig-us-east-1a.yaml
2apiVersion: crd.k8s.amazonaws.com/v1alpha1
3kind: ENIConfig
4metadata:
5  name: us-east-1a
6spec:
7  subnet: subnet-0aaaa1111bbbb2222
8  securityGroups:
9    - sg-0123456789abcdef0

1# eniconfig-us-east-1b.yaml
2apiVersion: crd.k8s.amazonaws.com/v1alpha1
3kind: ENIConfig
4metadata:
5  name: us-east-1b
6spec:
7  subnet: subnet-0bbbb2222cccc3333
8  securityGroups:
9    - sg-0123456789abcdef0

1# eniconfig-us-east-1c.yaml
2apiVersion: crd.k8s.amazonaws.com/v1alpha1
3kind: ENIConfig
4metadata:
5  name: us-east-1c
6spec:
7  subnet: subnet-0cccc3333dddd4444
8  securityGroups:
9    - sg-0123456789abcdef0

kubectl apply -f eniconfig-us-east-1a.yaml
kubectl apply -f eniconfig-us-east-1b.yaml
kubectl apply -f eniconfig-us-east-1c.yaml

The security group here is whatever you'd normally attach to pods in that AZ — it can be the node security group, or something narrower if you're already segmenting pod traffic.

Step 5: Confirm Nodes Are Mapped to the Right ENIConfig

Because ENI_CONFIG_LABEL_DEF points at topology.kubernetes.io/zone, and each ENIConfig is named exactly after an AZ, every node picks up the correct config automatically — no per-node kubectl label step required. Confirm the label exists and matches an ENIConfig name:

kubectl get nodes -L topology.kubernetes.io/zone

If you're on an older CNI setup that doesn't use ENI_CONFIG_LABEL_DEF (or you want to override the mapping for a specific node pool — say, a dedicated node group that should draw from a different subnet than its AZ default), fall back to labeling manually with the CNI's own label key:

kubectl label node ip-10-0-1-23.ec2.internal k8s.amazonaws.com/eniConfig=us-east-1a --overwrite

Step 6: Restart aws-node, Then Replace the Nodes

The DaemonSet needs to reconcile after the env var and ENIConfig changes:

kubectl rollout restart daemonset aws-node -n kube-system
kubectl rollout status daemonset aws-node -n kube-system

Restarting aws-node is not enough, and this is the step people skip. Custom networking only governs ENIs attached after it is enabled. Nodes that joined the cluster beforehand keep their existing ENIs and keep allocating pod IPs from the primary subnet indefinitely — new pods on those nodes included. Nothing errors; the feature simply appears not to work. You have to recycle the nodes:

# Managed node groups: force a rolling replacement.
aws eks update-nodegroup-version --cluster-name my-cluster --nodegroup-name my-nodegroup --force

# Or, node by node:
kubectl drain ip-10-0-1-23.ec2.internal --ignore-daemonsets --delete-emptydir-data

The replacement nodes also need a lower --max-pods. With custom networking the primary ENI no longer serves pod IPs, so the formula from Step 1 loses an ENI's worth of slots: ((ENIs − 1) × (IPs per ENI − 1)) + 2. For the m5.large in Step 1 that is ((3 − 1) × 9) + 2 = 20, down from 29. Leave the old value in place and the kubelet will happily schedule 29 pods onto a node that can only address 20 of them — the extras sit in ContainerCreating with IP allocation failures in the CNI logs. Set it explicitly on the new nodes — via the node group launch template's bootstrap arguments, or maxPods in the nodeadm kubelet config:

# Launch template user data, AL2023 / nodeadm
--kubelet-extra-args '--max-pods=20'

Existing pods keep the IPs they already have — none of this rebalances running workloads, it only changes new ENI allocations on new nodes.

Step 7: Add Prefix Delegation for More IPs per ENI

Custom networking fixes where pod IPs come from; it doesn't change how many an ENI can hand out. Prefix delegation is the separate, complementary lever for that: instead of an ENI holding one IP per address slot, each slot holds a /28 prefix (16 addresses), so the same ENI slot count yields far more usable pod IPs.

kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true
kubectl set env daemonset aws-node -n kube-system WARM_PREFIX_TARGET=1

WARM_PREFIX_TARGET=1 keeps one spare /28 warm per node so pod scheduling doesn't stall on an EC2 API call for a fresh prefix. This stacks directly on top of what you just configured — prefix delegation still respects whichever subnet the ENIConfig points at, so pods keep landing in the dedicated pod subnets, there are just more IPs available per ENI within them.

kubectl rollout restart daemonset aws-node -n kube-system

The same two caveats from Step 6 apply again, for the same reasons: on a managed addon these belong in configurationValues rather than kubectl set env, and prefix delegation only affects ENIs attached after it's enabled, so the nodes need recycling before it changes anything. --max-pods moves in the opposite direction this time — each slot now carries 16 addresses instead of one — so re-run the calculator with both flags rather than assuming the Step 6 number still holds.

Step 8: Security Groups for Pods (Related, Not Required)

If your next requirement is applying security groups at the individual pod level rather than the node level — separate database access rules per workload sharing a node, for instance — that's security groups for pods, a related but distinct VPC CNI feature built on the same branch-ENI machinery:

kubectl set env daemonset aws-node -n kube-system ENABLE_POD_ENI=true

It also requires the AmazonEKSVPCResourceController managed policy attached to the EKS cluster IAM role (not the node role), and pods get their security groups via a SecurityGroupPolicy CRD matching on pod labels rather than through the ENIConfig you just built. That's a big enough topic to warrant its own walkthrough — this tutorial stops at enabling the flag so you know it exists and where it sits relative to what you just configured.

Verify Pod IPs Moved to the Dedicated Subnet

Deploy a test pod and check where its IP actually came from:

Pin it to a node created after Step 6, otherwise this test proves nothing — a pod that lands on a surviving old node will show a primary-subnet IP no matter how correct your configuration is, and you'll spend the next hour debugging a working setup. Check node ages first:

kubectl get nodes --sort-by=.metadata.creationTimestamp

kubectl run netcheck --image=nginx:alpine --restart=Never \
  --overrides='{"spec":{"nodeName":"ip-10-0-1-45.ec2.internal"}}'
kubectl get pod netcheck -o wide

The IP column should show an address in 100.64.0.0/16 — the secondary CIDR from Step 3 — not the original VPC primary CIDR (e.g. 10.0.x.x) that the node itself sits in. kubectl describe pod netcheck also shows this in the IP: field, alongside the node it landed on. If the pod's IP is still coming from the primary subnet, work through it in this order: the node predates the config change (Step 6 — by far the most likely cause), the node never picked up an ENIConfig (Step 5), or aws-node isn't actually running with the new env vars (kubectl get daemonset aws-node -n kube-system -o jsonpath='{.spec.template.spec.containers[0].env}' — and if they're missing, the managed addon reverted them).

kubectl delete pod netcheck

Where to Go Next

• EKS Networking and VPC CNI Guide — the conceptual foundation this tutorial builds on, covering how VPC CNI allocation works before you start overriding it.
• AWS VPC Design for EKS — the subnet and CIDR planning this tutorial's secondary CIDRs depend on; worth reading before you pick sizes for a production rollout.
• EKS Cluster Autoscaler — IP exhaustion interacts directly with how many nodes the autoscaler can actually add before it's blocked on address space, not compute quota.
• Install AWS Load Balancer Controller on EKS — a related piece of the EKS networking stack, since ALB target group registration also depends on which subnets your pods are reachable in.