EKS Auto Mode for GPU Workloads

EKS Auto Mode for GPU Workloads

The most tedious part of running GPUs on Kubernetes has nothing to do with GPUs. It's keeping the driver stack alive: matching NVIDIA driver versions to CUDA versions to kernel versions, upgrading the GPU Operator, and discovering that a node came up without the device plugin so your pods sit Pending against a node with eight idle A100s.

EKS Auto Mode deletes that job. AWS builds the AMI, and the NVIDIA drivers and the Kubernetes device plugin are already in it — the same is true for AWS Neuron on Trainium and Inferentia. There is nothing to install and nothing to keep in sync.

That's a real saving. It comes with a node lifecycle you don't control, and whether that's fine or disqualifying depends almost entirely on whether you're doing inference or training.

The built-in NodePools will not give you a GPU

This is where everyone starts and stalls. An Auto Mode cluster comes with two NodePools, general-purpose and system, and you cannot modify either — they can only be enabled or disabled. Neither will provision an accelerated instance.

So a GPU pod scheduled onto a fresh Auto Mode cluster stays Pending forever, with no obvious error explaining why. You need a custom NodePool.

Auto Mode uses the Karpenter NodePool API, with its own label namespace (eks.amazonaws.com/* rather than karpenter.k8s.aws/* — this trips up anyone porting a config from self-managed Karpenter):

1apiVersion: karpenter.sh/v1
2kind: NodePool
3metadata:
4  name: gpu-inference
5spec:
6  template:
7    metadata:
8      labels:
9        workload: gpu
10    spec:
11      nodeClassRef:
12        group: eks.amazonaws.com
13        kind: NodeClass
14        name: default
15
16      requirements:
17        - key: "eks.amazonaws.com/instance-family"
18          operator: In
19          values: ["g6e", "g6", "g5"]
20        - key: "eks.amazonaws.com/instance-gpu-manufacturer"
21          operator: In
22          values: ["nvidia"]
23        - key: "karpenter.sh/capacity-type"
24          operator: In
25          values: ["on-demand"]
26
27      # Keep general workloads off expensive hardware.
28      taints:
29        - key: nvidia.com/gpu
30          effect: NoSchedule
31
32  disruption:
33    consolidationPolicy: WhenEmpty
34    consolidateAfter: 5m
35
36  limits:
37    cpu: "512"

Auto Mode exposes a useful set of GPU-aware labels for these requirements:

instance-gpu-memory is the most useful and least used. Rather than maintaining a list of instance types that fit your model, express the actual constraint: at least 40 GiB of VRAM. That keeps working when AWS releases a new family, which a hardcoded type list does not. Sizing that number is a VRAM and quantization question before it's an infrastructure one.

Auto Mode supports a broad set of accelerated families — p6-b200, p6-b300, p5, p5e, p5en, p4d, p4de, p3, p3dn, g7e, g6, g6e, gr6, g5, g5g, g4dn, g4ad, plus trn1/trn1n/trn2 and inf1/inf2 for Neuron. Instances must have more than 1 CPU and not be nano/micro/small, which never binds for GPU types.

Requesting the GPU in your pod is unchanged from any other cluster:

1resources:
2  limits:
3    nvidia.com/gpu: 1
4tolerations:
5  - key: nvidia.com/gpu
6    operator: Exists
7    effect: NoSchedule

The node lifecycle is the real trade

Everything above is a config detail. This section is the actual decision.

Auto Mode manages node lifecycle on your behalf, with defaults that are sensible for stateless services and hostile to long-running training jobs:

• Nodes are terminated after 336 hours — 14 days. Not configurable away by ignoring it; it's the default expireAfter.
• Nodes are replaced when a new AMI ships, which AWS does roughly weekly with CVE and security fixes. Your GPU node will be drained and replaced on someone else's schedule.
• Default disruption budget is 10% of nodes, so replacements roll rather than happening all at once.
• Default termination grace period is 24 hours on the NodeClaim when you haven't set one.

For inference this is close to ideal. Pods are replaceable, a rolling node replacement is a rolling deploy, and you get security patching for free — patching that, on self-managed GPU nodes, is a chore precisely because of the driver stack.

For training it's a live hazard. A fine-tuning run that takes eleven days has a meaningful chance of meeting a weekly AMI rollout. The 24-hour grace period saves a job that checkpoints; it does nothing for one that doesn't. If you train on Auto Mode, checkpoint to durable storage and make your job resumable — not as a best practice, as a functional requirement.

You can push back on the schedule per NodePool:

1spec:
2  template:
3    spec:
4      expireAfter: Never        # opt out of the 336h expiry
5  disruption:
6    consolidationPolicy: WhenEmpty
7    budgets:
8      - nodes: "0"
9        schedule: "0 9 * * mon-fri"   # no disruption during business hours
10        duration: 8h

expireAfter: Never removes the NodePool's age-based termination. It does not remove drift-based replacement when a new AMI ships — that's the part you can delay with budgets but not disable. And it does not buy a node unlimited life: AWS enforces a 21-day maximum node lifetime, and a node that PDBs or disruption budgets have held back until then gets disrupted regardless. If your workload genuinely cannot tolerate a node being replaced on AWS's cadence, that's the signal to run self-managed Karpenter instead, where you own the AMI and the timing. The Karpenter v1 deep dive covers what you take back on.

Set consolidationPolicy deliberately for GPU. The general-purpose default is fine for web services, but WhenEmptyOrUnderutilized will repack a partially-loaded inference node to save money — evicting a pod that holds a GPU and has a multi-minute model load on startup. WhenEmpty is almost always the right choice for GPU: reclaim nodes when nothing is on them, never shuffle live work. Balanced is a reasonable middle if you're cost-sensitive and your pods restart quickly.

What you give up beyond scheduling

Auto Mode instances are managed instances — EKS owns them, and several things you may rely on are simply not available.

No SSH, no SSM, and you cannot install software on the node. No nvidia-smi on the host, no manual driver swap, no DaemonSet that expects to write to the host filesystem. If your GPU debugging runbook starts with "SSH to the node," it needs rewriting around kubectl debug and pod-level tooling before you migrate.

You cannot pin a driver version. AWS chooses the AMI, so the NVIDIA driver underneath your containers — and with it the maximum CUDA version they can use — moves when the AMI moves. Your CUDA toolkit still ships in your own image; what you lose is control over the ceiling. If you have a framework build pinned to an exact driver, verify against a new AMI before it reaches production, and expect to track AWS's cadence rather than your own.

IMDSv2 is enforced with a hop limit of 1, and that is not configurable. A pod needing instance metadata must run with hostNetwork: true. This bites GPU workloads more than most because ML tooling reaches for IMDS to discover region and credentials — prefer EKS Pod Identity, which sidesteps it entirely.

Since April 22, 2026, new managed instances are hidden by default from EC2 console views and DescribeInstances list operations — accounts that already held managed resources before the change were left set to visible, so this bites new clusters and new accounts first. Your GPU fleet still runs and still bills; it just doesn't show up where your inventory, tagging, and CSPM tooling look for it. They remain visible through the EKS console, kubectl get nodes, a describe-instances call by explicit instance ID, or the include-managed-resources parameter. Check what your cost and compliance tooling actually queries before you assume your GPU inventory is complete.

One thing you gain quietly: if a GPU instance has local NVMe and your NodeClass requests less ephemeral storage than the NVMe provides, Auto Mode formats and configures it for you — including a RAID 0 array across multiple drives. For inference that's a fast local cache for model weights with no setup, and it meaningfully cuts cold-start time on large models.

Does it cost more?

Auto Mode charges a management fee on top of the EC2 instance price. On general-purpose nodes that's a visible percentage. On GPU nodes, where the instance itself dominates, the same fee is a much smaller proportion of the bill — the accelerator is the line item, not the management.

That reframes the question. The real comparison isn't fee versus no fee; it's the fee against the engineering time you currently spend on GPU Operator upgrades, driver-version incidents, and AMI patching for a fleet whose nodes are individually expensive. For most teams running fewer than a few dozen GPU nodes, Auto Mode wins that trade comfortably.

What actually determines your GPU bill is utilization, and Auto Mode doesn't help there — a node it provisions perfectly still bills at full rate while idle. Measuring cost per million tokens rather than cost per node is where the money is.

Choosing

Use Auto Mode for GPU when you're serving inference, your pods are replaceable, you'd rather not own a driver stack, and you don't need a pinned CUDA version. This is most inference deployments, and the operational saving is genuine.

Don't use it when you run multi-week training jobs that can't checkpoint, you need a specific driver or a custom AMI, you depend on host-level access for profiling, or you have compliance tooling that must see every instance through the EC2 API.

A reasonable middle: Auto Mode for the inference fleet, self-managed node groups or Karpenter for training. They coexist in one cluster — AWS labels every Auto Mode node eks.amazonaws.com/compute-type: auto, so a nodeSelector or a NotIn node affinity pins each workload deterministically to one side or the other. You get managed patching where nodes are cattle and full control where a node holds eleven days of work.

For the platform-level comparison, EKS Auto Mode vs GKE Autopilot covers how the two managed models differ. If you're staying on self-managed GPU nodes, the NVIDIA GPU Operator is the stack Auto Mode is replacing, and running GPU and ML workloads on Kubernetes covers the scheduling side that doesn't change either way.