Scale-to-Zero LLM Inference on Kubernetes

Scale-to-Zero LLM Inference on Kubernetes

An idle GPU pod costs exactly what a saturated one costs. A g6.12xlarge bills the same whether it is serving 400 requests per second or sitting at 0% SM utilization while your team sleeps.

That symmetry is the whole argument. For a production chatbot with steady traffic it barely matters. For everything else it is most of your bill. A dev endpoint serving 10am to 6pm on weekdays is idle roughly 75% of the hours in a week. Internal tools, sales demo endpoints, evaluation harnesses, per-team sandboxes — all holding a GPU 24/7 to serve a few hundred requests.

Measure it first. Tracking LLM inference cost with OpenCost gives you the dollars-per-namespace number that says whether this post is worth your afternoon. The usual finding: a handful of non-production endpoints quietly outspending the production one.

Why the cold start is minutes, not seconds

Scale-to-zero for a stateless HTTP API is boring — the pod comes back in five seconds. For LLM inference it is a chain of four serial stages, and every one of them is slow.

1. Node provisioning (0 to ~3 minutes). If there is no warm GPU node, Karpenter has to launch one. GPU instance families have thinner capacity pools than general-purpose ones, so a request can bounce across AZs or instance types before it lands. Then the NVIDIA driver initializes and the device plugin registers nvidia.com/gpu with the kubelet before your pod is even schedulable. Budget 90 seconds to 3 minutes.

2. Container image pull (45 seconds to ~2 minutes). CUDA inference images are enormous. AWS benchmarked the Deep Learning Container vLLM image at roughly 10 GB with a baseline pull of 1m52.876s; SOCI parallel pull mode dropped it to 45.121s. That is the good case, on a fat instance with high network bandwidth.

3. Model weight download (1 to 5+ minutes). An 8B model at fp16 is about 16 GB; a 70B is about 140 GB. If your pod pulls that from S3 or the Hugging Face Hub on every start, you pay for the transfer in wall-clock time on every cold start.

4. Weight load into VRAM and CUDA graph capture (30 to 90 seconds). vLLM moves weights onto the device, allocates the KV cache, and captures CUDA graphs. Largely irreducible.

Serially, an unoptimized cold start on a fresh node lands between five and eight minutes. That is the number that kills most scale-to-zero attempts before they start.

Attack the chain stage by stage

Fix the two that dominate your trace, not all four.

Weights are the biggest lever. Stop fetching from object storage per start. Put them on a ReadOnlyMany volume — an EFS or FSx for Lustre PVC shared across replicas, or better, a local NVMe cache on the instance store that survives pod restarts on the same node. A read from local NVMe is tens of seconds instead of minutes. If your model set is small and stable, baking weights into the image collapses stages 2 and 3 into one, at the cost of a 30 GB image and a rebuild on every model bump.

Image pull is a config change, not an architecture change. On EKS, SOCI parallel pull mode is built into Amazon Linux 2023 and Bottlerocket. For Bottlerocket, it is userdata:

[settings.container-runtime]
snapshotter = "soci"

[settings.container-runtime-plugins.soci-snapshotter]
pull-mode = "parallel-pull-unpack"

You get most of this for free from managed node lifecycles — EKS Auto Mode for GPU workloads uses SOCI parallel pull on GPU instances by default.

Node provisioning you cheat rather than optimize. See the warm-node section below.

Weight load into VRAM shrinks if the weights shrink. A 4-bit AWQ quantization of an 8B model is roughly 5.5 GB instead of 16 GB, cutting both the download and the VRAM load. Read the quantization and tensor parallelism trade-offs before picking a format — you are trading accuracy for start time, not getting it free.

The ScaledObject that actually scales to zero

Plain HPA cannot scale below one replica. KEDA can, because it splits the problem in two: KEDA's operator owns the 0↔1 transition, and hands 1↔N to a standard HPA it creates and manages.

1apiVersion: keda.sh/v1alpha1
2kind: ScaledObject
3metadata:
4  name: vllm-internal
5  namespace: llm-serving
6spec:
7  scaleTargetRef:
8    name: vllm-llama3-8b        # Deployment name
9
10  minReplicaCount: 0             # the whole point
11  maxReplicaCount: 4
12
13  pollingInterval: 15            # seconds; default is 30
14  cooldownPeriod: 900            # wait 15 min of quiet before going to 0
15  initialCooldownPeriod: 1800    # do not deactivate for 30 min after create
16
17  fallback:
18    failureThreshold: 3
19    replicas: 1                  # if Prometheus is down, hold one warm pod
20
21  advanced:
22    horizontalPodAutoscalerConfig:
23      behavior:
24        scaleDown:
25          stabilizationWindowSeconds: 600
26
27  triggers:
28    - type: prometheus
29      name: vllm-pending-and-running   # trigger-level; `metricName` was removed
30      metadata:
31        serverAddress: http://prometheus-operated.monitoring.svc:9090
32        query: |
33          sum(vllm:num_requests_waiting{namespace="llm-serving", model_name="meta-llama/Meta-Llama-3-8B-Instruct"})
34          +
35          sum(vllm:num_requests_running{namespace="llm-serving", model_name="meta-llama/Meta-Llama-3-8B-Instruct"})
36        threshold: "8"           # target per-replica load, drives 1 -> N
37        activationThreshold: "0" # any in-flight work at all drives 0 -> 1

Three fields carry the weight.

cooldownPeriod is "the period to wait after the last trigger reported active before scaling the resource back to 0", and defaults to 300 seconds. For GPU pods that is too aggressive — a five-minute lull during a working session drops the pod and charges the next user a full cold start. Fifteen minutes is a more honest starting point; tune it against your actual inter-request gap distribution.

fallback matters more than it looks. Without it, an erroring Prometheus scaler stops propagating a metric to the HPA and the workload stays wherever it is. replicas: 1 means a monitoring outage degrades a running endpoint to "warm and paying for it" rather than letting it drift. Note the limit: fallback works by feeding a synthesised AverageValue metric to the HPA, and the HPA does not own the 0↔1 transition — so it holds a pod up, it does not resurrect one that is already at zero.

The query sums waiting plus running, not waiting alone. Count only vllm:num_requests_waiting and a single in-flight request with an empty queue reads as zero — KEDA deactivates the pod out from under an active generation once the cooldown expires.

Activation is a separate decision from scaling

This is the part that produces the "KEDA won't scale my pod up from zero" bug report.

KEDA's docs draw the line clearly. The activating phase is "the moment when KEDA (operator) has to decide if the workload should be scaled from/to zero." The scaling phase is "the moment when KEDA has decided to scale out to 1 instance and now it is the HPA controller who takes the scaling decisions."

Different numbers govern each. threshold is the HPA target — the per-replica load KEDA aims for once a pod exists. activationThreshold is the 0→1 gate, and it is a strict greater-than: "activation only occurs when this value is greater than the set value; not greater than or equal to. ie, in the default case: activationThreshold: 0 will only activate when the metric value is 1 or more."

The trap is setting activationThreshold equal to threshold. Write threshold: "8" and activationThreshold: "8" and your endpoint stays at zero until nine requests pile up somewhere — and with zero pods running, there is nothing to pile them up in. Keep the activation gate at or near zero and let the threshold do the sizing. For the general pattern, see the KEDA event-driven autoscaling guide and the debugging walkthrough for HPAs that will not scale.

When there is no queue to measure

A Prometheus trigger has a chicken-and-egg problem: at zero replicas there is no vLLM process, so vllm:num_requests_waiting does not exist. You get away with it when something upstream — an SQS queue, a Kafka topic, a batch dispatcher — holds the work. For a plain HTTP endpoint, nothing does. The first request arrives, hits no backend, and 503s.

The KEDA HTTP add-on solves this with an interceptor in the request path that holds the connection open, signals KEDA out-of-band, and forwards once a pod is ready. As of v0.14 the API is InterceptorRoute (http.keda.sh/v1beta1); the older HTTPScaledObject (http.keda.sh/v1alpha1) is deprecated. The new shape splits routing from scaling — you write the route, and you own the ScaledObject:

1apiVersion: http.keda.sh/v1beta1
2kind: InterceptorRoute
3metadata:
4  name: vllm-internal
5  namespace: llm-serving
6spec:
7  target:
8    service: vllm-llama3-8b
9    port: 8000
10  rules:
11    - hosts:
12        - llm-dev.internal.example.com
13  scalingMetric:
14    concurrency:
15      targetValue: 8
16  timeouts:
17    readiness: 10m     # must exceed your worst cold start
18    request: 12m
19  coldStart:
20    placeholder:
21      response:
22        # served instead of hanging, if you would rather fail fast
23        statusCode: 503
24---
25apiVersion: keda.sh/v1alpha1
26kind: ScaledObject
27metadata:
28  name: vllm-internal
29  namespace: llm-serving
30spec:
31  scaleTargetRef:
32    name: vllm-llama3-8b
33  minReplicaCount: 0
34  maxReplicaCount: 4
35  cooldownPeriod: 900
36  triggers:
37    - type: external-push
38      metadata:
39        scalerAddress: keda-add-ons-http-external-scaler.keda:9090
40        interceptorRoute: vllm-internal

Set timeouts.readiness above your measured p99 cold start, or the interceptor gives up mid-warm-up and you pay for the start without getting the response. And be honest about what a held connection means downstream: a client with a 30-second HTTP timeout disconnects long before your model is up. Either your callers tolerate multi-minute waits, or you use coldStart.placeholder to return a fast 503 and make them retry.

The middle ground: pod to zero, node stays warm

Node provisioning is the slowest and least controllable stage, and it is the one you can simply skip.

Scale the pod to zero, keep one GPU node alive. You still pay for the instance, but you drop stages 1 and 2 entirely — the image is already in the node's content store — and a cold start becomes weight load plus VRAM load. That is 60 to 120 seconds instead of eight minutes.

The mechanism is a balloon pod at negative priority holding the GPU, which real workloads preempt:

1apiVersion: scheduling.k8s.io/v1
2kind: PriorityClass
3metadata:
4  name: gpu-balloon
5value: -10
6globalDefault: false
7description: "Holds a GPU node warm; preempted by any real workload."
8---
9apiVersion: apps/v1
10kind: Deployment
11metadata:
12  name: gpu-balloon
13  namespace: llm-serving
14spec:
15  replicas: 1
16  selector:
17    matchLabels: { app: gpu-balloon }
18  template:
19    metadata:
20      labels: { app: gpu-balloon }
21    spec:
22      priorityClassName: gpu-balloon
23      terminationGracePeriodSeconds: 0
24      tolerations:
25        - key: nvidia.com/gpu
26          operator: Exists
27          effect: NoSchedule
28      containers:
29        - name: pause
30          image: registry.k8s.io/pause:3.10
31          resources:
32            requests:
33              nvidia.com/gpu: "1"
34            limits:
35              nvidia.com/gpu: "1"

Pair it with a Karpenter NodePool that will not consolidate the node away the moment the real pod leaves:

1apiVersion: karpenter.sh/v1
2kind: NodePool
3metadata:
4  name: gpu-inference
5spec:
6  template:
7    spec:
8      expireAfter: 720h        # v1 moved this out of spec.disruption
9      nodeClassRef:
10        group: karpenter.k8s.aws
11        kind: EC2NodeClass
12        name: gpu
13  disruption:
14    consolidationPolicy: WhenEmpty
15    consolidateAfter: 30m

WhenEmpty with a 30-minute consolidateAfter means Karpenter only reclaims a node with no workload pods on it, and only after half an hour. The balloon pod keeps it non-empty indefinitely. Watch the expireAfter placement: in Karpenter v1 it lives under spec.template.spec, not spec.disruption, and 720h is already the default — it is in the example so you notice the warm node still gets recycled every 30 days. Check the Karpenter v1 disruption model before tuning these — WhenEmptyOrUnderutilized will happily bin-pack your warm node away.

The economics: you save nothing on the node and everything on the additional nodes. One shared endpoint is a wash. Twelve per-team endpoints collapsing onto three warm nodes is a 4x reduction.

When not to do this

Say this part plainly: scale-to-zero trades p99 latency for cost. Not p50 — p50 is unaffected, because the pod is usually warm when traffic is steady. It is the tail, and the tail is where a real user is sitting.

Do not scale to zero when:

• The endpoint is user-facing. A 90-second first-token latency on a customer-visible product is not a latency problem, it is an outage your dashboards will report as healthy.
• Traffic is steady. If the pod is up 80% of the time anyway, you added cold-start risk to save 20%.
• You are under an SLO with an error budget. Cold starts consume it. Model the burn before you enable this, not after.
• The model is large. 140 GB of fp16 weights takes minutes to load even from local NVMe. The economics get worse as models get bigger, which is backwards from where you want them.

The correct home for this is bursty and internal: dev and staging endpoints, per-team sandboxes, evaluation harnesses, demo endpoints, and anything batch where the work sits in a queue and nobody is watching a spinner.

Measure the thing you just built

Two numbers, and you need both.

Instrument cold start end to end: from the KEDA activation event to the first successful /health response, not just container start. Emit it as a histogram and alert on p99 drift — a base image bump or a model swap will silently add two minutes.

Then re-run the cost attribution. The saving should show up at the namespace level within a billing cycle. If it does not, you have a cooldownPeriod that is too long, a fallback stuck on, or a balloon pod on a node nobody shares.

For the deployment side — GPU node pools, the NVIDIA device plugin, vLLM configuration, and the storage layout the weight cache depends on — start from deploying an LLM on Kubernetes and layer scale-to-zero on once the warm path is stable. Adding it to a workload that was not reliable to begin with just moves the blame around.