Kubernetes Cluster Upgrades: Zero-Downtime Strategy for Production

Kubernetes releases a new minor version every four months. Standard support covers three minor versions — the current release plus two prior. Fall behind more than three versions and you're running unsupported Kubernetes with unpatched security vulnerabilities, no upstream bug fixes, and an increasingly wide delta to close when you finally do upgrade.

The good news: Kubernetes upgrades on managed platforms (EKS, GKE, AKS) are well-understood operations with clear failure modes. The bad news: the failure modes are predictable enough that organisations keep hitting them anyway — deprecated APIs, add-on version drift, nodes that don't drain cleanly.

This post covers the upgrade process end to end: pre-upgrade preparation, control plane upgrade, node upgrade strategies, add-on management, and the rollback plan you need before starting.

The Upgrade Model

Before diving into process, understand what gets upgraded and in what order:

Control Plane (API server, etcd, scheduler, controller manager)
    ↓
Node Groups (kubelet, kube-proxy)
    ↓
Add-ons (CoreDNS, kube-proxy, VPC CNI, CSI drivers)
    ↓
In-cluster tools (Argo CD, Prometheus, cert-manager, Kyverno)

Rule: Never upgrade nodes to a version higher than the control plane. The Kubernetes version skew policy allows kubelet (nodes) to be up to three minor versions behind the kube-apiserver (e.g., a 1.32 control plane supports nodes running 1.32, 1.31, 1.30, or 1.29). This three-version skew applies to Kubernetes 1.28+; on 1.27 and earlier the allowed skew was two minor versions. Upgrade the control plane first, then nodes.

Rule: Add-ons must be compatible with the new Kubernetes version. Many add-ons are sensitive to API group changes — kube-proxy and CoreDNS are bundled, but AWS VPC CNI, EBS CSI driver, cert-manager, and Kyverno are not.

The skew policy isn't arbitrary — each minor version removes APIs deprecated two versions earlier, so the allowed skew is exactly the window you have to migrate workloads off them. Skipping versions means skipping that migration window and hitting every removal at once. Check your current skew before planning an upgrade:

# Control plane version
kubectl version --short

# Node kubelet versions
kubectl get nodes -o custom-columns='NAME:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion'

If any node's kubelet is already more than three minor versions behind the API server, fix that node's version before touching the control plane — it won't reconnect cleanly after the upgrade.

Pre-Upgrade Checklist

Run this checklist at least one sprint before the planned upgrade date.

1. Check for Deprecated API Usage

Kubernetes deprecates and removes API versions on a defined schedule. Using a removed API version after upgrade causes workloads to fail to deploy. Check for deprecated APIs in use:

1# Install pluto (deprecated API checker)
2brew install FairwindsOps/tap/pluto
3
4# Scan running resources in cluster
5pluto detect-all-in-cluster --target-versions k8s=v1.32
6
7# Scan Helm releases
8pluto detect-helm --target-versions k8s=v1.32
9
10# Scan local manifests
11pluto detect-files -d ./manifests --target-versions k8s=v1.32

Common API removals to watch:

• networking.k8s.io/v1beta1 Ingress (removed in 1.22)
• policy/v1beta1 PodDisruptionBudget (removed in 1.25)
• policy/v1beta1 PodSecurityPolicy (removed in 1.25)
• autoscaling/v2beta2 HorizontalPodAutoscaler (removed in 1.26)
• batch/v1beta1 CronJob (removed in 1.25)

Fix any deprecated API usage before upgrading. For Helm charts, this means upgrading the chart to a version that uses the current API version.

2. Verify PodDisruptionBudgets

Node drain during upgrade will evict pods. If a PDB blocks eviction (minAvailable can't be satisfied), the drain hangs:

1# Check all PDBs
2kubectl get pdb -A
3
4# Identify PDBs that would block drain
5kubectl get pdb -A -o json | jq '.items[] |
6  select(.status.disruptionsAllowed == 0) |
7  {name: .metadata.name, namespace: .metadata.namespace, reason: "0 disruptions allowed"}'

A PDB with minAvailable: 100% or maxUnavailable: 0 on a single-replica Deployment blocks all evictions. Before upgrading, either scale the Deployment to 2+ replicas or temporarily relax the PDB. Set a reminder to restore it after upgrade.

3. Verify Add-on Version Compatibility

For each add-on, check compatibility with the target Kubernetes version:

1# EKS: check current add-on versions
2aws eks list-addons --cluster-name my-cluster --region us-east-1
3
4# Get the latest compatible version for each add-on for the target K8s version
5aws eks describe-addon-versions \
6  --kubernetes-version 1.32 \
7  --addon-name aws-ebs-csi-driver \
8  --query 'addons[0].addonVersions[0].addonVersion' \
9  --output text

Key add-ons to verify:

• aws-vpc-cni — must be >= minimum version for target K8s
• aws-ebs-csi-driver — CSI API changes between K8s versions
• coredns — bundled, but may have configuration changes
• kube-proxy — bundled, pinned to K8s version
• cert-manager — check cert-manager compatibility matrix
• kyverno — check Kyverno release notes for K8s version support
• metrics-server — check API server compatibility

4. Take a Backup

Before any upgrade, take a Velero backup:

velero backup create pre-upgrade-$(date +%Y%m%d) \
  --include-namespaces "*" \
  --include-cluster-scoped-resources=true \
  --wait

On EKS, also enable automated etcd backups via AWS Backup if not already configured.

5. Check Node Health

# Nodes not in Ready state will cause issues during upgrade
kubectl get nodes | grep -v Ready

# Check for nodes with resource pressure
kubectl describe nodes | grep -A 5 "Conditions:"

Resolve any unhealthy nodes before upgrading.

6. Set minReadySeconds on Critical Deployments

minReadySeconds controls how long a new pod must stay ready before it counts as available. Without it, a pod is considered available the moment its readiness probe passes once — even if it crashes a few seconds later:

1apiVersion: apps/v1
2kind: Deployment
3spec:
4  minReadySeconds: 30   # pod must be healthy for 30s before counted available
5  strategy:
6    rollingUpdate:
7      maxUnavailable: 0
8      maxSurge: 1

This matters specifically during node drains: without it, Kubernetes can count a not-yet-stable new pod as available and evict an old one before the new pod has proven itself, turning a routine drain into a brief capacity gap.

7. Upgrade in Non-Production First

Run the upgrade in staging/dev before production. The upgrade process itself should be tested even if you've run it before — add-on compatibility, PDB behaviour, and deprecated API removal all manifest during the actual upgrade, not during planning.

Control Plane Upgrade

EKS

1# Upgrade control plane (only one minor version at a time)
2aws eks update-cluster-version \
3  --name my-cluster \
4  --kubernetes-version 1.32 \
5  --region us-east-1
6
7# Monitor upgrade progress
8aws eks describe-update \
9  --name my-cluster \
10  --update-id <update-id> \
11  --region us-east-1
12
13# Or watch via kubectl (API server briefly unavailable during upgrade)
14watch kubectl get nodes

EKS control plane upgrade takes 15–30 minutes. The API server is briefly unavailable (typically <30 seconds) as the API server rolls over. Existing workloads continue running — only new API calls fail during this window. Plan upgrades outside peak traffic windows.

After control plane upgrade, update EKS managed add-ons:

1# Update each managed add-on to the latest compatible version
2aws eks update-addon \
3  --cluster-name my-cluster \
4  --addon-name aws-vpc-cni \
5  --addon-version $(aws eks describe-addon-versions --addon-name aws-vpc-cni --kubernetes-version 1.32 --query 'addons[0].addonVersions[0].addonVersion' --output text) \
6  # Check the current version with: aws eks describe-addon-versions --addon-name aws-vpc-cni before upgrading
7  --resolve-conflicts OVERWRITE \
8  --region us-east-1
9
10# Repeat for coredns, kube-proxy, aws-ebs-csi-driver

--resolve-conflicts OVERWRITE replaces any custom addon configuration with AWS defaults. If you've customized an addon (a custom CoreDNS Corefile, VPC CNI env vars), use PRESERVE instead — it keeps your divergent field values and completes the update rather than failing on them. NONE returns an error on any conflict, which is useful for detecting configuration drift before it's silently overwritten.

EKS supports roughly 4-5 Kubernetes versions at a time, each for about 14 months of standard support, and it will auto-upgrade you past your end-of-support date whether you're ready or not. After standard support ends, EKS offers 12 months of extended support at an extra $0.60/cluster/hour on top of the normal control-plane fee — a real, recurring cost of falling behind, not just a risk. Upgrading on the quarterly cadence in this post's Upgrade Frequency Best Practices section keeps you inside the free support window.

GKE

1# Upgrade control plane
2gcloud container clusters upgrade my-cluster \
3  --master \
4  --cluster-version 1.32 \
5  --region us-central1
6
7# If on a release channel, trigger upgrade manually
8gcloud container clusters upgrade my-cluster \
9  --master \
10  --region us-central1

On GKE, if you use release channels (Regular, Stable, Rapid), Google schedules control plane upgrades automatically within your maintenance window. For production clusters, configure a maintenance window and exclusion windows:

gcloud container clusters update my-cluster \
  --maintenance-window-start "2026-05-09T02:00:00Z" \
  --maintenance-window-end "2026-05-09T06:00:00Z" \
  --maintenance-window-recurrence "FREQ=WEEKLY;BYDAY=SA,SU" \
  --region us-central1

AKS

1# Upgrade control plane only first
2az aks upgrade \
3  --resource-group my-rg \
4  --name my-cluster \
5  --kubernetes-version 1.32.0 \
6  --control-plane-only
7
8# After verifying control plane, upgrade node pools

AKS also supports auto-upgrade channels:

az aks update \
  --resource-group my-rg \
  --name my-cluster \
  --auto-upgrade-channel stable

Node Upgrade Strategies

This is where the most risk lives. Nodes hold your workloads. The goal is to replace nodes running the old kubelet version with new nodes without dropping requests.

Surge Upgrade (Managed Node Groups)

The default on managed node groups (EKS, GKE, AKS): the platform provisions new nodes, drains old nodes, and removes them.

EKS managed node group upgrade:

aws eks update-nodegroup-version \
  --cluster-name my-cluster \
  --nodegroup-name workers \
  --kubernetes-version 1.32 \
  --region us-east-1

EKS uses a surge upgrade strategy by default: it provisions max_surge new nodes (default: 1), drains and terminates an equal number of old nodes, then repeats. Configure the surge size:

# Terraform EKS module
update_config = {
  max_unavailable_percentage = 33   # Up to 33% of nodes unavailable during upgrade
}

Higher surge = faster upgrade, more temporary overcapacity cost. Lower surge = slower upgrade, less disruption to workloads.

Before node group upgrade, ensure:

• HPA minimum replicas ≥ 2 for all critical Deployments
• PDBs are set on all Deployments that can't tolerate a single pod being evicted

Blue-Green Node Upgrade (Self-Managed Nodes)

For self-managed node groups or when you want maximum control:

1. Create a new node group with the target Kubernetes version
2. Cordon all nodes in the old node group (prevent new scheduling)
3. Drain nodes one at a time
4. Verify pods are running on new nodes
5. Terminate old nodes

1# Cordon all old nodes
2kubectl get nodes -l node-group=workers-v1.31 -o name | \
3  xargs -I {} kubectl cordon {}
4
5# Drain one node at a time
6kubectl drain <node-name> \
7  --ignore-daemonsets \
8  --delete-emptydir-data \
9  --grace-period=300
10
11# Verify
12kubectl get pods -A -o wide | grep <node-name>
13# Should show no pods (DaemonSet pods are excluded by --ignore-daemonsets)

--delete-emptydir-data is required for pods using emptyDir volumes. These are ephemeral by definition — the pod knows this data is not durable. Don't omit this flag to "protect" data in emptyDir; the pod contract is that emptyDir is lost on pod termination.

EKS: A Separate Node Group Instead of In-Place Surge

The Surge Upgrade above replaces nodes within the same managed node group — simplest, but you're validating the new version's nodes in production as they come up. For a genuine blue-green (new node group fully proven before any old node is touched), create a second EKS-managed node group at the target version instead of bumping the existing one in place:

1# Create the new node group at the target version, alongside the old one
2aws eks create-nodegroup \
3  --cluster-name my-cluster \
4  --nodegroup-name workers-v1-33 \
5  --node-role arn:aws:iam::123456789:role/eks-node-role \
6  --subnets subnet-abc123 subnet-def456 \
7  --scaling-config minSize=3,maxSize=10,desiredSize=3 \
8  --kubernetes-version 1.33 \
9  --region us-east-1
10
11aws eks wait nodegroup-active \
12  --cluster-name my-cluster \
13  --nodegroup-name workers-v1-33
14
15# Once new nodes are healthy, cordon and drain the old group (see kubectl steps above),
16# then remove it once nothing is scheduled there
17aws eks delete-nodegroup \
18  --cluster-name my-cluster \
19  --nodegroup-name workers-v1-32 \
20  --region us-east-1

The trade-off versus Surge Upgrade: double the node capacity (and cost) for the migration window, in exchange for a rollback that's a routing decision — cordon the new group and uncordon the old one — rather than an in-progress node replacement you have to fix forward.

Karpenter Node Replacement

With Karpenter, node upgrades work differently. You update the EC2NodeClass AMI configuration and Karpenter gradually replaces nodes via its disruption mechanism:

1apiVersion: karpenter.k8s.aws/v1
2kind: EC2NodeClass
3metadata:
4  name: default
5spec:
6  amiFamily: AL2023   # Karpenter resolves the latest AMI for this family automatically
7  # Or pin a specific AMI version:
8  amiSelectorTerms:
9    - alias: al2023@v20261201   # Specific AMI version

To force node replacement:

# Remove do-not-disrupt protection so Karpenter's drift mechanism can replace the node
kubectl annotate node <node-name> karpenter.sh/do-not-disrupt-

# Or let Karpenter's drift detection handle it:
# When EC2NodeClass AMI changes, Karpenter marks affected nodes as "drifted"
# and gradually replaces them according to disruption budgets

Karpenter's drift detection makes node upgrades largely automatic — when you update the EC2NodeClass AMI family or alias, Karpenter identifies nodes running the old AMI as "drifted" and gradually replaces them, respecting disruption budgets and PDBs. This is a separate mechanism from the consolidation policy.

Terraform-Managed EKS Upgrades

If the cluster is managed with the terraform-aws-eks module, the upgrade is a version bump applied in stages rather than a sequence of aws eks CLI calls:

1module "eks" {
2  source  = "terraform-aws-modules/eks/aws"
3  version = "~> 21.0"    # v21 is the current major series
4
5  kubernetes_version = "1.33"    # Was "1.32"
6
7  eks_managed_node_groups = {
8    default = {
9      min_size     = 3
10      max_size     = 10
11      desired_size = 3
12      ami_type     = "AL2023_x86_64_STANDARD"
13      # ami_release_version = "1.33.x-eksbuild.y"   # leave unset for latest
14    }
15  }
16}

1# Control plane first
2terraform apply -target module.eks.aws_eks_cluster.this
3
4# Verify control plane, then addons
5terraform apply -target module.eks.aws_eks_addon.this
6
7# Finally, node groups
8terraform apply

If you're still on the ~> 20.0 series, note that v21 renamed the variables this example uses — cluster_version became kubernetes_version and cluster_name became name. That's a breaking major, not a drop-in bump: read the module's UPGRADE-21.0.md first, and do the version-constraint bump as its own terraform apply separate from the Kubernetes upgrade itself. Two breaking changes landing in the same plan makes it much harder to tell which one caused a given diff.

Add-on Upgrades After Node Upgrade

After nodes are on the new Kubernetes version, upgrade in-cluster tools:

cert-manager

1# Check current version
2kubectl get deployment -n cert-manager cert-manager -o jsonpath='{.spec.template.spec.containers[0].image}'
3
4# Upgrade via Helm
5helm repo update
6helm upgrade cert-manager jetstack/cert-manager \
7  --namespace cert-manager \
8  --version v1.21.x \
9  --reuse-values

Kyverno

helm upgrade kyverno kyverno/kyverno \
  --namespace kyverno \
  --version 3.x.x \
  --reuse-values

Check Kyverno's migration guides between major versions — ClusterPolicy API versions and webhook configurations sometimes change in major releases.

Prometheus / kube-prometheus-stack

helm upgrade kube-prometheus-stack \
  prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --version <target-version> \
  --reuse-values

The kube-prometheus-stack upgrade path has historically been sensitive to CRD changes. Check the chart's migration guide for any manual CRD upgrade steps.

Rollback Plan

Before starting any upgrade, document the rollback plan. You will not have time to reason through it if something goes wrong at 2am.

Control Plane Rollback

You cannot downgrade a Kubernetes control plane version. If the control plane upgrade fails:

• EKS: AWS Support can restore from their internal snapshots (contact AWS Support immediately)
• GKE: Google manages control plane — contact Cloud Support
• AKS: Azure manages control plane — contact Azure Support

This is why the backup before upgrade is non-negotiable. If you have a Velero backup and the cluster is irrecoverably broken, you can provision a new cluster on the old version and restore.

Node Rollback

Node upgrades are reversible. If the new node group has issues:

1. Stop the upgrade
2. Keep the old node group running
3. Scale the old node group back up
4. Remove the cordon from old nodes
5. Drain and terminate new nodes

For EKS managed node groups, the old AMI is still available — you can create a new node group with the previous Kubernetes version and shift pods back while you investigate.

Application Rollback

For application-level breakages (deprecated API usage, changed behaviour), roll back via GitOps:

# Argo CD: sync to previous revision
argocd app history my-app
argocd app rollback my-app <revision-number>

# Or kubectl: if using Deployments directly
kubectl rollout undo deployment/my-app -n production

Post-Upgrade Verification

After the upgrade completes, verify before declaring success:

1# All nodes on new version
2kubectl get nodes -o wide
3
4# All system pods running
5kubectl get pods -n kube-system
6
7# All application pods running
8kubectl get pods -A | grep -v Running | grep -v Completed
9
10# HPA functioning
11kubectl get hpa -A
12
13# Ingress controller responding
14curl -I https://my-service.example.com/healthz
15
16# Check for any admission webhook failures in recent events
17kubectl get events -A --sort-by='.lastTimestamp' | tail -20

Let the upgraded cluster run through at least one business day before closing the upgrade incident. Admission webhook failures, API deprecation errors in logs, and Ingress controller regressions sometimes surface only under real traffic.

Upgrade Frequency Best Practices

Running at N-1 (one version behind latest) is a reasonable steady-state for production clusters on EKS and AKS, where extended support is available. Running at N-2 (two versions behind) should be the maximum acceptable — it means one upgrade separates you from end-of-standard-support.

Build a quarterly upgrade cadence:

• Q1: Upgrade non-production clusters to latest minor version
• Q2: Upgrade production clusters
• Q3: Upgrade non-production to next minor version
• Q4: Upgrade production again

This keeps production at most two minor versions behind latest, gives you 3+ months of non-production validation before production upgrades, and prevents the "haven't upgraded in 18 months and now we're 4 versions behind" scenario that requires multi-step upgrades to close.

See also

• Zero-Downtime Deployments with Rolling Updates and Readiness Probes

Frequently Asked Questions

Can I skip minor versions when upgrading?

No. Kubernetes (and all managed services) require upgrading one minor version at a time. To go from 1.28 to 1.32, you must upgrade 1.28 → 1.29 → 1.30 → 1.31 → 1.32. Each hop is a full upgrade cycle. This is why staying current (upgrading every 2-3 versions rather than waiting) significantly reduces the total upgrade burden.

How long does the API server outage last during control plane upgrade?

On EKS, typically 30–60 seconds. On GKE regional clusters, the control plane upgrade is HA and doesn't cause API server downtime — the upgrade is rolling across the control plane replicas. AKS is similar to EKS (brief unavailability). Existing workloads continue running during the API server outage; only new API calls (deployments, scaling) fail.

What if node drain hangs?

A hung drain is almost always a PDB that prevents eviction. To identify the blocking PDB:

kubectl get events -n <namespace> | grep "disruption budget"

Options:

1. Scale up the Deployment so the PDB can be satisfied
2. Temporarily delete the PDB (kubectl delete pdb <name> -n <namespace>) — restore it after drain completes
3. Force-evict (only as a last resort — it bypasses the PDB): kubectl delete pod <pod> -n <namespace> --grace-period=0 --force

How do I handle StatefulSets during node drain?

StatefulSets with ordered pod management (podManagementPolicy: OrderedReady) drain correctly — the StatefulSet controller ensures orderly shutdown. Ensure your StatefulSet's PDB allows at least one pod to be disrupted; minAvailable: n-1 is the standard pattern.

Should I upgrade Kubernetes before or after upgrading Helm charts?

Kubernetes first. Once the control plane is upgraded, Helm chart upgrades can happen at any pace. The risk is upgrading Helm charts (e.g., cert-manager, Kyverno) while the old Kubernetes version is running — chart updates sometimes add features that require the new K8s version and fail on the old one.

For GitOps-managed upgrades and sync policies, see GitOps with Argo CD: Production Setup Guide. For Karpenter node management, see How to Install Karpenter on EKS.

Planning a major Kubernetes version upgrade? Talk to us at Coding Protocols — we've run these upgrades on production clusters handling millions of requests and can help you avoid the common failure modes.

Official References

• Operating etcd for Kubernetes — backup, restore and cluster sizing
• Deprecated API migration guide — which APIs are removed in which release