Karpenter IAM Deadlock: How We Broke Our EKS Cluster with a Terraform Apply

Running Karpenter on EKS and managing its IAM role with Terraform is a common setup. It's also one that can silently break your cluster in a way that isn't immediately obvious — new nodes launch but never join, pods sit in Pending, and the Karpenter logs show IAM permission errors that weren't there an hour ago.

This post documents the failure mode in detail: what causes it, what it looks like, how to recover, and how to prevent it from happening again.

Background: How Karpenter Uses IAM

Karpenter's controller pod needs IAM permissions to call EC2 APIs — RunInstances, TerminateInstances, DescribeInstances, and others. In a standard setup, these permissions are granted via either IRSA (IAM Roles for Service Accounts) or EKS Pod Identity.

In addition, every node Karpenter launches assumes a separate IAM role — the KarpenterNodeRole — which gives the node enough permissions to join the cluster (pull from ECR, describe cluster, etc.).

Two IAM roles. Two separate concern surfaces. The deadlock involves the controller role.

The Failure Mode

What Triggers It

The race condition occurs when you run a terraform apply that modifies the Karpenter controller IAM role or its attached policies while the Karpenter controller is actively making EC2 API calls.

The most common triggers:

• Adding or removing an IAM policy attachment from the Karpenter controller role
• Modifying inline policies on the controller role
• Rotating the role (delete + recreate via Terraform's create_before_destroy)
• Changing the trust policy on the role

The Race

IAM changes in AWS are not instant. When Terraform updates an IAM role or policy, there's a propagation window — typically 15–60 seconds — during which the role's effective permissions may be inconsistent across AWS's internal IAM systems.

If Karpenter's controller is mid-flight during this window (it has called ec2:RunInstances and is waiting for the response, for example), the following can happen:

1. Terraform modifies the policy attachment
2. AWS's IAM propagation is in-flight
3. Karpenter's next API call is evaluated against a partially-updated policy state
4. The call is denied with AccessDeniedException
5. Karpenter logs the error and retries
6. Retries also fail — the policy state is now fully updated, but the role's session token was issued before the policy update and may still be cached

The last point is the key: IAM session tokens are cached. When Karpenter's pod (via IRSA or Pod Identity) retrieves temporary credentials, those credentials are valid for up to an hour. If the policy was updated after the credentials were issued but before they expired, the credentials may not reflect the new policy state — or in some race conditions, they reflect a partial state that denies calls they shouldn't.

What You See

Karpenter controller logs:

{"level":"error","time":"2026-03-14T14:23:11Z","message":"launching node",
"error":"AccessDeniedException: User: arn:aws:sts::123456789:assumed-role/KarpenterControllerRole-my-cluster/... 
is not authorized to perform: ec2:RunInstances"}

Or more confusingly:

{"level":"error","message":"syncing machines",
"error":"AccessDeniedException: not authorized to perform: ec2:DescribeInstances"}

ec2:DescribeInstances was working fine before the apply. It's in the policy. But after the apply, it fails intermittently.

Node state: Nodes that were provisioning before the apply may launch the EC2 instance but fail at the RunInstances tagging step or the node registration step. These show up as EC2 instances in the console that never appear as Kubernetes nodes.

Pod state: Pods that need new nodes are stuck in Pending with 0/N nodes available: N Insufficient cpu.

What makes it confusing: The policy itself is correct. If you run aws iam simulate-principal-policy, it shows the permission is allowed. But the live Karpenter pod is still getting denied.

Root Cause: Credential Caching

Karpenter (like all AWS SDK clients) caches temporary credentials. With IRSA, the pod's service account token is exchanged for temporary IAM credentials via sts:AssumeRoleWithWebIdentity. With Pod Identity, the Pod Identity Agent handles this. Either way, the credentials have a validity window.

When IAM policies are updated, AWS doesn't invalidate in-flight credential sessions. The session continues to use the permissions it was granted at issuance time — but in some edge cases during policy propagation, the permissions evaluated against the session reflect an inconsistent state.

The fix is to force Karpenter to acquire fresh credentials. There are two ways to do that.

Recovery

Option 1: Restart the Karpenter Controller Pod (Fastest)

Restarting the controller pod forces it to acquire new temporary credentials on startup. If the IAM change has fully propagated, the new credentials will reflect the correct policy state.

kubectl rollout restart deployment/karpenter -n kube-system

Wait for the new pod to become ready:

kubectl rollout status deployment/karpenter -n kube-system

Then verify the error is gone:

kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter -c controller --tail=50 -f

If EC2 API calls succeed without AccessDeniedException, recovery is complete. Clean up any orphaned EC2 instances (nodes that launched but never joined):

# List instances tagged with your cluster that aren't registered as nodes
aws ec2 describe-instances \
  --filters "Name=tag:karpenter.sh/managed-by,Values=my-cluster" \
             "Name=instance-state-name,Values=running" \
  --query "Reservations[].Instances[].{ID:InstanceId,State:State.Name,Launch:LaunchTime}" \
  --output table

Compare against kubectl get nodes. Any EC2 instance running for more than 10 minutes that hasn't joined is orphaned — Karpenter will eventually terminate it when it reconciles, but you can terminate it manually to stop the billing.

Option 2: Wait for Credential Expiry

If you don't want to restart the controller (e.g., you're mid-investigation or the restart would disrupt an in-progress scale event), temporary credentials expire on their own. IRSA credentials are valid for 1 hour by default; Pod Identity credentials are also short-lived.

The controller will naturally refresh credentials before expiry. If the IAM propagation completes within the credential refresh window, the problem resolves on its own. In practice this takes 15–60 minutes from the time of the Terraform apply.

This is the option when you're not in active production impact — let it resolve and use the time to implement the prevention below.

Option 3: If the Policy Was Actually Broken (Not Just a Race)

If the AccessDeniedException persists after a pod restart, the policy change itself removed a permission that Karpenter needs. Check what changed:

# See recent IAM role policy changes
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=KarpenterControllerRole-my-cluster \
  --start-time $(date -u -d '2 hours ago' +%Y-%m-%dT%H:%M:%SZ) \
  --output json | jq '.Events[].CloudTrailEvent' | python3 -m json.tool

Compare the current policy against the official Karpenter controller policy for your version. Restore any missing permissions, run terraform apply again (carefully — see Prevention below), and then restart the controller.

Prevention

1. Drain Karpenter Before IAM Changes

Before any Terraform apply that touches the Karpenter controller role, scale the controller to zero:

kubectl scale deployment karpenter -n kube-system --replicas=0

Apply your Terraform changes. Wait for IAM propagation (30–60 seconds is usually sufficient; aws iam simulate-principal-policy returning correct results is a better signal).

Then scale back up:

kubectl scale deployment karpenter -n kube-system --replicas=2

This eliminates the race entirely — there are no in-flight credentials when the policy changes.

The downside: Karpenter is unavailable during the apply window. If your cluster is actively auto-scaling, this can cause a brief scaling gap. For most clusters, the 60–120 second window is acceptable. If it isn't, use a maintenance window.

2. Separate IAM Terraform from Cluster Terraform

Structure your Terraform so that IAM resources are in a separate state/workspace from the live cluster resources:

terraform/
├── iam/                # IAM roles, policies — no live cluster resources
│   ├── main.tf
│   └── karpenter.tf
├── eks-cluster/        # EKS cluster, node groups — no IAM role bodies
│   ├── main.tf
│   └── karpenter.tf    # only references IAM role ARNs, no policy content
└── karpenter-config/   # K8s resources: NodePool, EC2NodeClass
    └── main.tf

When you need to change the Karpenter policy, you apply terraform/iam/ — which has no live cluster resources to race against. The controller role update happens independently of any in-flight EC2 operations.

This separation also gives you:

• Clear blast radius for IAM changes
• Ability to apply IAM changes without cluster access (useful if the cluster is unhealthy)
• Cleaner state management

3. Use create_before_destroy = false on Role Resources

If your Terraform uses create_before_destroy on the IAM role (to avoid a brief gap when the role is recreated), this actually makes the race worse — the old role is still being used while the new role is being created, and the switchover happens mid-flight.

For Karpenter's controller role, prefer to recreate in place (update the existing role) rather than delete-and-recreate. If you must recreate, drain Karpenter first.

4. Add a Post-Apply Health Check

Add a null_resource or terraform_data block that verifies Karpenter is healthy after any IAM change:

1resource "terraform_data" "karpenter_health_check" {
2  depends_on = [aws_iam_role_policy_attachment.karpenter]
3
4  provisioner "local-exec" {
5    command = <<EOF
6      echo "Waiting for IAM propagation..."
7      sleep 30
8      kubectl rollout restart deployment/karpenter -n kube-system
9      kubectl rollout status deployment/karpenter -n kube-system --timeout=120s
10    EOF
11  }
12}

This is a blunt instrument — it always restarts the controller after an IAM change, whether or not a race condition occurred. For teams that can't or don't want to drain before apply, this at least ensures fresh credentials after every IAM change.

The Broader Pattern

This race condition is a specific instance of a broader anti-pattern: modifying the IAM trust or permission boundary of a running AWS service client while it has active in-flight requests.

The same failure mode can affect:

• Cluster Autoscaler (also assumes an IAM role for EC2 APIs)
• AWS Load Balancer Controller (IAM role for ELB/ALB APIs)
• External Secrets Operator (IAM role for Secrets Manager/Parameter Store)
• Velero (IAM role for S3 backup operations)

The fix is the same in each case: drain the controller before changing its IAM role, or accept a potential brief credential inconsistency after the change and restart the pod to recover.

Frequently Asked Questions

Does this affect EKS Pod Identity differently from IRSA?

The race condition can affect both, but the credential caching mechanics differ. With IRSA, the pod's projected service account token is refreshed by kubelet every ~50% of the token's TTL (usually 24h by default, though EKS sets this lower). The STS credentials derived from it are typically cached for ~1 hour.

With Pod Identity, the Pod Identity Agent handles credential refresh. The agent rotates credentials proactively before expiry, which slightly reduces the window for a stale-credential race. In practice, both are susceptible; the recovery (pod restart) is the same.

Can I use IAM policy conditions to avoid needing to update the policy?

In some cases, yes. If you're adding a new S3 bucket that Karpenter needs to access, and your policy already uses a wildcard ("s3:GetObject" on "arn:aws:s3:::karpenter-*"), adding the new bucket doesn't require a policy change. Design your policies with enough flexibility that routine infrastructure changes don't require Karpenter IAM updates.

What if the IAM deadlock also breaks existing nodes?

Existing nodes are unaffected by changes to the Karpenter controller role — they use the KarpenterNodeRole, which is separate. If existing nodes start failing, the KarpenterNodeRole has been modified, not the controller role. Check aws-auth ConfigMap or Access Entries to ensure the node role is still mapped, and check the KarpenterNodeRole policy for missing EC2/ECR permissions.

Does Karpenter have a health endpoint I can monitor?

Yes. Karpenter exposes /healthz and /readyz endpoints. The controller deployment's readiness probe uses /readyz. Monitoring the Karpenter deployment's ready replicas in your alerting stack gives you a signal when the controller is unhealthy — including during a credential deadlock scenario.

For the full Karpenter setup guide, see How to Install Karpenter on EKS. For Terraform-driven EKS cluster management patterns more broadly, see the platform engineering resources at Coding Protocols.

Hit this deadlock in production? Talk to us at Coding Protocols — we've recovered from this more than once and can help you structure your Terraform to prevent it.

Official References

• Karpenter NodePool — the v1 NodePool spec, disruption budgets and consolidation
• Karpenter upgrade guide — the per-release API transitions
• Terraform documentation — configuration language, state and provider behaviour
• Terraform state — remote backends, locking and drift