Automate Secret Rotation with External Secrets Operator

A secret rotates in AWS Secrets Manager right on schedule — the Lambda runs, the new value is written, everything looks green in the AWS console. Twenty minutes later a pod is still authenticating with the old password because it read that value into an environment variable at container start and nothing has told it otherwise since. The Kubernetes Secret that External Secrets Operator manages did update; the process that already loaded it into memory has no idea. Rotation without propagation is not rotation — it's just a second place the credential now silently disagrees with the first.

This tutorial assumes ESO is already installed and a SecretStore is already syncing secrets into the cluster. It skips straight to the two things that actually determine whether rotation reaches a running workload: how tightly you poll, and what happens to a pod that already read the old value.

What You'll Build

• An ExternalSecret with refreshInterval deliberately tuned to your rotation cadence, not left at a default
• A Secret consumed as a volume mount instead of environment variables, so kubelet propagates updates without a restart
• Stakater Reloader wired to a Deployment via annotation, so apps that only read secrets at startup get a rolling restart when the Secret changes
• A verification pass that confirms the new value actually reaches the pod's filesystem — not just that ESO says it synced

Step 1: Confirm Your SecretStore Is Already Wired Up

You should already have a ClusterSecretStore (or namespaced SecretStore) pointing at AWS Secrets Manager, something like:

1apiVersion: external-secrets.io/v1
2kind: ClusterSecretStore
3metadata:
4  name: aws-secrets-manager
5spec:
6  provider:
7    aws:
8      service: SecretsManager
9      region: us-east-1
10      auth:
11        jwt:
12          serviceAccountRef:
13            name: external-secrets
14            namespace: external-secrets

If you don't have this yet, Install External Secrets walks through installing ESO and getting IRSA authentication working — do that first. From here on, this tutorial assumes kubectl get clustersecretstore aws-secrets-manager already returns Valid / Ready: True.

Step 2: Set refreshInterval to Match the Rotation Schedule, Not the Default

ExternalSecret is pull-based — ESO doesn't get notified when the source secret changes, it polls on spec.refreshInterval and pulls if the value differs. That interval is the entire propagation budget: if your AWS rotation Lambda runs and produces a new value, the Kubernetes Secret stays stale for up to one full refreshInterval before ESO even notices.

1apiVersion: external-secrets.io/v1
2kind: ExternalSecret
3metadata:
4  name: db-credentials
5  namespace: default
6spec:
7  refreshInterval: 1h
8  secretStoreRef:
9    name: aws-secrets-manager
10    kind: ClusterSecretStore
11  target:
12    name: db-credentials
13    creationPolicy: Owner
14  data:
15    - secretKey: DB_PASSWORD
16      remoteRef:
17        key: prod/app/database
18        property: password
19    - secretKey: DB_USERNAME
20      remoteRef:
21        key: prod/app/database
22        property: username

refreshInterval: 1h here is a deliberate choice, not a placeholder. If AWS Secrets Manager rotates this secret every 24 hours, polling hourly means the worst case is a 1-hour window of staleness — acceptable for most workloads. If you rotate every 30 minutes because the credential is short-lived and high-value, 1h is wrong: you'd serve a stale value for a full rotation cycle, sometimes two. Set refreshInterval to a fraction of your actual rotation period, not a value you picked because it looked reasonable. There's no cross-cluster default that's correct for every secret — check the rotation schedule on the AWS side (aws secretsmanager describe-secret --secret-id prod/app/database --query RotationRules) and work backward from it.

ExternalSecret is the pull direction — a PushSecret resource exists for syncing a Kubernetes-native Secret out to a provider, but that's the reverse flow and not what rotation propagation needs here.

Step 3: The Gap ESO Doesn't Close: Env Vars vs. Volume Mounts

Once ESO polls and updates the Kubernetes Secret, the object in etcd is current. Whether a running pod sees the new value depends entirely on how the pod consumes it, and this is the part most rotation setups get wrong.

Environment variables are fixed at container start. If a Deployment loads the Secret via envFrom or env.valueFrom.secretKeyRef, the value is read once when the container process starts and never again. ESO can refresh the underlying Secret every minute and the running container will still be holding the value from whenever it last restarted.

Volume-mounted Secrets update in place. kubelet watches mounted Secrets and rewrites the files on disk when the Secret's contents change, independent of the pod's lifecycle — no restart needed. The catch is a sync delay: kubelet's cache TTL plus its periodic sync means updates typically land within about a minute, not instantly, and your application has to actually notice the file changed (poll its mtime, use an fsnotify-based watcher, or reload periodically) rather than reading it once at startup and caching it forever.

1apiVersion: apps/v1
2kind: Deployment
3metadata:
4  name: app
5  namespace: default
6spec:
7  replicas: 2
8  selector:
9    matchLabels: { app: app }
10  template:
11    metadata:
12      labels: { app: app }
13    spec:
14      containers:
15        - name: app
16          image: your-org/app:v1.2.0
17          volumeMounts:
18            - name: db-credentials
19              mountPath: /etc/secrets/db
20              readOnly: true
21      volumes:
22        - name: db-credentials
23          secret:
24            secretName: db-credentials

One gotcha worth flagging explicitly: if you mount an individual key with subPath, kubelet does not propagate updates to that file — subPath breaks the live-update mechanism. Mount the whole Secret as a directory and have the app read /etc/secrets/db/DB_PASSWORD rather than mounting a single subPath'd file.

Step 4: For Apps That Only Read Secrets at Startup, Trigger a Restart with Reloader

Volume mounts solve rotation for anything that watches its own filesystem. A lot of real workloads don't — they read the config once at boot and hold it in memory for the life of the process, by design or by whatever framework they're built on. For those, the only real fix is restarting the pod when the Secret changes.

Stakater Reloader watches Secrets and ConfigMaps for changes and triggers a rolling restart of any Deployment, StatefulSet, or DaemonSet that references them and opts in via annotation. It does this by patching a checksum annotation onto the pod template on each change — Kubernetes treats that as a spec change and rolls the pods, same mechanism as any other rolling update, so it respects maxUnavailable/maxSurge rather than killing everything at once.

Install it:

helm repo add stakater https://stakater.github.io/stakater-charts
helm repo update
helm install reloader stakater/reloader --namespace kube-system

Then annotate the Deployment that consumes the rotated Secret:

1apiVersion: apps/v1
2kind: Deployment
3metadata:
4  name: app
5  namespace: default
6  annotations:
7    reloader.stakater.com/auto: "true"
8spec:
9  replicas: 2
10  selector:
11    matchLabels: { app: app }
12  template:
13    metadata:
14      labels: { app: app }
15    spec:
16      containers:
17        - name: app
18          image: your-org/app:v1.2.0
19          envFrom:
20            - secretRef:
21                name: db-credentials

reloader.stakater.com/auto: "true" tells Reloader to watch every Secret and ConfigMap this Deployment references and restart it on any change — no need to name db-credentials explicitly. If you'd rather scope it to one specific Secret instead of everything the Deployment mounts, use secret.reloader.stakater.com/reload: "db-credentials" on the Deployment instead.

Step 5: Verify Rotation Actually Reaches the Pod

Rotate the value manually so you don't have to wait for the AWS-side schedule:

aws secretsmanager update-secret \
  --secret-id prod/app/database \
  --region us-east-1 \
  --secret-string '{"username":"appuser","password":"rotated-value-01"}'

Confirm ESO picked it up within one refreshInterval — check the Secret's resourceVersion changed, or decode the value directly:

kubectl get secret db-credentials -n default \
  -o jsonpath='{.data.DB_PASSWORD}' | base64 -d

If you're on the volume-mount path, exec into the pod and read the file rather than trusting the Secret object alone — this is the step that actually proves propagation, not just that ESO synced:

kubectl exec -n default deploy/app -- cat /etc/secrets/db/DB_PASSWORD

If you're on the Reloader path, confirm a rollout actually happened after the Secret changed:

kubectl rollout history deployment/app -n default
kubectl get pods -n default -l app=app

New pod names and a recent AGE mean Reloader fired. If the pods are the same age as before the rotation, check that the annotation is on the Deployment (not the Pod template only) and that Reloader's own pod in kube-system is running and has RBAC to watch Secrets in the target namespace.

Where to Go Next

• Install External Secrets — the installation and IRSA setup this tutorial assumes is already done
• External Secrets Operator for Kubernetes — the conceptual background on ESO's sync model
• AWS Secrets Manager and Parameter Store — the rotation Lambda mechanics on the AWS side that feed into what this tutorial propagates
• Install Vault — swap the backend: the same refreshInterval and propagation mechanics apply if ESO is pulling from Vault instead of Secrets Manager