HashiCorp Vault on Kubernetes: Dynamic Secrets, PKI, and the Transit Engine

Most comparisons of Kubernetes secrets tooling treat Vault as a fixed point — "Vault vs External Secrets Operator," "Vault vs native Secrets" — without explaining what Vault actually does that a plain secret store doesn't. The honest answer is: a KV store that happens to be encrypted at rest isn't the interesting part of Vault. The interesting part is that Vault can generate credentials on demand, encrypt data without ever handing you the key, and issue short-lived certificates as its own CA — three capabilities that have nothing to do with "storing secrets" in the conventional sense.

This post is about Vault itself: how a pod actually authenticates to it, what dynamic secrets look like in practice, and the real operational cost of running it — because that cost is the main reason teams reach for External Secrets Operator instead.

Static Secrets vs Dynamic Secrets

Vault's KV secrets engine (kv-v2) is a versioned, access-controlled key-value store. If that's all you use, Vault is functionally a fancier version of Kubernetes' own Secret objects — better audit logging, better access policies, but the same fundamental model: a human or a pipeline writes a credential, and it sits there until someone rotates it.

Dynamic secrets are a different model entirely. Instead of storing a long-lived database password, you configure Vault's database secrets engine to connect to the database with admin credentials, and every time a workload requests a credential, Vault creates a brand-new, short-lived database user on the spot:

1# One-time setup: Vault connects to Postgres with admin creds
2vault write database/config/payments-db \
3    plugin_name=postgresql-database-plugin \
4    connection_url="postgresql://{{username}}:{{password}}@payments-db.internal:5432/payments?sslmode=require" \
5    allowed_roles="payments-readonly" \
6    username="vault-admin" \
7    password="$VAULT_ADMIN_DB_PASSWORD"
8
9# Define a role: what SQL runs to create a credential, and its TTL
10vault write database/roles/payments-readonly \
11    db_name=payments-db \
12    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
13        GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
14    default_ttl="1h" \
15    max_ttl="24h"

1# Every request gets its own, unique, time-boxed credential
2vault read database/creds/payments-readonly
3# Key                Value
4# ---                -----
5# lease_id           database/creds/payments-readonly/8f3a2c1e-...
6# lease_duration      1h
7# username            v-root-payments-readonly-a1b2c3
8# password            A1a2B3b4-generated-by-vault

When the lease expires, Vault connects back to the database and revokes that specific role. There is no credential sitting in a Kubernetes Secret for an attacker to find six months after it should have been rotated — because it never lives longer than an hour. This is the actual differentiator between Vault and "a secret store with good RBAC."

The Kubernetes Auth Method

For a pod to request anything from Vault — a dynamic credential, a KV read, an encryption operation — it needs a Vault token. The Kubernetes auth method is how a pod gets one without a human typing a password.

Setup, once per cluster:

1vault auth enable kubernetes
2
3vault write auth/kubernetes/config \
4    kubernetes_host="https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT" \
5    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
6    token_reviewer_jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token
7
8# Bind a Vault role to a specific ServiceAccount + namespace
9vault write auth/kubernetes/role/payments-api \
10    bound_service_account_names=payments-api \
11    bound_service_account_namespaces=payments \
12    policies=payments-readonly \
13    ttl=1h

At login time, the pod presents its own mounted ServiceAccount JWT to Vault. Vault validates that token against the Kubernetes API's TokenReview endpoint (using the reviewer token configured above), confirms the service account and namespace match a bound role, and issues a Vault token scoped to that role's policies:

# Inside the pod
curl --request POST \
    --data "{\"jwt\": \"$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\", \"role\": \"payments-api\"}" \
    https://vault.internal:8200/v1/auth/kubernetes/login

One thing worth getting right: Kubernetes 1.21+ defaults ServiceAccount tokens to expiring (bound, time-limited JWTs via TokenRequest), which breaks the old pattern of storing a single long-lived reviewer JWT. Current Vault versions handle this by reading Vault's own pod's token from disk and re-reading it as it rotates, rather than relying on a stored copy — if you're configuring this from scratch today, use that pattern, not a JWT pasted in once and forgotten.

In practice, most teams don't call this API directly — they run the Vault Agent sidecar, the Vault CSI Provider (mounts secrets as a volume, closer semantics to how ESO or the Secrets Store CSI Driver work), or Vault Secrets Operator (VSO), which syncs Vault data into native Kubernetes Secret objects on a watch-and-reconcile loop. VSO is the one most directly comparable to ESO — see Vault Secrets Operator vs External Secrets Operator for that specific comparison.

Transit: Encryption as a Service

The Transit secrets engine does something neither a KV store nor a dynamic-secrets engine does: it encrypts and decrypts data on your behalf without ever giving you the encryption key.

1vault secrets enable transit
2vault write -f transit/keys/payments-pii
3
4# Your application sends plaintext, Vault returns ciphertext — the key never leaves Vault
5vault write transit/encrypt/payments-pii plaintext=$(base64 <<< "4111-1111-1111-1111")
6# ciphertext: vault:v1:8SDd3WHDOjf7mq69CyC...
7
8vault write transit/decrypt/payments-pii ciphertext="vault:v1:8SDd3WHDOjf7mq69CyC..."
9# plaintext: NDExMS0xMTExLTExMTEtMTExMQ==   (base64 of the original)

This matters for a specific class of problem: application-level encryption of sensitive fields (PII, card numbers, tokens) where you want the encryption key's lifecycle — rotation, access control, audit trail — managed entirely separately from the application that uses it. If your database is compromised but Vault isn't, the encrypted fields are still useless to an attacker. Transit also supports key rotation with vault write -f transit/keys/payments-pii/rotate, after which old ciphertext still decrypts (Vault tracks key versions), while new encryption uses the latest version.

PKI: Vault as Your Internal CA

The PKI secrets engine turns Vault into a certificate authority that issues short-lived X.509 certificates on demand, the same dynamic-credential pattern as the database engine applied to TLS:

1vault secrets enable pki
2vault secrets tune -max-lease-ttl=87600h pki
3
4# Generate (or import) a root CA
5vault write -field=certificate pki/root/generate/internal \
6    common_name="internal.codingprotocols.com" \
7    ttl=87600h > ca_cert.crt
8
9vault write pki/roles/payments-api \
10    allowed_domains="payments.svc.cluster.local" \
11    allow_subdomains=true \
12    max_ttl="720h"
13
14# Issue a cert — valid for a few days, not years
15vault write pki/issue/payments-api common_name="payments-api.payments.svc.cluster.local"

This is not a competitor to cert-manager — it's a backend cert-manager can use. cert-manager is the automation layer that watches Certificate resources, requests certs from some issuer, and handles renewal before expiry; Vault's PKI engine is one of several issuer types cert-manager natively supports (alongside Let's Encrypt, a self-signed CA, or your cloud provider's CA). If you already run Vault for secrets, pointing cert-manager's Issuer at Vault's PKI engine means one fewer CA to operate — but you don't need Vault at all just to get automated TLS; cert-manager with Let's Encrypt covers that case on its own.

Running Vault Itself on Kubernetes

Vault's own data needs somewhere to live, and a sealed Vault is useless until it's unsealed:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault \
    --namespace vault --create-namespace \
    --set "server.ha.enabled=true" \
    --set "server.ha.raft.enabled=true"

Unsealing: Vault's storage is encrypted with a master key, which is itself split into shares via Shamir's Secret Sharing — by default you need a quorum (e.g., 3 of 5) of key-holders to unseal Vault after every restart. This is deliberately manual in the simplest setup, and it's the single biggest operational surprise for teams running Vault for the first time: restart the pod, and it comes back up sealed, serving zero traffic, until someone runs vault operator unseal enough times. Production deployments almost always configure auto-unseal via a cloud KMS (AWS KMS, GCP Cloud KMS, Azure Key Vault) instead, which removes the manual step but means Vault's own availability now depends on that KMS being reachable.

HA with integrated storage: Modern Vault deployments use the Raft integrated storage backend for high availability — no separate Consul cluster required, Vault nodes form their own Raft quorum and elect a leader. This is simpler to operate than the old Vault-plus-Consul pattern, but it's still a stateful, quorum-based system: losing a majority of Raft nodes loses the cluster, the same failure mode as etcd.

When Vault Is Overkill

None of this is free. Running Vault well means: operating a highly-available, quorum-based cluster; having an actual unseal/auto-unseal strategy and testing it; upgrading carefully (Vault upgrades have historically required real attention to release notes); and understanding Shamir shares, leases, and token TTLs well enough to debug a 2am page when a lease didn't renew.

If your actual requirement is "get secrets from AWS Secrets Manager or GCP Secret Manager into Kubernetes Secrets," you don't need any of Vault's dynamic-secrets or PKI machinery — External Secrets Operator talking directly to your cloud provider's native secrets manager is less infrastructure to run and fewer new concepts for the team. Reach for Vault specifically when you need what only Vault does: credentials generated per-request with automatic expiry, encryption-as-a-service decoupled from the application, or a single internal CA across multiple clusters and clouds. If none of those apply, Vault is a second production-grade distributed system you now have to keep alive for no capability you're actually using.

Frequently Asked Questions

Do I need Vault if I already use AWS Secrets Manager or GCP Secret Manager?

Not necessarily. Cloud secrets managers already give you encrypted storage, access control, and (in AWS's case) automatic rotation for specific resource types like RDS. Vault adds value on top of that mainly through dynamic secrets for backends your cloud provider doesn't natively rotate, cross-cloud consistency if you run multi-cloud, and the Transit/PKI engines. If you're single-cloud and your rotation needs are met by what your provider already offers, Vault is additional operational surface for marginal benefit.

What happens if Vault goes down?

Any workload that already holds a valid lease keeps working until that lease expires — Vault being down doesn't instantly revoke live credentials. New logins and new dynamic-secret requests fail until Vault is back. This is why Vault's own availability (HA Raft cluster, tested unseal procedure) matters more as more things depend on it; a single-node dev-mode Vault in production is a single point of failure for everything that authenticates against it.

Is Vault Agent, the CSI provider, or Vault Secrets Operator the right choice for getting secrets into a pod?

Vault Agent (sidecar, renders secrets to a file or environment via templating) gives the most control and works everywhere but adds a sidecar per pod. The Vault CSI Provider mounts secrets as a volume without a sidecar, closest in shape to the Kubernetes Secrets Store CSI Driver pattern. Vault Secrets Operator syncs Vault data into native Kubernetes Secret objects on a reconcile loop — the easiest mental model if your applications already expect a Secret, and the one most directly comparable to External Secrets Operator.

Can Vault's PKI engine replace my public-facing TLS certificates?

No — Vault's PKI engine issues certificates from a CA you control, which is only trusted by clients that trust that CA (internal services, your own cluster). It's for internal/service-to-service TLS, not public-facing certificates a browser needs to trust, which still need a publicly trusted CA like Let's Encrypt.

For the operator-based pattern of getting Vault secrets into native Kubernetes Secrets, see Vault Secrets Operator vs External Secrets Operator. For the broader decision between Vault and other secrets backends, see Secrets Management in Kubernetes: Native Secrets, ESO, Vault, and SOPS Compared. For automated TLS that can use Vault's PKI engine as one of several issuers, see cert-manager: Automated TLS for Kubernetes.

Deciding whether Vault is worth the operational overhead for your team, or need help running it safely in production? Talk to us at Coding Protocols — we help platform teams implement secrets management that matches their actual threat model, not just the most feature-complete option.

Official References

• Kubernetes Auth Method — configuration, the login flow, and token reviewer options
• Database Secrets Engine — dynamic credential generation and lease management
• Transit Secrets Engine — encryption-as-a-service and key rotation