Kubernetes ConfigMap and Secret Best Practices: What Most Teams Get Wrong

Kubernetes ConfigMap and Secret are the two built-in mechanisms for injecting configuration into pods. They work. They're also misused in ways that create security exposures and operational debt — secrets stored in Git, environment variables that appear in crash logs, base64 values that developers treat as encrypted.

This post covers the mechanics, the correct usage patterns, and the specific mistakes that cause the most problems in production.

ConfigMap vs Secret: The Actual Difference

Both ConfigMap and Secret store key-value data and can be injected into pods as environment variables or mounted as files. The differences:

Secret data is base64-encoded at rest in the Kubernetes API. This is encoding, not encryption. Anyone who can read the Secret object (via kubectl get secret -o yaml or the API) sees the decoded value. base64 provides no confidentiality.

Secrets get separate RBAC treatment. You can grant a service account access to configmaps without granting access to secrets. This allows meaningful RBAC separation — application code that only needs configuration doesn't need secrets read access.

Secrets can be encrypted at rest via Kubernetes' encryption configuration (EncryptionConfig), which encrypts Secret data in etcd using AES-GCM or envelope encryption with an external KMS. ConfigMaps are not encrypted at rest even if you configure Secret encryption.

The operational implication: never put sensitive values in ConfigMaps. Database URLs without credentials are fine in ConfigMaps. Database credentials belong in Secrets. The distinction matters because Secret encryption, Secret RBAC, and external secret management tools (Vault, ESO) all target the Secret resource type.

ConfigMap Patterns

Basic Configuration

1apiVersion: v1
2kind: ConfigMap
3metadata:
4  name: app-config
5  namespace: production
6data:
7  LOG_LEVEL: "info"
8  MAX_CONNECTIONS: "100"
9  FEATURE_FLAGS: |
10    new_checkout=true
11    dark_mode=false
12  app.properties: |
13    server.port=8080
14    server.timeout=30s
15    database.pool.min=5
16    database.pool.max=20

Mounting as Files vs Environment Variables

Environment variables are simple but have limitations:

spec:
  containers:
    - name: app
      envFrom:
        - configMapRef:
            name: app-config

envFrom injects all keys as environment variables. Individual keys:

env:
  - name: LOG_LEVEL
    valueFrom:
      configMapKeyRef:
        name: app-config
        key: LOG_LEVEL

Limitations of env vars:

• Changes to the ConfigMap do not automatically update env vars in running pods — the pod must be restarted
• Env vars appear in process listings (/proc/<pid>/environ) and can leak into logs via error messages or debug output
• The 1MB ConfigMap size limit affects how much you can inject via envFrom

File mounts update automatically. When a ConfigMap is mounted as a volume, Kubernetes updates the file contents when the ConfigMap changes (within ~1–2 minutes, via the kubelet's sync period). Applications that watch config files for changes can reload configuration without pod restarts.

1spec:
2  containers:
3    - name: app
4      volumeMounts:
5        - name: config
6          mountPath: /etc/app/config
7          readOnly: true
8  volumes:
9    - name: config
10      configMap:
11        name: app-config
12        items:
13          - key: app.properties
14            path: app.properties

For configuration that changes without redeployment (feature flags, tunable thresholds), file mounts with application-side file watching are the right pattern.

Immutable ConfigMaps

Mark ConfigMaps as immutable once they're finalised:

1apiVersion: v1
2kind: ConfigMap
3metadata:
4  name: app-config-v2
5immutable: true
6data:
7  # ...

Immutable ConfigMaps:

• Cannot be modified after creation — changes require creating a new ConfigMap and updating pod references
• Are not watched by the kubelet, reducing API server load at scale
• Prevent accidental in-place modifications of configuration that should be versioned

This pattern works well combined with versioned naming (app-config-v1, app-config-v2) and GitOps — each config change creates a new versioned ConfigMap and updates the pod reference in the same commit.

Secret Patterns

Creating Secrets Correctly

1# From literal values — safe, no value in shell history if using --from-literal
2kubectl create secret generic db-credentials \
3  --namespace production \
4  --from-literal=username=myapp \
5  --from-literal=password='s3cr3t-p@ssword'
6
7# From files — useful for certificates, SSH keys
8kubectl create secret generic tls-cert \
9  --namespace production \
10  --from-file=tls.crt=./cert.pem \
11  --from-file=tls.key=./key.pem

Do not create Secrets from YAML manifests committed to Git. The base64 values in the manifest are trivially decoded:

echo "czNjcjN0LXBAc3N3b3Jk" | base64 -d
# s3cr3t-p@ssword

Anyone with read access to the repository has the credentials. Use sealed-secrets, SOPS, or an external secret management system (ESO, Vault) for any Secret that needs to be represented in Git.

Injecting Secrets into Pods

Prefer volume mounts over environment variables for secrets.

Environment variables are accessible to all processes in the container, appear in crash reports, and are logged by various runtime tools. A secret mounted as a file is readable only when the application explicitly reads it.

1# Avoid for sensitive values
2env:
3  - name: DB_PASSWORD
4    valueFrom:
5      secretKeyRef:
6        name: db-credentials
7        key: password

1# Prefer — secret mounted as file, read by application
2spec:
3  containers:
4    - name: app
5      volumeMounts:
6        - name: db-secret
7          mountPath: /run/secrets/db
8          readOnly: true
9  volumes:
10    - name: db-secret
11      secret:
12        secretName: db-credentials
13        defaultMode: 0400   # Read-only by owner

The application reads /run/secrets/db/password at startup or on-demand. The value is not in the process environment. defaultMode: 0400 ensures only the process owner can read the file.

Disabling Automatic Service Account Token Mounting

Every pod gets a mounted service account token by default at /var/run/secrets/kubernetes.io/serviceaccount/token. If your application doesn't call the Kubernetes API, this token is unnecessary and is an attack surface if the pod is compromised.

spec:
  automountServiceAccountToken: false
  containers:
    # ...

Or on the ServiceAccount itself:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app
  namespace: production
automountServiceAccountToken: false

Immutable Secrets

Same pattern as ConfigMaps — mark production secrets as immutable once set, use versioned naming for rotation:

1apiVersion: v1
2kind: Secret
3metadata:
4  name: db-credentials-v3
5  namespace: production
6immutable: true
7type: Opaque
8data:
9  username: bXlhcHA=
10  password: czNjcjN0

The 1MB Limit

ConfigMaps and Secrets are stored in etcd, which has a default value size limit of 1.5MB (configurable, but often left at default). The Kubernetes API imposes a 1MB limit on ConfigMap and Secret objects.

This becomes relevant when:

• Storing large configuration files (ML model configs, exhaustive lookup tables)
• Storing certificate bundles with many certificates
• Using ConfigMaps as a makeshift key-value store

Solutions for data exceeding 1MB:

• Store in an object store (S3, GCS) and have the application fetch at startup
• Split into multiple ConfigMaps and mount multiple volumes
• Use a proper configuration management system (Consul, AWS AppConfig)

Secret Rotation

Secrets should be rotated. The mechanism:

1. Create a new Secret version: db-credentials-v4 with the new password
2. Update the database to accept both the old and new password simultaneously
3. Update the pod volumeMount or secretKeyRef to reference db-credentials-v4
4. Trigger a rolling restart: kubectl rollout restart deployment/my-app -n production
5. Verify pods are running and using the new credentials
6. Remove the old password from the database
7. Delete the old Secret: kubectl delete secret db-credentials-v3 -n production

The critical step is maintaining backward compatibility in the database (step 2) so there's no gap where old pods use an invalid credential while new pods start.

For automated rotation, External Secrets Operator with Vault or AWS Secrets Manager handles this lifecycle — the external secret manager rotates the value and ESO syncs the updated value into the Kubernetes Secret.

External Secret Management

For production, Kubernetes Secrets should be sourced from an external secret manager rather than managed directly. The two primary tools:

External Secrets Operator (ESO) syncs secrets from external stores (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, HashiCorp Vault, 1Password) into Kubernetes Secrets on a schedule:

1apiVersion: external-secrets.io/v1beta1
2kind: ExternalSecret
3metadata:
4  name: db-credentials
5  namespace: production
6spec:
7  refreshInterval: 1h
8  secretStoreRef:
9    name: aws-secrets-manager
10    kind: ClusterSecretStore
11  target:
12    name: db-credentials        # Creates/updates this Kubernetes Secret
13    creationPolicy: Owner
14  data:
15    - secretKey: password
16      remoteRef:
17        key: production/db/credentials
18        property: password

Vault Secrets Operator (VSO) does the same for HashiCorp Vault, with native Vault dynamic secret support (credentials generated on-demand with short TTLs, automatically rotated).

For the detailed comparison see Secrets Management in Kubernetes: Vault vs ESO vs SOPS.

The key principle: the source of truth for secrets is the external secret manager, not Kubernetes. Kubernetes Secrets are a cache. If a Kubernetes Secret is deleted, ESO or VSO recreates it from the external store on the next sync.

Secrets in GitOps Pipelines

The challenge: GitOps (Argo CD, Flux) syncs everything in Git to the cluster. Secrets can't be stored in Git in plaintext. Three approaches:

Sealed Secrets (Bitnami) — encrypts the Secret with a cluster-specific public key. The encrypted SealedSecret object is safe to commit to Git. The Sealed Secrets controller decrypts it in the cluster.

kubeseal --cert <cluster-cert.pem> \
  --format yaml < secret.yaml > sealed-secret.yaml
# sealed-secret.yaml is safe to commit

SOPS — encrypts Secret values in YAML manifests using AWS KMS, GCP KMS, Azure Key Vault, or PGP. The manifest is committed encrypted; Argo CD or Flux decrypt it during apply using the cluster's KMS access.

External Secrets Operator — don't store secrets in Git at all. Store ExternalSecret manifests (which contain no sensitive data, just references to the external secret manager). ESO fetches and syncs the actual values at runtime.

The recommended approach for new setups: ESO with a cloud-native secret manager (AWS Secrets Manager, GCP Secret Manager). It has the clearest security model — secrets never touch Git, the cluster is a consumer, not a store.

Auditing What's in Your Secrets

1# List all secrets in a namespace with their types and ages
2kubectl get secrets -n production -o wide
3
4# Check which secrets are referenced by pods
5kubectl get pods -n production -o json | \
6  jq '.items[].spec | {
7    name: .containers[].name,
8    envSecrets: [.containers[].env[]? | select(.valueFrom.secretKeyRef) | .valueFrom.secretKeyRef.name],
9    volSecrets: [.volumes[]? | select(.secret) | .secret.secretName]
10  }'
11
12# Find secrets not referenced by any pod (candidates for cleanup)
13kubectl get secrets -n production -o name | while read secret; do
14  name=$(echo $secret | cut -d/ -f2)
15  refs=$(kubectl get pods -n production -o json | \
16    jq --arg s "$name" '[.items[].spec.volumes[]? | select(.secret.secretName == $s)] | length')
17  if [ "$refs" -eq "0" ]; then
18    echo "Unreferenced: $name"
19  fi
20done

Frequently Asked Questions

Is it safe to use kubectl create secret in CI/CD?

If the CI/CD system has credentials to call kubectl create secret, those credentials have access to create arbitrary Secrets in the cluster. This is a significant privilege. Prefer ESO or Vault — your CI system pushes to the external secret manager, and the cluster pulls from it. The CI system never needs direct Kubernetes API access to manage secrets.

Why does kubectl describe secret not show values?

describe masks Secret values by design. Use kubectl get secret <name> -o jsonpath='{.data.<key>}' | base64 -d to retrieve a specific value. This is intentional friction — it forces you to be deliberate about extracting secret values.

Can I use a ConfigMap for feature flags?

Yes, and file-mounted ConfigMaps with application-side file watching are a reasonable feature flag implementation for teams that don't need a dedicated feature flag service. The limitation: ConfigMap changes take up to the kubelet sync period (~1–2 minutes) to propagate to mounted files, and longer for env var-based injection (requires pod restart). For sub-second flag propagation, use a dedicated feature flag service.

What's the difference between Opaque and other Secret types?

Opaque is the generic type — arbitrary key-value data. Other types have specific semantics:

• kubernetes.io/tls — expects tls.crt and tls.key keys; used by Ingress and Gateway API for TLS termination
• kubernetes.io/dockerconfigjson — registry credentials for private image pulls
• kubernetes.io/service-account-token — service account token (managed automatically)

Using the correct type helps tools and controllers identify secret purpose, but doesn't add any enforcement beyond the expected key structure.

For external secret management, see Secrets Management in Kubernetes: Vault vs ESO vs SOPS. For RBAC controls on secret access, see Kubernetes RBAC in Practice.

Building a secrets management strategy for a multi-team Kubernetes platform? Talk to us at Coding Protocols — we help platform teams move from ad-hoc Secret management to a defensible, auditable approach.