AWS Secrets Manager and Parameter Store: Secrets Management on AWS

Environment variables are the wrong place for secrets. They show up in process listings, get logged in crash dumps, and leak through debug endpoints. The right place for secrets on AWS is either Secrets Manager or Parameter Store — both serve as centralized secret stores, but they have different capabilities, pricing, and integration patterns.

Secrets Manager vs Parameter Store

Use Secrets Manager when: you need automatic rotation (database credentials, API keys), you have secrets that need multi-region replication, or the secrets are accessed frequently and you need managed lifecycle.

Use Parameter Store when: you have configuration values (non-secrets) alongside secrets, you want to avoid per-secret pricing, or you're using SSM Parameter Store for hierarchical configuration management across environments.

Many teams use both: Parameter Store for configuration and non-sensitive values, Secrets Manager for credentials that require rotation.

Secrets Manager

Creating and Retrieving Secrets

1# Create a secret
2aws secretsmanager create-secret \
3  --name prod/payments-api/database \
4  --description "Production payments database credentials" \
5  --secret-string '{"username":"payments","password":"initial-password","host":"prod-db.cluster-abc.us-east-1.rds.amazonaws.com","port":5432,"dbname":"payments"}' \
6  --kms-key-id arn:aws:kms:us-east-1:012345678901:key/mrk-abc123
7
8# Retrieve the secret value
9aws secretsmanager get-secret-value \
10  --secret-id prod/payments-api/database \
11  --query SecretString \
12  --output text | jq .
13
14# Retrieve a specific version
15aws secretsmanager get-secret-value \
16  --secret-id prod/payments-api/database \
17  --version-stage AWSPREVIOUS

Secret names use path-style naming (prod/payments-api/database) for organizational hierarchy. This is convention, not structure — Secrets Manager doesn't treat / as a hierarchy delimiter in access control. Use the full secret name or ARN in IAM policies.

Automatic Rotation

Secrets Manager triggers a Lambda function to rotate the secret. For RDS secrets, AWS provides pre-built rotation Lambda functions you can deploy directly:

1# Enable rotation using the AWS-provided RDS single-user rotation Lambda
2aws secretsmanager rotate-secret \
3  --secret-id prod/payments-api/database \
4  --rotation-lambda-arn arn:aws:lambda:us-east-1:012345678901:function:SecretsManagerRDSPostgreSQLRotationSingleUser \
5  --rotation-rules AutomaticallyAfterDays=30
6
7# Trigger immediate rotation (test that rotation works)
8aws secretsmanager rotate-secret \
9  --secret-id prod/payments-api/database \
10  --rotate-immediately

Rotation Lambda invocation stages:

1. createSecret — creates a new version of the secret with a new password
2. setSecret — sets the new password in the database
3. testSecret — tests that the new credentials work
4. finishSecret — moves the AWSCURRENT label to the new version; the old version becomes AWSPREVIOUS

Applications should retrieve the secret at startup and ideally refresh periodically — during a rotation window, the old credentials remain valid (AWSPREVIOUS) until the next rotation cycle.

Resource Policy (Cross-Account Access)

1# Allow another AWS account to read this secret
2aws secretsmanager put-resource-policy \
3  --secret-id prod/payments-api/database \
4  --resource-policy '{
5    "Version": "2012-10-17",
6    "Statement": [
7      {
8        "Effect": "Allow",
9        "Principal": {
10          "AWS": "arn:aws:iam::111111111111:role/CrossAccountReader"
11        },
12        "Action": [
13          "secretsmanager:GetSecretValue",
14          "secretsmanager:DescribeSecret"
15        ],
16        "Resource": "*"
17      }
18    ]
19  }'

The cross-account principal also needs access to the KMS key used to encrypt the secret:

1{
2  "Effect": "Allow",
3  "Principal": {
4    "AWS": "arn:aws:iam::111111111111:role/CrossAccountReader"
5  },
6  "Action": ["kms:Decrypt", "kms:DescribeKey"],
7  "Resource": "*"
8}

Multi-Region Replication

1# Replicate a secret to another region
2aws secretsmanager replicate-secret-to-regions \
3  --secret-id prod/payments-api/database \
4  --add-replica-regions Region=us-west-2,KmsKeyId=arn:aws:kms:us-west-2:012345678901:key/mrk-replica-key
5
6# List replicas
7aws secretsmanager describe-secret \
8  --secret-id prod/payments-api/database \
9  --query ReplicationStatus

Replicas are read-only. Write operations (rotation, updates) only happen to the primary. The replica is synchronized automatically.

Parameter Store

Hierarchy and Types

1# Create a SecureString parameter (encrypted with KMS)
2aws ssm put-parameter \
3  --name /prod/payments-api/database-url \
4  --value "postgresql://payments:password@prod-db.cluster.us-east-1.rds.amazonaws.com/payments" \
5  --type SecureString \
6  --key-id arn:aws:kms:us-east-1:012345678901:key/mrk-abc123 \
7  --description "Production payments database connection URL"
8
9# Create a String parameter (unencrypted — use for non-sensitive config)
10aws ssm put-parameter \
11  --name /prod/payments-api/feature-flags \
12  --value '{"new-checkout":true,"legacy-payments":false}' \
13  --type String
14
15# Retrieve a parameter
16aws ssm get-parameter \
17  --name /prod/payments-api/database-url \
18  --with-decryption \
19  --query Parameter.Value \
20  --output text
21
22# Retrieve all parameters under a path
23aws ssm get-parameters-by-path \
24  --path /prod/payments-api/ \
25  --with-decryption \
26  --recursive

Parameter Store hierarchies allow fetching all config for an environment in one API call. Organize by /env/service/key — this enables IAM policies that grant access to arn:aws:ssm:*:*:parameter/prod/payments-api/* without listing individual parameters.

Versioning

Parameter Store retains all versions. The current version is always retrieved by default:

1# Retrieve a specific version
2aws ssm get-parameter \
3  --name /prod/payments-api/database-url:5 \    # Version 5
4  --with-decryption
5
6# List version history
7aws ssm get-parameter-history \
8  --name /prod/payments-api/database-url

To roll back to a previous value, read the previous version and put-parameter with --overwrite.

EKS Integration

External Secrets Operator

External Secrets Operator (ESO) synchronizes secrets from Secrets Manager and Parameter Store into Kubernetes Secrets. ESO uses IRSA for authentication.

1# ClusterSecretStore — cluster-wide store (or SecretStore for namespace-scoped)
2apiVersion: external-secrets.io/v1beta1
3kind: ClusterSecretStore
4metadata:
5  name: aws-secrets-manager
6spec:
7  provider:
8    aws:
9      service: SecretsManager
10      region: us-east-1
11      auth:
12        jwt:
13          serviceAccountRef:
14            name: external-secrets
15            namespace: external-secrets

1# ExternalSecret — syncs from Secrets Manager to Kubernetes Secret
2apiVersion: external-secrets.io/v1beta1
3kind: ExternalSecret
4metadata:
5  name: payments-database-credentials
6  namespace: payments
7spec:
8  refreshInterval: 1h    # How often ESO re-syncs the secret
9  secretStoreRef:
10    name: aws-secrets-manager
11    kind: ClusterSecretStore
12  target:
13    name: payments-database-secret    # Kubernetes Secret to create/update
14    creationPolicy: Owner
15  data:
16    - secretKey: username             # Key in the Kubernetes Secret
17      remoteRef:
18        key: prod/payments-api/database    # Secrets Manager secret name
19        property: username                 # JSON key within the secret
20    - secretKey: password
21      remoteRef:
22        key: prod/payments-api/database
23        property: password

ESO creates and updates the Kubernetes Secret on a schedule. When Secrets Manager rotates the secret, ESO picks up the new value within refreshInterval and updates the Kubernetes Secret. Pods that mount the Secret as a volume see the updated file on disk automatically (without pod restart, if using projected volumes) — but applications that read and cache the value at startup will still need to re-read the file or restart to use the new credentials. Pods that consume the Secret as environment variables always need to restart to pick up the new value.

Secrets Store CSI Driver

The AWS Secrets Store CSI Driver mounts secrets directly as files into pods from Secrets Manager or Parameter Store, without creating a Kubernetes Secret object:

1# SecretProviderClass — defines what to fetch
2apiVersion: secrets-store.csi.x-k8s.io/v1
3kind: SecretProviderClass
4metadata:
5  name: payments-secrets
6  namespace: payments
7spec:
8  provider: aws
9  parameters:
10    objects: |
11      - objectName: "prod/payments-api/database"
12        objectType: "secretsmanager"
13        jmesPath:
14          - path: username
15            objectAlias: db-username
16          - path: password
17            objectAlias: db-password
18  secretObjects:
19    - secretName: payments-database-secret    # Also sync to Kubernetes Secret
20      type: Opaque
21      data:
22        - objectName: db-username
23          key: username
24        - objectName: db-password
25          key: password

1# Pod configuration to use the CSI driver
2spec:
3  volumes:
4    - name: secrets-store
5      csi:
6        driver: secrets-store.csi.k8s.io
7        readOnly: true
8        volumeAttributes:
9          secretProviderClass: payments-secrets
10  containers:
11    - name: payments-api
12      volumeMounts:
13        - name: secrets-store
14          mountPath: /mnt/secrets
15          readOnly: true
16      # Files available at /mnt/secrets/db-username and /mnt/secrets/db-password

ESO vs CSI Driver:

ESO is simpler to operate and integrates better with existing workflows that expect Kubernetes Secrets. CSI driver avoids creating K8s Secret objects in etcd (relevant for compliance requirements around secret storage).

Audit Logging

Every Secrets Manager and SSM Parameter Store API call is logged in CloudTrail:

# Find who accessed a secret in the last 24 hours
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=prod/payments-api/database \
  --start-time "$(date -u -d '-24 hours' +%Y-%m-%dT%H:%M:%SZ)" \
  --query 'Events[].{Time:EventTime,User:Username,Action:EventName,Source:SourceIPAddress}'

CloudTrail logs include: who accessed the secret, from which IP, when, and what action (GetSecretValue, RotateSecret, DeleteSecret, etc.). Set up CloudWatch alarms or Security Hub alerts for unexpected GetSecretValue calls from unusual principals or IPs.

Frequently Asked Questions

How do I prevent secrets from appearing in CloudTrail logs?

CloudTrail logs the API call metadata but not the secret value itself — GetSecretValue is logged, but the actual password is not included in the CloudTrail event. The call metadata (who called, when, from where) is logged, which is the intended behavior for auditing.

Secrets Manager does not log the secret value to CloudTrail. Parameter Store with GetParameter --with-decryption also does not log the decrypted value.

What's the difference between AWSCURRENT and AWSPREVIOUS?

Secrets Manager uses staging labels to track versions:

• AWSCURRENT: the active version — always returned by default
• AWSPREVIOUS: the previous version — remains valid during rotation so applications can continue working
• AWSPENDING: created during rotation before it's promoted to AWSCURRENT

During a rotation cycle, the finishSecret step moves the AWSCURRENT label to the newly rotated version — the previous AWSCURRENT version receives the AWSPREVIOUS label, and whatever version previously held AWSPREVIOUS becomes unlabeled. This means two credential versions are simultaneously valid during the brief window between setSecret and finishSecret, giving applications that cache credentials time to continue working without an immediate restart.

How do I handle secret rotation in applications that cache credentials?

Two patterns:

1. Retry on auth failure: when the application gets an authentication error (database auth failure, API 401), it fetches fresh credentials from Secrets Manager and retries. This handles rotation gracefully without needing to know when rotation happens.

2. Background refresh: a background goroutine/thread refreshes credentials from Secrets Manager every N minutes (set N to less than the rotation interval). This proactively stays ahead of rotation.

The retry pattern is simpler and more resilient. The background refresh is appropriate when auth failures are expensive (e.g., database connection pools that take time to establish).

Can I use Parameter Store for Lambda environment variables?

Yes — and this is a common pattern for Lambda configuration. Reference Parameter Store values in CloudFormation or Terraform:

# CloudFormation — resolve at deploy time
DatabaseUrl: "{{resolve:ssm:/prod/payments-api/database-url}}"

# CloudFormation — resolve SecureString at deploy time
DatabasePassword: "{{resolve:ssm-secure:/prod/payments-api/db-password:5}}"

The {{resolve:ssm-secure:...}} dynamic reference resolves the parameter at CloudFormation deployment time and injects it as the Lambda environment variable value — the value is stored encrypted in the Lambda configuration.

For runtime access to Parameter Store from Lambda, use the Lambda Extensions for Parameter Store and Secrets Manager — a Lambda layer that provides a local cache, reducing API calls and latency.

For EKS pods that authenticate to Secrets Manager using IRSA, see AWS IAM: Roles, Policies, Permission Boundaries, and IRSA for EKS. For syncing Secrets Manager secrets into Kubernetes Secrets automatically, see External Secrets Operator on Kubernetes.

Setting up automatic rotation for RDS credentials, integrating Secrets Manager into EKS with External Secrets Operator, or designing a cross-account secret sharing architecture? Talk to us at Coding Protocols — we help platform teams implement secrets management that rotates credentials automatically without disrupting running services.