AWS Secrets Manager Rotation Patterns: Zero-Downtime Credential Rotation at Scale

A security audit found that 67% of our production database credentials had not been rotated in over 18 months. Several API keys dated back to the initial provisioning three years earlier. The risk was clear: any leaked credential provided indefinite access. But manual rotation was operationally terrifying — rotate the wrong credential at the wrong time, and a production database becomes unreachable.

We implemented automated rotation for 143 secrets using AWS Secrets Manager with custom rotation Lambda functions. Credentials now rotate every 30 days with zero application downtime. The key was the multi-user rotation strategy that keeps both old and new credentials valid during the transition window.

The Problem: Rotation Fear Creates Stale Credentials

Teams avoid credential rotation because the risk of breaking production outweighs the security benefit in their minds. The common failure modes:

• Application caches old credential — Database password rotates, but the app holds the old connection string in memory. Next reconnection attempt fails.
• Deployment timing collision — Credential rotates mid-deployment. Blue-green environments use different credential versions.
• Downstream dependency cascade — Service A's credential rotates, but Service B still references the old version through a hardcoded parameter.
• Rotation function failure — The Lambda function that performs rotation errors out, leaving the credential in a half-rotated state.

Each of these has caused production incidents at our organization. The solution is not to avoid rotation — it is to make rotation safe by design.

Architecture: Multi-User Rotation Strategy

For database credentials, we use the alternating user strategy. Instead of a single database user whose password rotates in-place, we maintain two users that alternate:

• User A (current) — Active credential that applications use
• User B (alternate) — Previous credential that remains valid during transition

When rotation occurs:

1. User B's password is changed to a new value
2. The secret's AWSCURRENT label moves to User B's new credential
3. Applications pick up User B on next secret fetch
4. User A remains valid (now labeled AWSPREVIOUS) until the next rotation

This means at no point is there a window where the active credential is invalid.

Implementation: Custom Rotation Lambda

import {
  SecretsManagerClient,
  GetSecretValueCommand,
  PutSecretValueCommand,
  UpdateSecretVersionStageCommand,
  DescribeSecretCommand,
} from '@aws-sdk/client-secrets-manager';
import { Client } from 'pg';
import { randomBytes } from 'crypto';

interface SecretValue {
  engine: string;
  host: string;
  port: number;
  dbname: string;
  username: string;
  password: string;
  masterarn?: string;
}

type RotationStep = 'createSecret' | 'setSecret' | 'testSecret' | 'finishSecret';

export async function handler(event: {
  SecretId: string;
  ClientRequestToken: string;
  Step: RotationStep;
}): Promise<void> {
  const client = new SecretsManagerClient({ region: 'us-east-1' });
  const { SecretId, ClientRequestToken, Step } = event;

  // Verify the secret is enabled for rotation
  const metadata = await client.send(
    new DescribeSecretCommand({ SecretId })
  );
  if (!metadata.RotationEnabled) {
    throw new Error(`Secret ${SecretId} is not enabled for rotation`);
  }

  switch (Step) {
    case 'createSecret':
      await createSecret(client, SecretId, ClientRequestToken);
      break;
    case 'setSecret':
      await setSecret(client, SecretId, ClientRequestToken);
      break;
    case 'testSecret':
      await testSecret(client, SecretId, ClientRequestToken);
      break;
    case 'finishSecret':
      await finishSecret(client, SecretId, ClientRequestToken);
      break;
  }
}

async function createSecret(
  client: SecretsManagerClient,
  secretId: string,
  token: string
): Promise<void> {
  // Get current secret value
  const current = await getSecretValue(client, secretId, 'AWSCURRENT');
  const currentSecret: SecretValue = JSON.parse(current);

  // Generate new password
  const newPassword = generateSecurePassword(32);

  // Determine alternate username
  const alternateUsername = currentSecret.username.endsWith('_a')
    ? currentSecret.username.replace('_a', '_b')
    : currentSecret.username.replace('_b', '_a');

  // Create new secret version with alternate user
  const newSecret: SecretValue = {
    ...currentSecret,
    username: alternateUsername,
    password: newPassword,
  };

  await client.send(
    new PutSecretValueCommand({
      SecretId: secretId,
      ClientRequestToken: token,
      SecretString: JSON.stringify(newSecret),
      VersionStages: ['AWSPENDING'],
    })
  );
}

async function setSecret(
  client: SecretsManagerClient,
  secretId: string,
  token: string
): Promise<void> {
  // Get the pending secret (new credentials)
  const pending = await getSecretValue(client, secretId, 'AWSPENDING');
  const pendingSecret: SecretValue = JSON.parse(pending);

  // Get master credentials to perform the password change
  const masterArn = JSON.parse(
    await getSecretValue(client, secretId, 'AWSCURRENT')
  ).masterarn;
  const masterSecret: SecretValue = JSON.parse(
    await getSecretValue(client, masterArn!, 'AWSCURRENT')
  );

  // Connect as master and change the alternate user's password
  const db = new Client({
    host: masterSecret.host,
    port: masterSecret.port,
    database: masterSecret.dbname,
    user: masterSecret.username,
    password: masterSecret.password,
    ssl: { rejectUnauthorized: true },
  });

  await db.connect();
  await db.query(
    `ALTER USER ${pendingSecret.username} WITH PASSWORD '${pendingSecret.password}'`
  );
  await db.end();
}

async function testSecret(
  client: SecretsManagerClient,
  secretId: string,
  token: string
): Promise<void> {
  const pending = await getSecretValue(client, secretId, 'AWSPENDING');
  const pendingSecret: SecretValue = JSON.parse(pending);

  // Verify the new credentials work
  const db = new Client({
    host: pendingSecret.host,
    port: pendingSecret.port,
    database: pendingSecret.dbname,
    user: pendingSecret.username,
    password: pendingSecret.password,
    ssl: { rejectUnauthorized: true },
  });

  await db.connect();
  const result = await db.query('SELECT 1 AS connected');
  if (result.rows[0].connected !== 1) {
    throw new Error('Connection test failed');
  }
  await db.end();
}

async function finishSecret(
  client: SecretsManagerClient,
  secretId: string,
  token: string
): Promise<void> {
  // Move AWSCURRENT to the new version
  await client.send(
    new UpdateSecretVersionStageCommand({
      SecretId: secretId,
      VersionStage: 'AWSCURRENT',
      MoveToVersionId: token,
      RemoveFromVersionId: await getCurrentVersionId(client, secretId),
    })
  );
}

function generateSecurePassword(length: number): string {
  const chars =
    'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789!@#$%^&*';
  const bytes = randomBytes(length);
  return Array.from(bytes)
    .map((b) => chars[b % chars.length])
    .join('');
}

Application-Side: Handling Rotation Gracefully

Applications must handle the moment when the cached credential becomes the previous version. We use a retry-with-refresh pattern:

import {
  SecretsManagerClient,
  GetSecretValueCommand,
} from '@aws-sdk/client-secrets-manager';
import { Client } from 'pg';

class ResilientDatabaseConnection {
  private client: Client | null = null;
  private cachedSecret: string | null = null;
  private lastFetchTime = 0;
  private readonly secretId: string;
  private readonly secretsClient: SecretsManagerClient;
  private readonly cacheTtlMs = 300_000; // 5 minutes

  constructor(secretId: string) {
    this.secretId = secretId;
    this.secretsClient = new SecretsManagerClient({ region: 'us-east-1' });
  }

  async query(sql: string, params?: unknown[]): Promise<any> {
    try {
      const conn = await this.getConnection();
      return await conn.query(sql, params);
    } catch (error: any) {
      if (this.isAuthError(error)) {
        // Credential may have rotated — force refresh and retry
        this.cachedSecret = null;
        this.client = null;
        const conn = await this.getConnection();
        return await conn.query(sql, params);
      }
      throw error;
    }
  }

  private async getConnection(): Promise<Client> {
    const secret = await this.getSecret();

    if (!this.client || this.client.ended) {
      const parsed = JSON.parse(secret);
      this.client = new Client({
        host: parsed.host,
        port: parsed.port,
        database: parsed.dbname,
        user: parsed.username,
        password: parsed.password,
        ssl: { rejectUnauthorized: true },
      });
      await this.client.connect();
    }

    return this.client;
  }

  private async getSecret(): Promise<string> {
    const now = Date.now();
    if (this.cachedSecret && now - this.lastFetchTime < this.cacheTtlMs) {
      return this.cachedSecret;
    }

    const response = await this.secretsClient.send(
      new GetSecretValueCommand({ SecretId: this.secretId })
    );
    this.cachedSecret = response.SecretString!;
    this.lastFetchTime = now;
    return this.cachedSecret;
  }

  private isAuthError(error: any): boolean {
    return (
      error.code === '28P01' || // PostgreSQL authentication failed
      error.message?.includes('password authentication failed')
    );
  }
}

The 5-minute cache TTL means an application will pick up the new credential within 5 minutes of rotation. The retry-on-auth-error pattern handles the edge case where rotation occurs between cache refresh and query execution.

Rotation Schedule Strategy

Not all secrets rotate at the same frequency:

Secret Type	Rotation Interval	Strategy	Count
Database credentials	30 days	Multi-user alternating	24
API keys (internal)	90 days	Single-user with overlap	47
API keys (third-party)	Manual trigger	Single-user with notification	18
Service account tokens	7 days	Single-user immediate	32
TLS certificates	60 days	ACM auto-renewal	22
Total	—	—	143

We stagger rotations so no more than 5 secrets rotate on the same day. This limits blast radius if a rotation function has a bug.

Monitoring: Catching Rotation Failures

Failed rotations leave secrets in a AWSPENDING state. We monitor for this:

Monitor	Threshold	Action
Secret age > rotation period + 7 days	Any secret	P2 alert to security team
Rotation Lambda errors	> 0	P1 alert to platform team
AWSPENDING state duration > 1 hour	Any secret	P1 alert (stuck rotation)
Application auth failures spike	3x baseline	Automatic secret refresh trigger

Results: 180-Day Transformation

Metric	Before	After	Change
Secrets never rotated	96 (67%)	0	Eliminated
Average credential age	14.2 months	22 days	-95%
Rotation-caused incidents	N/A	0 (180 days)	Zero
Manual rotation effort	4 hrs/month	0	Eliminated
Compliance audit findings	12	0	Eliminated
Monthly Secrets Manager cost	$57	$143 (more secrets managed)	+$86

The $86/month increase covers the rotation Lambda invocations and additional secret versions stored. Trivial compared to the compliance risk of stale credentials.

Lessons Learned

Multi-user rotation is mandatory for databases. Single-user rotation has a brief window where the old password is invalid but cached by applications. Even with sub-second rotation, this window causes errors under high concurrency. Multi-user eliminates the window entirely.

Test rotation in staging weekly. Our rotation Lambda has caught 3 bugs in 180 days. All were discovered in staging during scheduled test rotations, not in production. The staging rotation schedule is 7 days for all secrets regardless of production cadence.

Secrets Manager caching SDK reduces API calls by 99%. Without caching, every Lambda invocation calls GetSecretValue. With the caching client, it calls once per TTL period. This reduced our Secrets Manager API costs from $42/month to $0.40/month.

Document the rotation runbook before automating. We wrote manual rotation procedures for every secret type first. The automation Lambda is just that runbook encoded. When automation fails, engineers can follow the manual runbook without deciphering Lambda code under incident pressure.

Separate master credentials from application credentials. The rotation Lambda needs a master credential to change other users' passwords. This master credential is itself rotated but on a longer schedule (90 days) and stored in a separate secret with restricted access.

Conclusion

Automated secret rotation eliminates the tension between security requirements and operational stability. The multi-user rotation pattern ensures zero-downtime transitions, the retry-with-refresh application pattern handles edge cases gracefully, and comprehensive monitoring catches failures before they impact production. For any organization with compliance requirements around credential rotation, this pattern transforms a quarterly fire drill into an invisible automated process running 143 times per month without a single incident.