Secrets Rotation: Why Rotating Credentials Should Be Automatic

A credential that never changes is a credential that, once compromised, stays compromised indefinitely. The attacker who obtained your database password six months ago can still access your database today, if you have not rotated it. The API key leaked in a git commit three years ago — before you caught it — may still be valid. The former employee who memorized the staging database password can still use it.

Secrets rotation limits the window of exposure. If a credential is compromised and you rotate every 90 days, the maximum exposure window is 90 days, not infinite. If you rotate on a 30-day schedule, it is 30 days. If you have automated rotation with fresh credentials every day, it is 24 hours.

The catch is that manual rotation is operationally painful enough that it does not happen consistently. You write it down as something to do quarterly, the quarter ends, you put it on next quarter's list, and two years later the credentials have never changed. Automation solves this.

What Gets Rotated and How Often

Database passwords: rotate quarterly minimum. Monthly for sensitive systems. AWS RDS, Supabase, and most managed database providers have mechanisms for rotation without taking the database offline.

JWT signing secrets: rotate annually or whenever team membership changes significantly. Rotation invalidates existing tokens — plan for a graceful transition period where both old and new signing keys are valid, allowing active sessions to transition naturally.

API keys for third-party services: rotate when team members with access leave, when you suspect compromise, and annually as a baseline. Use service accounts with limited permissions rather than personal API keys — service account keys can be rotated without affecting personal access.

Internal service-to-service credentials: rotate frequently, ideally with short-lived credentials generated on demand. AWS IAM roles with instance profiles are better than long-lived access keys because they rotate automatically.

TLS certificates: Let's Encrypt certificates expire every 90 days. Automate renewal via Certbot or your platform's certificate management. Set up monitoring that alerts 14 days before expiry as a backup to catch any automation failure.

Encryption keys: key rotation for data encryption is complex because rotating the key requires re-encrypting all encrypted data. Implement key rotation through envelope encryption — rotate the key encryption key (KEK) without re-encrypting the data encryption keys immediately. Build a background re-encryption job that gradually migrates to the new KEK.

Rotating Database Passwords Without Downtime

The challenge with database password rotation is that your application needs to connect to the database. Changing the password while the application is running causes connection failures until you restart the application with the new password. On a single-instance deployment, there is a brief outage. On a multi-instance deployment, instances get inconsistent credentials.

The solution is a rotation strategy that maintains both old and new credentials briefly:

Two-phase rotation:

Generate a new password
Add the new password as an alternative authentication credential for the database user
Update your secrets manager with the new password
Deploy/restart your application so it picks up the new password
Remove the old password from the database user

This means the database accepts both passwords during the transition window. No connection failures during rotation.

AWS Secrets Manager automates this for RDS databases with Lambda rotation functions:

{
  "SecretId": "production/api/database-password",
  "RotationLambdaARN": "arn:aws:lambda:us-east-1:123456789:function:SecretsManagerRDSRotation",
  "RotationRules": {
    "AutomaticallyAfterDays": 30
  }
}

The Lambda function handles the two-phase rotation automatically. Your application uses Secrets Manager's SDK to fetch the current secret — Secrets Manager transparently serves the current valid credential.

Rotating JWT Signing Secrets

JWT rotation requires careful handling because existing tokens are signed with the old secret. If you immediately invalidate the old secret, every user is logged out and must re-authenticate.

Graceful JWT rotation uses a key identifier (kid) claim to allow multiple valid signing keys simultaneously:

interface KeyPair {
  kid: string;
  secret: string;
  createdAt: Date;
  expiresAt: Date;
}

class JwtKeyManager {
  private keys: Map<string, KeyPair> = new Map();

  constructor() {
    this.loadKeys();
  }

  private loadKeys() {
    // Load current and previous key from secrets manager
    const keys = JSON.parse(process.env.JWT_KEYS!);
    keys.forEach((key: KeyPair) => this.keys.set(key.kid, key));
  }

  sign(payload: object): string {
    const currentKey = this.getCurrentKey();
    return jwt.sign(payload, currentKey.secret, {
      algorithm: "HS256",
      keyid: currentKey.kid,
      expiresIn: "15m",
    });
  }

  verify(token: string): jwt.JwtPayload {
    const decoded = jwt.decode(token, { complete: true });
    if (!decoded || typeof decoded === "string") {
      throw new Error("Invalid token");
    }

    const kid = decoded.header.kid as string;
    const key = this.keys.get(kid);

    if (!key || new Date() > key.expiresAt) {
      throw new Error("Invalid or expired signing key");
    }

    return jwt.verify(token, key.secret) as jwt.JwtPayload;
  }

  private getCurrentKey(): KeyPair {
    // Return the newest non-expired key
    return Array.from(this.keys.values())
      .sort((a, b) => b.createdAt.getTime() - a.createdAt.getTime())[0];
  }
}

The rotation process:

Generate a new signing key with a new kid
Add it to your secrets manager alongside the current key
Restart your application — new tokens are signed with the new key
Old tokens signed with the old key are still valid because the old key is still in your key set
After your token expiry period (say, 15 minutes), all active tokens have been re-issued with the new key
Remove the old key from your key set after the expiry period

Automating Rotation with AWS Secrets Manager

For applications in AWS, Secrets Manager provides the most complete rotation automation:

// Application reads from Secrets Manager at startup and refreshes periodically
import { SecretsManagerClient, GetSecretValueCommand } from "@aws-sdk/client-secrets-manager";

class SecretManager {
  private client: SecretsManagerClient;
  private cache: Map<string, { value: string; fetchedAt: number }> = new Map();
  private ttl = 5 * 60 * 1000; // 5 minute cache

  constructor() {
    this.client = new SecretsManagerClient({ region: "us-east-1" });
  }

  async get(secretId: string): Promise<string> {
    const cached = this.cache.get(secretId);
    if (cached && Date.now() - cached.fetchedAt < this.ttl) {
      return cached.value;
    }

    const response = await this.client.send(
      new GetSecretValueCommand({ SecretId: secretId })
    );

    const value = response.SecretString ?? "";
    this.cache.set(secretId, { value, fetchedAt: Date.now() });
    return value;
  }
}

Caching secrets with a short TTL means your application periodically refreshes from Secrets Manager. When a rotation occurs, your application picks up the new secret within five minutes without a restart.

Rotation for Self-Hosted Deployments

Without a managed service like AWS Secrets Manager, implement rotation as a scheduled job:

// Rotation script run weekly via cron
async function rotateApiDatabasePassword() {
  const newPassword = generateStrongPassword();

  // Phase 1: Add new password to database user (keep old password)
  await db.execute(
    `ALTER USER api_app PASSWORD '${newPassword}' VALID UNTIL 'now' + interval '7 days'`
  );

  // Phase 2: Update secrets in your secrets store
  await doppler.updateSecret("DATABASE_PASSWORD", newPassword);

  // Phase 3: Notify that restart is needed (or trigger rolling restart)
  await notifySlack("#ops", "Database password rotated. Rolling restart initiated.");
  await triggerRollingRestart("api");

  // Phase 4 (after restart completes): Remove old password alternative
  // ... implementation depends on your database
}

The gap in self-hosted rotation is that phase 4 (removing old credentials after the application has transitioned) requires careful timing. Managed rotation services handle this automatically.

The Rotation Audit Trail

Every rotation should be logged: what was rotated, when, by what process (automated or manual), and who triggered it if manual. This audit trail is essential for:

Incident response: if a credential was compromised, knowing exactly when it was last rotated tells you the exposure window
Compliance: many frameworks require evidence of credential rotation
Debugging: if an authentication failure appears after a rotation, knowing the rotation timestamp helps narrow the cause

Store rotation logs in your centralized logging system, separate from the systems the credentials access. Logs stored on the system protected by the credentials can be tampered with if those credentials are compromised.

If you want help designing an automated rotation strategy for your application's credentials or need to audit your current rotation practices, book a session at https://calendly.com/jamesrossjr.