nick-doc/08 - Operations/Secret Rotation Runbook - 2026-05-30.md

title, tags, created, status, source

title	tags	created	status	source
Secret Rotation Runbook — 2026-05-30	operations security secrets incident	2026-05-30	action-required	Full Codebase Audit - 2026-05-30

Secret Rotation Runbook — 2026-05-30

The 2026-05-30 full codebase audit found live credentials committed to the repos and, in some cases, baked into container images. The audit's no-brainer fixes replaced the committed values with placeholders in the working tree, but the real credentials are still valid and must be rotated by a human — replacing a string in git does not invalidate a leaked key.

Treat every credential below as compromised. Anyone with repo (or image) access has had these values. Rotate first, then scrub history.

Related issues: ISSUE-074, ISSUE-075, ISSUE-079, ISSUE-115 and decisions DEC-49, DEC-50, DEC-56, DEC-74, DEC-75, DEC-78.

---

Order of operations (per credential)

1. Rotate — generate a new value at the provider.
2. Inject at runtime — put the new value in the deployment secret store (Arcane env / compose secrets), never back into a committed file.
3. Deploy — roll the new value out and confirm the service is healthy.
4. Revoke — invalidate the old value at the provider.
5. Scrub — remove the secret from git history (see "History scrub" at the bottom).

Do these one credential at a time and verify the dependent service after each.

---

Credentials to rotate

#	Credential	Where it leaked	Blast radius	How to rotate
1	Telegram bot token	backend/.env.development, backend/.env.example, frontend/.gitleaks.toml	Full control of the bot: read/send messages, hijack the login widget, phish users	BotFather → /revoke → new token. Update TELEGRAM_BOT_TOKEN.
2	Resend SMTP / API key	backend/.env.development, backend/.env.example	Send email as the platform (phishing, OTP spoofing), read sending logs	Resend dashboard → API Keys → delete + create. Update RESEND_API_KEY / SMTP creds.
3	JWT signing secret	backend/.env.example	Forge any user/admin session token — critical	Generate 32+ random bytes (openssl rand -hex 32). Update JWT_SECRET. Rotating invalidates all sessions (users re-login). Consider also adding a separate REFRESH_TOKEN_SECRET (see DEC-26).
4	Admin bootstrap password	backend/.env.example, was also a hardcoded fallback in init-admin.ts (removed by NB-20)	Direct admin login	Set a strong ADMIN_PASSWORD secret; change the admin account password in-app; confirm init-admin no longer has a fallback.
5	Request Network API key	backend/.env.example	Act against the RN account; manipulate payment intents	RN dashboard → rotate key. Update REQUEST_NETWORK_API_KEY.
6	Request Network webhook secret	backend/.env.example	Forge RN webhooks → mark payments paid (this is the HMAC secret the backend verifies)	Rotate at RN; update REQUEST_NETWORK_WEBHOOK_SECRET.
7	Telegram webhook secret token	backend/.env.example	Forge Telegram webhook calls	Reset via setWebhook with a new secret_token; update the env var.
8	Google OAuth client secret	backend/.env.example	Impersonate the OAuth app	Google Cloud Console → Credentials → reset client secret. Update GOOGLE_CLIENT_SECRET.
9	Alchemy API key(s)	frontend/Dockerfile ARG defaults (removed by NB-10)	Quota theft / RPC abuse on your account	Alchemy dashboard → rotate app key. Supply via CI build-arg / runtime, not a default.
10	TG_NOTIFY_BOT_TOKEN (ops alert bot)	backend startup notification (committed env)	Spoof ops alerts; spam the ops channel	BotFather → revoke → new token. Update TG_NOTIFY_BOT_TOKEN. See telegram_notify_no_parse_mode.
11	Frontend test account password (Moji6364)	frontend/scripts/show-credentials.sh (DEC-75)	Login as that test user if it exists in any real env	Delete the script (or env-prompt it); rotate the account password if real.

Public-by-design (lower priority, but make explicit)

• WalletConnect project ID, Google OAuth client ID — frontend/Dockerfile ARG defaults (DEC-74). These are public values, but remove the baked defaults and pass them via CI build-args so forks don't reuse the production IDs.

---

Stop re-leaking (pairs with rotation)

These are the structural fixes (tracked as decisions) that stop the secrets coming back:

• DEC-50 / ISSUE-075 — backend/.dockerignore whitelists .env.development into the prod image. Remove the !.env.development line so no env file is ever copied into an image; inject secrets at runtime.
• DEC-49 / ISSUE-101 — backend/src/shared/config/index.ts loads .env.development unconditionally. Load .env.<NODE_ENV> (or nothing in production) and never fall back to the dev file.
• DEC-56 / ISSUE-074 — untrack backend/.env.development entirely (git rm --cached) and add it to .gitignore.
• DEC-78 / ISSUE-079 — frontend/.gitleaks.toml allowlists the bot token by value. Switch to a path/fingerprint-based allowlist after scrubbing, so gitleaks stops "approving" the secret. See the handle-gitleaks skill.

Runtime injection point for this stack: the Arcane env / project config (see arcane_dev_stack, arcane_cli_usage) for dev, and the production secret store for prod. After changing any backend secret, remember the dev redeploy caveat: restart nickDev-nginx (see devEscrow_nginx_after_redeploy).

---

History scrub (after rotation + revocation)

Only after the old values are revoked, purge them from history so they can't be mined from old commits:

1. Use git filter-repo (preferred) or BFG to remove the affected files/blobs from each repo's history: backend/.env.development, the historical backend/.env.example, frontend/.gitleaks.toml values, frontend/scripts/show-credentials.sh.
2. Force-push the rewritten history and have all collaborators re-clone. Coordinate — per parallel_agents_on_escrow another agent pushes to these branches; a history rewrite mid-flight will conflict badly. Pick a quiet window.
3. Re-run gitleaks to confirm the working tree and history are clean.

---

Verification checklist

• Each credential rotated at the provider and old value revoked.
• New values present only in the runtime secret store (no committed file holds a real value).
• Backend boots; /api/health green; login, email send, Telegram login, and an RN webhook all succeed with new secrets.
• .env.development untracked; .dockerignore no longer whitelists it; config no longer loads it in prod.
• gitleaks passes on working tree; history scrubbed and force-pushed in a coordinated window.