--- title: Secret Rotation Runbook — 2026-05-30 tags: [operations, security, secrets, incident] created: 2026-05-30 status: action-required source: Full Codebase Audit - 2026-05-30 --- # Secret Rotation Runbook — 2026-05-30 The 2026-05-30 full codebase audit found live credentials committed to the repos and, in some cases, baked into container images. The audit's no-brainer fixes **replaced the committed values with placeholders in the working tree**, but the *real* credentials are still valid and must be **rotated by a human** — replacing a string in git does not invalidate a leaked key. > Treat every credential below as **compromised**. Anyone with repo (or image) access has > had these values. Rotate first, then scrub history. Related issues: ISSUE-074, ISSUE-075, ISSUE-079, ISSUE-115 and decisions DEC-49, DEC-50, DEC-56, DEC-74, DEC-75, DEC-78. --- ## Order of operations (per credential) 1. **Rotate** — generate a new value at the provider. 2. **Inject at runtime** — put the new value in the deployment secret store (Arcane env / compose secrets), **never** back into a committed file. 3. **Deploy** — roll the new value out and confirm the service is healthy. 4. **Revoke** — invalidate the old value at the provider. 5. **Scrub** — remove the secret from git history (see "History scrub" at the bottom). Do these one credential at a time and verify the dependent service after each. --- ## Credentials to rotate | # | Credential | Where it leaked | Blast radius | How to rotate | |---|-----------|-----------------|--------------|---------------| | 1 | **Telegram bot token** | `backend/.env.development`, `backend/.env.example`, `frontend/.gitleaks.toml` | Full control of the bot: read/send messages, hijack the login widget, phish users | BotFather → `/revoke` → new token. Update `TELEGRAM_BOT_TOKEN`. | | 2 | **Resend SMTP / API key** | `backend/.env.development`, `backend/.env.example` | Send email as the platform (phishing, OTP spoofing), read sending logs | Resend dashboard → API Keys → delete + create. Update `RESEND_API_KEY` / SMTP creds. | | 3 | **JWT signing secret** | `backend/.env.example` | Forge **any** user/admin session token — critical | Generate 32+ random bytes (`openssl rand -hex 32`). Update `JWT_SECRET`. **Rotating invalidates all sessions** (users re-login). Consider also adding a separate `REFRESH_TOKEN_SECRET` (see DEC-26). | | 4 | **Admin bootstrap password** | `backend/.env.example`, was also a hardcoded fallback in `init-admin.ts` (removed by NB-20) | Direct admin login | Set a strong `ADMIN_PASSWORD` secret; change the admin account password in-app; confirm `init-admin` no longer has a fallback. | | 5 | **Request Network API key** | `backend/.env.example` | Act against the RN account; manipulate payment intents | RN dashboard → rotate key. Update `REQUEST_NETWORK_API_KEY`. | | 6 | **Request Network webhook secret** | `backend/.env.example` | Forge RN webhooks → mark payments paid (this is the HMAC secret the backend verifies) | Rotate at RN; update `REQUEST_NETWORK_WEBHOOK_SECRET`. | | 7 | **Telegram webhook secret token** | `backend/.env.example` | Forge Telegram webhook calls | Reset via `setWebhook` with a new `secret_token`; update the env var. | | 8 | **Google OAuth client secret** | `backend/.env.example` | Impersonate the OAuth app | Google Cloud Console → Credentials → reset client secret. Update `GOOGLE_CLIENT_SECRET`. | | 9 | **Alchemy API key(s)** | `frontend/Dockerfile` ARG defaults (removed by NB-10) | Quota theft / RPC abuse on your account | Alchemy dashboard → rotate app key. Supply via CI build-arg / runtime, not a default. | | 10 | **TG_NOTIFY_BOT_TOKEN** (ops alert bot) | backend startup notification (committed env) | Spoof ops alerts; spam the ops channel | BotFather → revoke → new token. Update `TG_NOTIFY_BOT_TOKEN`. See [[telegram_notify_no_parse_mode]]. | | 11 | **Frontend test account password** (`Moji6364`) | `frontend/scripts/show-credentials.sh` (DEC-75) | Login as that test user if it exists in any real env | Delete the script (or env-prompt it); rotate the account password if real. | ### Public-by-design (lower priority, but make explicit) - **WalletConnect project ID**, **Google OAuth *client ID*** — `frontend/Dockerfile` ARG defaults (DEC-74). These are public values, but remove the baked defaults and pass them via CI build-args so forks don't reuse the production IDs. --- ## Stop re-leaking (pairs with rotation) These are the structural fixes (tracked as decisions) that stop the secrets coming back: - **DEC-50 / ISSUE-075** — `backend/.dockerignore` whitelists `.env.development` *into the prod image*. Remove the `!.env.development` line so no env file is ever copied into an image; inject secrets at runtime. - **DEC-49 / ISSUE-101** — `backend/src/shared/config/index.ts` loads `.env.development` unconditionally. Load `.env.` (or nothing in production) and never fall back to the dev file. - **DEC-56 / ISSUE-074** — untrack `backend/.env.development` entirely (`git rm --cached`) and add it to `.gitignore`. - **DEC-78 / ISSUE-079** — `frontend/.gitleaks.toml` allowlists the bot token *by value*. Switch to a path/fingerprint-based allowlist after scrubbing, so gitleaks stops "approving" the secret. See the `handle-gitleaks` skill. Runtime injection point for this stack: the **Arcane** env / project config (see [[arcane_dev_stack]], [[arcane_cli_usage]]) for dev, and the production secret store for prod. After changing any backend secret, remember the dev redeploy caveat: restart `nickDev-nginx` (see [[devEscrow_nginx_after_redeploy]]). --- ## History scrub (after rotation + revocation) Only after the old values are revoked, purge them from history so they can't be mined from old commits: 1. Use `git filter-repo` (preferred) or BFG to remove the affected files/blobs from each repo's history: `backend/.env.development`, the historical `backend/.env.example`, `frontend/.gitleaks.toml` values, `frontend/scripts/show-credentials.sh`. 2. Force-push the rewritten history and have all collaborators re-clone. **Coordinate** — per [[parallel_agents_on_escrow]] another agent pushes to these branches; a history rewrite mid-flight will conflict badly. Pick a quiet window. 3. Re-run gitleaks to confirm the working tree and history are clean. --- ## Verification checklist - [ ] Each credential rotated at the provider and old value **revoked**. - [ ] New values present only in the runtime secret store (no committed file holds a real value). - [ ] Backend boots; `/api/health` green; login, email send, Telegram login, and an RN webhook all succeed with new secrets. - [ ] `.env.development` untracked; `.dockerignore` no longer whitelists it; config no longer loads it in prod. - [ ] gitleaks passes on working tree; history scrubbed and force-pushed in a coordinated window.