---
title: Secret Rotation Runbook — 2026-05-30
tags: [operations, security, secrets, incident]
created: 2026-05-30
status: action-required
source: Full Codebase Audit - 2026-05-30
---

# Secret Rotation Runbook — 2026-05-30

The 2026-05-30 full codebase audit found live credentials committed to the repos and, in
some cases, baked into container images. The audit's no-brainer fixes **replaced the
committed values with placeholders in the working tree**, but the *real* credentials are
still valid and must be **rotated by a human** — replacing a string in git does not
invalidate a leaked key.

> Treat every credential below as **compromised**. Anyone with repo (or image) access has
> had these values. Rotate first, then scrub history.

Related issues: ISSUE-074, ISSUE-075, ISSUE-079, ISSUE-115 and decisions DEC-49, DEC-50,
DEC-56, DEC-74, DEC-75, DEC-78.

---

## Order of operations (per credential)

1. **Rotate** — generate a new value at the provider.
2. **Inject at runtime** — put the new value in the deployment secret store (Arcane env /
   compose secrets), **never** back into a committed file.
3. **Deploy** — roll the new value out and confirm the service is healthy.
4. **Revoke** — invalidate the old value at the provider.
5. **Scrub** — remove the secret from git history (see "History scrub" at the bottom).

Do these one credential at a time and verify the dependent service after each.

---

## Credentials to rotate

| # | Credential | Where it leaked | Blast radius | How to rotate |
|---|-----------|-----------------|--------------|---------------|
| 1 | **Telegram bot token** | `backend/.env.development`, `backend/.env.example`, `frontend/.gitleaks.toml` | Full control of the bot: read/send messages, hijack the login widget, phish users | BotFather → `/revoke` → new token. Update `TELEGRAM_BOT_TOKEN`. |
| 2 | **Resend SMTP / API key** | `backend/.env.development`, `backend/.env.example` | Send email as the platform (phishing, OTP spoofing), read sending logs | Resend dashboard → API Keys → delete + create. Update `RESEND_API_KEY` / SMTP creds. |
| 3 | **JWT signing secret** | `backend/.env.example` | Forge **any** user/admin session token — critical | Generate 32+ random bytes (`openssl rand -hex 32`). Update `JWT_SECRET`. **Rotating invalidates all sessions** (users re-login). Consider also adding a separate `REFRESH_TOKEN_SECRET` (see DEC-26). |
| 4 | **Admin bootstrap password** | `backend/.env.example`, was also a hardcoded fallback in `init-admin.ts` (removed by NB-20) | Direct admin login | Set a strong `ADMIN_PASSWORD` secret; change the admin account password in-app; confirm `init-admin` no longer has a fallback. |
| 5 | **Request Network API key** | `backend/.env.example` | Act against the RN account; manipulate payment intents | RN dashboard → rotate key. Update `REQUEST_NETWORK_API_KEY`. |
| 6 | **Request Network webhook secret** | `backend/.env.example` | Forge RN webhooks → mark payments paid (this is the HMAC secret the backend verifies) | Rotate at RN; update `REQUEST_NETWORK_WEBHOOK_SECRET`. |
| 7 | **Telegram webhook secret token** | `backend/.env.example` | Forge Telegram webhook calls | Reset via `setWebhook` with a new `secret_token`; update the env var. |
| 8 | **Google OAuth client secret** | `backend/.env.example` | Impersonate the OAuth app | Google Cloud Console → Credentials → reset client secret. Update `GOOGLE_CLIENT_SECRET`. |
| 9 | **Alchemy API key(s)** | `frontend/Dockerfile` ARG defaults (removed by NB-10) | Quota theft / RPC abuse on your account | Alchemy dashboard → rotate app key. Supply via CI build-arg / runtime, not a default. |
| 10 | **TG_NOTIFY_BOT_TOKEN** (ops alert bot) | backend startup notification (committed env) | Spoof ops alerts; spam the ops channel | BotFather → revoke → new token. Update `TG_NOTIFY_BOT_TOKEN`. See [[telegram_notify_no_parse_mode]]. |
| 11 | **Frontend test account password** (`Moji6364`) | `frontend/scripts/show-credentials.sh` (DEC-75) | Login as that test user if it exists in any real env | Delete the script (or env-prompt it); rotate the account password if real. |

### Public-by-design (lower priority, but make explicit)
- **WalletConnect project ID**, **Google OAuth *client ID*** — `frontend/Dockerfile` ARG
  defaults (DEC-74). These are public values, but remove the baked defaults and pass them
  via CI build-args so forks don't reuse the production IDs.

---

## Stop re-leaking (pairs with rotation)

These are the structural fixes (tracked as decisions) that stop the secrets coming back:

- **DEC-50 / ISSUE-075** — `backend/.dockerignore` whitelists `.env.development` *into the
  prod image*. Remove the `!.env.development` line so no env file is ever copied into an
  image; inject secrets at runtime.
- **DEC-49 / ISSUE-101** — `backend/src/shared/config/index.ts` loads `.env.development`
  unconditionally. Load `.env.<NODE_ENV>` (or nothing in production) and never fall back to
  the dev file.
- **DEC-56 / ISSUE-074** — untrack `backend/.env.development` entirely (`git rm --cached`)
  and add it to `.gitignore`.
- **DEC-78 / ISSUE-079** — `frontend/.gitleaks.toml` allowlists the bot token *by value*.
  Switch to a path/fingerprint-based allowlist after scrubbing, so gitleaks stops
  "approving" the secret. See the `handle-gitleaks` skill.

Runtime injection point for this stack: the **Arcane** env / project config (see
[[arcane_dev_stack]], [[arcane_cli_usage]]) for dev, and the production secret store for
prod. After changing any backend secret, remember the dev redeploy caveat:
restart `nickDev-nginx` (see [[devEscrow_nginx_after_redeploy]]).

---

## History scrub (after rotation + revocation)

Only after the old values are revoked, purge them from history so they can't be mined from
old commits:

1. Use `git filter-repo` (preferred) or BFG to remove the affected files/blobs from each
   repo's history: `backend/.env.development`, the historical `backend/.env.example`,
   `frontend/.gitleaks.toml` values, `frontend/scripts/show-credentials.sh`.
2. Force-push the rewritten history and have all collaborators re-clone. **Coordinate** —
   per [[parallel_agents_on_escrow]] another agent pushes to these branches; a history
   rewrite mid-flight will conflict badly. Pick a quiet window.
3. Re-run gitleaks to confirm the working tree and history are clean.

---

## Verification checklist

- [ ] Each credential rotated at the provider and old value **revoked**.
- [ ] New values present only in the runtime secret store (no committed file holds a real value).
- [ ] Backend boots; `/api/health` green; login, email send, Telegram login, and an RN webhook all succeed with new secrets.
- [ ] `.env.development` untracked; `.dockerignore` no longer whitelists it; config no longer loads it in prod.
- [ ] gitleaks passes on working tree; history scrubbed and force-pushed in a coordinated window.
