Skip to main content

Secret Rotation

Written 2026-07-25 from the full-history audit.

:::tip Rotation completed 2026-08-17 The five secrets below have been rotated by the owner. The keys embedded in origin/main history are now dead, so this is no longer a live credential exposure.

What remains is cleanup, not an incident: the blobs are still fetchable by anyone who can clone the repo, and Secret Scan (full history) stays red until the history is rewritten. Treat the rewrite as a scheduled maintenance task — it is a force-push that invalidates every existing clone, so coordinate it.

Precedent: the UNFORCE repos were scrubbed and force-pushed on 2026-08-16 and all eleven verify clean, re-checked 2026-08-20 across all 406 commits of unforce-gpu-backend.

Two lessons from that run:

  • git filter-repo will silently rewrite brand-guard sentinel strings too, so re-check them afterwards.
  • Verify by scanning the rewritten history with value-bearing patterns. A first re-check here reported 47 dirty commits and was wrong: it matched the bare field name "private_key_id" in a docs example whose value is literally "...". A secret-shaped field name is not a secret. Require the value — a PEM block, rpa_ + 20 chars, sk-ant-api + 20, a hex key id — or the check will cry wolf and get switched off. :::

What leaked​

Five live secrets are in origin/main history on GitHub. Working-tree deletion did nothing — the blobs remain fetchable by anyone who can clone the repo.

#SecretHistorical pathAdded
1acesense-prod GCP service-account private key (runpod@acesense-prod.iam.gserviceaccount.com, key id 360da8c7…a33a)firebasecred/acesense-prod-360da8c714c2.json94dfdf0, 2026-01-14
2acesense-6fd55 Firebase Admin SDK key (key id 1551dbe9…9bf4)firebasecred/acesense-6fd55-firebase-adminsdk-fbsvc-1551dbe9c7.json8ab7984, 2025-12-18
3Anthropic API key (sk-ant-api…9gAA)firebasecred/anthropic-api, anthropic-api8ab7984 / d1492ee
4RunPod API key (rpa_4CYYOA…x1w8)22 script/doc paths (see below)fb979a0, 2025-11-09
5RunPod API key (rpa_56OV7U…u3hq)scripts/deploy_test.sh, archive/docker/scripts/deploy_test.sh1360c2f

Commit 94dfdf0 is confirmed an ancestor of origin/main (git merge-base --is-ancestor → true), and the blob genuinely contains "project_id": "acesense-prod" plus a real BEGIN PRIVATE KEY block. This is the production project's key — treat it as the priority.

Paths that ever contained an rpa_ literal (22, all in this repo's history):

CREDENTIALS_BAKED_IN.md              docs/CREDENTIALS_BAKED_IN.md
FINAL_SUMMARY.md                     docs/FINAL_SUMMARY.md
FIREBASE_DEPLOYED.md                 docs/FIREBASE_FUNCTION.md
FIREBASE_FUNCTION_SETUP.md           docs/archive/FIREBASE_DEPLOYED.md
archive/docker/scripts/deploy_test.sh docs/archive/FIREBASE_FUNCTION_SETUP.md
firebasefunction/functions/README.md  functions/README.md
firebasefunction/functions/main.py    functions/main.py
monitor_job.sh                        scripts/deploy_test.sh
scripts/monitor_job.sh                scripts/test_simple.sh
test_new_endpoint.sh                  test_runpod_quick.sh
test_simple.sh                        test_with_cloud.sh

Step 1 — Rotate FIRST. Do not reorder​

The keys are already public. Scrubbing history before rotating only removes your ability to see what leaked; it does not un-leak anything. Anyone who cloned the repo already has them.

  1. GCP → IAM & Admin → Service Accounts, for project acesense-prod: delete key id 360da8c714c2…a33a on runpod@acesense-prod.iam.gserviceaccount.com, issue a replacement, store it at ~/Developer/secrets/ (never in the repo). Repeat for project acesense-6fd55, key id 1551dbe9…9bf4.
  2. RunPod console → Settings → API Keys: revoke both keys, mint one replacement, put it in the Firebase secret RUNPOD_API_KEY (already declared in acesense-auth-function/video/gpu/secrets.ts — the declaration pattern is correct, only the value needs replacing).
  3. Anthropic console → API Keys: revoke sk-ant-api…9gAA, mint a replacement.
  4. Audit usage logs from 2025-11-09 onward (earliest leak) on all three providers. For GCP that is Cloud Logging + IAM "key last used"; unrecognised activity from unexpected IPs is what you are looking for. This is the step people skip; it is the one that tells you whether the leak was actually exploited.

Do not proceed to step 2 until every key above is dead. A revoked key in git history is an embarrassment; a live one is an incident.

Step 2 — Scrub history​

Requires git-filter-repo (brew install git-filter-repo — this is a tool, not a runtime, so brew is correct here per the workstation conventions).

Work on a fresh mirror clone, never your working checkout:

cd ~/Developer/caches
git clone --mirror git@github-acesense:Acesense/acesense-gpu-backend.git scrub.git
cd scrub.git

Generate the replacement list from history rather than pasting secrets into a file that could itself get committed:

git rev-list --all | \
  xargs -P4 -I{} git grep -h -I -o -E 'rpa_[A-Za-z0-9]{40,}' {} -- 2>/dev/null | \
  sort -u | sed 's/$/==>REDACTED_ROTATED_KEY/' > /tmp/replacements.txt
wc -l /tmp/replacements.txt   # expect 2

Then rewrite:

git filter-repo --force \
  --path firebasecred --path anthropic-api --invert-paths \
  --replace-text /tmp/replacements.txt

Verify before pushing — this should print nothing:

git rev-list --all | xargs -P4 -I{} git grep -l -I -e 'rpa_' -e 'BEGIN PRIVATE KEY' {} -- 2>/dev/null

Then force-push every ref and tag:

git push --force --all && git push --force --tags
rm -f /tmp/replacements.txt

Step 3 — After the rewrite​

  • Every collaborator must re-clone. A normal git pull on an old clone will reintroduce the old objects. There is no polite way around this.

  • Open a GitHub Support ticket asking them to expire cached views of the old commit SHAs. Force-push alone leaves the old commits reachable by direct SHA URL (github.com/Acesense/acesense-gpu-backend/commit/94dfdf0) more or less indefinitely. Support is the only way to purge those.

  • Your local working checkout still has the old history. Re-clone it too, or the next push re-uploads what you just removed. The checkout currently holds ~16.5k lines of uncommitted refactor work.

    That work is already snapshotted twice: as ref backup/audit-2026-07-25 inside each repo, and as standalone bundles in ~/Developer/caches/acesense-worktree-backups/ (2.8 MB total, all 10 repos). The in-repo ref does not survive a re-clone; the bundles do — that is why they exist. Caveat: each bundle is incremental against its current base commit (gpu-backend's is 9559bdf), and filter-repo rewrites those base SHAs, so a bundle cannot be replayed onto post-scrub history. Concretely: land the refactor before you scrub, or re-export the work as a plain patch/tarball first. Restore from a bundle looks like:

    git fetch ~/Developer/caches/acesense-worktree-backups/acesense-gpu-backend-2026-07-25.bundle \
      refs/heads/backup/audit-2026-07-25:refs/heads/recovered

Step 4 — Close the detection gap​

The retired per-push workflow scanned only BASE_SHA..HEAD_SHA. That range scoping is why five secrets sat in main for eight months without a single CI failure — it structurally could not see a commit outside the PR range.

Since GitHub Actions was retired (2026-08-24) both detection modes live in the local sweep: the default ci-local.sh run scans a HEAD export per repo (stops the next credential), and FULL_HISTORY=1 ./ci-local.sh runs gitleaks over all history so reintroduction and pre-existing debt both surface. Run the full-history mode after any incident and on a regular cadence. The retired scheduled-workflow shape, for the record:

  full-history-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - uses: gitleaks/gitleaks-action@v2
        env: { GITLEAKS_CONFIG: .gitleaks.toml }

with on: schedule: [{ cron: "0 6 * * 1" }].

.gitleaks.toml has no allowlist for the leaked rpa_ values, so it will correctly trip if either is ever reintroduced. Leave it that way.