Observability Playbook

AceSense's non-GPU control plane is observed through Cloud Logging, Firebase consoles, Firestore job/audit data, client error events, and the admin UI. There is no repository-backed evidence for the old fixed latency and success rate numbers, so this page does not present targets as measured baselines.

Start with a correlation key

Use the most specific identifier available:

• jobId for upload, dispatch, processing, report, and deletion;
• Firebase uid for account, entitlement, quota, and consent;
• API keyId and job ID for REST/MCP requests;
• sessionId for Storage object paths;
• function name and deployment revision for release failures.

Do not paste ID tokens, session cookies, raw API keys, signed URLs, or private video links into tickets or chat.

Where to look

Signal	Source
function errors and structured actions	Google Cloud Logging / Firebase Functions logs
current job state and timestamps	jobs/{jobId} or Admin → Jobs
queue/backlog summary	Admin → Operations
admin mutations	admin_audit / Admin → Audit Log
signed-URL issuance and revocation	signed_url_audit
moderation decisions	moderation_audit
device-side upload failures	client_errors
public API liveness	GET /v1/healthz (also /healthz)
docs/launchpad auth and serving errors	docsServer / launchpadServer logs

The functions logger emits structured records through logAction; filter by function, action, UID, job ID, or session ID instead of scanning unstructured messages.

Healthy invariants

• A successful upload moves through known states and ends in done.
• A terminal job releases any reserved concurrency slot exactly once.
• Commerce cost reservation is either committed or refunded exactly once.
• Old pending/uploaded work is reconciled after two hours and old processing work after six hours.
• Private docs and launchpad assets are never served before session authorization.
• Job and user writes rejected by rules remain rejected in emulator tests.

Triage sequence

1. Establish affected users, first occurrence, and last known-good release.
2. Read the job or account document without editing it.
3. Filter logs using a correlation key and capture the first failing action.
4. Check task retries and scheduled-reconciliation output.
5. Check Firebase/GCP service health and current billing/quota state.
6. Use the narrowest runbook. Escalate to an external provider only after the control-plane handoff is confirmed.

Alert design

Maintain alerts for sustained function errors, task exhaustion, stale-job growth, auth-denial spikes on private sites, retention failures, and budget thresholds. Choose thresholds from observed traffic and error budgets in the monitoring system; keep the alert definition linked from its runbook.

Privacy and retention

Logs should contain opaque identifiers and bounded error text, not uploaded content or credentials. Completed job artifacts are eligible for the 30-day retention sweep, and generated account-export bundles for the seven-day sweep. Audit retention requirements may differ; follow the compliance policy rather than deleting audit rows ad hoc.

GPU implementation metrics are outside this audit. Provider-specific evidence can be joined to the control plane by job ID.