Skip to main content

ADR 0006: Allow all crawlers — search, AI search, training, agents

  • Status: ✅ Accepted for public marketing pages; superseded for docs.acesense.io on 2026-06-21
  • Date: 2026-04-26
  • Deciders: Akshay Sarode
  • Tags: seo, distribution, content-licensing

Context​

The earlier robots.txt on acesense-landing blocked training-class crawlers (GPTBot, Google-Extended, CCBot, Applebot-Extended, Meta-ExternalAgent, etc.) on the assumption that "training crawlers should be opt-in pending a content-licensing decision."

Two problems with that posture for AceSense's stage:

  1. AI search is the strategic channel. The discovery strategy targets AI-search citation as the primary acquisition lever (Perplexity, ChatGPT, Claude). Search bots and training crawlers are operated by the same companies and increasingly share corpora. Blocking the training crawler reduces the chance our content lands in the model — which reduces the chance AI search surfaces it later.
  2. No paid licensing deal exists today. Withholding access doesn't earn us a content-licensing payout — it just removes us from the corpus. We have no contractual basis to charge for it; blocking is a pure cost.

A separate Cloudflare-managed override was also in play: Cloudflare's "Manage AI Bots" feature was blocking GPTBot/CCBot/Google-Extended/etc. by default at the edge.

Options considered​

Option A — Block training crawlers, allow search-only​

  • Pros: Conservative; preserves the option to license later.
  • Cons: No realistic licensing pipeline today. Reduces our presence in the AI corpora that will be cited tomorrow.
  • Cost: Opportunity cost — invisible.

Option B — Allow everything​

  • Pros: Maximum visibility. Same content already public on acesense.io anyway. Cheapest to operate.
  • Cons: No licensing leverage. Once content is in a model, you can't pull it out.
  • Cost: Zero implementation; theoretical content-licensing forgone.

Option C — Cloudflare-managed (default block training)​

  • Pros: Single-pane management at the CDN.
  • Cons: Cloudflare's defaults are overly restrictive for a small-stage product. Was actively hurting our discovery surface.

Decision​

We will allow every crawler — search, AI search, training, and agent — on every public page of acesense.io. The repo's public/robots.txt enumerates each known bot UA explicitly with Allow: / so a future audit can show the intent.

As of 2026-06-21, this ADR no longer applies to docs.acesense.io: the docs host is private, Firebase Auth gated, and served with X-Robots-Tag: noindex, nofollow, noarchive.

We will also flip every category to "Allowed" inside Cloudflare's "Manage AI Bots" / "Content Signals" dashboard so the served robots.txt at the edge matches the origin.

If a content-licensing deal ever becomes plausible (negotiated 1:1, payable, with a clear scope), we revisit this ADR and selectively block.

Consequences​

Positive​

  • Maximum citation surface for AI search.
  • No mismatch between intent (open) and served policy (was blocked at the edge).
  • Less ongoing maintenance — we just need to ensure new bot UAs don't get default-blocked.

Negative​

  • We cannot later argue "the bot didn't have permission" if a model trains on our content. We accept this — the business value of the citation outweighs.
  • Some training crawlers are noisy and slightly increase Cloudflare bandwidth. Negligible cost on Firebase Hosting + Cloudflare free tier.

Neutral​

  • The acesense-prod-app.web.app (Flutter app) hosting target is not in scope here — it serves authenticated user content and is not crawled in any meaningful sense.
  • Authenticated routes (app.acesense.io, admin.acesense.io) are on separate domains and are not addressable from this robots.txt.

Follow-up actions​

  • Update acesense-landing/public/robots.txt to allow every known UA.
  • Flip each entry to Allowed in Cloudflare → Security → Bots → Manage AI Bots.
  • Confirm https://acesense.io/robots.txt (Cloudflare-served) matches the repo file at next deploy.
  • Re-check at every major model-vendor announcement; new UAs may need explicit allow rules.

References​