fix(marketing): RFC 9309 robots.txt semantics + lazy marketing import

- Remove redundant named User-agent blocks (Googlebot, Bingbot, ClaudeBot,
  GPTBot, PerplexityBot, Applebot) that per RFC 9309 §2.2 overrode the
  wildcard and granted those bots access to /api/, /admin, /account.
- Add explicit Google-Extended and ChatGPT-User blocks (AI opt-in
  signaling) with full Allow/Disallow rule sets.
- Fix /blog → /blog/ for prefix-match consistency.
- Move src.marketing.routes import inside recordings.index() function
  to localize cross-blueprint dependency (was at module top, inverting
  initialization order).
- Add shadow-warning comment at marketing_bp registration site.
This commit is contained in:
Allison
2026-04-27 16:43:02 -04:00
parent af2953995c
commit 08318a946f
3 changed files with 47 additions and 17 deletions

View File

@@ -641,6 +641,8 @@ app.register_blueprint(docs_bp)
# - marketing_bp at "/" (placeholder; coexists with recordings_bp.index, resolved in B-1.3)
# - billing_bp at /checkout/* (routes added in B-2.7 and B-2.8)
# - legal_bp at /legal/* (routes added in B-2.9)
# NOTE: marketing_bp.landing at "/" is shadowed by recordings.index (registered
# earlier above). recordings.index dispatches anonymous users to landing() directly.
app.register_blueprint(marketing_bp)
app.register_blueprint(billing_bp)
app.register_blueprint(legal_bp)