fix(marketing): RFC 9309 robots.txt semantics + lazy marketing import

- Remove redundant named User-agent blocks (Googlebot, Bingbot, ClaudeBot,
  GPTBot, PerplexityBot, Applebot) that per RFC 9309 §2.2 overrode the
  wildcard and granted those bots access to /api/, /admin, /account.
- Add explicit Google-Extended and ChatGPT-User blocks (AI opt-in
  signaling) with full Allow/Disallow rule sets.
- Fix /blog → /blog/ for prefix-match consistency.
- Move src.marketing.routes import inside recordings.index() function
  to localize cross-blueprint dependency (was at module top, inverting
  initialization order).
- Add shadow-warning comment at marketing_bp registration site.
This commit is contained in:
Allison
2026-04-27 16:43:02 -04:00
parent af2953995c
commit 08318a946f
3 changed files with 47 additions and 17 deletions

View File

@@ -12,7 +12,7 @@ Allow: /tarifs
Allow: /fonctionnalites
Allow: /conformite
Allow: /contact
Allow: /blog
Allow: /blog/
Allow: /legal/
Disallow: /api/
Disallow: /admin
@@ -26,23 +26,46 @@ Disallow: /oublie
Disallow: /verifier-email
Disallow: /webhooks/
# Search/AI crawlers explicitly allowed on public marketing surface
User-agent: Googlebot
# Google-Extended (Bard/Gemini training): explicit opt-in to public marketing
User-agent: Google-Extended
Allow: /
Allow: /tarifs
Allow: /fonctionnalites
Allow: /conformite
Allow: /contact
Allow: /blog/
Allow: /legal/
Disallow: /api/
Disallow: /admin
Disallow: /account
Disallow: /share/
Disallow: /app/
Disallow: /checkout
Disallow: /login
Disallow: /signup
Disallow: /oublie
Disallow: /verifier-email
Disallow: /webhooks/
User-agent: Bingbot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: GPTBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Applebot
# ChatGPT-User (on-demand browsing): explicit opt-in to public marketing
User-agent: ChatGPT-User
Allow: /
Allow: /tarifs
Allow: /fonctionnalites
Allow: /conformite
Allow: /contact
Allow: /blog/
Allow: /legal/
Disallow: /api/
Disallow: /admin
Disallow: /account
Disallow: /share/
Disallow: /app/
Disallow: /checkout
Disallow: /login
Disallow: /signup
Disallow: /oublie
Disallow: /verifier-email
Disallow: /webhooks/
Sitemap: https://dictia.pages.dev/sitemap.xml