fix(marketing): RFC 9309 robots.txt semantics + lazy marketing import
- Remove redundant named User-agent blocks (Googlebot, Bingbot, ClaudeBot, GPTBot, PerplexityBot, Applebot) that per RFC 9309 §2.2 overrode the wildcard and granted those bots access to /api/, /admin, /account. - Add explicit Google-Extended and ChatGPT-User blocks (AI opt-in signaling) with full Allow/Disallow rule sets. - Fix /blog → /blog/ for prefix-match consistency. - Move src.marketing.routes import inside recordings.index() function to localize cross-blueprint dependency (was at module top, inverting initialization order). - Add shadow-warning comment at marketing_bp registration site.
This commit is contained in:
@@ -12,7 +12,7 @@ Allow: /tarifs
|
||||
Allow: /fonctionnalites
|
||||
Allow: /conformite
|
||||
Allow: /contact
|
||||
Allow: /blog
|
||||
Allow: /blog/
|
||||
Allow: /legal/
|
||||
Disallow: /api/
|
||||
Disallow: /admin
|
||||
@@ -26,23 +26,46 @@ Disallow: /oublie
|
||||
Disallow: /verifier-email
|
||||
Disallow: /webhooks/
|
||||
|
||||
# Search/AI crawlers explicitly allowed on public marketing surface
|
||||
User-agent: Googlebot
|
||||
# Google-Extended (Bard/Gemini training): explicit opt-in to public marketing
|
||||
User-agent: Google-Extended
|
||||
Allow: /
|
||||
Allow: /tarifs
|
||||
Allow: /fonctionnalites
|
||||
Allow: /conformite
|
||||
Allow: /contact
|
||||
Allow: /blog/
|
||||
Allow: /legal/
|
||||
Disallow: /api/
|
||||
Disallow: /admin
|
||||
Disallow: /account
|
||||
Disallow: /share/
|
||||
Disallow: /app/
|
||||
Disallow: /checkout
|
||||
Disallow: /login
|
||||
Disallow: /signup
|
||||
Disallow: /oublie
|
||||
Disallow: /verifier-email
|
||||
Disallow: /webhooks/
|
||||
|
||||
User-agent: Bingbot
|
||||
Allow: /
|
||||
|
||||
User-agent: ClaudeBot
|
||||
Allow: /
|
||||
|
||||
User-agent: GPTBot
|
||||
Allow: /
|
||||
|
||||
User-agent: PerplexityBot
|
||||
Allow: /
|
||||
|
||||
User-agent: Applebot
|
||||
# ChatGPT-User (on-demand browsing): explicit opt-in to public marketing
|
||||
User-agent: ChatGPT-User
|
||||
Allow: /
|
||||
Allow: /tarifs
|
||||
Allow: /fonctionnalites
|
||||
Allow: /conformite
|
||||
Allow: /contact
|
||||
Allow: /blog/
|
||||
Allow: /legal/
|
||||
Disallow: /api/
|
||||
Disallow: /admin
|
||||
Disallow: /account
|
||||
Disallow: /share/
|
||||
Disallow: /app/
|
||||
Disallow: /checkout
|
||||
Disallow: /login
|
||||
Disallow: /signup
|
||||
Disallow: /oublie
|
||||
Disallow: /verifier-email
|
||||
Disallow: /webhooks/
|
||||
|
||||
Sitemap: https://dictia.pages.dev/sitemap.xml
|
||||
|
||||
Reference in New Issue
Block a user