08318a946f
- Remove redundant named User-agent blocks (Googlebot, Bingbot, ClaudeBot, GPTBot, PerplexityBot, Applebot) that per RFC 9309 §2.2 overrode the wildcard and granted those bots access to /api/, /admin, /account. - Add explicit Google-Extended and ChatGPT-User blocks (AI opt-in signaling) with full Allow/Disallow rule sets. - Fix /blog → /blog/ for prefix-match consistency. - Move src.marketing.routes import inside recordings.index() function to localize cross-blueprint dependency (was at module top, inverting initialization order). - Add shadow-warning comment at marketing_bp registration site.
72 lines
1.5 KiB
Plaintext
72 lines
1.5 KiB
Plaintext
# DictIA - robots.txt
|
|
# Updated 2026-04-27 for marketing redesign (Task B-1.3)
|
|
#
|
|
# Public marketing pages (root, /tarifs, /fonctionnalites, /conformite,
|
|
# /contact, /blog) and legal pages (/legal/*) are indexable.
|
|
# Application routes (/api, /admin, /account, /share, /app, /checkout,
|
|
# /login, /signup, /webhooks) remain blocked.
|
|
|
|
User-agent: *
|
|
Allow: /
|
|
Allow: /tarifs
|
|
Allow: /fonctionnalites
|
|
Allow: /conformite
|
|
Allow: /contact
|
|
Allow: /blog/
|
|
Allow: /legal/
|
|
Disallow: /api/
|
|
Disallow: /admin
|
|
Disallow: /account
|
|
Disallow: /share/
|
|
Disallow: /app/
|
|
Disallow: /checkout
|
|
Disallow: /login
|
|
Disallow: /signup
|
|
Disallow: /oublie
|
|
Disallow: /verifier-email
|
|
Disallow: /webhooks/
|
|
|
|
# Google-Extended (Bard/Gemini training): explicit opt-in to public marketing
|
|
User-agent: Google-Extended
|
|
Allow: /
|
|
Allow: /tarifs
|
|
Allow: /fonctionnalites
|
|
Allow: /conformite
|
|
Allow: /contact
|
|
Allow: /blog/
|
|
Allow: /legal/
|
|
Disallow: /api/
|
|
Disallow: /admin
|
|
Disallow: /account
|
|
Disallow: /share/
|
|
Disallow: /app/
|
|
Disallow: /checkout
|
|
Disallow: /login
|
|
Disallow: /signup
|
|
Disallow: /oublie
|
|
Disallow: /verifier-email
|
|
Disallow: /webhooks/
|
|
|
|
# ChatGPT-User (on-demand browsing): explicit opt-in to public marketing
|
|
User-agent: ChatGPT-User
|
|
Allow: /
|
|
Allow: /tarifs
|
|
Allow: /fonctionnalites
|
|
Allow: /conformite
|
|
Allow: /contact
|
|
Allow: /blog/
|
|
Allow: /legal/
|
|
Disallow: /api/
|
|
Disallow: /admin
|
|
Disallow: /account
|
|
Disallow: /share/
|
|
Disallow: /app/
|
|
Disallow: /checkout
|
|
Disallow: /login
|
|
Disallow: /signup
|
|
Disallow: /oublie
|
|
Disallow: /verifier-email
|
|
Disallow: /webhooks/
|
|
|
|
Sitemap: https://dictia.pages.dev/sitemap.xml
|