1071e56173
- add_no_crawl_headers now skips marketing.*, legal.*, billing.success, static, and robots_txt endpoints via _is_public_indexable_endpoint helper; all other routes keep the X-Robots-Tag noindex header - recordings.index drops @login_required and instead redirects anonymous users to marketing.landing, resolving the URL-map collision between recordings_bp and marketing_bp at "/" - robots.txt rewritten: public marketing pages and /legal/* allowed, /api/, /admin, /account, /share/, /app/, /checkout, /login, /signup, /webhooks/ disallowed; Googlebot, Bingbot, ClaudeBot, GPTBot, PerplexityBot, Applebot explicitly allowed - New tests/test_no_crawl_headers.py (14 tests) covers exemption helper + integration on /, /robots.txt, /static, /admin, /login - New tests/test_marketing_root_redirect.py (4 tests) verifies anonymous users at / never get a /login redirect Tests verified via AST + logic walkthrough; pytest blocked on Windows by pre-existing fcntl import in src/init_db.py (B-1.2 limitation).
49 lines
968 B
Plaintext
49 lines
968 B
Plaintext
# DictIA - robots.txt
|
|
# Updated 2026-04-27 for marketing redesign (Task B-1.3)
|
|
#
|
|
# Public marketing pages (root, /tarifs, /fonctionnalites, /conformite,
|
|
# /contact, /blog) and legal pages (/legal/*) are indexable.
|
|
# Application routes (/api, /admin, /account, /share, /app, /checkout,
|
|
# /login, /signup, /webhooks) remain blocked.
|
|
|
|
User-agent: *
|
|
Allow: /
|
|
Allow: /tarifs
|
|
Allow: /fonctionnalites
|
|
Allow: /conformite
|
|
Allow: /contact
|
|
Allow: /blog
|
|
Allow: /legal/
|
|
Disallow: /api/
|
|
Disallow: /admin
|
|
Disallow: /account
|
|
Disallow: /share/
|
|
Disallow: /app/
|
|
Disallow: /checkout
|
|
Disallow: /login
|
|
Disallow: /signup
|
|
Disallow: /oublie
|
|
Disallow: /verifier-email
|
|
Disallow: /webhooks/
|
|
|
|
# Search/AI crawlers explicitly allowed on public marketing surface
|
|
User-agent: Googlebot
|
|
Allow: /
|
|
|
|
User-agent: Bingbot
|
|
Allow: /
|
|
|
|
User-agent: ClaudeBot
|
|
Allow: /
|
|
|
|
User-agent: GPTBot
|
|
Allow: /
|
|
|
|
User-agent: PerplexityBot
|
|
Allow: /
|
|
|
|
User-agent: Applebot
|
|
Allow: /
|
|
|
|
Sitemap: https://dictia.pages.dev/sitemap.xml
|