# Combinate Robots.txt # ---- AI training crawlers and AI search indexers ---- # These bots may only access public marketing pages (homepage, FAQ, # resources, for-brands/for-manufacturers, pricing, legal pages). # Everything else — dashboards, profiles, messaging, billing, admin — # remains off-limits to them. # # NOT listed (intentionally): Claude-User, ChatGPT-User. These are # on-demand user-initiated fetches (agent mode, e.g. when a logged-in # user asks Claude/ChatGPT to read a specific URL). They behave like a # browser request on behalf of a real person, not a crawler, and are # governed by the generic `User-agent: *` block below. User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Claude-SearchBot User-agent: GPTBot User-agent: OAI-SearchBot User-agent: Google-Extended User-agent: PerplexityBot User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: Bytespider User-agent: CCBot Disallow: / Allow: /$ Allow: /about/ Allow: /contact/ Allow: /how-it-works/ Allow: /for-brands/ Allow: /for-manufacturers/ Allow: /faq/ Allow: /pricing/ Allow: /resources/ Allow: /capabilities/ Allow: /manufacturers-in/ Allow: /privacy-policy/ Allow: /terms-of-service/ Allow: /cookie-policy/ Allow: /sitemap.xml Allow: /robots.txt Allow: /static/ # ---- Search engines and everyone else ---- User-agent: * # Disallow private/admin areas Disallow: /admin/ Disallow: /dashboard/ Disallow: /profile/ Disallow: /inbox/ Disallow: /conversations/ Disallow: /notifications/ Disallow: /account/ Disallow: /accounts/ Disallow: /billing/ Disallow: /checkout/ Disallow: /payments/ Disallow: /ai/ Disallow: /analytics/ Disallow: /media/private/ Disallow: /logout/ Disallow: /api/ Disallow: /stripe/webhook/ Disallow: /data-request/ Disallow: /r/ Disallow: /*/edit/ Disallow: /*/delete/ # Disallow search parameters and filters to avoid duplicate content Disallow: /*?*page=* Disallow: /*?*sort=* Disallow: /*?*filter=* Disallow: /*?*query=* Disallow: /*search*?* # Allow static resources so Google can render pages correctly Allow: /static/ # Sitemap location Sitemap: https://www.combinate.co.uk/sitemap.xml