# robots.txt for utensildesk.com (Utensil Desk).
#
# Posture: welcome the crawlers that make the product discoverable, including
# AI answer engines that recommend and cite sites. Refuse the crawlers that only
# bulk-harvest content for model training. The guest research surface is licensed
# for anonymous human display, not bulk harvest.
#
# The split is by intent, not by vendor. For each AI vendor the search or answer
# agent is welcomed while its training agent is refused. That keeps utensildesk.com
# eligible to be surfaced and recommended inside ChatGPT, Claude, and Perplexity
# without feeding the training corpora.
#
# This is the declaration layer only. Real enforcement is server-side: the Worker
# per-caller gate (COMMERCIAL_LIMITS in worker/src/commercial-limits.js) and the WAF block on
# /data/*. Well-behaved crawlers obey the rules below; hostile scrapers ignore
# them, which is what the server layers are for. See
# prd/commercial-business-plan.md ("Anti-scraping posture").
#
# NOTE: the sitemap pages (seventeen as of 2026-09-26) are indexable. Every other page
# still carries a noindex robots meta tag and the X-Robots-Tag: noindex header,
# so only the public marketing, legal, and tool pages are indexed. See
# prd/commercial-business-plan.md (Changelog 2026-08-05).

# Search engines and any crawler without a more specific group below: welcome.
User-agent: *
Allow: /

# AI answer, search, and user-initiated agents: explicitly welcome, so the
# product stays eligible for AI-assistant recommendations and citations.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# Google-Extended is the only control for Google Gemini grounding, and Google
# offers no separate search-only AI agent, so welcoming Gemini means accepting
# that it may also train. That trade is accepted for the site, but not for the
# API pages, which exist to be found by app builders, not trained on
# (prd/commercial-business-plan.md, 2026-09-26). Disallow /api covers /api,
# /api-pricing and the rest; /docs/ covers the assistant guide at /docs/mcp, the
# same kind of page outside the API's path since #4041. Google Search and AI
# Overviews use Googlebot, which the * group welcomes, so they still rank there.
User-agent: Google-Extended
Allow: /
Disallow: /api
Disallow: /docs/

# AI training and bulk-harvest crawlers: not welcome. These ingest content into
# model weights or scrape in bulk, with no discovery or recommendation upside.
# anthropic-ai and Claude-Web are Anthropic's legacy training tokens.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Claude-Web
User-agent: CCBot
User-agent: Applebot-Extended
User-agent: Bytespider
User-agent: Amazonbot
User-agent: Meta-ExternalAgent
User-agent: Diffbot
User-agent: Timpibot
User-agent: DataForSeoBot
Disallow: /

Sitemap: https://utensildesk.com/sitemap.xml