What is Claude-SearchBot?
Claude-SearchBot is Anthropic's search crawler. Anthropic's crawler documentation describes it plainly: “Claude-SearchBot navigates the web to improve search result quality for users.” It reads pages so Claude's web search can surface and cite them. Anthropic documents 3 bots, each with a different job: ClaudeBot collects content that may contribute to model training, Claude-SearchBot indexes for search, and Claude-User fetches a page when a person asks Claude to look at it. All 3 announce themselves by name, and Anthropic says its bots honor industry standard robots.txt directives. Block the wrong one and you disappear from Claude's cited answers.
What Claude-SearchBot actually does
When Claude answers a question with web search turned on, it does not invent its sources. It pulls from pages that have been fetched and evaluated. Claude-SearchBot is the crawler doing that groundwork. Per Anthropic's documentation, it analyzes online content specifically to enhance the relevance and accuracy of search responses. In plain terms, it decides what Claude's search layer knows about, and a page it cannot reach is a page Claude's search cannot rank or cite.
This is the retrieval side of the house, not the training side. What Claude-SearchBot fetches feeds search results users see now. It follows the same pattern OpenAI set with its search crawler: a dedicated bot whose whole job is finding pages worth citing, kept separate from the crawler that gathers training data.
ClaudeBot vs Claude-SearchBot vs Claude-User
Anthropic runs 3 documented bots. Treating them as one is the mistake that costs sites their citations:
- ClaudeBot is the training crawler. The docs say it “helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training.” Blocking it opts you out of training future Claude models.
- Claude-SearchBot is the search crawler. It indexes pages so Claude's search results are relevant and accurate. This is the one that earns you a citation.
- Claude-User serves live requests. Per the docs, when individuals ask questions to Claude, it may access websites using a Claude-User agent. It fetches one page because one person asked, rather than building an index.
Each has its own user-agent token: ClaudeBot, Claude-SearchBot, Claude-User. That means your robots.txt can say no to training and yes to search, or any other combination. Anthropic states that its bots respect do-not-crawl signals by honoring industry standard directives in robots.txt, so the file is your actual control surface here.
What blocking each one costs you
Blocking ClaudeBot keeps your content out of future Claude training runs. That is a real choice some publishers want, and it costs you nothing in today's search results.
Blocking Claude-SearchBot removes you from the pool of pages Claude's web search can index and cite. If people in your market ask Claude questions your site answers, this block hands your citations to competitors.
Blocking Claude-User stops fetches that a human directly requested. That usually punishes your own prospects, the people interested enough to point Claude at your site.
The pattern to avoid is the blanket rule written in one bad afternoon. A robots.txt that disallows everything with "Claude" in the name, intended to keep training bots out, also deletes you from a search engine. The same failure happens with OpenAI's bots, which we break down in What is GPTBot?, and the general repair is covered in how to fix AI crawler access.
How to allow it in robots.txt
Name each bot and give each its own rule. To opt out of training while staying visible in Claude's search:
# No training, yes citations User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: /
Then verify two things. First, that no wildcard rule higher in the file overrides these lines. Second, that your actual content is in the raw HTML the server returns, because a crawler that is allowed in but receives an empty JavaScript shell still has nothing to cite. The step-by-step check lives in how to fix robots.txt blocking AI crawlers.
See also
Every major AI company now splits its bots along the same 3 lines: training, search, user fetches. OpenAI's version of the search crawler is covered in What is OAI-SearchBot?, and Perplexity's indexing crawler in What is PerplexityBot?. The names change. The rule does not: the search crawler is the one that pays you back for letting it in.