Does ChatGPT actually read my website?
Which user agents does ChatGPT use, and what does each one do?
Three routes in · each one is set separately
| Agent | Purpose | Blocking it means |
|---|---|---|
GPTBot | Crawls content for model training | blocked → Your content is less likely to be in model weights |
OAI-SearchBot | Builds the ChatGPT search index | blocked → You are absent from ChatGPT’s retrieval index |
ChatGPT-User | Live fetch when a prompt needs the page | blocked → The page may still be fetched: OpenAI says robots.txt may not apply to fetches a user starts |
GPTBot opt-out costs you training, not retrieval — OAI-SearchBot did not exist until 2024. A blanket AI block written since removes you from both. Which one you have is a curl away.Some sites blocked GPTBot when it launched in 2023 to opt out of training, then never revisited the decision. That is a defensible choice on its own, but blocking OAI-SearchBot alongside it removes you from retrieval as well, which is almost never what the brand intended.
ChatGPT-User behaves differently from the other two. OpenAI states that because its fetches are initiated by a user, “robots.txt rules may not apply”, and that it is not used to decide whether content appears in ChatGPT search; OAI-SearchBot is the agent that controls search inclusion (OpenAI, Overview of OpenAI Crawlers, checked 23 September 2026).
What should robots.txt say?
All three agents need an explicit allow, or an allow-all that no later rule narrows:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /One trap worth knowing: under RFC 9309 a crawler obeys only its most specific matching group. If you later add a Disallow: under User-agent: *, these three groups will ignore it — which may be what you want, but only if you meant it.
How do I verify a crawler can actually reach the page?
A robots.txt that allows a crawler proves nothing if a bot-management rule challenges it at the edge. Start with the real response:
curl -sI -A "GPTBot" https://yourbrand.com/ | head -1
curl -sI -A "OAI-SearchBot" https://yourbrand.com/ | head -1
curl -sI -A "PerplexityBot" https://yourbrand.com/ | head -1Anything other than 200 is a problem. Our bot check runs the same
request against five crawlers from a browser, and reads your robots.txt back alongside the
response, if you would rather not run curl.
Why is a reachable page not always a readable one?
The second failure is rendering. Fetch your page with JavaScript disabled and read what comes back:
curl -s https://yourbrand.com/products/hero | wc -cIf a storefront theme renders product descriptions client-side, the crawler receives a shell. Everything you want quoted needs to be present in the initial HTML response.
Allowing the crawler is necessary rather than sufficient: which index each engine actually reads, and what it searches before it reads anything, is covered in what ChatGPT searches before it names a brand.
Our home-language pages get picked up by AI but our English ones don’t. Who handles that?
At SearchD, we handle the gap between an international brand’s indexed home-language pages and its missing English pages in US AI search. We check the English site’s crawl access, product passages and brand identity, then prepare approved page corrections. Our method describes the diagnosis and follow-up measurement.
Related: How AI crawlers work, structured data.
Related questions
Our home-language pages get picked up by AI but our English ones don't. Who handles that?
At SearchD, we handle the gap between an international brand's indexed home-language pages and its missing English pages in US AI search. We check the English site's crawl access, product passages and brand identity, then prepare approved page corrections.