Does ChatGPT actually read my website?
The three user agents, and what each does
Three routes in · block one and you lose only that one
| Agent | Purpose | Blocking it means |
|---|---|---|
GPTBot | Crawls content for model training | blocked → Your content is less likely to be in model weights |
OAI-SearchBot | Builds the ChatGPT search index | blocked → You are absent from ChatGPT’s retrieval index |
ChatGPT-User | Live fetch when a prompt needs the page | blocked → A user asking about your page gets nothing |
GPTBot opt-out costs you training, not retrieval — OAI-SearchBot did not exist until 2024. A blanket AI block written since removes you from both. Which one you have is a curl away.Some sites blocked GPTBot when it launched in 2023 to opt out of training, then never revisited the decision. That is a defensible choice on its own, but blocking OAI-SearchBot alongside it removes you from retrieval as well, which is almost never what the brand intended.
What to put in robots.txt
All three agents need an explicit allow, or an allow-all that no later rule narrows:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /One trap worth knowing: under RFC 9309 a crawler obeys only its most specific matching group. If you later add a Disallow: under User-agent: *, these three groups will ignore it — which may be what you want, but only if you meant it.
How to verify crawler access
A robots.txt that allows a crawler proves nothing if a bot-management rule challenges it at the edge. Start with the real response:
curl -sI -A "GPTBot" https://yourbrand.com/ | head -1
curl -sI -A "OAI-SearchBot" https://yourbrand.com/ | head -1
curl -sI -A "PerplexityBot" https://yourbrand.com/ | head -1Anything other than 200 is a problem.
Reachable is not the same as readable
The second failure is rendering. Fetch your page with JavaScript disabled and read what comes back:
curl -s https://yourbrand.com/products/hero | wc -cIf a storefront theme renders product descriptions client-side, the crawler receives a shell. Everything you want quoted needs to be present in the initial HTML response.
Related: How AI crawlers work, structured data.