What Is GPTBot? OpenAI's Web Crawler Explained
What is GPTBot? It's OpenAI's crawler for AI training data, separate from ChatGPT's live search bot — and controlled via robots.txt.
· Updated · By Rogier Bruggeman, Founder of KinetixSEO
What is GPTBot?
GPTBot is OpenAI's web crawler that collects publicly available content from the internet to train its AI models. It works like any other automated crawler: it requests pages, reads their content, and follows the rules set in a site's robots.txt file. If you allow or disallow other bots today, blocking or permitting GPTBot uses the same mechanism — a disallow rule under its own user-agent name.
Measure actual visibility
Want to see this on your own site?
Use your own site as the evidence. Get a free SEO and AI-citation readiness baseline, then monitor what changes.
GPTBot vs. OpenAI's other crawler
GPTBot is not the same bot that powers ChatGPT's live search feature. OpenAI runs a separate crawler for real-time web retrieval during a ChatGPT search session — the one that fetches a page so ChatGPT can cite or summarize it in an answer right now. GPTBot, by contrast, gathers content for future model training and has nothing to do with what ChatGPT can cite today. The two bots use different user-agent names in robots.txt, so a rule written for one does not affect the other.
A concrete example
A site owner who wants both outcomes at once — no training use, but full eligibility for ChatGPT citations — needs two separate robots.txt entries, not one blanket rule. One entry disallows GPTBot's user-agent to stop training access. A second, distinct entry explicitly allows the search-retrieval bot's user-agent so live citations still work. Treating the two crawlers as one either blocks citations the owner wanted to keep or leaves training access open when the goal was to shut it off.
For the exact syntax and directives, see this technical guide to blocking AI crawlers with robots.txt. If you're deciding whether opting out of AI training is right for your site, check the glossary entry on training-data opt-outs for the tradeoffs involved.
Measure actual visibility
See how your own site scores on SEO and AI-search visibility — free report, no signup.
Use your own site as the evidence. Get a free SEO and AI-citation readiness baseline, then monitor what changes.