What is Googlebot?

Googlebot is Google's own web crawler — the software that discovers and downloads pages so they can be processed and added to Google's search index. It's the crawler that classic SEO has always been optimized for: when you write title tags, build internal links, or fix crawl errors in Search Console, you're managing how Googlebot sees your site and how that translates into rankings on google.com. Googlebot also renders JavaScript before indexing and crawls primarily as a mobile client under Google's mobile-first indexing, which is why pages that hide content behind client-side scripts or serve different markup to mobile agents can index differently than expected. Verifying a bot is actually Googlebot — rather than a spoofed user agent — requires a reverse DNS lookup against Google's published IP ranges, not just trusting the user-agent string in a log file.

Google-Extended controls AI training, not search crawling

Google-Extended is a separate robots.txt token that governs only whether your content can be used to train Gemini models or populate AI Overviews, and it has no effect on search crawling at all. Blocking Google-Extended opts a site out of that AI training-data use while leaving Googlebot's search crawling and indexing completely untouched — your pages keep getting crawled, indexed, and ranked in Google Search exactly as before. Disallowing Googlebot itself, by contrast, would pull your site out of Google Search entirely, since that's the crawler feeding the index in the first place. Mixing up the two directives is one of the more common robots.txt mistakes site owners make when trying to manage AI exposure without realizing the tokens don't overlap.

The two tokens in practice

A robots.txt file that disallows Google-Extended but allows Googlebot is the standard way to separate these concerns: search visibility stays intact, AI training exposure goes away. The web crawler glossary entry covers how crawlers request, parse, and prioritize pages more broadly, and the same guide walks through the exact robots.txt syntax for blocking Google-Extended and other AI crawlers without harming Googlebot's access.

What Is Googlebot? Definition vs. Google-Extended