---
title: "What Is Googlebot? Definition vs. Google-Extended"
description: "What is Googlebot? Google's own web crawler for search indexing — distinct from Google-Extended, which governs AI training-data permissions."
canonical_url: "https://kinetixseo.nl/learn/googlebot"
published_at: "2026-08-26T07:18:43+00:00"
updated_at: "2026-08-26T07:18:43+00:00"
author: "Rogier Bruggeman"
category: "Glossary"
---
# What Is Googlebot? Definition vs. Google-Extended

What is Googlebot? Google's own web crawler for search indexing — distinct from Google-Extended, which governs AI training-data permissions.

## What is Googlebot?

Googlebot is Google's own web crawler — the software that discovers and downloads pages so they can be processed and added to Google's search index. It's the crawler that classic SEO has always been optimized for: when you write title tags, build internal links, or fix crawl errors in Search Console, you're managing how Googlebot sees your site and how that translates into rankings on google.com. Googlebot also renders JavaScript before indexing and crawls primarily as a mobile client under Google's mobile-first indexing, which is why pages that hide content behind client-side scripts or serve different markup to mobile agents can index differently than expected. Verifying a bot is actually Googlebot — rather than a spoofed user agent — requires a reverse DNS lookup against Google's published IP ranges, not just trusting the user-agent string in a log file.

## Google-Extended controls AI training, not search crawling

Google-Extended is a separate robots.txt token that governs only whether your content can be used to train Gemini models or populate [AI Overviews](https://kinetixseo.nl/learn/google-ai-overviews), and it has no effect on search crawling at all. Blocking Google-Extended opts a site out of that AI training-data use while leaving Googlebot's search crawling and indexing completely untouched — your pages keep getting crawled, indexed, and ranked in Google Search exactly as before. Disallowing Googlebot itself, by contrast, would pull your site out of Google Search entirely, since that's the crawler feeding the index in the first place. Mixing up the two directives is one of the more common robots.txt mistakes site owners make when trying to manage AI exposure without realizing the tokens don't overlap.

## The two tokens in practice

A robots.txt file that disallows Google-Extended but allows Googlebot is the standard way to separate these concerns: search visibility stays intact, AI training exposure goes away. The [web crawler glossary entry](https://kinetixseo.com/articles/does-robots-txt-block-ai-crawlers-a-practical-guide) covers how crawlers request, parse, and prioritize pages more broadly, and the same guide walks through the exact robots.txt syntax for blocking Google-Extended and other AI crawlers without harming Googlebot's access.