What is crawl budget?

Crawl budget is the number of pages on a site that a search engine's crawler is willing and able to crawl within a given window of time. It's shaped by two things: the site's perceived crawl health (how fast it responds, how many errors it throws, how much low-value content it serves up) and the crawler's own capacity constraints (how much it wants to spend on your site without overloading your servers or wasting its own resources). Crawl budget isn't infinite, and it isn't evenly distributed across every URL on a site — a search engine effectively rations its attention, deciding how many requests to make and which pages get them.

Where crawl budget gets wasted

Faceted navigation is a common way crawl budget gets drained by pages that don't need it. An e-commerce category page that lets visitors filter by size, color, price, and brand can generate thousands of near-duplicate URLs from a handful of underlying products. Each filter combination is technically a unique URL, and each one competes for the same limited pool of crawl attention as the product pages, category pages, and landing pages that actually need to be indexed and ranked. A crawler that spends most of its budget on faceted-navigation permutations has less left over for the pages a site depends on for organic traffic, so on large sites new or updated pages can sit uncrawled for longer than they should.

Crawl budget vs. crawl rate and crawl demand

Crawl budget is not the same thing as crawl rate or crawl demand. Rate is how fast a crawler can fetch pages without straining a server, demand is how much a search engine actually wants those pages based on their perceived value, and budget is the practical result of the two working together.

This is a definition, not a fix — for the tactics that actually reduce crawl waste on large sites, including how to read a crawl stats report and prioritize fixes, see Crawl Budget Optimization: What Actually Matters. If you need the broader context first, on how crawling, indexing, and ranking fit together, start with What Is SEO? A Beginner's Guide to Search Basics before diving into crawl budget specifics.

What Is Crawl Budget? A Quick Definition