Crawl Budget
Crawl budget is the amount of crawling Google is willing and able to spend on a site in a given time, combining how much it wants to crawl with how much the site can be crawled efficiently.
Start here
- Open Google Search Console → Crawl Stats to see your current crawl activity.
- Audit for duplicate content, soft 404s, and redirect chains — fix those first.
- Update your XML sitemap to only include canonical, high-value pages with accurate <lastmod>.
- Improve server response time to under 200ms for faster crawling.
- If you have under 10,000 pages, stop worrying about crawl budget — focus on content quality instead.
Think of crawl budget like a limited number of invites Googlebot has to visit your site each day — you want to make sure it spends them on your best pages.
What I'd do first
- Check your Crawl Stats report in Google Search Console to see how many URLs Googlebot is hitting and how fast.
- Identify and remove or consolidate duplicate content — Google says wasted crawling on duplicates can make it decide not to spend as much time on the rest of your site.
- Fix soft 404s and long redirect chains — these are silent budget killers.
- Update your XML sitemap with only canonical, high-value URLs and accurate
<lastmod>timestamps. - Ensure your server responds quickly (under 200ms) so Googlebot can crawl more pages per visit.
Plain-English take
Crawl budget is just the number of pages Googlebot will crawl on your site in a given time. It's a combination of two things:
- Crawl rate: How fast Googlebot can crawl without overwhelming your server.
- Crawl demand: How much Google wants to crawl your site based on content quality, freshness, and popularity.
If your site is small (under a few thousand pages), you probably don't need to worry about it. But if you have tens of thousands of pages — or your site changes frequently — wasted crawling on junk pages means Google might miss your important content.
When it actually matters
- Large sites (10,000+ pages): Google can't crawl everything every day, so you need to prioritize.
- Frequently updated sites (news, ecommerce, blogs): Fresh content needs to be discovered quickly.
- Sites with technical debt: Duplicate URLs, soft 404s, redirect chains, or slow servers all eat budget.
- After a site migration: Crawl patterns reset, so you want to guide Googlebot to the right pages.
- If you notice important pages not indexed: Crawl budget might be the bottleneck.
What I got wrong
- I used to think crawl budget was a direct ranking factor. It's not — it's an efficiency and discovery issue.
- I focused on crawl rate fixes (like server speed) before cleaning up duplicates and soft 404s. That's backwards: remove the waste first, then optimize speed.
- I left old, low-value URLs in my sitemap thinking it wouldn't hurt. It does — it tells Google to crawl pages you don't care about.
- I blocked URLs in robots.txt when the real problem was poor internal linking or duplicate content. Blocking can actually make things worse by hiding signals.
Next step
Quick answers
What is crawl budget?
Crawl budget is the number of URLs Googlebot will crawl on your site within a given time. It combines crawl rate (how fast Google can crawl without overloading your server) and crawl demand (how much Google wants to crawl your site based on content quality and popularity).
How to check crawl budget?
You can check crawl budget using Google Search Console's Crawl Stats report, which shows total crawl requests, response times, and errors. Log file analysis also provides detailed crawl patterns.
Does crawl budget affect SEO?
Yes, but primarily for large sites (over 10,000 pages) or sites with frequent updates. For small sites, crawl budget is rarely a limiting factor.
Sources
Primary documentation is linked directly. Anything commercial is marked nofollow.
- Google Search Central: Crawl budget management — Primary source for Google's definition, causes of wasted crawl budget, and optimization guidance.
- Google Search Central: Manage crawling and indexing of site URLs — Useful supporting Google documentation on controlling crawl and index behavior through technical SEO signals.
- Google Search Central: Crawl stats report — Authoritative reference for measuring crawl activity in Google Search Console.
Notes from Callum Bennett.