Crawling SEO

Think of crawling like a librarian walking through a library. They start at the front desk (your homepage), then follow signs (links) to different sections. If a book is in a locked room (blocked by robots.txt) or has no signs pointing to it (orphan page), the librarian never finds it. Crawling is just the walking part; indexing is cataloguing what they find. But here's where I made a mistake early on: I assumed a sitemap would solve all discovery problems. It won't, because a sitemap tells the librarian about a room, but if the door is locked, they still can't enter. I now check crawl stats in Google Search Console before anything else. On one client site with 2,000 product pages, Googlebot was crawling only 45 per day. The crawl budget was wasteful because the site had a chain of [301 redirects](/301-redirect/) on category pages. After fixing those and improving internal linking, the crawl rate tripled to 135 per day within a week. If I had to choose between submitting a sitemap and fixing internal links, I would fix internal links first — every orphan page is dead weight.

Crawling issues become visible when you launch a new site. Without crawl coverage, your pages don't exist in the index. I worked on a site migration last year: the client changed all URLs but kept old redirects in place. Googlebot followed the 301s and eventually hit a 404 loop on some pages. Crawl errors spiked, and traffic dropped by 60% for three weeks. That's when it matters most: during a migration or redesign. But also when you update content regularly — if a blog post goes live but the homepage doesn't link to it, Googlebot may not find it until the next sitemap push, which could be days. A counter-argument I hear is that crawl budget only matters for sites with millions of pages. Not true. A small ecommerce site with 500 pages can still have crawl issues if [robots.txt](/robots-txt/) blocks product images. Googlebot needs to render the page; if assets are blocked, the page may appear incomplete and fail a quality threshold. The decision rule I use: monitor crawl stats weekly and set an alert if crawl rate drops by more than 20% in a day. That caught a misconfigured server for me once — the site was returning 503 errors for six hours before I noticed.

I used to think crawling and indexing were the same. They are not. Googlebot can crawl a page and decide not to index it if it's thin or has a noindex tag. I once had a site where all pages were crawled but only half were indexed. The other half had duplicate content — I hadn't set [canonical tags](/canonical-tag/). That was a lesson. I also blocked CSS and JavaScript in robots.txt because I thought it would speed up crawling. Instead, Google couldn't render the pages properly, and they ranked poorly for months. Now I never block any resource unless I'm sure it's not needed for rendering. Another mistake: I assumed that submitting a [sitemap](/sitemap/) would guarantee all URLs are crawled. It doesn't — Google still needs to see links to those URLs, or they might not be crawled at all. I now use a combination: a clean sitemap, a flat internal link structure, and weekly crawl report reviews. One thing I still struggle with: balancing crawl budget for large sites with frequent content updates. I haven't found a perfect formula, but I've learned to prioritise pages that generate revenue.