Chicago Technical SEO Audit

I always start a Chicago technical SEO audit by establishing a crawl baseline. I fire up Screaming Frog with the site's full URL list, set a high crawl limit, and let it run. For a typical Chicago e-commerce site with 15,000 product pages, that takes about an hour. I then export the crawl and filter by HTTP status. Looking at 4xx and 5xx responses first tells me where the site is broken. I recently found 47 broken internal links on a client's site — each one wasted crawl budget and frustrated users.

Next, I examine the robots.txt file. I use the crawl tool's built-in checker to see which resources it blocks. A common mistake is accidentally blocking CSS or JS files, which prevents Google from rendering the page correctly. I check that robots.txt doesn't contain a 'Disallow: /' unless intentional. Then I move to the XML sitemap. I open the sitemap URL and scan for non-canonical URLs, redirects, or pages that return 3xx. Any URL in the sitemap that resolves to a different canonical URL is wasted effort. I also verify the sitemap is referenced in the [robots.txt](/robots-txt/) and submitted to Google Search Console.

From there, I open Search Console's Index Coverage report. I sort by the 'Excluded' tab and read each reason. Common entries are 'Crawled – currently not indexed' and 'Page with redirect'. I've seen sites where 30% of pages were excluded due to a soft 404 or noindex tag. For each exclusion reason, I decide whether to fix the underlying issue or accept the decision. A counter-argument I hear is to start with Core Web Vitals because Google prioritises user experience. But I argue that if your pages aren't indexed, speed is irrelevant. I prioritise crawl and indexation as the non-negotiable first step. Only after these pass do I move to performance. For a Chicago client with a JavaScript-heavy single-page app, I had to check rendering via the URL Inspection Tool first — crawl alone wasn't enough. That's an edge case where you need to ensure Google can fully render the page before debugging indexation. Decision rule: if your site has more than 5% of pages excluded due to a noindex directive, investigate whether those pages should be visible. If not, keep the noindex; if yes, remove it and request reindexing.

The biggest watch-out is treating a technical audit like a content audit. Keywords don't matter here. I once had a client ask me to optimise their meta descriptions during a technical audit. I had to explain that technical SEO is about infrastructure, not copy. You don't care about keyword density; you care about crawl budget, indexability, and rendering.

Don't skip foundational signals: robots.txt, sitemaps, noindex, [canonical tags](/canonical-tag/). These are the most common blockers. I see audits that dive straight into page speed optimisation while ignoring that half the site is blocked by a misplaced noindex tag. That's a waste of everyone's time. I have a decision rule: if the site has any pages with a 200 status but a 'noindex' directive, I either remove the noindex or redirect the URL. Leaving it in limbo only confuses Google.

A specific number: on a Chicago restaurant chain site, I found 80% of category pages had a self-referencing canonical but the paginated pages /page/2/ had no canonical at all. That caused [duplicate content](/duplicate-content/) issues and diluted link equity. Always check [pagination](/pagination/) handling.

Another watch-out: soft 404s. These are not caught by standard crawl tools because they return a 200 OK. Example: a search results page that says 'no results' but returns 200. Use Search Console's coverage report to spot 'soft 404' errors. I missed this on a client for two weeks. Don't fix page speed without first checking crawlability. A fast page that Google can't crawl is invisible. I once saw a Chicago agency advertise a 'technical SEO audit' that only ran Lighthouse scores. That's not a technical audit.

Also, be wary of relying solely on automated tools. They miss nuances like JavaScript rendering issues or mobile-specific blocks. For a recent audit, Screaming Frog showed all pages as crawlable, but the mobile version had a different robots.txt meta tag that blocked indexing. I caught it only by checking the mobile HTML source.

I used to think a technical SEO audit was just checking page speed and fixing broken links. That was early in my career. I worked with a Chicago-based B2B software company for months. I spent time optimising their content, improving load times from 5s to 2.5s, but organic traffic stayed flat. When I finally looked at their robots.txt, I found it contained 'Disallow: /' for the entire site. Google couldn't crawl any of the optimised content. That cost us four months of work. Now I always start with crawl and indexation before touching performance.

I also underestimated the importance of [mobile SEO](/mobile-seo/). Another client had a responsive site but the mobile version was blocked by a separate robots.txt directive specific to mobile user-agents. I assumed responsiveness meant mobile was fine. I missed that for two weeks. The lesson: check mobile crawlability explicitly.

Another mistake: I used to check [Core Web Vitals](/core-web-vitals/) before confirming indexability. On one client, I reduced LCP from 4.2s to 2.1s, but traffic didn't improve because 30% of pages were not indexed due to a bad sitemap. That was a painful lesson. Now I have a strict rule: crawl and indexation first, Core Web Vitals second.

I also got wrong the priority between soft 404s and 500 errors. I used to fix 500 errors first, but soft 404s can affect more pages. On the same Chicago client, there were 400 soft 404s and only 20 server errors. I should have fixed the soft 404s first because they wasted more crawl budget.