Advanced Crawl Optimisation

Executive summary

Crawl budget matters most on large, frequently changing sites, where every crawler request spent on a junk URL is a request not spent on a revenue page. This guide uses server log analysis as ground truth to reveal where bots actually spend their time, then prescribes robots controls, redirect cleanups and architecture changes that redirect crawl attention to the pages that pay.

What this helps you decide

Which URL patterns to open, block, or restructure so crawlers concentrate their limited budget on high-value, indexable pages.

Business problem

On large sites, crawlers spend a finite budget and squander much of it on parameters, redirects, thin pages and infinite spaces, leaving important pages crawled slowly or stale. This delays indexing of new content and lets updates languish, capping how fast the site can grow.

Step-by-step process

1. 1
   Establish whether crawl budget is even your constraint

   Crawl optimisation matters chiefly for large sites (tens of thousands of URLs or more) and fast-changing catalogues. For a small brochure site it is rarely the bottleneck. Confirm the problem is real before investing, using site size, publish cadence and index-lag symptoms.

2. 2
   Analyse server logs as ground truth

   Export raw server logs and isolate verified search-engine bot requests. Logs show what crawlers actually fetch, not what you assume; they reveal the true distribution of crawl budget across URL patterns, status codes and directories.

3. 3
   Quantify wasted crawl

   Bucket bot requests by response: 200 valuable, 200 low-value, 3xx redirects, 4xx errors, and parameter or faceted noise. The share hitting redirects, errors and junk parameters (SS-PT-22, SS-PT-23) is your recoverable budget.

4. 4
   Eliminate redirect chains and error hits

   Every crawl of a redirect or 404 is budget burned reaching nothing. Update internal links to point at final destinations, collapse redirect chains to single hops, and fix or remove the error URLs bots repeatedly fetch.

5. 5
   Contain infinite and low-value spaces

   Faceted navigation, calendars, session parameters and internal search create effectively infinite URL spaces. Use robots.txt disallows, parameter controls and nofollow judiciously to stop bots wandering into spaces that will never rank.

6. 6
   Prioritise freshness routing

   Ensure new and updated pages are surfaced through frequently crawled hubs, XML sitemaps with accurate lastmod, and strong internal links so crawlers discover changes quickly. Crawl budget is only useful if it reaches what actually changed.

7. 7
   Decide AI crawler policy deliberately

   Modern crawl policy includes AI bots. Decide per bot whether to allow access (SS-DE-029), balancing the visibility upside in AI answers against server load, and encode the decision explicitly in robots directives rather than leaving it to default.

8. 8
   Re-pull logs to confirm the shift

   After changes, re-analyse logs and confirm the share of budget on valuable 200s has risen and waste on redirects, errors and parameters has fallen. Faster indexing of new pages is the downstream proof.

Worked example

Checklist

• Confirmed the site is large or fast-changing enough for crawl budget to be a real constraint
• Server logs analysed with verified bot requests isolated
• Bot requests bucketed into valuable 200s, low-value 200s, redirects, errors, and parameter noise
• Redirect chains collapsed to single hops and internal links repointed to final URLs
• Infinite and faceted spaces contained via robots and parameter controls
• New and updated pages routed through frequently crawled hubs and accurate sitemaps
• AI crawler access decided per bot and encoded explicitly
• Follow-up log pull confirms valuable-200 crawl share rose and waste fell

Common mistakes

• Optimising crawl budget on a small site where it is not the constraint, at the expense of content and links that are
• Blocking a parameter space in robots.txt that also carries the only path to pages you want indexed
• Fixing redirects in the sitemap but leaving internal links pointing at the old chained URLs, so bots keep crawling them
• Relying on crawl-stats summaries instead of raw logs, missing the true pattern-level distribution of wasted budget

Best practices

• Use raw server logs as the single source of truth for crawl behaviour; assumptions about where bots go are frequently wrong
• Repoint internal links to final destination URLs so you fix the cause of redirect crawling, not just the sitemap symptom
• Contain infinite spaces at the robots layer but verify no unique valuable page is reachable only through the blocked pattern
• Route freshness through hubs and accurate lastmod so recovered budget lands on the pages that actually changed
• Make AI crawler access an explicit, reviewed decision rather than an accidental default, and revisit it as AI referral value shifts

Troubleshooting

30-minute experiment

KPIs to track

• Share of crawl budget spent on valuable 200-status pages
• Median time-to-index for newly published pages
• Percentage of bot requests hitting redirects, errors, or parameter noise

FAQs