How to optimise crawl budget on a large site

Executive summary

Crawl budget matters once a site has tens of thousands of URLs or publishes frequently. The lever is not asking crawlers to work harder but removing the low-value URLs they waste time on, so the same budget covers more of what counts. Use log files to prove where the effort actually goes.

What this helps you decide

Where crawlers are wasting effort and which patterns to prune, block or consolidate first.

Business problem

On large sites, search engines spend a finite amount of crawling per day. When that budget is burned on faceted URLs, redirects and duplicates, important pages are crawled slowly or not at all, so updates take weeks to show.

Step-by-step process

1. 1
   Measure current crawl distribution

   Analyse server logs to see which URL patterns crawlers actually request, and what share of hits land on low-value paths.

2. 2
   Identify the waste

   Flag faceted-navigation combinations, session parameters, infinite calendars, soft 404s and redirect hops that consume crawls without earning traffic.

3. 3
   Block or canonicalise the dross

   Use robots.txt to stop crawling of genuinely worthless patterns, and canonical tags to consolidate near-duplicates that must stay accessible.

4. 4
   Flatten redirect chains

   Collapse multi-hop redirects to a single hop so each crawl resolves in one request instead of three.

5. 5
   Strengthen internal links to priorities

   Ensure high-value pages are within a few clicks of the home page and well linked, so crawlers reach and revisit them often.

6. 6
   Keep sitemaps clean and current

   List only canonical, indexable URLs with accurate lastmod dates so crawlers prioritise genuinely updated pages.

7. 7
   Re-measure after changes

   Recheck logs a few weeks later to confirm the share of crawl hits on valuable pages has risen.

Worked example

Checklist

• Use logs to quantify where crawl effort currently lands
• Identify and list the low-value URL patterns eating budget
• Block worthless patterns and canonicalise necessary duplicates
• Reduce every redirect chain to a single hop
• Confirm priority pages are shallow and well linked
• Re-measure the crawl distribution after the changes

Common mistakes

• Blocking a pattern in robots.txt that also carries a canonical or noindex you still need read
• Optimising crawl budget on a small site where it makes no measurable difference
• Leaving stale, non-canonical URLs in the sitemap and diluting the signal

30-minute experiment

KPIs to track

• Share of crawl requests hitting valuable pages
• Average time for new pages to be crawled
• Number of low-value URLs crawled per day

FAQs