Overview
A large enterprise site recovered indexation of important pages by eliminating crawl waste and resolving systemic technical blockers.
Business context
An enterprise technology company running a sprawling site of tens of thousands of URLs across products, support and resources. A significant share of commercially important pages were not being indexed, while crawlers spent budget on low-value parameter and faceted URLs.
Starting metrics
- Share of important pages indexed
- well below where it should be
- Crawl budget spent on low-value URLs
- high, dominated by parameters and facets
- Discovered-but-not-indexed pages
- a large and growing bucket
- Internal link depth to key pages
- too deep for reliable discovery
Problems identified
- Pages not indexed pattern affecting a meaningful share of revenue-relevant URLs.
- Crawl budget waste pattern as bots repeatedly fetched low-value parameterised URLs.
- Redirect issues and chains adding friction to crawling and consolidation.
- Orphan pages that received little or no internal linking and were rarely discovered.
Actions taken
-
1
Ran a technical audit and log analysis
Combined the Technical Audit and Log File Analysis frameworks to see where crawlers actually spent time versus where value lay.
-
2
Contained low-value URL sprawl
Applied crawl controls and canonicalisation to parameter and faceted URLs so budget flowed to important pages.
-
3
Resolved indexability blockers
Fixed the directives, redirects and templates that were keeping important pages out of the index.
-
4
Reduced click depth to key pages
Reworked internal linking so priority pages sat closer to the surface and orphan pages were connected.
-
5
Monitored recovery
Tracked indexation and crawl distribution over the engagement to confirm budget was reallocated to value.
Results
- Share of important pages indexed
- moved substantially closer to full coverage (illustrative)
- Crawl budget on low-value URLs
- materially reduced through controls and canonicalisation
- Discovered-but-not-indexed bucket
- shrank as blockers were resolved
- Discovery of priority pages
- improved as click depth fell
Timeline: roughly 4 months
Lessons learned
- On large sites, indexation problems are usually crawl-budget problems in disguise.
- Log files reveal what audits alone cannot: where crawlers genuinely spend their time.
- Reducing click depth to priority pages does as much for discovery as any directive fix.