SEOgirl

Practical guide · Practical SEO guide

Crawl Budget: Discovery, Waste and Priorities

Find crawl waste and improve discovery without assuming every small site has a crawl-budget problem.

By Ameneh Saeednia · · 11 pages

Download guide PDF
Cover of Crawl Budget: Discovery, Waste and Priorities

Inside the guide

Separate the stages

Discovery, fetching, indexing and ranking are different problems. First establish which stage is failing using available first-party evidence. A page that was crawled but not selected for indexing needs a different investigation from one a crawler never found.

Reduce avoidable URL work

Review parameter spaces, duplicates, redirect chains, errors and slow responses. Link to final canonical URLs and use accurate sitemaps. Prioritize changes from the size and behavior of the real site, not an arbitrary crawl target.

Work through the guide

Use the downloadable guide for its complete examples, audit checks and reference list. Record the page or entity reviewed, the evidence available, unresolved questions, an action owner and a review date. Keep assumptions separate from observed findings.

Related resources