Optimise Your Crawl Budget: Get Google to Index Your Site Quicker
Key Takeaways
- Crawl budget is the total number of pages Google will crawl on your website within a specific period, a limited resource particularly for larger sites.
- For sites with fewer than 10,000 pages, crawl budget seldom presents an issue. However, for extensive websites such as e-commerce platforms, publishers, or SaaS providers, it's absolutely crucial.
- The primary culprits for wasted crawl budget include faceted navigation, parameter URLs, duplicate content, redirect chains, and soft 404s.
- Optimise your crawl budget by guiding Googlebot towards high-value pages while preventing access to pages of low value, duplicate material, or technical nature.
- The speed of your server's response directly affects the crawl rate, meaning quicker servers will experience more frequent crawling..
Table of Contents
Whenever Googlebot accesses your site, it operates with finite resources. It's simply not possible for it to crawl every single URL indefinitely, so it has to make choices. For smaller websites, this seldom poses an issue. However, for expansive sites featuring thousands, or even millions, of pages, effective crawl budget management can dictate whether your critical content is indexed within hours or remains overlooked for weeks.
What Exactly Is Crawl Budget?
Your crawl budget is shaped by two distinct factors:
- Crawl rate limit: This refers to the highest crawl speed Google employs, ensuring your server remains unburdened. It automatically adapts according to your server's response times and overall capacity.
- Crawl demand: This indicates Google's desire to crawl your site, influenced by factors such as URL popularity, content recency, and significant site-wide occurrences, for example, a sitemap update or a substantial content revision.
The actual crawl budget you receive represents where these two elements converge. Should your server operate slowly, Google will decrease its crawl rate to prevent any operational problems. Similarly, if your content is updated infrequently, Google lessens its crawl demand, as there's little new information for it to find.
Is Crawl Budget Important for Your Website?
| Site Size | Crawl Budget Priority | Action Required |
|---|---|---|
| Fewer than 1,000 pages | Low | Prioritise content quality and link building instead |
| 1,000 to 10,000 pages | Low to Medium | Straightforward optimisation: tidy sitemaps, resolve errors |
| 10,000 to 100,000 pages | Medium to High | Ongoing crawl budget management is essential |
| 100,000 pages or more | Critical | Advanced optimisation is vital for proper indexing |
How Google Determines What to Crawl
Google's crawling approach relies on a number of key indicators:
- URL popularity: URLs attracting more inbound links and user visits get crawled more regularly
- Content freshness: Pages that update regularly receive more frequent crawl visits
- Sitemap hints: XML sitemap entries showing recent lastmod timestamps receive preferential crawling
- Internal link structure: Pages positioned near your homepage (requiring fewer clicks) get found sooner
- Page value indicators: Google applies PageRank assessment and comparable metrics to prioritise pages of genuine value
Top 10 Crawl Budget Offenders
- Faceted navigation: Attribute filters (colour, size, price) producing numerous URL permutations
- URL parameters: Analytics codes, session markers, and ordering options producing redundant URLs
- Repeated pages: Multiple URLs presenting the same content lacking canonical indicators
- Redirect sequences: A→B→C→D chains squander crawl resources with each step
- Incorrect 404 responses: Pages that return success codes but contain error messages or no actual content
- Site search output pages: Numerous search query URLs with minimal value being indexed
- Never-ending calendar pages: Calendar tools generating URLs for past and upcoming dates indefinitely
- Tag and category page proliferation: Numerous shallow tag pages featuring repetitive subject matter
- Sluggish server performance: Lengthy TTFB values result in fewer pages being crawled per session
- Compromised and malicious content: Malicious content uses up crawl allocation that should be reserved for authentic material
Crawl Budget Optimisation Approaches
Block Low-Value URLs
- Disallow faceted navigation, internal search, and parameter URLs via robots.txt
- Apply a 'noindex' tag to any thin or low-value pages you wish to retain but don't want ranking in search results.
- Define how parameters should be handled using the URL Parameters tool within Google Search Console.
Optimise Server Performance
- Aim for a TTFB below 200ms, as Google's crawlers operate more quickly on highly responsive servers.
- Implement HTTP/2 to enable multiplexed connections.
- Deploy server-side caching, which helps cut down response times for recurring requests.
- Think about using a CDN to minimise latency for Googlebot, given it typically crawls from US-based IP addresses.
Optimise Internal Linking
- Make sure all crucial pages are accessible within three clicks from your homepage.
- Address or delete any broken internal links, as these simply squander crawl budget on dead ends.
- Employ breadcrumbs and contextual links to establish streamlined crawl paths.
Keep Sitemaps Tidy
- Your sitemap should only ever contain canonical, indexable URLs.
- Only update 'lastmod' dates when content genuinely changes, avoid adding false dates to your sitemap.
- Take out any 404, redirected, or noindexed URLs from your sitemaps.
- Divide larger sitemaps into sitemap index files, categorised appropriately.
Monitoring Your Crawl Budget
| Tool | What It Displays | Cost |
|---|---|---|
| Google Search Console (Crawl Stats) | Total requests, response codes, various file types, Googlebot classifications, and average response time. | Free |
| Server Log Analysis | Precise pages crawled, frequency of crawls, user agents involved, and their corresponding status codes. | Free (Do It Yourself) |
| Screaming Frog Log Analyzer | Visual log analysis, mapping of crawl frequency, and bot identification capabilities. | From £99/yr |
| Oncrawl | Enterprise-level log analysis, complete with valuable crawl budget insights. | From $69/mo |
In Summary
Optimising your crawl budget means ensuring every single visit from Googlebot is productive. Guide the bot towards your most important pages, cut out any wasted effort on low-value URLs, and confirm your server can adeptly manage Google's crawling requirements. For substantial websites, this crucial step can mean the difference between pages being indexed in mere hours or taking several months.
Require Assistance with Technical SEO?
Our team carries out thorough crawl audits and puts in place optimisations designed to get your key pages indexed more quickly. We specialise in larger e-commerce and publisher websites.
Request a Complimentary Crawl Audit →
Comments
Have a question or a first-hand experience with this? Join the conversation. Your email is never shown or shared.
Great article. Any case studies showing ROI?
Good point. The combination of proper structure and testing makes all the difference for this strategy.