Log File Analysis: Follow the Bots

By Firdaus Nagree

TL;DR: Log file analysis shows which URLs verified crawlers actually requested, when they requested them and how the server responded.

Key takeaways

  • Raw access logs record requests that reporting interfaces may only summarise.
  • Verify crawler identity before treating a user-agent string as Googlebot.
  • Classify URLs by template, status, response time, indexability and business value.
  • A crawled URL is not automatically indexed, ranked or commercially useful.
  • Change one verified problem pattern at a time and measure the result.

What is log file analysis for SEO?

Log file analysis for SEO is the study of server access records to understand crawler behaviour. A useful record usually includes the requested URL, timestamp, response status, bytes served, user agent, referrer and request IP, although the exact format depends on the server and infrastructure.

Google calls raw access logs the best source for analysing which resources it is crawling in its Crawling December guidance. Search Console Crawl Stats is useful context, but logs let an analyst examine individual paths and patterns.

Crawler request streams entering a server and analytical grid

Logs show requests, not motives. They are evidence that a server interaction happened, not a tiny signed letter explaining Google's ranking plans.

How do you verify a crawler?

Verify a crawler with the search engine's documented method rather than trusting the user-agent string alone. User agents are easy to imitate.

Google publishes crawler user agents, IP ranges and reverse DNS patterns in its common crawler documentation. An audit should preserve the raw request evidence and apply the current verification procedure before labelling traffic as Googlebot.

Separate Googlebot Smartphone, Googlebot Desktop, image and other documented fetchers where the distinction matters. Also keep non-Google crawlers in their own groups. One bucket labelled bots is convenient, but convenience is not classification.

Server log cards sorted into response status groups

Which fields should an SEO audit classify?

Classify requests by crawler, URL pattern, template, status code, response time, content type and canonical or indexability state. Then connect each URL group to a business purpose, such as product, category, article, account, filter, internal search, asset or error.

Useful questions include:

  • Which priority URLs were requested during the period?
  • Which query parameters or filter paths consumed repeated requests?
  • How often did crawlers meet redirects, server errors or soft 404 candidates?
  • Were HTML, JavaScript, CSS and images served quickly enough for rendering?
  • Did retired URLs keep receiving requests because internal links or sitemaps still exposed them?

The goal is not to celebrate a large crawl count. A million requests to duplicate filters is still a very energetic way to avoid the important pages.

How are logs different from Search Console?

Logs are the server's request record, while Search Console provides Google's aggregated reports and inspection tools. Use both because they answer different questions.

Conceptual crawler passing through a double identity checkpoint

Google's crawling error guide says Search Console does not provide crawl history filtered by URL or path, and recommends server logs for checking whether specific URLs were crawled. It also warns that crawling and indexing are separate outcomes.

Compare log evidence with Crawl Stats, URL Inspection, indexing reports, sitemaps and analytics. A page may be fetched but excluded from indexing, indexed but rarely shown, or shown without producing qualified visits.

What patterns usually deserve investigation?

Investigate repeated requests to low-value parameter combinations, redirect chains, obsolete URLs, server errors, slow templates and resources required for rendering. Prioritise patterns that affect commercially important pages or consume substantial server effort.

SAGEO's crawl budget guide explains that advanced crawl-budget work mainly matters for large, rapidly changing or heavily undiscovered sites. Smaller sites can still use logs to diagnose access and errors, but they should not invent a crawl emergency merely because the data looks technical.

For JavaScript sites, compare requests with the rendered result. The JavaScript SEO guide separates the initial fetch from rendering and indexing, which helps explain why a requested application shell may still expose little useful content.

URL paths classified into products, filters, errors and assets

How should fixes be prioritised?

Prioritise the largest verified failure that has a safe, reviewable remedy. Examples include removing internal links to redirecting URLs, fixing a template returning 500 errors, closing an endless filter path or restoring a blocked rendering resource.

Before changing canonicals, redirects, robots rules or templates, record the exact target, current response, intended change, test set, monitoring window and rollback. Then change one pattern at a time where practical.

Measure request mix, host errors, response time and crawling of priority URLs after release. Fewer requests can be a positive outcome when more of them reach useful pages.

Limitations

Logs may be incomplete because of retention, sampling, content delivery networks, load balancers or missing fields. They do not prove indexing, ranking, citation or conversion. Crawler verification methods and user agents can change, so reopen current official documentation during each audit.

Monitoring room prepared to review crawl trends and rollback

What is the next sensible step?

Export a representative log period, preserve the raw file and classify requests without changing the live site. Ask SAGEO to turn verified patterns into a prioritised technical brief with exact targets, acceptance tests and rollback.

Sources