Log file analysis for SEO and AI crawlers.

What is log file analysis for SEO?

Log file analysis for SEO is the practice of reading your server’s request records to see which URLs search and AI bots fetch, how often they return, and what response each request received. It tells you whether the right bots are reaching the pages that matter to the business.

An access log records requests handled at one layer of your infrastructure. Depending on configuration, each line can hold the client IP address, timestamp, requested URL, user agent, status code and bytes sent.

Logs answer questions such as whether Googlebot requested your product pages after a migration, or kept receiving server errors from an important category. They cannot confirm indexing, rankings, rendering or AI citations. That gap shapes everything that follows.

How do logs differ from analytics, Search Console and a site crawl?

They measure different things, so a mismatch between them is a prompt to investigate, not a sign that one is broken.

The Evidence Ladder

Most bad log file decisions share one mistake: claiming a result from request evidence. I use a five-rung ladder to stop it. Each rung needs its own proof, and your logs only reach the first two.

Name the rung in every recommendation. “Googlebot received 150 server errors on /services/” sits on rung 2. “Fixing it will lift leads” sits on rung 5 and needs separate proof, collected after the change.

Is log file analysis worth doing for your site?

Do it when you have a specific crawl question, not as a routine report. Google’s crawl budget guidance is aimed at very large or fast-changing sites. A small site whose important pages are fetched promptly rarely needs a crawl budget programme.

Choose the question before you open any tool. “Why are our new service pages not being fetched?” produces a better audit than “How do we get more bot hits?”

How do you get your server log files?

Ask your hosting provider, infrastructure team, control panel or CDN for an export. You cannot retrieve a private site’s logs by entering its public URL into a crawler.

• Apache: ask which files the CustomLog directive writes. /var/log/apache2/ and /var/log/httpd/ are common, but installs vary.
• NGINX: ask for the configured access_log destination and log_format. /var/log/nginx/access.log is a common location, not a guarantee.
• cPanel: look for Raw Access under Metrics.
• CDN: if a CDN answers from cache, those requests never reach your origin. Request edge logs too, with cache and security outcomes where the plan provides them.

Send this to your developer. It asks for every field the rest of this guide needs.

Please export access logs for the agreed hostname and date range, including rotated files. Include timestamp and timezone, client IP, host, request path and query string, method, status, bytes, and user agent. Where configured, include request duration, cache outcome and request ID. Confirm any sampling, any missing periods, and whether the export comes from the CDN, load balancer or origin.

Copy request

Store exports in approved, access-controlled storage. Redact credentials, tokens and personal information before sharing examples, and keep the untouched original for verification.

How do you analyse server logs, step by step?

Fix the dataset first, verify the bot second, then read status codes, crawl waste and coverage in that order. Skipping ahead is how teams end up fixing a chart instead of a problem.

For a first investigation I suggest 14 to 30 complete days. That is a working recommendation, not a search engine rule. Check timezone consistency, log rotation, missing days, sampling and hostname coverage. Keep raw URLs before you create grouped fields, because query strings and letter case can be the whole story.

Do not add CDN and origin totals together. One external request can appear in both. Pick one primary reporting layer and use request IDs to study the overlap.

Log formats vary. Confirm your parser and any custom fields before you rely on command-line shortcuts.

A user agent containing “Googlebot” is a claim, not proof. Run a reverse DNS lookup on the IP in your log, check that the hostname ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup on that hostname and confirm it returns the original IP.

Google documents this method in verifying requests from Google crawlers. This example uses the address and hostname from that page:

host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

For large datasets, automate the check by matching IPs against the published Google ranges in common-crawlers.json. Behind a proxy or CDN, confirm which log field holds the real client IP. A proxy address or an untrusted forwarded header invalidates the check.

Which grep or awk filter finds claimed Googlebot hits?

For a conventional combined-format Apache or NGINX log, this prints the prefix, path and status for rows whose user-agent field contains Googlebot:

awk -F'"' '$6 ~ /Googlebot/ {split($2,r," "); split($3,s,/ +/); print $1, r[2], s[2]}' access.log | head -n 20

A simpler exploratory filter is grep -i 'Googlebot' access.log | head -n 20. Neither command authenticates anything. They list claims, and you verify the source IPs separately. Adjust for JSON, CSV, custom, rotated or compressed formats, and never treat partial output as a complete audit.

Separate verified Googlebot from Bingbot, AI crawlers, monitoring tools and unknown automation. Keep an “unverified” bucket for anything you cannot prove.

Within verified Google traffic, split Googlebot Smartphone from Googlebot Desktop using the recorded user agent. Because Google uses mobile-first indexing, the smartphone crawler is generally the main one for Search, and a small share of desktop requests can be normal. If a mobile-only redirect, navigation or access rule affects important pages, test both versions instead of reading a count gap as the fault. Both variants share one robots.txt token, Googlebot, so you cannot control them separately there.

Then group URLs into sections, and separate HTML documents from images, JavaScript and CSS. For each segment review requests per day, unique URLs requested, status mix and last observed request. A flood of image hits should never hide a service directory that gets almost no HTML requests. Compare every segment with its own history. There is no universal healthy number of Googlebot visits per day.

Start with failures on valuable pages, then redirects, then speed. Google says server errors and 429 responses lower its crawl capacity limit, so repeated failures on commercially important URLs come before any cosmetic cleanup.

Timing fields need care. NGINX’s $request_time measures request processing time. It is not a Core Web Vitals metric. Compare like with like, and inspect the slow tail as well as the average.

Crawl budget is the set of URLs Google can and wants to crawl, shaped by crawl capacity and crawl demand. Waste is the share of that effort spent on URLs you have reviewed and decided have no search value.

Inspect repeated requests to internal search results, endless filter combinations, tracking variants, outdated redirect chains and duplicate URL patterns. Do not assume that every low-value-looking URL is waste. A filtered category may carry real search demand, and a JavaScript file may be essential for rendering. Define the approved URL inventory before you propose any control.

• Fix the source: correct the links that generate unwanted variants, shorten redirect paths and consolidate duplicates.
• A canonical tag signals a preferred URL. It does not stop Googlebot requesting the other variants.
• Robots.txt controls crawling, not removal from search. Google must fetch a page to see its noindex, so plan the sequence before you combine blocking with deindexing.
• Do not promise that cutting wasted hits will lift crawling of priority pages. Google says freed budget moves elsewhere only if the site is already at its crawl capacity limit.

Orphan pages are URLs with no discoverable internal links from the pages in scope. Logs alone cannot prove that links are missing, so compare URLs from logs, XML sitemaps and your CMS with a properly configured site crawl.

A URL present in logs but absent from your internal crawl is an orphan candidate. Check crawl exclusions, authentication, JavaScript navigation and crawl coverage before you declare it orphaned. For valuable pages with no observed bot requests, inspect internal links, sitemap inclusion, canonical tags, robots rules, response status and publication timing. The correct label is “not observed in this dataset”.

Keep a URL-level record of the evidence, likely cause, recommended change, owner and validation date. That turns an interesting chart into a task someone can ship.

Recheck the same URL groups in a comparable window after release. Did verified bots receive successful responses? Were the unwanted variants reduced? Did priority pages get their first observed fetch? Only then review indexing and search performance, which sit on rungs 3 to 5 of the ladder. My Search Console API and BigQuery guide is a starting point for the page and query data, and you should check current API limits before you scale it.

More crawling does not automatically mean more leads. Tie the change to relevant impressions, clicks, qualified enquiries or revenue, and record the other launches and seasonal swings in the same period.

What does a real investigation look like?

Suppose valuable service pages are in your XML sitemap but rarely appear in verified Googlebot HTML requests. Do not call them orphans, and do not blame crawl budget, on that evidence alone.

1. Check the dataset. Confirm date range, hostname, log layer, verified bot identity, HTML filtering and missing records. Label zero-hit URLs “not observed in this dataset”.
2. Compare the inventory. Match the URLs against the sitemap, the internal link crawl, canonical status, robots rules and the live response. Look for differences by page type and by Googlebot variant.
3. Prioritise fixes. Repair broken internal links or blocked resources, correct template-level causes and resolve verified 429 or 5xx problems. Never add redirects or indexing controls just to move a chart.
4. Validate. Repeat the same collection and grouping after release, then review indexing, organic performance and conversions on their own.

Which AI crawlers should you look for in your logs?

Separate search discovery, potential model training and user-triggered fetching. Calling all three “AI visibility” hides what each one can tell you.

OpenAI documents its roles independently, with separate search and training controls. Anthropic and Perplexity publish their own guidance. Each publishes IP ranges too. A user-triggered fetch is a person asking a question, so those fetchers may ignore robots.txt. Do not assume every bot behaves like Googlebot.

What does a verified AI bot request prove?

It proves a fetch occurred at the layer you observed. It does not reveal the user’s prompt, prove training use, or show that your page was quoted or cited in an answer. A citation can also use content retrieved earlier, with no new request inside your measurement window. Measure crawler access, observed citations, referral visits and conversions as four separate things.

For AEO and GEO, split access checks from content checks. Can the system retrieve the page, and does the page answer the question clearly? Use logs for the first. For the second, record prompts, dates, platforms and cited URLs. For Google’s AI Overviews and AI Mode, focus on Googlebot access and search eligibility: a supporting page must be indexed and eligible for a snippet, and logs alone can establish neither. No special schema guarantees inclusion.

Which SEO log file analyser should you choose?

Choose by data volume, review frequency, integrations, bot verification and security, not by feature count. A dedicated SEO product is usually easier to run. A general log platform is more flexible but needs SEO-specific setup.

The Screaming Frog Log File Analyser is a separate application from the SEO Spider. A typical workflow is to import logs, verify bots, filter the segment, then match a URL export from the SEO Spider. For recurring enterprise work, compare candidates on the same representative log sample, URL inventory and approval criteria. Confirm capabilities in your own account, because plans differ. Splunk is a general search and observability platform, so verified bot filters, URL grouping and indexing joins remain your team’s job.

How often should you review log files?

Match the schedule to how fast the site changes and how costly a miss would be. Record deployments and configuration changes beside your charts so every pattern has context.

Where does log file analysis break down?

It breaks wherever the log does not see the whole request path. These are the limits I check before I trust a finding.

• Cached traffic. When a CDN answers first, origin logs undercount. Edge logs fix coverage but bring different IP and timestamp fields.
• Rendering. A log shows that a JavaScript file was fetched. It does not show that the page rendered. Pair it with URL Inspection’s rendered HTML, and see my note on rendering in JavaScript SEO.
• Thin data. On a low-traffic site a 14-day window produces noisy zero-hit lists. Extend the window before you diagnose anything.
• No log access. Some managed platforms give you only Crawl Stats and URL Inspection. Work from those and state the lower certainty in your report.
• Wrong IP field. If your proxy rewrites the client IP, every verification result is void.

What should you record before recommending a fix?

Five checks decide whether a log finding is strong enough to send to a developer. Tick them off as you go.

Start with one question you can resolve

Choose a valuable section of your site, get a reliable dataset, and investigate one measurable problem. For every recommendation, write down the evidence, the expected technical change, and the outcome you will assess separately.

Want help choosing that first investigation? Message me on LinkedIn with your website, the affected section and the issue you want to solve. Share raw logs only through a secure channel agreed with your infrastructure team.