Screaming Frog V25+: every new feature, tested with Claude MCP.

Screaming Frog V24 vs V25+: what’s actually different?

Many posts credit V25 with the Claude integration. That is wrong: the official MCP server arrived in version 24.0 on 19 May 2026 ↗. V25+ builds on it so the assistant can configure the crawl as well as drive it.

How does Claude work with Screaming Frog V25+?

The SEO Spider ships its own MCP server. Claude connects to it and calls tools to configure, crawl, export and analyse. It needs a paid licence and database storage mode, and Node tools stay off until you enable them (MCP server guide ↗).

Set it up in five steps

1. In the SEO Spider, open File › Settings › MCP Server, enable the server and accept the Node.js runtime.
2. Download the official Claude Desktop extension (.mcpb) for STDIO or Streamable HTTP.
3. In Claude Desktop, go to Settings › Extensions › Advanced settings › Install extension.
4. Quit Claude fully (on Windows, from the system tray), reopen it and check Settings › Developer shows the server running.
5. Ask Claude to list the SEO Spider config categories. A list back means you are connected.

What Claude can now configure

Asked for its configuration categories, the V25.1 MCP returned 37, from crawl scope, user agent and speed to issue overrides, embeddings, custom JavaScript and prompts for OpenAI, Gemini, Ollama and Anthropic.

// sf_list_config_categories (excerpt, V25.1)
{ "name": "linkPathExcludeManager",
  "description": "Link path exclude: regexes that stop links being followed by where they sit in the page." }
{ "name": "performanceConfig",
  "description": "Performance ('Speed'): how fast the crawl may go and how it reacts when a host pushes back." }

How it helpsAgencies can keep one config per client and let Claude adapt it per audit, such as switching on rendering for a JavaScript site, without rebuilding settings by hand.

Let Claude change settings the safe way

One trap: partial updates leave other fields alone, but lists are replaced, not merged. If Claude adds one exclude rule without resending the existing ones, the old rules disappear.

Before changing any SEO Spider setting, read its schema and current values.
Change only what this task needs, resend every entry of any list you edit,
then show me the values before and after.

When a crawl fails

The progress tool flags a crawl that stopped after the start URL as a likely failure, and an FAQ tool gives Claude Screaming Frog’s official answers on 403s, 429s, Cloudflare and rendering before it diagnoses.

How it helpsA blocked or failed crawl gets flagged and explained before anyone reads the data, so you don’t send a client an audit built on three crawled pages.

Two safety habits: API keys stay in the app (the connect tool never accepts one), and because Claude Desktop uses tokens fast, save exports to files and start a new chat per job.

The new V25+ features, one by one

For each feature: what it does, where to find it, and how it helps in a real audit.

JSON config

Saved configurations are now JSON, readable in a text editor and editable by Claude. Old .seospiderconfig files still load. JSON file and CLI usage aren’t fully documented yet, so test before switching scheduled jobs. One catch I found: an API key pasted into a custom JavaScript snippet travels inside the config, so check it before sharing or committing.

How it helpsTeams can review config changes like code: compare two versions, see exactly what changed between audits, and roll back when a crawl suddenly behaves differently.

Configurable Issues

Right-click any issue in the Issues tab to disable it or change its type (Issue, Warning, Opportunity) and priority. You can select several at once.

How it helpsYour issue list matches your own audit priorities, so clients see high-priority items first instead of a wall of generic warnings to triage.

Auto Throttle

The crawler slows down when a host returns 429 or 503 responses or drops connections, recovers gradually, and pauses if the host stops responding. A new Connection Failure Retries setting sits under Spider › Advanced.

How it helpsYou can crawl a fragile staging server or a rate-limited ecommerce site without straining it, and without reporting 5xx errors your own crawler caused.

Markdown view and export

With Store HTML or Store Rendered HTML enabled, the View Source tab gains a Markdown option, and Bulk Export › Web › All Page Markdown exports every page. Through MCP, page content can be requested as MARKDOWN.

How it helpsYou can give Claude one page, or feed hundreds into embeddings, without HTML noise, and test whether each page answers its question clearly.

Custom Dashboard

The new Dashboard tab takes chart, counter and table widgets. Click + to start, then save, share or export the view. It presents crawl data; it does not add new data. The MCP has no dashboard tool, so build these in the app.

How it helpsLeadership gets a one-screen health view of indexability, errors and priority issues, without an analyst exporting CSVs and building charts after every crawl.

Social tab

The Social tab checks Open Graph and X Card tags: missing, incomplete or duplicate tags, image errors, sizes and tags placed outside the head. No source shows it affects AI answers.

How it helpsMarketing teams catch broken LinkedIn or Slack previews before a campaign link goes out, rather than finding a missing image after it has been shared.

Exclude links by link path

Stop links being followed based on where they sit in the page, such as a filter block, even when their URLs share no pattern. A page linked only from that block drops out of the crawl.

How it helpsOn faceted ecommerce sites you can stop the crawler following filter links, so crawl time and internal-link data focus on the pages that matter.

Network Idle rendering

Network Idle is the new default wait for JavaScript rendering. It waits until network activity and the event loop go quiet, which Screaming Frog describes as closer to how Google decides a page has finished rendering.

How it helpsContent that JavaScript adds a moment after load is less likely to be missed, so you don’t flag thin content or missing links that users actually see.

Filename Patterns, scheduling and CLI

A new Export Filename setting defines the default name for scheduled tasks, multi-exports and tab exports. Insert Token offers Date, Date and Time, Domain, Crawl Name, Project and Report Name; the default is {report}. Schedules gain enable, disable, Run Now and pause. The CLI adds pause, resume and delete; check –help for exact syntax, as some commands aren’t documented yet.

How it helpsWeekly exports arrive with predictable names, so scripts, dashboards and crawl comparisons can pick them up automatically without anyone renaming files.

New issues

Four checks join the Issues tab: Hreflang Self Reference Only, Hreflang Mismatch with HTML Lang Attribute, Pages With Empty Anchor Tags and Missing Image Title. The two hreflang checks matter most on multilingual sites. Missing image titles are low priority for most audits.

How it helpsInternational sites can catch pages that declare one language in hreflang and another in the HTML lang attribute, a mix-up that muddles language targeting.

How to use Screaming Frog V25+ for AEO and GEO audits

Screaming Frog doesn’t simulate AI answers. It shows whether pages can be found, read and trusted, which is what Google says matters: “There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary.” A page must be indexed and eligible for a snippet, and you don’t need AI text files or special markup (Google: AI features and your website ↗). Google’s newer guide adds that the site must be included in Search generative AI features in Search Console (AI optimization guide ↗).

For deeper render checks, see my JavaScript SEO render gap guide, and for bot traffic, log file analysis for SEO and AI crawlers.

The Two-Ledger Audit: a repeatable AI-search workflow

V25+ makes it easy to let Claude crawl and judge in one go. The risk is one score that mixes measured facts with model opinion, so I keep two ledgers.

1. Step 1
   

   Pick the URLs

   10 to 50 priority pages from Search Console, AI referral traffic or your content plan.

2. Step 2
   

   Let Claude configure

   Rendering, Store HTML and Store Rendered HTML, user agent. Use the safe read-then-write loop.

3. Step 3
   

   Crawl in List Mode

   Wait until the crawl and the post-crawl analysis both finish before reading results.

4. Step 4
   

   Export Ledger 1

   Response codes, indexability, structured data, hreflang and Social issues.

5. Step 5
   

   Build Ledger 2

   Export Markdown and ask Claude to grade each page on answer, entities and evidence.

6. Step 6
   

   Act in order

   Fix Ledger 1 first: a page that can’t be indexed can’t be cited. Ledger 2 guides editorial work.

Switch to List Mode and crawl only these URLs. Store rendered HTML.
When the crawl and post-crawl analysis are complete, give me two sections per URL:
1) Spider facts: status, indexability, canonical, structured-data errors, hreflang.
2) Your judgement of the Markdown: direct answer, definitions, entities, evidence.
Never mix the two. Report missing values as missing.

What V25+ can’t do, and should you upgrade now?

Upgrade advice: one user reported crashes on launch day, and Screaming Frog offered V24 or a beta with a likely fix. That is one report, not a pattern. Upgrade interactive work now; for scheduled or CLI jobs, re-run one known crawl on V25.1 first and back up your V24 configs.