Screaming Frog SEO Spider: Install, Configure, and the Crawls Worth Saving

Screaming Frog SEO Spider is a desktop crawler from Screaming Frog Ltd, a UK agency. You install it on your own Windows, macOS or Linux machine, point it at a site or a list of URLs, and it fetches every page it can reach, much as a search engine bot follows links. It reports status codes, redirects, titles, canonicals, directives, hreflang and a long tail of technical data. It is built for anyone who runs technical SEO audits, plans a migration or checks a release before and after it ships.

The verdict: if you do technical SEO on sites with more than a few hundred URLs, this is the crawler to license. The free version is capped at 500 URLs and has configuration, saving and JavaScript rendering locked, so it cannot do the jobs on this page. The value of the paid version is less the single audit than the crawls you keep and compare later.

Price, checked 10 October 2026: the Screaming Frog pricing page lists one licence at £199, $279 or €245 per year, depending on the currency you select. Bulk discounts start at 5 licences (£189, £179 and £169 per licence at the 5, 10 and 20 tiers). Licences are per user: “2 users will require 2 individual licences.” The SEO Spider FAQ says EU business buyers can enter a VAT number at checkout, while buyers without a tax ID are told “tax will be added to the order”. The current version is 24.3, released 29 June 2026.

Installing Screaming Frog and entering a licence

Screaming Frog installs as a normal desktop application and needs no account for the free version. Its general user guide sets the minimum at “a 64-bit OS with at least 4gb of RAM available”. It recommends 8 GB of RAM for crawls under 200,000 URLs, and an SSD with 16 GB or more for crawls over 1 million.

1. Download the build for your system from the SEO Spider page. As of 10 October 2026 that means Windows, macOS (Apple Silicon or Intel), and Ubuntu or Fedora in both amd64/x86_64 and arm64/aarch64 builds.
2. Install. On Windows, run ScreamingFrogSEOSpider-VERSION.exe and accept the defaults. The guide also documents a silent install with /VERYSILENT.
3. Enter the licence under Licence > Enter Licence Key. The FAQ notes that the licence username “isn’t always the same as your ‘account username’”, and that this mismatch is the most common reason a key is rejected.
4. Check the storage mode under File > Settings > Storage Mode (Screaming Frog SEO Spider > Settings > Storage Mode on macOS). Choose Database Storage if the machine has an SSD.
5. Set memory under File > Settings > Memory Allocation, then restart. To confirm the change took effect, open Help > Debug and read the Max= figure on the Memory line.
6. Save your configuration once it is set up, with Configuration > Profiles > Save Current Configuration As Default, or with Save As to keep a named profile per client.

What the free version can and cannot do

The pricing page compares the two versions in a 31-row table. Read with its tick and cross markers, the free column has 7 ticks and 24 crosses, plus the 500-URL limit.

The settings that change your results

Most of Screaming Frog’s configuration can stay at its defaults. The settings below change what the crawl finds, how much it can hold, or how hard it hits the server. Every path comes from the configuration user guide, checked 10 October 2026, and each default is either stated there or marked as not stated.

One default most guides miss is that “Unlimited*” on the pricing page carries an asterisk. The configuration guide says that on a licensed copy the 500-URL cap “will be replaced with 5 million URLs” in Limit Crawl Total. The memory guide confirms that 5 million is a default rather than a hard ceiling. A crawl that stops at exactly 5,000,000 URLs has hit that setting, not the edge of the site.

Crawl politeness is documented rather than guessed. Screaming Frog says the 5-thread default exists “to not overload servers”. It warns that high thread counts “can impact a site’s response times” and that in extreme cases you “could overload a server and crash it”. It recommends “approving a crawl rate and time with the webmaster first”. This page gives no target number because the vendor publishes none: the right rate depends on the server, and you agree it with whoever runs the server.

Leave the rest alone until you have a reason to change it, including the window sizes, the preference tabs and the crawl-analysis list. Check that Configuration > Crawl Analysis is set to “Auto Analyse At End of Crawl”. Some filters, the orphan-page and sitemap filters among them, only fill once that analysis has run.

The crawls worth saving

A crawl you do not keep can only tell you about today. The four crawls below are worth storing because each becomes evidence later, and each ends in a decision. All four need a licence and database storage. Screaming Frog’s crawl comparison tutorial states that comparing crawls requires both.

1. The baseline crawl before a migration

Crawl the live site in full before anything changes. Store HTML, connect Search Console and GA4 under Configuration > API Access, and save the crawl in a named project under File > Crawls. The GA4 integration fetches 7 metrics by default, including sessions, engaged sessions and views. The Search Console integration fetches clicks, impressions, CTR and position for the last 30 days, so every URL in the baseline carries its traffic.

Decision: any URL in the baseline with clicks or sessions that has no row in your redirect map goes into the map before launch. Any non-indexable or canonicalised URL you find gets fixed or deliberately left out now, so you do not carry the problem to the new site.

2. The list-mode crawl of the old URL set

After launch, export the baseline’s URLs and upload them in list mode. Screaming Frog’s redirect audit tutorial warns that list mode “works at a 0 crawl depth” by default. Without ticking Always Follow Redirects in the Advanced tab of the spider configuration, you only see the first hop. With it ticked, the crawl follows each redirect to its final destination, and the All Redirects report under the Reports menu lists every uploaded URL with its start and final status.

Decision: a final status other than 200, or a final URL that differs from the mapped target, gets corrected in the redirect rules. A chain of more than one hop gets collapsed to a single redirect. Save this crawl too, because it is your proof of what the redirects did on the day.

3. The JavaScript-rendered crawl next to the raw one

Crawl once in Text Only mode and once with JavaScript rendering, both with HTML and rendered HTML stored. The JavaScript crawling tutorial describes a JavaScript tab with 15 filters, among them “Contains JavaScript Links”, “Noindex Only in Original HTML” and “Canonical Mismatch”. The comparison tutorial lists “raw HTML Vs JavaScript enabled crawl” as a parity check you can run in compare mode.

Decision: if “Contains JavaScript Links” fires on navigation, category or pagination pages, ask the developers to put those links in the server-sent HTML. A hit on “Noindex Only in Original HTML” is fixed at the source: the tutorial notes that Googlebot skips rendering when it meets a noindex, so removing it with JavaScript will not work. A canonical mismatch is fixed in the raw HTML.

4. The scheduled crawl you compare against

Set a recurring crawl under File > Scheduling, with a project, an interval, a saved configuration and any APIs. Version 24.0 (19 May 2026) added “Auto Compare Crawls”, which compares the last two crawls in a project and can email the changes in issues. Open any pair manually under Mode > Compare. Configure change detection under Config > Compare to catch edits that trigger no filter, such as a rewritten title.

Decision: URLs in the “Missing” column that had traffic in the baseline get investigated the same week. A key page whose crawl depth increased gets its internal links restored. Title or word-count changes nobody asked for go back to whoever shipped them.

Checklist before you press Start

1. Is the storage mode Database, and is memory allocation confirmed in Help > Debug?
2. Is the rendering mode right for this site? If you have not checked, crawl once in each mode.
3. Are robots.txt handling and the user-agent the ones you meant? Write them in the crawl name.
4. Have you agreed speed with the site owner, and are you watching response times?
5. Are HTML and rendered HTML stored if you will compare content later?
6. Are the GSC and GA4 APIs connected, if traffic decides what matters?
7. Is the crawl going into a named project so it can be compared in three months?

Custom extraction and the AI integrations

Custom extraction (Configuration > Custom > Extraction) pulls any value from a page by XPath, CSS Path or regex. By default it runs on the static HTML of internal 2XX pages; switch to JavaScript rendering to extract client-side data. Use it for author names, prices, schema fields or a count such as count(//h1).

The AI integrations sit under Configuration > API Access > AI and cover OpenAI, Gemini, Ollama and Anthropic. The configuration guide says each lets you set “up to 100 prompts against crawl data”, with results shown in an AI tab. Using page text or HTML requires Store HTML. You supply your own API key and pay the provider. For OpenAI the guide is explicit that this “is different to a ChatGPT subscription”.

Version 24.0 also added an MCP server that lets assistants such as Claude run and analyse crawls. The release post says “Not every feature and function is currently supported”. Setting up that connector is the subject of a Level 8 lesson in the course, lesson 106 on the Screaming Frog MCP, which is coming soon.

What Screaming Frog cannot tell you

Screaming Frog reports what its crawler reached from your machine, with your settings, on the day. That is a different thing from what Google has crawled, indexed or ranks, and from what real users experienced.

• Not Google’s index. The crawler uses its own user-agent and your rules. Index status arrives only through the Search Console URL Inspection API, which the configuration guide caps at “up to 2,000 URLs per property a day”.
• Not real users. Sessions and clicks come only from the GA4 and Search Console APIs. Field data from the Chrome UX Report arrives only when PageSpeed Insights runs in “remote” mode, and running Lighthouse locally gives no CrUX data. The guide puts the PageSpeed Insights API at 25,000 queries a day, which it estimates at just over 4.5 hours for 10,000 URLs. How much those metrics weigh in ranking is an open question the course has not settled.
• A memory ceiling. The memory guide starts the tool with 2 GB allocated. Memory storage “typically” holds 10,000 to 100,000 URLs, while database storage on an SSD with 4 GB reaches roughly 2 million. The guide recommends against hard disk drives in database mode.
• Not Google’s renderer. The configuration guide says the in-built Chromium captures pages up to 8,192 px tall, against 12,140 px for Google, and takes the snapshot at the configured AJAX timeout.
• Tied to one machine. A scheduled crawl starts a new instance of the program on the computer it is set up on, and overlapping schedules run in parallel. Moving a schedule means exporting a .task file and re-authenticating its APIs.
• Comparisons depend on stored crawls. Delete either crawl and the comparison “will not be accessible anymore”.

Screaming Frog’s own documentation contradicts itself on two points, and that is a defect: a reader following one page gets a different answer from the next, and none of the four pages shows a date in its visible text. The general user guide calls database storage “the default mode”, while the crawl comparison tutorial still refers to “default memory storage”. The configuration user guide puts the user-agent under Configuration > User-Agent, but the JavaScript crawling tutorial says Configuration > HTTP Header > User-Agent. Their last-modified dates sit only in the page metadata (general guide 1 February 2024, comparison tutorial 31 January 2024, configuration guide 22 May 2026, JavaScript tutorial 29 July 2026), so a reader cannot see which page is current. Trust what your installed version shows.

Alternatives and companions

Screaming Frog’s nearest like-for-like competitor in this section is Sitebulb, another desktop crawler, which gets its own page later in the series. The site-audit modules inside the large SEO suites crawl from the vendor’s cloud instead of your machine, and they get their own pages too.

The real companions are Google’s own tools, because they show the part a crawler cannot. Google Search Console shows what Google indexed and which queries reached each URL. Google Analytics 4 shows what users did after they arrived. Connecting both inside Screaming Frog is what turns a list of URLs into a list of priorities. The rest of the section is collected on the SEO tools hub.

Where the course goes deeper

This page covers the tool. The method behind each crawl, meaning why crawl depth matters, how canonicals and redirects interact and what a migration plan contains, lives in the course. Start with the technical SEO level. If you want to script crawl analysis, the Level 8 lesson on Claude Code for technical SEO picks up where Screaming Frog’s exports end. The background on how a crawler differs from Google is in SEO fundamentals. The dedicated crawl lesson in the technical level and Level 8 lesson 106 on the Screaming Frog MCP are coming soon.

Sources

• Screaming Frog, SEO Spider Pricing: free versus paid table, prices in GBP, USD and EUR, volume tiers, per-user licences. Undated page, checked 10 October 2026.
• Screaming Frog, SEO Spider Tool: free-version restrictions, £199 per year, operating-system downloads, current version 24.3. Undated, checked 10 October 2026.
• Screaming Frog, SEO Spider General user guide: installation, minimum specification, storage modes, memory, saving, scheduling, exporting. Undated, checked 10 October 2026.
• Screaming Frog, SEO Spider Configuration user guide: rendering, limits, robots.txt, user-agent, speed, include/exclude, custom extraction, API and AI integrations. Undated, checked 10 October 2026.
• Screaming Frog, How To Compare Crawls. Undated, checked 10 October 2026.
• Screaming Frog, How To Audit Redirects In A Site Migration. Undated, checked 10 October 2026.
• Screaming Frog, How To Crawl JavaScript Websites. Undated, checked 10 October 2026.
• Dan Sharp, Screaming Frog, SEO Spider Update – Version 24.0, 19 May 2026, with notes for 24.1 (8 June), 24.2 (22 June) and 24.3 (29 June 2026). Checked 10 October 2026.
• Screaming Frog, SEO Spider FAQ: licence keys, one user per licence, VAT at checkout. Undated, checked 10 October 2026.