Screaming Frog

New crawl

Configure a crawl. The target must be on the server allowlist.

Target

Crawl mode

Start URL

Name (optional)

Include URLs (regex, one per line)

Exclude URLs (regex, one per line)

User agent

Settings

Max URLs

Max depth

Concurrency

Per host / sec

Respect robots.txt

Accessibility (axe-core)

Follow links

Rendering & audits

JavaScript rendering

Captures the post-JS DOM and raises the JAVASCRIPT_* checks.

PageSpeed (Lighthouse)

Runs Lighthouse and raises the PAGESPEED_* checks.

AI content review (OmniRoute)

Uses OMNIROUTE_API_KEY from the server environment; pages are flagged AI_UNAVAILABLE when it is unset.

Pages per integration pass

Bounds rendering, Lighthouse, AI, GA and GSC work. Default 25, maximum 200.

Search & analytics data

Google Analytics 4

Search Console

Tokens come from the server environment (GA_TOKEN / GSC_TOKEN / GOOGLE_APPLICATION_CREDENTIALS). Missing credentials flag pages ANALYTICS_UNAVAILABLE or SEARCH_CONSOLE_UNAVAILABLE instead of failing.

Custom extraction, search & JavaScript

Extraction — name | selector | attribute

Search — name | contains|does_not_contain|regex | value

JavaScript — name | expression (needs rendering)

JavaScript rules run in the rendered page, so they require JavaScript rendering.