Crawlie MCP Server

The fast, free, open-source technical SEO + GEO crawler - built for humans and agents. Its 5 documented tools cluster into audit, crawl, explain, read & search. It is available as a remote MCP endpoint.

Teams that want a hosted web scraping endpoint instead of running a local process. The project is written in Rust.

VERIFIED ACTIVE

LAST COMMIT 2026-07-18 · ★ 99 · #55 OF 78 MAINTAINED WEB SCRAPING · VERIFIED 2026-08-25

NOASSERTION · Rust servers · how we verify → /methodology

01 · Install Crawlie

Claude Code

claude mcp add spronta-crawlie --transport http https://crawlie.app/mcp

Claude Desktop / Cursor / VS Code - add to config

{
  "mcpServers": {
    "spronta-crawlie": {
      "url": "https://crawlie.app/mcp"
    }
  }
}

Same JSON for Cursor. For VS Code, rename the top-level key from `mcpServers` to `servers`.

Using another client? Same JSON, different key

Claude Desktop · mcpServers

Cursor · mcpServers

VS Code · servers

Windsurf · mcpServers

Zed · context_servers

Cline · mcpServers

Roo Code · mcpServers

Continue · mcpServers

LibreChat · mcpServers

Gemini CLI · mcpServers

Codex CLI · mcp_servers

Full setup guides: every client.

02 · Evidence

https://crawlie.app/mcp

transport: streamable-http

endpoint alive - authentication required · probed 2026-08-25

Security posture

What to check before giving this server access to your agent - from the registry, GitHub, and our own probes. We don't score safety; we show what's verifiable.

runs as remote - your agent's requests go to crawlie.app

endpoint auth requires authentication - rejected our unauthenticated MCP initialize

license no standard license detected - usage rights unclear; check the repo before commercial use

registry vendor namespace app.crawlie - domain-verified with the official MCP registry

03 · What Crawlie can do

Prose above is summarized from the project's README and registry record - no invented capabilities.

What you can build

An agent gets 5 documented tools across audit, crawl, explain, including crawl_site, audit_url, audit_urls, explain_issue.

Capability map

Tools grouped from the project's README - what Crawlie lets an agent do.

Audit

2 tools - e.g. Audit a single page; Audit an explicit list of pages

audit_url · audit_urls

Crawl

1 tool - e.g. Crawl + audit a whole site (SEO + GEO), returns scores, issues, per-page data

crawl_site

Explain

1 tool - e.g. Why a rule matters + how to fix it

explain_issue

read & search

1 tool - e.g. The full catalogue of checks

list_rules

Latest releases

v0.5.3 · 2026-06-29

CLI + MCP: npm i -g crawlie. Desktop app: download the signed .dmg below.

v0.5.2 · 2026-06-28

CLI + MCP: npm i -g crawlie. Desktop app: download the signed .dmg below.

v0.5.1 · 2026-06-26

CLI + MCP: npm i -g crawlie. Desktop app: download the signed .dmg below.

04 · Who maintains Crawlie

Crawlie is maintained by spronta. We track 2 MCP servers from spronta - 2 actively maintained, 100 combined GitHub stars, oldest repo from Mar 2026.

05 · Facts

repository: github.com/spronta/crawlie
website: https://crawlie.app/
category: web scraping - ranked #55 of 78 actively-maintained web scraping servers as of 2026-08-25.
release cadence: 10+ releases in the last 90 days (latest 2026-06-29)
registry: app.crawlie/crawlie (active, first published 2026-07-11)

06 · Crawlie FAQ

Is Crawlie still maintained?

Yes - as of 2026-08-25, its last commit was 2026-07-18 and it shipped 10+ releases in the last 90 days. We re-verify nightly.

What can Crawlie do?

An agent gets 5 documented tools across audit, crawl, explain, including crawl_site, audit_url, audit_urls, explain_issue.

How do I install Crawlie?

Run `claude mcp add spronta-crawlie --transport http https://crawlie.app/mcp`. You can also paste the ready-made client config above.

Does Crawlie require authentication?

Yes - when we probed the endpoint on 2026-08-25, it rejected an unauthenticated MCP initialize; you'll need credentials to connect.

07 · Alternatives to Crawlie

Alternatives to Crawlie

Maintained web scraping servers if Crawlie isn't the fit.

Pairs well with

Servers that cover what Crawlie doesn't - only shown when the pairing reason fits the companion.

More web scraping MCP servers · Search1 API · Indonesia Civic Stack · Read GZH · Charlotte

More Rust MCP servers · Lain · Dbx · Memkeeper · Pond · Raven Nest MCP · see all