The Crawler MCP Server

Scrape web pages, run LLM-powered structured extraction, or diagnose whether URLs are ready for a built-in extraction contract before spending LLM tokens. Open source engine (AGPL-3.0). $0.005 per successfully scraped page on Apify. It runs locally over stdio via the published package.

People connecting browser automation tools to Claude, Cursor, VS Code, or another MCP client. The project is written in TypeScript.

VERIFIED ACTIVE

LAST COMMIT 2026-07-02 · ★ 2 · #152 OF 196 MAINTAINED BROWSER AUTOMATION · VERIFIED 2026-08-25

AGPL-3.0 · TypeScript servers · how we verify → /methodology

01 · Install The Crawler

before you install - you'll need

Set THECRAWLER_API_KEY before connecting.

Claude Code

claude mcp add manchittlab-thecrawler -- npx -y thecrawler

Claude Desktop / Cursor / VS Code - add to config

{
  "mcpServers": {
    "manchittlab-thecrawler": {
      "command": "npx",
      "args": [
        "-y",
        "thecrawler"
      ]
    }
  }
}

Same JSON for Cursor. For VS Code, rename the top-level key from `mcpServers` to `servers`.

Using another client? Same JSON, different key

Claude Desktop · mcpServers

Cursor · mcpServers

VS Code · servers

Windsurf · mcpServers

Zed · context_servers

Cline · mcpServers

Roo Code · mcpServers

Continue · mcpServers

LibreChat · mcpServers

Gemini CLI · mcpServers

Codex CLI · mcp_servers

Full setup guides: every client.

02 · Evidence

Security posture

What to check before giving this server access to your agent - from the registry, GitHub, and our own probes. We don't score safety; we show what's verifiable.

runs as local process (stdio) - runs on your machine with your user's permissions

license AGPL-3.0 - declared in the repository

npm package thecrawler - unscoped; check the name against the project README before installing

registry namespace io.github.manchittlab is GitHub-verified and matches the repo owner

03 · Who maintains The Crawler

TheCrawler is maintained by manchittlab. It's the only MCP server we track from this author; the repo dates to Apr 2026.

04 · Facts

repository: github.com/manchittlab/thecrawler
category: browser automation - ranked #152 of 196 actively-maintained browser automation servers as of 2026-08-25.
registry: io.github.manchittlab/thecrawler (active, first published 2026-04-18 · 5 versions)
packages: npm:thecrawler

05 · The Crawler FAQ

What is The Crawler?

Is The Crawler still maintained?

Yes - as of 2026-08-25, its last commit was 2026-07-02. We re-verify nightly.

How do I install The Crawler?

Run `npx -y thecrawler`. The README documents one environment variable (THECRAWLER_API_KEY) to set first. Set THECRAWLER_API_KEY before connecting. You can also paste the ready-made client config above.

Does The Crawler run locally?

Yes - it's a stdio server: it runs on your machine (via npx) with your user's permissions. Your data stays local unless the server itself calls external APIs.

06 · Alternatives to The Crawler

Alternatives to The Crawler

Maintained browser automation servers if The Crawler isn't the fit.

Pairs well with

Servers that cover what The Crawler doesn't - only shown when the pairing reason fits the companion.

More browser automation MCP servers · Manovagyanik1 MCP Server · Android (martingeidobler) · Pagecast · App Publish

More TypeScript MCP servers · Devkit Server · Mapbox MCP Server · Oracle · Qnap · Sap Docs · see all