M
MCP Directory CURATED DIRECTORY
911 MCP servers indexed
中文
← Back to directory
C

CRW Web Scraper

Community
Open-source web scraping MCP server for AI agents with scrape, crawl, and map capabilities
How to install ↓ Compare GitHub source repository ↗
Category
Dev Tools #199 of 438
Stars
★ 1.2k Very popular
Transport
stdio (local process)
Runtime
Node.js · Prebuilt binary · Rust 1.85+ · Python · Hosted (no local runtime)
Credentials
Optional API key
License
AGPL-3.0
Last commit
54FMRS · D

CRW Web Scraper (fastCRW) is an open-source web scraping and search MCP server written in Rust, offering scrape, crawl, map, search, and extract capabilities with a Firecrawl-compatible /scrape, /crawl, /search API. It emphasizes a single low-memory binary and self-hosting, while also offering a managed cloud option, making it a fit for developers who want data control and AI-agent integration. Note that it is not an official Firecrawl, Tavily, or Anthropic project, the engine is AGPL-3.0 licensed, and usage must respect target site policies and robots.txt.

Strongest · Maintenance 12/20 Weakest · Reliability 10/20

Reliability
10/20
Security and permissions
11/20
Maintenance
12/20
Documentation
10/20
Setup experience
11/20
Why each score
Reliability 10/20
The manifest declares stdio transport launching an OCI image (ghcr.io/us/crw:latest) with a positional 'crw-mcp' argument, and the README claims five operations (scrape, crawl, map, search, extract) with published crates.io/PyPI/npm artifacts and a CI badge, so a reproducible build path is plausible. However, this is a static review: no source was available showing tool implementations, parameter schemas, or error-handling code; the MCP tool input/output contracts, timeouts, retries, and rate-limit behavior are undisclosed. The manifest title/description emphasizes scrape/crawl/map while the README adds extract, so declared-tools-vs-behavior alignment cannot be confirmed. No committed tests covering the server's key paths are visible. Per static calibration, reliability is capped at 12 without such evidence; scoring 10 for a self-consistent capability surface with real package/CI signals, deducting for unverifiable tool semantics and edge-case handling.
Security and permissions 11/20
Positive signals: AGPL-3.0 license, crawl/map documented to honor robots.txt by default, the API key stored in ~/.config/crw/config.toml rather than copied into each AI tool's config, and automatic agent registration limited to already-configured tools with a CRW_NO_AGENTS=1 opt-out, which reflects reasonable least-privilege and data-boundary thinking. Deductions: the primary install method is 'curl -fsSL https://fastcrw.com/install | sh', piping a remote script straight into a shell, and it can be invoked with a real-format CRW_API_KEY=crw_live_..., an unauditable high-risk default path; the README repeatedly funnels users to registration/credits links, yet no data-flow disclosure exists for the managed mode (scraped content, keys, telemetry); MCP tools perform outbound fetching/crawling with side effects but no confirmation or scope-limiting is described. No malware, credential theft, fabricated maintainer identity, real leaked secrets, or unconfirmed critical vulnerabilities were found, so no red line is triggered; 11 reflects visible-but-incomplete controls and undisclosed data flows.
Maintenance 12/20
Clear AGPL-3.0 licensing for the engine and MCP server and MIT for SDKs; multi-registry releases on crates.io/PyPI/npm; version 0.34.0 indicating sustained iteration; a CI workflow badge; a contributor list with several external contributors; a non-archived repo with 967 stars and only 9 open issues — all consistent with active maintenance. Gaps: no security-response or vulnerability-disclosure channel in the README, no stated release cadence or semantic-versioning policy, no visible dependency-update mechanism, an 'embedding license' pointing to an email rather than a clear license text, and limited governance/ownership transparency (multiple packages under a personal-looking 'us' account). Static evidence supports 12: active, but with governance and versioning gaps.
Documentation 10/20
Documentation is reasonably layered: the README covers installation, CLI usage, Python/Node SDKs, MCP installation, managed-vs-self-hosted differences, migration and self-hosting links, plus pointers to /v1/capabilities and response shapes, and it discloses that robots.txt is followed by default. Key gaps remain: the five MCP tools' parameters, return shapes, quotas/cost, rate limits, timeouts, and troubleshooting are not documented in-repo and require external sites; benchmark claims are presented as self-authored SVGs with external links and no reproducible methodology in the supplied material; comparative numbers such as '2.3x faster than Tavily' lack in-repo verification; manual MCP client setup is only a link with no config examples. Usable, but with hidden assumptions and troubleshooting blind spots: 10.
Setup experience 11/20
The install path is clear and short: a single curl command, local mode needs no account, and the same command with CRW_API_KEY can configure the key and register the MCP server, with automatic detection of Claude Code, Cursor, Codex, Gemini CLI, OpenCode, and Windsurf; an 'npx -y crw-mcp@latest install' path and interactive 'crw setup' also exist. macOS and Linux on Intel and ARM are supported (Rust 1.85+ only for building from source). Deductions: the headline method pipes a remote script into a shell, which is fragile and unauditable; the manifest offers only an OCI image with stdio and no per-client JSON configuration examples; Windows support is unmentioned; and default auto-registration mutates multiple AI tool configs (opt-out exists, but the default writes). Per static calibration setup cannot exceed 15; scoring 11.

Static review · not runListed 2026-09-10

Read the FMRS scoring method →

Fit and risk

What it can accessUses the network

Best for

  • Teams that want self-hosted web scraping with data control
  • Developers using Claude Code, Cursor, Codex, Gemini CLI, OpenCode, or Windsurf
  • Scenarios needing a low-resource single Rust binary
  • Migration projects requiring a Firecrawl-compatible REST API

Not for

  • Users who need official Firecrawl, Tavily, or Anthropic backing or support
  • Scraping needs that will not respect site policies and robots.txt
  • Teams needing a fully managed SaaS without deploying anything themselves

Required permissions

  • Network access: outbound HTTP/HTTPS requests to scrape, crawl, and search the web
  • Local filesystem read/write: to store configuration such as ~/.config/crw/config.toml
  • The install script may modify MCP configuration of detected AI tools
  • A CRW_API_KEY credential is required when using the managed cloud

Risks and side effects

  • Scraping third-party sites may conflict with their terms of service, copyright, or robots.txt rules; the project states crawl and map follow robots.txt by default
  • The one-command install downloads and executes a remote script, which carries supply-chain risk
  • The API key is stored in ~/.config/crw/config.toml, so local file permissions and leakage matter
  • The container image uses the latest tag, which may make versions non-reproducible
  • Managed cloud and local capabilities or response shapes can differ; check /v1/capabilities and response shapes before migrating

Setup

Before you start

Runtime：Node.js · Prebuilt binary · Rust 1.85+ · Python · Hosted (no local runtime)

CRW_API_KEY optionalsecret API key (crw_live_...) for fastCRW Cloud, available free at fastcrw.com/register; not needed for local mode.
Other optional settings (1)
CRW_NO_AGENTS optional Set to 1 to skip auto-registering the MCP server into detected AI tools during install.

Option 1 (one-command install): run curl -fsSL https://fastcrw.com/install | sh; local use requires no account. To use the cloud, pass your key in the same command: curl -fsSL https://fastcrw.com/install | CRW_API_KEY=crw_live_... sh, which also registers the MCP server with detected AI coding tools. Option 2 (MCP-only install): run npx -y crw-mcp@latest install to install the CRW skill and MCP server in detected tools. You can also run crw setup on its own for interactive configuration. Container option: use image ghcr.io/us/crw:latest with stdio transport and the positional argument crw-mcp.

Check that it works

After running npx -y crw-mcp@latest install, check your client's tool list for CRW's scrape/crawl/map tools, or issue a scrape request and confirm markdown content is returned.

Troubleshooting

  1. Command not found: confirm the crw binary is installed via the install script or a package manager and is on PATH
  2. Search unavailable: search requires configuration via `crw setup` first
  3. Cloud calls failing: verify CRW_API_KEY is exported or present in the config file and is valid
  4. Do not want automatic AI tool registration: set CRW_NO_AGENTS=1 to skip that step
  5. MCP client does not detect the server: re-run `npx -y crw-mcp@latest install` or `crw setup`, and consult the mcp-clients manual setup docs
  6. Build from source fails: confirm Rust 1.85 or newer and run `make check-fast`

Things to try

Once connected, you can ask your AI assistant things like:

  • Scrape https://example.com and return it as markdown
  • Crawl this site and list all its page URLs
  • Search for rust async runtime and summarize the top results
  • Extract product names and prices from this page

Use cases

Let AI agents scrape a single URL into Markdown or structured JSON
Follow a bounded site crawl and collect page content
Discover site URLs without scraping every page
Run a web search and optionally scrape selected results
Extract structured fields from one or many URLs
Serve as a drop-in Firecrawl-compatible alternative for /scrape, /crawl, /search calls

Overview

CRW Web Scraper (fastCRW) is an open-source web scraper, crawler, and search tool written in Rust, packaged with an MCP server for AI agents. It turns URLs into clean Markdown or structured JSON and covers scrape, crawl, map, search, and extract operations. It exposes a Firecrawl-compatible API (/scrape, /crawl, /search) and can be self-hosted as a single binary (about 6 MB RAM) or used through a managed cloud service. The engine and MCP server are licensed AGPL-3.0, while the Python and TypeScript SDKs are MIT. It is not maintained by Firecrawl, Tavily, or Anthropic.

Similar servers

Context7 80 · B

Upstash's official server providing up-to-date third-party library docs for AI coding assistants

★ 63.3k · Tools 2 Compare with this →
Compare with other MCP servers

Source revision ede27192236c Data synced 2026-10-19 Read the FMRS scoring method