← Back to directory
C

CRW Web Scraper

Community
Open-source web scraping MCP server for AI agents with scrape, crawl, and map capabilities
GitHub source repository ↗
★ 967 Stars Category · Dev Tools Very popular
54FMRS · D

CRW Web Scraper (fastCRW) is an open-source web scraping and search MCP server written in Rust, offering scrape, crawl, map, search, and extract capabilities with a Firecrawl-compatible /scrape, /crawl, /search API. It emphasizes a single low-memory binary and self-hosting, while also offering a managed cloud option, making it a fit for developers who want data control and AI-agent integration. Note that it is not an official Firecrawl, Tavily, or Anthropic project, the engine is AGPL-3.0 licensed, and usage must respect target site policies and robots.txt.

Reliability
10/20
Security and permissions
11/20
Maintenance
12/20
Documentation
10/20
Setup experience
11/20
Read the FMRS scoring method →

CRW Web Scraper (fastCRW) is an open-source web scraper, crawler, and search tool written in Rust, packaged with an MCP server for AI agents. It turns URLs into clean Markdown or structured JSON and covers scrape, crawl, map, search, and extract operations. It exposes a Firecrawl-compatible API (/scrape, /crawl, /search) and can be self-hosted as a single binary (about 6 MB RAM) or used through a managed cloud service. The engine and MCP server are licensed AGPL-3.0, while the Python and TypeScript SDKs are MIT. It is not maintained by Firecrawl, Tavily, or Anthropic.

Setup

Option 1 (one-command install): run curl -fsSL https://fastcrw.com/install | sh; local use requires no account. To use the cloud, pass your key in the same command: curl -fsSL https://fastcrw.com/install | CRW_API_KEY=crw_live_... sh, which also registers the MCP server with detected AI coding tools. Option 2 (MCP-only install): run npx -y crw-mcp@latest install to install the CRW skill and MCP server in detected tools. You can also run crw setup on its own for interactive configuration. Container option: use image ghcr.io/us/crw:latest with stdio transport and the positional argument crw-mcp.

Fit and risk

Best for

  • Teams that want self-hosted web scraping with data control
  • Developers using Claude Code, Cursor, Codex, Gemini CLI, OpenCode, or Windsurf
  • Scenarios needing a low-resource single Rust binary
  • Migration projects requiring a Firecrawl-compatible REST API

Not for

  • Users who need official Firecrawl, Tavily, or Anthropic backing or support
  • Scraping needs that will not respect site policies and robots.txt
  • Teams needing a fully managed SaaS without deploying anything themselves

Required permissions

  • Network access: outbound HTTP/HTTPS requests to scrape, crawl, and search the web
  • Local filesystem read/write: to store configuration such as ~/.config/crw/config.toml
  • The install script may modify MCP configuration of detected AI tools
  • A CRW_API_KEY credential is required when using the managed cloud

Risks and side effects

  • Scraping third-party sites may conflict with their terms of service, copyright, or robots.txt rules; the project states crawl and map follow robots.txt by default
  • The one-command install downloads and executes a remote script, which carries supply-chain risk
  • The API key is stored in ~/.config/crw/config.toml, so local file permissions and leakage matter
  • The container image uses the latest tag, which may make versions non-reproducible
  • Managed cloud and local capabilities or response shapes can differ; check /v1/capabilities and response shapes before migrating

Troubleshooting

  1. Command not found: confirm the crw binary is installed via the install script or a package manager and is on PATH
  2. Search unavailable: search requires configuration via `crw setup` first
  3. Cloud calls failing: verify CRW_API_KEY is exported or present in the config file and is valid
  4. Do not want automatic AI tool registration: set CRW_NO_AGENTS=1 to skip that step
  5. MCP client does not detect the server: re-run `npx -y crw-mcp@latest install` or `crw setup`, and consult the mcp-clients manual setup docs
  6. Build from source fails: confirm Rust 1.85 or newer and run `make check-fast`

Use cases

Let AI agents scrape a single URL into Markdown or structured JSON
Follow a bounded site crawl and collect page content
Discover site URLs without scraping every page
Run a web search and optionally scrape selected results
Extract structured fields from one or many URLs
Serve as a drop-in Firecrawl-compatible alternative for /scrape, /crawl, /search calls