M
MCP Directory CURATED DIRECTORY
911 MCP servers indexed
中文
← Back to directory
W

Webclaw

Community
Turn any URL into clean markdown, JSON, and LLM-ready context.
How to install ↓ Compare GitHub source repository ↗
Category
Dev Tools #50 of 438
Stars
★ 2.4k Very popular
Transport
stdio (local process)
Runtime
Node.js · Rust · Docker
Credentials
Optional API key
License
AGPL-3.0
Last commit
Tools
10
65FMRS · C

webclaw is a robust web extraction tool tailored for AI/LLM workflows. It offers a wide range of formats and tools, with a compelling local-first approach. Be aware of hosted API dependencies and the legal/ethical risks of web scraping.

Strongest · Documentation 15/20 Weakest · Reliability 10/20

Reliability
10/20
Security and permissions
12/20
Maintenance
14/20
Documentation
15/20
Setup experience
14/20
Why each score
Reliability 10/20
Static review only; the MCP init, tool-list handshake, and actual tool behavior cannot be verified from the README and manifest alone. The stdio + npx path looks plausible on paper, but no CI workflow or committed tests are present in the supplied evidence. The manifest version 0.6.17 vs npm package version 0.6.16 mismatch also reduces confidence.
Security and permissions 12/20
No red-line issues found: no real tokens in examples, no malicious downloads, and no covert exfiltration path. Credentials are passed via environment variables (e.g., WEBCLAW_API_KEY), and the README clearly distinguishes local extraction from hosted API usage. Deductions: no SSRF/network policy is documented, there is no security policy or data-flow disclosure, and permission scoping/confirmation of risky operations is not addressed.
Maintenance 14/20
The repository is not archived and has only 2 open issues; release badges, versioning, AGPL-3.0 license, contributor list, and Discord presence indicate an active project with a visible update path. Deductions: commit frequency, dependency update history, and a dedicated security-response channel are not evidenced in the supplied material.
Documentation 15/20
The README is comprehensive and layered: install options, quick start, MCP client config example, tool table, output formats, environment variables, example directories, and contribution guidance are all present. Deductions: detailed MCP tool parameter schemas, rate limits/cost notes, and a dedicated troubleshooting section are missing, and some sections are promotional.
Setup experience 14/20
Setup is clear and multi-path: one-line npx config, create-webclaw auto-detection for supported clients, Homebrew, Docker, prebuilt binaries, and Cargo. A mainstream-client config example is included. Deductions: static review cannot confirm execution, building from source requires native toolchain prerequisites, and the package/manifest version mismatch slightly weakens setup expectations.

Static review · not runListed 2026-08-07

Read the FMRS scoring method →

Fit and risk

What it can accessUses the network

Best for

  • Developers building AI agents or RAG pipelines needing clean web content.
  • Users who prefer local-first, no-account extraction for most sites.
  • Those needing fast, Rust-powered extraction and a scriptable CLI.

Not for

  • Users needing large-scale, hosted extraction without self-hosting infrastructure.
  • Users who need to handle heavily protected or JavaScript-rendered pages without a hosted API key.
  • Non-technical users looking for a GUI tool with no configuration.

Required permissions

  • File system access: the CLI reads local files for diff operations and writes output files.
  • Network access: to fetch web pages and optionally connect to the hosted API.
  • Environment variables: may read API keys (WEBCLAW_API_KEY, OPENAI_API_KEY, etc.) and proxy settings.

Risks and side effects

  • Scraping websites may violate their terms of service or robots.txt; use responsibly.
  • Hosted API requires network requests and data may be sent to remote servers.
  • Self-hosting requires maintenance and monitoring.
  • Incorrect proxy configuration can cause connection failures.

Setup

Before you start

Runtime：Node.js · Rust · Docker

WEBCLAW_API_KEY optionalsecret Hosted API key from webclaw.io for protected sites, JS rendering, search, and research; optional.
OPENAI_API_KEY optionalsecret OpenAI-compatible LLM provider key; optional.
ANTHROPIC_API_KEY optionalsecret Anthropic-compatible LLM provider key; optional.
Other optional settings (5)
OLLAMA_HOST optional Local Ollama URL for local LLM features (extract/summarize); optional.
OPENAI_BASE_URL optional OpenAI-compatible base URL; optional.
ANTHROPIC_BASE_URL optional Anthropic-compatible base URL; optional.
WEBCLAW_PROXY optional Single proxy URL; optional.
WEBCLAW_PROXY_FILE optional Proxy pool file path; optional.

For MCP clients, configure the server using the npx launcher as shown in the JSON example, or run npx create-webclaw to auto-detect and configure. The CLI can also be installed via Homebrew (brew tap 0xMassi/webclaw && brew install webclaw), prebuilt binaries, Docker (docker run --rm ghcr.io/0xmassi/webclaw https://example.com), or Cargo.

claude_desktop_config.json 
{
  "mcpServers": {
    "webclaw": {
      "command": "npx",
      "args": [
        "-y",
        "@webclaw/mcp"
      ]
    }
  }
}

Shown for Claude Desktop. Other clients may use a different file or key (VS Code uses "servers") — the configurator below converts it.

.vscode/mcp.json
{
  "servers": {
    "webclaw": {
      "command": "npx",
      "args": [
        "-y",
        "@webclaw/mcp"
      ]
    }
  }
}

Goes in your project's .vscode/mcp.json (VS Code uses a "servers" key).

Terminal
claude mcp add webclaw -- npx -y @webclaw/mcp

Run it in a terminal; replace any <…> placeholders with your own values first.

Generate and validate client config

Check that it works

Confirm the webclaw tools (scrape, crawl, map, batch, extract, summarize, diff, brand, search, research) appear in your MCP client's tool list, then ask it to scrape https://example.com and return markdown.

Troubleshooting

  1. Ensure Cargo build dependencies are installed (e.g., pkg-config, libssl-dev, cmake).
  2. Check WEBCLAW_API_KEY is set for hosted features.
  3. Verify the target URL is publicly accessible and not blocking requests.
  4. Inspect CLI output and logs for detailed extraction errors.

Things to try

Once connected, you can ask your AI assistant things like:

  • Scrape these competitor pricing pages and summarize the differences.
  • Crawl this documentation site and prepare clean context for a RAG index.
  • Extract the brand colors, fonts, and logos from this company website.
  • Convert this page into clean markdown.

Tools 10

scrape read-only
Extract a single URL as markdown, text, JSON, LLM format, or HTML.
crawl read-only
Follow same-origin links and extract discovered pages.
map read-only
Discover URLs without extracting every page.
batch read-only
Scrape multiple URLs in parallel.
extract read-only
Convert page content into structured data.
summarize read-only
Summarize a page.
diff read-only
Compare page content snapshots.
brand read-only
Extract colors, fonts, logos, and metadata.
Show 2 more tools
search read-only
Search the web and scrape results (hosted API only).
research read-only
Multi-source research workflow (hosted API only).

Use cases

Give AI agents like Claude or Cursor clean page context.
Crawl docs, help centers, and knowledge bases for RAG ingestion.
Track competitor pricing pages, changelogs, and product pages.
Turn messy pages into typed JSON for automations.
Research workflows: search, scrape, summarize multiple sources.

Supported clients

Claude Desktop
Claude Code
Cursor
Windsurf
OpenCode
Codex

Listed from the project's documentation, not tested by this site.

Overview

webclaw is a fast, local-first web content extraction tool designed for LLMs and AI agents. It provides scraping, crawling, and structured data extraction via CLI, REST API, and MCP server. The core extraction engine is written in Rust and works locally without an account for most websites. For bot-protected or JavaScript-rendered pages, an optional hosted API (webclaw.io) is available.

Similar servers

Context7 80 · B

Upstash's official server providing up-to-date third-party library docs for AI coding assistants

★ 63.3k · Tools 2 Compare with this →
Compare with other MCP servers

Source revision 3c32041967c2 Data synced 2026-10-19 Read the FMRS scoring method