Best for
- Developers building AI agents or RAG pipelines needing clean web content.
- Users who prefer local-first, no-account extraction for most sites.
- Those needing fast, Rust-powered extraction and a scriptable CLI.
W
Webclaw
Community
Turn any URL into clean markdown, JSON, and LLM-ready context.
GitHub source repository ↗
65FMRS · C
webclaw is a robust web extraction tool tailored for AI/LLM workflows. It offers a wide range of formats and tools, with a compelling local-first approach. Be aware of hosted API dependencies and the legal/ethical risks of web scraping.
webclaw is a fast, local-first web content extraction tool designed for LLMs and AI agents. It provides scraping, crawling, and structured data extraction via CLI, REST API, and MCP server. The core extraction engine is written in Rust and works locally without an account for most websites. For bot-protected or JavaScript-rendered pages, an optional hosted API (webclaw.io) is available.
Tools
scrape
Extract a single URL as markdown, text, JSON, LLM format, or HTML.
crawl
Follow same-origin links and extract discovered pages.
map
Discover URLs without extracting every page.
batch
Scrape multiple URLs in parallel.
extract
Convert page content into structured data.
summarize
Summarize a page.
diff
Compare page content snapshots.
brand
Extract colors, fonts, logos, and metadata.
search
Search the web and scrape results (hosted API only).
research
Multi-source research workflow (hosted API only).
Setup
For MCP clients, configure the server using the npx launcher as shown in the JSON example, or run npx create-webclaw to auto-detect and configure. The CLI can also be installed via Homebrew (brew tap 0xMassi/webclaw && brew install webclaw), prebuilt binaries, Docker (docker run --rm ghcr.io/0xmassi/webclaw https://example.com), or Cargo.
claude_desktop_config.json
{
"mcpServers": {
"webclaw": {
"command": "npx",
"args": [
"-y",
"@webclaw/mcp"
]
}
}
}
Fit and risk
Not for
- Users needing large-scale, hosted extraction without self-hosting infrastructure.
- Users who need to handle heavily protected or JavaScript-rendered pages without a hosted API key.
- Non-technical users looking for a GUI tool with no configuration.
Required permissions
- File system access: the CLI reads local files for diff operations and writes output files.
- Network access: to fetch web pages and optionally connect to the hosted API.
- Environment variables: may read API keys (WEBCLAW_API_KEY, OPENAI_API_KEY, etc.) and proxy settings.
Risks and side effects
- Scraping websites may violate their terms of service or robots.txt; use responsibly.
- Hosted API requires network requests and data may be sent to remote servers.
- Self-hosting requires maintenance and monitoring.
- Incorrect proxy configuration can cause connection failures.
Troubleshooting
- Ensure Cargo build dependencies are installed (e.g., pkg-config, libssl-dev, cmake).
- Check WEBCLAW_API_KEY is set for hosted features.
- Verify the target URL is publicly accessible and not blocking requests.
- Inspect CLI output and logs for detailed extraction errors.
Use cases
Give AI agents like Claude or Cursor clean page context.
Crawl docs, help centers, and knowledge bases for RAG ingestion.
Track competitor pricing pages, changelogs, and product pages.
Turn messy pages into typed JSON for automations.
Research workflows: search, scrape, summarize multiple sources.
Supported clients
Claude DesktopFull support
Claude CodeFull support
CursorFull support
WindsurfFull support
OpenCodeFull support
Codex CLIFull support