Best for
- AI developers needing real-time web data.
- Users integrating live data into Claude, Cursor, or Windsurf.
- Scraping tasks requiring structured outputs (HTML, Markdown, screenshots).
- Users needing async large-scale crawling with cloud storage.
C
Crawlbase MCP Server
Community
Connect AI agents to real-time web data with JS rendering and anti-bot protection.
GitHub source repository ↗
50FMRS · D
Crawlbase MCP Server provides real-time web data scraping for AI agents. It integrates easily with multiple clients and offers async storage. It suits developers needing fresh data, but note dependencies on external services and token configuration.
Crawlbase MCP is a Model Context Protocol (MCP) server that bridges AI agents and the live web. Instead of outdated training data, your LLMs can fetch fresh, structured, real-time content powered by Crawlbase's proven crawling infrastructure trusted by 70,000+ developers. It handles complexity: JavaScript rendering, proxy rotation & anti-bot evasion, and structured outputs (HTML, Markdown, screenshots). Supports stdio and HTTP transport, with async crawling and cloud storage.
Tools
crawl
Fetch raw HTML from a URL.
crawl_markdown
Extract clean Markdown from a webpage.
crawl_screenshot
Capture a screenshot of a webpage.
crawl_async
Push crawl results to Crawlbase Cloud Storage asynchronously instead of returning directly.
list_stored_pages
List pages stored in Crawlbase Cloud Storage.
read_stored_page
Read a stored page from Crawlbase Cloud Storage.
delete_stored_page
Delete a stored page from Crawlbase Cloud Storage.
Setup
- Sign up at Crawlbase to get free tokens. 2. Configure the MCP server in your client (Claude Desktop, Claude Code, Cursor, or Windsurf). 3. Add the server configuration to your client's config file, replacing
your_token_hereandyour_js_token_here. 4. For HTTP mode, clone the repo, runnpm install, then start withnpm run start:httpsetting tokens as environment variables.
claude_desktop_config.json
{"mcpServers":{"crawlbase":{"type":"stdio","command":"npx","args":["@crawlbase/mcp@latest"],"env":{"CRAWLBASE_TOKEN":"your_token_here","CRAWLBASE_JS_TOKEN":"your_js_token_here"}}}}
Fit and risk
Not for
- Scenarios not requiring real-time data and relying only on training data.
- Users wanting fully local processing without external API dependencies.
- Users needing complex custom crawler logic beyond simple fetching.
- Strict compliance environments that do not allow third-party services.
Required permissions
- Requires Crawlbase tokens (normal and JS) to access the API.
- Network access: server must connect to Crawlbase services for crawling.
- In HTTP mode, tokens can be provided via environment variables or request headers.
Risks and side effects
- Dependency on Crawlbase service availability and API limits.
- Traffic to target sites may be blocked or violate their terms of service.
- Misconfiguration or invalid tokens can cause failures.
- Free tokens may have rate limits affecting large-scale use.
Troubleshooting
- Check that tokens are correctly set, ensure `CRAWLBASE_TOKEN` and `CRAWLBASE_JS_TOKEN` environment variables are configured.
- Verify the client config file JSON format is correct.
- For HTTP mode, check server port accessibility, `GET /health` returns healthy.
- If crawling a specific site fails, the target may be blocking; try reducing request frequency or check robots.txt.
- Ensure `npx` is available and can reach the npm registry.
Use cases
Market research: pull live data from competitors, news, and reports.
E-commerce monitoring: track products, reviews, and prices in real time.
News & finance feeds: keep AI agents up-to-date with live events.
Autonomous AI agents: give them vision to act on fresh web data.
Supported clients
Claude DesktopFull support
Claude CodeFull support
CursorFull support
WindsurfFull support