M
MCP Directory CURATED DIRECTORY
911 MCP servers indexed
中文
← Back to directory
C

Crawlbase MCP Server

Community
Connect AI agents to real-time web data with JS rendering and anti-bot protection.
How to install ↓ Compare GitHub source repository ↗
Category
Other #150 of 230
Stars
★ 60 Popular
Transport
stdio (local process) · Streamable HTTP
Runtime
Node.js
Credentials
API key / credential required
Last commit
⚠
Tools
7
50FMRS · D

Crawlbase MCP Server provides real-time web data scraping for AI agents. It integrates easily with multiple clients and offers async storage. It suits developers needing fresh data, but note dependencies on external services and token configuration.

Strongest · Documentation 12/20 Weakest · Reliability 8/20

Reliability
8/20
Security and permissions
10/20
Maintenance
10/20
Documentation
12/20
Setup experience
10/20
Why each score
Reliability 8/20
README describes tools like crawl, crawl_markdown, crawl_screenshot and an HTTP transport mode (POST /mcp, GET /health), which are plausible. However, no tests or CI workflows are present in the repository, so static review cannot verify that declared tools match actual behavior.
Security and permissions 10/20
Tokens are passed via environment variables or HTTP headers, and no real tokens appear in the README examples. Async storage and delete operations are mentioned without explicit confirmation prompts. Data boundaries and external network usage are partially disclosed, but least privilege is not fully demonstrated.
Maintenance 10/20
The repository is not archived and has zero open issues, but no license or security response channel is provided. README mentions 'Copyright 2026' but no explicit license. Maintenance activity is unclear but not abandoned.
Documentation 12/20
README includes installation configuration examples for Claude, Cursor, Windsurf, HTTP mode, tool listing, usage examples, and async storage description. However, it lacks detailed tool parameters, limits, costs, and troubleshooting guidance.
Setup experience 10/20
Installation via npx or npm run start:http is documented with multiple client config examples. However, no verifiable tests or CI evidence, and reliance on npm package availability.

Static review · not runListed 2026-08-07

Read the FMRS scoring method →

Fit and risk

What it can accessUses the network

Best for

  • AI developers needing real-time web data.
  • Users integrating live data into Claude, Cursor, or Windsurf.
  • Scraping tasks requiring structured outputs (HTML, Markdown, screenshots).
  • Users needing async large-scale crawling with cloud storage.

Not for

  • Scenarios not requiring real-time data and relying only on training data.
  • Users wanting fully local processing without external API dependencies.
  • Users needing complex custom crawler logic beyond simple fetching.
  • Strict compliance environments that do not allow third-party services.

Required permissions

  • Requires Crawlbase tokens (normal and JS) to access the API.
  • Network access: server must connect to Crawlbase services for crawling.
  • In HTTP mode, tokens can be provided via environment variables or request headers.

Risks and side effects

  • Dependency on Crawlbase service availability and API limits.
  • Traffic to target sites may be blocked or violate their terms of service.
  • Misconfiguration or invalid tokens can cause failures.
  • Free tokens may have rate limits affecting large-scale use.

Setup

Before you start

Runtime：Node.js

CRAWLBASE_TOKEN requiredsecret Crawlbase Normal token used for HTML crawls; get it from your Crawlbase dashboard after signing up.
CRAWLBASE_JS_TOKEN requiredsecret Crawlbase JavaScript token used for JS-rendered pages and screenshots; get it from your Crawlbase dashboard.
Other optional settings (1)
MCP_PORT optional Custom port for HTTP transport mode (default 3000); only used when running npm run start:http.
  1. Sign up at Crawlbase to get free tokens.
  2. Configure the MCP server in your client (Claude Desktop, Claude Code, Cursor, or Windsurf).
  3. Add the server configuration to your client's config file, replacing your_token_here and your_js_token_here.
  4. For HTTP mode, clone the repo, run npm install, then start with npm run start:http setting tokens as environment variables.
claude_desktop_config.json 
{"mcpServers":{"crawlbase":{"type":"stdio","command":"npx","args":["@crawlbase/mcp@latest"],"env":{"CRAWLBASE_TOKEN":"your_token_here","CRAWLBASE_JS_TOKEN":"your_js_token_here"}}}}

Shown for Claude Desktop. Other clients may use a different file or key (VS Code uses "servers") — the configurator below converts it.

.vscode/mcp.json
{
  "servers": {
    "crawlbase": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "@crawlbase/mcp@latest"
      ],
      "env": {
        "CRAWLBASE_TOKEN": "your_token_here",
        "CRAWLBASE_JS_TOKEN": "your_js_token_here"
      }
    }
  }
}

Goes in your project's .vscode/mcp.json (VS Code uses a "servers" key).

Terminal
claude mcp add crawlbase -e CRAWLBASE_TOKEN=your_token_here -e CRAWLBASE_JS_TOKEN=your_js_token_here -- npx @crawlbase/mcp@latest

Run it in a terminal; replace any <…> placeholders with your own values first.

Generate and validate client config

Check that it works

After configuring the client, confirm the crawlbase server appears in your MCP tool list with tools like crawl, crawl_markdown, and crawl_screenshot; then run a prompt such as "Crawl Hacker News and return top stories in markdown" to verify live data is returned.

Troubleshooting

  1. Check that tokens are correctly set, ensure `CRAWLBASE_TOKEN` and `CRAWLBASE_JS_TOKEN` environment variables are configured.
  2. Verify the client config file JSON format is correct.
  3. For HTTP mode, check server port accessibility, `GET /health` returns healthy.
  4. If crawling a specific site fails, the target may be blocking; try reducing request frequency or check robots.txt.
  5. Ensure `npx` is available and can reach the npm registry.

Things to try

Once connected, you can ask your AI assistant things like:

  • Crawl Hacker News and return top stories in markdown.
  • Take a screenshot of TechCrunch homepage.
  • Fetch Tesla investor relations page as HTML.
  • How many pages do I have stored in Crawlbase right now? Show me the most recent 20.

Tools 7

crawl read-only
Fetch raw HTML from a URL.
crawl_markdown read-only
Extract clean Markdown from a webpage.
crawl_screenshot read-only
Capture a screenshot of a webpage.
crawl_async writes
Push crawl results to Crawlbase Cloud Storage asynchronously instead of returning directly.
list_stored_pages read-only
List pages stored in Crawlbase Cloud Storage.
read_stored_page read-only
Read a stored page from Crawlbase Cloud Storage.
delete_stored_page destructive
Delete a stored page from Crawlbase Cloud Storage.

Use cases

Market research: pull live data from competitors, news, and reports.
E-commerce monitoring: track products, reviews, and prices in real time.
News & finance feeds: keep AI agents up-to-date with live events.
Autonomous AI agents: give them vision to act on fresh web data.

Supported clients

Claude Desktop
Claude Code
Cursor
Windsurf

Listed from the project's documentation, not tested by this site.

Overview

Crawlbase MCP is a Model Context Protocol (MCP) server that bridges AI agents and the live web. Instead of outdated training data, your LLMs can fetch fresh, structured, real-time content powered by Crawlbase's proven crawling infrastructure trusted by 70,000+ developers. It handles complexity: JavaScript rendering, proxy rotation & anti-bot evasion, and structured outputs (HTML, Markdown, screenshots). Supports stdio and HTTP transport, with async crawling and cloud storage.

Similar servers

BioMCP 75 · B

One binary. One grammar. Evidence from the biomedical sources you already trust.

★ 659 · Tools 11 Compare with this →
Compare with other MCP servers

Source revision 66e16b730580 Data synced 2026-10-19 Read the FMRS scoring method