Best for
- Developers needing robust web scraping
- Researchers scraping data in adversarial environments
- Teams wanting AI agents to handle scraping tasks autonomously
A
Anansi MCP Server
Community
Self-healing, conversational scraper for hostile sites
GitHub source repository ↗
59FMRS · C
Anansi is a powerful self-healing scraper with advanced anti-bot features, ideal for highly adversarial sites. Its MCP server enables any LLM to drive crawls, simplifying data collection for AI agents, but users must assume compliance responsibility.
Anansi is a self-healing web scraper for hostile sites. It automatically repairs broken CSS selectors, switches to a stealth browser when needed, and presents a coherent identity with matched TLS fingerprints, personas, and vendor-aware handling (Cloudflare, Akamai, DataDome) to slip past bot detection. Ships with an MCP server so any LLM can drive a full crawl through conversation, with incremental re-crawls and Pydantic-based data validation.
Tools
fetch
Fetch and process a URL with automatic browser upgrades and anti-bot identity.
extract
Extract structured data from a URL using configured selectors.
crawl
Crawl from one or more start URLs within a spider, following links and page rules.
screenshot
Take a screenshot of a URL or a page during crawl.
run_spider
Run a configured spider and output the scraped data.
list_spiders
List all registered spiders with their configurations.
get_status
Get the current status and summary of the running spider.
pause
Pause the currently running spider.
resume
Resume a paused spider.
set_proxy
Set proxy configuration for subsequent crawls.
get_stats
Get crawler statistics such as request counts, cache hits, etc.
export_data
Export scraped data to CSV/JSON/database files.
clear_cache
Clear cached entries for unchanged pages.
validate_item
Validate an item against the configured schema.
configure_anti_bot
Adjust anti-bot settings such as vendor-level behavior and proxy scoring.
set_rate_limit
Set adaptive rate limiting parameters.
get_logs
Retrieve crawler logs for debugging.
Setup
- Install core: pip install "git+https://github.com/mdowis/anansi"
- For browser-based fetching (Cloudflare bypass, JS rendering): playwright install chromium
- For TLS-fingerprint mimicry (curl-cffi impersonation): pip install "anansi-scraper[tls] @ git+https://github.com/mdowis/anansi"
- Register MCP: claude mcp add anansi -- anansi-mcp
claude_desktop_config.json
{
"mcpServers": {
"anansi": {
"command": "anansi-mcp",
"args": []
}
}
}
Fit and risk
Not for
- Developers needing legal, low-risk scraping
- Users targeting only static sites without anti-bot challenges
- Scenarios with strict legal constraints on scraping
Required permissions
- Making network requests and scraping public data
- Using a browser for rendering via Playwright
- Using proxies that provide access to a proxy service
- Writing exported files locally
Risks and side effects
- May violate a site's Terms of Service or applicable laws (e.g., CFAA, GDPR)
- Anti-bot features could be interpreted as circumventing access controls
- Proxies and IPs may get blocked by target sites
- Overly aggressive crawling may overload target servers
Troubleshooting
- CLI not working after install: ensure Python 3.11+ and dependencies installed
- Cloudflare bypass failing: run 'playwright install chromium'
- TLS fingerprint not active: install the [tls] extra
- MCP connection failure: ensure anansi-mcp is in PATH and check client config
Use cases
Scraping sites with aggressive anti-bot measures
Crawling frequently redesigned sites without maintaining selectors
Letting AI agents autonomously perform data collection through conversation
Reducing data collection time with incremental re-crawls
Supported clients
Claude CodeFull support
Claude DesktopFull support
CursorFull support
WindsurfFull support
ChatGPTFull support
LangChainFull support
OpenAI Agents SDKFull support