M
MCP Directory CURATED DIRECTORY
911 MCP servers indexed
中文
← Back to directory
C

Crawlie

Community
Fast, free, open-source website audit MCP server for technical SEO and Generative Engine Optimization (GEO).
How to install ↓ Compare GitHub source repository ↗
Category
Dev Tools #258 of 438
Stars
★ 114 Popular
Transport
stdio (local process) · Streamable HTTP
Runtime
Node.js · Rust · Hosted (no local runtime)
Credentials
Optional API key
License
Other / unspecified
Last commit
Tools
12
50FMRS · D

Crawlie provides a powerful, free, open-source MCP server for website audits, combining technical SEO and GEO checks with agent-friendly integration. It is ideal for developers and marketers looking to automate search optimization, but users should be mindful of privacy implications of hosted services and compliance responsibilities when crawling sites.

Strongest · Setup experience 13/20 Weakest · Reliability 8/20

Reliability
8/20
Security and permissions
9/20
Maintenance
8/20
Documentation
12/20
Setup experience
13/20
Why each score
Reliability 8/20
The manifest and README describe a coherent MCP server (local stdio via crawlie-mcp, hosted Streamable HTTP endpoint) with a defined tool set. No source code, tests, CI workflow, or run logs were supplied, so actual init/tool handshake and error behavior cannot be verified. The README also has inconsistencies: the hosted tool list mentions tools (top_fixes, geo_gaps, affected_urls, diff_reports, crawl_status) not present in the local 'Tools exposed' table, and the manifest claims report diffs. Deductions: no verifiable execution evidence and contradictory documentation. Score: 8.
Security and permissions 9/20
No red-line issues found: no malware, credential theft, real secrets in install examples, or destructive default operations. The README discloses that the hosted MCP endpoint uses a Bearer API key and sends data to crawlie.app; the local crawler makes outbound network requests by design. Deductions: without source code, least privilege, sandboxing, request scoping, and data retention cannot be verified; there is no explicit confirmation step for crawl operations or key-storage guidance. Score: 9.
Maintenance 8/20
The repository is not archived, has one open issue, and the README shows a CI badge, changelog, discussion link, and a clear owner (Spronta Ltd / Sean Ryan). Deductions: GitHub metadata reports License: NOASSERTION despite the README claiming MIT; no LICENSE file is supplied; there are no commit/release dates, dependency-update policy, or security-response channel; version is only 0.1.0. Score: 8.
Documentation 12/20
The README is detailed: install/setup, CLI usage, MCP configurations for Claude Desktop and Claude Code, hosted endpoint authentication example, plugin/skills, tool table, example agent prompts, checks, architecture, and roadmap. Deductions: no detailed tool parameter schemas, rate/cost limits, or troubleshooting guide; the hosted and local tool lists are inconsistent; the server manifest does not include a local command field; the source docs are only linked externally. Score: 12.
Setup experience 13/20
The easiest path is a one-line npm global install, then adding 'crawlie-mcp' to a client config; README provides concrete examples for Claude Desktop, Claude Code, and the hosted HTTP endpoint. Building from source requires Rust and pnpm. Deductions: no committed CI workflows or tests are supplied to verify executability, capping the score at 15 under static-calibration rules; Windows compatibility and binary requirements are not explicit. Score: 13.

Static review · not runListed 2026-08-07

Read the FMRS scoring method →

Fit and risk

What it can accessUses the network

Best for

  • Developers and marketers needing a free, local-first SEO/GEO audit tool.
  • Teams wanting to integrate website audits into agent workflows.
  • Users needing CI/CD integration and structured JSON output.
  • SEO professionals aiming to optimize for AI search (like ChatGPT, Perplexity).

Not for

  • Users requiring enterprise support or a hosted cloud service (Crawlie Cloud is closed-source).
  • Users needing additional plugin ecosystems beyond the 57 built-in rules.
  • Beginners who prefer a pure GUI and no CLI or agent interaction (desktop app available but separate download).

Required permissions

  • Local network access to crawl target websites.
  • Execution of the binary (installed automatically via npm).
  • For hosted endpoint: API key and network access.

Risks and side effects

  • Crawling websites may violate their robots.txt or terms of service; ensure you have permission to crawl the target site.
  • The MCP server is open-source and may not have undergone extensive security audits; assess your own risk.
  • For hosted services, your data is sent to Crawlie's servers; be aware of privacy implications.

Setup

Before you start

Runtime：Node.js · Rust · Hosted (no local runtime)

Install the npm package globally (includes CLI and MCP server): npm i -g crawlie. Then crawlie-mcp is on your PATH. For Claude Desktop, add the configuration in claude_desktop_config.json (see install_config). For Claude Code, run claude mcp add crawlie crawlie-mcp. Alternatively, use the hosted endpoint https://crawlie.app/mcp with an API key (create one in the dashboard).

claude_desktop_config.json 
{
  "mcpServers": {
    "crawlie": {
      "command": "crawlie-mcp"
    }
  }
}

Shown for Claude Desktop. Other clients may use a different file or key (VS Code uses "servers") — the configurator below converts it.

.vscode/mcp.json
{
  "servers": {
    "crawlie": {
      "command": "crawlie-mcp"
    }
  }
}

Goes in your project's .vscode/mcp.json (VS Code uses a "servers" key).

Terminal
claude mcp add crawlie -- crawlie-mcp

Run it in a terminal; replace any <…> placeholders with your own values first.

Generate and validate client config

Check that it works

After npm i -g crawlie, configure the client with command crawlie-mcp (or run claude mcp add crawlie crawlie-mcp); restart the client and confirm tools like crawl_site, audit_url, and explain_issue appear, then run a crawl_site request to prove the connection works.

Troubleshooting

  1. Ensure `crawlie-mcp` is on your PATH, or use the absolute path if building from source.
  2. For Claude Desktop, verify `claude_desktop_config.json` is correctly formatted and valid JSON.
  3. Check network connectivity and that the target site is reachable and not blocked by firewalls.
  4. If crawl results are unexpected, try disabling features like `--no-robots` or `--no-sitemap`.
  5. Consult official docs at https://crawlie.dev/docs or GitHub discussions for help.

Things to try

Once connected, you can ask your AI assistant things like:

  • Crawl example.com and give me the top 5 fixes that would most improve my GEO score, with the exact change for each.
  • Audit these three landing pages and tell me which is least ready to be cited by AI search, and why.
  • Explain the geo-not-answerable rule: why it matters and how to fix it.
  • List the URLs affected by broken links in my last crawl report.

Tools 12

crawl_site read-only
Crawl and audit an entire site (SEO + GEO), returning scores, issues, and per-page data.
audit_url read-only
Audit a single page.
audit_urls read-only
Audit an explicit list of pages.
explain_issue read-only
Explain why a rule matters and how to fix it.
list_rules read-only
List the full catalogue of checks.
list_reports read-only
List saved crawl history.
get_report read-only
Retrieve a saved report.
top_fixes read-only
Get the top N fixes that would most improve your GEO score.
Show 4 more tools
geo_gaps read-only
Identify gaps in AI-search citation readiness.
affected_urls read-only
List URLs affected by a specific issue.
diff_reports read-only
Compare differences between two reports.
crawl_status read-only
Poll the status of a long-running crawl.

Use cases

Pre-launch QA: Catch broken links, redirects, 4xx/5xx, and missing metadata before you ship.
GEO optimization: Make pages citable by AI search: structured data, semantic HTML, answer-ready content, authorship/E-E-A-T.
Agent workflows: Let a marketing/SEO agent audit a site and propose fixes autonomously via MCP.
CI/CD gating: Use `crawlie crawl --fail-on error` in a pipeline to block regressions.
Client reporting: Generate a polished, shareable HTML report in one command.
Auditing AI-generated sites: Verify that the site your agent just built is actually built for search.

Supported clients

Claude Desktop
Claude Code
Cursor
Cline

Listed from the project's documentation, not tested by this site.

Overview

Crawlie is a fast, free, open-source technical SEO + GEO crawler built for humans and agents. It can crawl any site for broken links, redirects, missing metadata, and 40+ SEO & Generative-Engine checks, with plain-English guidance on every fix. It runs locally, ships a CLI and an MCP server, and costs nothing.

Similar servers

Context7 80 · B

Upstash's official server providing up-to-date third-party library docs for AI coding assistants

★ 62.9k · Tools 2 Compare with this →
Compare with other MCP servers

Source revision 810e83687421 Data synced 2026-10-12 Read the FMRS scoring method