← Back to directory

Crawlie

Community

Fast, free, open-source website audit MCP server for technical SEO and Generative Engine Optimization (GEO).

GitHub source repository ↗

★ 100 Stars Category · Dev Tools Popular Source revision 810e83687421

50FMRS · D

Reliability

8/20

Security and permissions

9/20

Maintenance

8/20

Documentation

12/20

Setup experience

13/20

Crawlie provides a powerful, free, open-source MCP server for website audits, combining technical SEO and GEO checks with agent-friendly integration. It is ideal for developers and marketers looking to automate search optimization, but users should be mindful of privacy implications of hosted services and compliance responsibilities when crawling sites.

Read the FMRS scoring method →

Crawlie is a fast, free, open-source technical SEO + GEO crawler built for humans and agents. It can crawl any site for broken links, redirects, missing metadata, and 40+ SEO & Generative-Engine checks, with plain-English guidance on every fix. It runs locally, ships a CLI and an MCP server, and costs nothing.

Tools

crawl_site

Crawl and audit an entire site (SEO + GEO), returning scores, issues, and per-page data.

audit_url

Audit a single page.

audit_urls

Audit an explicit list of pages.

explain_issue

Explain why a rule matters and how to fix it.

list_rules

List the full catalogue of checks.

list_reports

List saved crawl history.

get_report

Retrieve a saved report.

top_fixes

Get the top N fixes that would most improve your GEO score.

geo_gaps

Identify gaps in AI-search citation readiness.

affected_urls

List URLs affected by a specific issue.

diff_reports

Compare differences between two reports.

crawl_status

Poll the status of a long-running crawl.

Setup

Install the npm package globally (includes CLI and MCP server): npm i -g crawlie. Then crawlie-mcp is on your PATH. For Claude Desktop, add the configuration in claude_desktop_config.json (see install_config). For Claude Code, run claude mcp add crawlie crawlie-mcp. Alternatively, use the hosted endpoint https://crawlie.app/mcp with an API key (create one in the dashboard).

claude_desktop_config.json

{
  "mcpServers": {
    "crawlie": {
      "command": "crawlie-mcp"
    }
  }
}

Generate and validate client config

Fit and risk

Best for

Developers and marketers needing a free, local-first SEO/GEO audit tool.
Teams wanting to integrate website audits into agent workflows.
Users needing CI/CD integration and structured JSON output.
SEO professionals aiming to optimize for AI search (like ChatGPT, Perplexity).

Not for

Users requiring enterprise support or a hosted cloud service (Crawlie Cloud is closed-source).
Users needing additional plugin ecosystems beyond the 57 built-in rules.
Beginners who prefer a pure GUI and no CLI or agent interaction (desktop app available but separate download).

Required permissions

Local network access to crawl target websites.
Execution of the binary (installed automatically via npm).
For hosted endpoint: API key and network access.

Risks and side effects

Crawling websites may violate their robots.txt or terms of service; ensure you have permission to crawl the target site.
The MCP server is open-source and may not have undergone extensive security audits; assess your own risk.
For hosted services, your data is sent to Crawlie's servers; be aware of privacy implications.

Troubleshooting

Ensure `crawlie-mcp` is on your PATH, or use the absolute path if building from source.
For Claude Desktop, verify `claude_desktop_config.json` is correctly formatted and valid JSON.
Check network connectivity and that the target site is reachable and not blocked by firewalls.
If crawl results are unexpected, try disabling features like `--no-robots` or `--no-sitemap`.
Consult official docs at https://crawlie.dev/docs or GitHub discussions for help.

Use cases

Pre-launch QA: Catch broken links, redirects, 4xx/5xx, and missing metadata before you ship.

GEO optimization: Make pages citable by AI search: structured data, semantic HTML, answer-ready content, authorship/E-E-A-T.

Agent workflows: Let a marketing/SEO agent audit a site and propose fixes autonomously via MCP.

CI/CD gating: Use `crawlie crawl --fail-on error` in a pipeline to block regressions.

Client reporting: Generate a polished, shareable HTML report in one command.

Auditing AI-generated sites: Verify that the site your agent just built is actually built for search.

Supported clients

Claude DesktopFull support

Claude CodeFull support

CursorFull support

ClineFull support

Related MCP servers

Compare with other MCP servers