MCP Observatory

Community

CI-native security testing for MCP servers: attack simulation, schema drift detection, and health scoring.

How to install ↓ Compare GitHub source repository ↗

Category: Dev Tools #243 of 438
Stars: ★ 141 Popular
Transport: stdio (local process)
Runtime: Node.js 20+
Credentials: No credential needed
License: MIT
Last commit: 10 days ago
Tools: 10

51FMRS · D

MCP Observatory is a powerful security scanner and testing tool for MCP servers. It offers unique attack simulation and schema drift detection, and can act as an MCP server itself. This is a valuable tool for teams looking to integrate security testing into their MCP development workflow.

Strongest · Documentation 14/20 Weakest · Reliability 6/20

Reliability

6/20

Security and permissions

8/20

Maintenance

10/20

Documentation

14/20

Setup experience

13/20

Why each score

Reliability 6/20: The manifest is coherent (v1.36.1, npm, stdio) and the README describes a plausible CLI/MCP feature set with ten tools, but no source code, tests, CI workflows, or committed test results were supplied. There is no verifiable evidence that the server starts, completes the initialize/tools handshake, or that the declared tools match real behavior. Deducted heavily for missing execution evidence and thin error/edge-case documentation; uncertainty lowers the score.
Security and permissions 8/20: README shows deliberate controls: a command allowlist for MCP mode, path validation to runs/cassettes, and safe-only tool invocation. However, no implementation source was provided to verify credential handling, least privilege, network egress, or side-effect confirmation. The CLI is intentionally unrestricted and scans multiple local configs, and there is no disclosed data-flow/privacy detail in the supplied material. No red-line issue (malware, token exfiltration, destructive defaults, real secrets in examples) is apparent, so the score stays mid-range rather than 0-4.
Maintenance 10/20: Evidence: not archived, MIT license, package version 1.36.1, and README claims CI, CodeQL, coverage, Dependabot, npm provenance, and contributors. Deductions: no commit history, release dates, issue-response metrics, dependency-update records, or security-response channel were provided; README badges are untrusted evidence, and 50 open issues is not evaluated either way. Score reflects plausible activity but unverified governance.
Documentation 14/20: The supplied README is layered and thorough: quick start, command table, architecture diagram, CI action inputs, target config examples with env placeholders, compatibility matrix, limitations, and contributing guide. Deductions: most linked docs (methodology, known issues, compatibility, PRIVACY) were not supplied, so claimed depth cannot be verified; MCP tool parameters are only summarized, and troubleshooting is delegated to linked pages. Good examples and disclosed limitations, but not full marks.
Setup experience 13/20: Setup is described as a low-friction npx one-liner plus a Claude Code registration command and a manual JSON config example; Node >= 20 is the main prerequisite. This is a clear, few-step path. However, static calibration caps setup at 15 without verifiable execution evidence (committed CI workflows and tests), and none was supplied; package contents, exact CLI entry points, and platform behavior could not be confirmed. Rounded down for that missing evidence.

Static review · not runListed 2026-08-07

Read the FMRS scoring method →

Fit and risk

What it can accessRuns commands or codeUses the network

Best for

Development teams building custom MCP servers.
Security teams needing to validate MCP server security in the AI supply chain.
DevOps engineers looking for CI integration for MCP server testing.

Not for

Servers requiring interactive OAuth (like Google Drive) that need pre-authentication.
Servers using custom WebSocket transports (e.g., BrowserTools MCP) are not supported.
Performance benchmarking of MCP servers (use MCPBench).

Required permissions

Requires access to the local filesystem to read configs and write reports.
Can spawn subprocesses to launch MCP servers.
Can post GitHub comments and set commit statuses (when configured).
Can upload SARIF results to GitHub Code Scanning (when configured).

Risks and side effects

May execute unsafe tool calls, but has an allowlist for base commands by default.
Requires network access to install npm packages and reach remote servers.
Generated reports may contain sensitive information; handle them carefully.

Setup

Before you start

Runtime：Node.js 20+

MCP_SERVER_API_KEY optionalsecret Only needed when testing an authenticated remote HTTP/SSE target via a target config's authToken/headers; obtain it from the target service provider.

Other optional settings (2)

MCP_OBSERVATORY_ORG optional Optional; sets internal account attribution in CI, e.g. your-company.com.

MCP_OBSERVATORY_CONTACT optional Optional; team contact used in enterprise reports.

Ensure Node.js 20 or later is installed.
For CLI usage, run npx @kryptosai/mcp-observatory or npx @kryptosai/mcp-observatory demo.
To use as an MCP server, add a mcpServers entry in your client config, e.g.,

   {
     "mcpServers": {
       "mcp-observatory": {
         "command": "npx",
         "args": ["-y", "@kryptosai/mcp-observatory", "serve"]
       }
     }
   }

In Claude Code, use claude mcp add mcp-observatory -- npx -y @kryptosai/mcp-observatory serve.
For CI, use setup-ci --all --command "npx -y my-mcp-server" --sarif to generate a workflow.

claude_desktop_config.json

{
  "mcpServers": {
    "mcp-observatory": {
      "command": "npx",
      "args": [
        "-y",
        "@kryptosai/mcp-observatory",
        "serve"
      ]
    }
  }
}

Shown for Claude Desktop. Other clients may use a different file or key (VS Code uses "servers") — the configurator below converts it.

.vscode/mcp.json

{
  "servers": {
    "mcp-observatory": {
      "command": "npx",
      "args": [
        "-y",
        "@kryptosai/mcp-observatory",
        "serve"
      ]
    }
  }
}

Goes in your project's .vscode/mcp.json (VS Code uses a "servers" key).

Terminal

claude mcp add mcp-observatory -- npx -y @kryptosai/mcp-observatory serve

Run it in a terminal; replace any <…> placeholders with your own values first.

Generate and validate client config

Check that it works

The client's tool list should show the 10 tools (scan, check_server, score_server, etc.); asking "scan my configured MCP servers" and getting a health score and verdict confirms the connection works.

Troubleshooting

If a server times out, check the `timeoutMs` setting in the target config.
If HTTP/SSE connections fail, verify the URL and authentication tokens.
Ensure servers use standard transports (stdio, HTTP/SSE) and are pre-authenticated.

Things to try

Once connected, you can ask your AI assistant things like:

Scan all my configured MCP servers and show their health scores
Test the @modelcontextprotocol/server-everything server before I install it
Compare my last two scan runs and report any schema drift or regressions
Suggest MCP servers that match my project's tech stack

Tools 10

scan read-only

Check if all your configured MCP servers are healthy.

check_server read-only

Test a specific server before installing or after updating.

score_server read-only

Get a quick health score and grade for a server.

record read-only

Capture a baseline of a working server for future comparison.

replay read-only

Test against a recorded session — no live server needed.

verify read-only

Confirm a server update didn't break anything.

watch read-only

Check a server and see what changed since the last check.

diff_runs read-only

Find regressions between two check results.

Show 2 more tools

get_last_run read-only

Retrieve previous check results for a server.

suggest_servers read-only

Discover MCP servers that match your project stack.

Use cases

Automatically detect regressions and schema drift in MCP servers within CI pipelines.

Perform security audits and attack simulations on MCP servers before deployment.

Generate health score badges for public MCP servers.

Record and replay MCP server sessions for offline testing.

Use MCP server mode to let AI agents automatically verify other MCP servers.

Supported clients

Claude Desktop

Listed from the project's documentation, not tested by this site.

Overview

MCP Observatory is a CI-native security tool for testing custom MCP servers. It detects schema drift, simulates attacks, generates compliance evidence (e.g., SARIF reports), and provides health scoring before agents depend on your servers. It can run as a CLI or as an MCP server, allowing AI agents to directly test other MCP servers.

Similar servers

Context7 80 · B

Upstash's official server providing up-to-date third-party library docs for AI coding assistants

★ 62.9k · Tools 2 Compare with this →

NPM Sentinel MCP Server 79 · B

AI-powered NPM package analysis MCP server

★ 18 · Tools 19 Compare with this →

MartinLoop MCP Server 79 · B

An independent control layer that gives AI coding agents budgets, brakes, and receipts

★ 409 Compare with this →

GitHub MCP Server 76 · B

GitHub's official MCP server for managing repos, issues, PRs, and workflows via natural language

★ 33.5k · Tools 15 Compare with this →

Compare with other MCP servers

Source revision 8f148e6fa035 Data synced 2026-10-11 Read the FMRS scoring method