- Reliability 8/20
- The manifest is well-formed with a schema reference, stdio transport is simple, and the four optional secret-marked environment variables are clearly declared. However, the manifest contains no tool definitions, and multiple internal contradictions exist between the README and the manifest: version 1.14.0 vs the v2.3.1 banner; 457 checks in the manifest vs 449 in several README sections; 12,132 tests/380 suites in one section vs 11,206 tests/344 suites in another. No CI workflow files or committed tests are present in the supplied evidence to verify the handshake or execution paths, and the static calibration caps reliability at 12 with uncertainty lowering the score. Deductions are for unverifiable claims and self-contradictory evidence.
- Security and permissions 17/20
- The security posture is strong on paper: MCP defaults to SAFE_MODE=true, blocking provision/destroy/restore/snapshot-delete/restart/maintain; non-TTY destructive commands require explicit --force; API tokens come only from environment variables and are never persisted; zero telemetry is claimed; MCP inputs use Zod schema validation and backup restore has path-traversal protection; SSH uses accept-new host-key verification with IP octet validation, and KASTELL_STRICT_HOST_KEY promotes TOFU warnings to hard errors; shell injection is mitigated by using spawn/spawnSync rather than execSync; error output sanitization redacts tokens/IPs/paths; config files use 0o600 permissions with Windows DACL equivalents; installation examples use 'your-token' placeholders, not real secrets. No red-line condition is triggered (no malware, no covert exfiltration, destructive operations have confirmation gates). Deductions: the safe-mode gate can be overridden via KASTELL_SAFE_MODE=false, and most protections are README claims that cannot be statically verified from the supplied files; the confirmation chain for destructive operations depends on runtime behavior not shown in the manifest.
- Maintenance 13/20
- Apache-2.0 license is clear; the repository is not archived; npm version badges, CI/coverage/Snyk/Socket badges, a referenced CHANGELOG, and a public roadmap (v1.14/v2.0/v3.0) indicate ongoing releases and automation. However, governance gaps exist: the README's version claim (v2.3.1) conflicts with the published version (1.14.0); test-count and check-count figures are inconsistent across sections, suggesting process sloppiness; no SECURITY.md or dedicated vulnerability-response channel is mentioned beyond GitHub Issues. Star counts were treated as discovery signals only and add no points. Commit frequency and release cadence cannot be independently verified from the supplied evidence, so points are deducted.
- Documentation 14/20
- Documentation is layered and fairly complete: quick start, command reference organized by category, provider and platform comparison tables, MCP client configuration snippets (Claude Code/Desktop, VS Code, Cursor), troubleshooting, a CI/CD integration example, safety-mode explanation, and pricing/template information, plus references to llms.txt and the MCP Registry. Gaps and contradictions lower the score: per-tool input parameter schemas and limit/error documentation for the MCP tools are absent; the tool table lists 14 tools while the comparison table claims 17; the check count is 449 in two README locations but 457 in the manifest, and test counts disagree between sections. These numeric inconsistencies undermine the README's reliability as reviewable evidence.
- Setup experience 13/20
- The install path is clear and short: Node.js 20+ plus an npm package; a standard MCP client JSON snippet is provided (`npx -y -p kastell kastell-mcp`); the four cloud-provider tokens are optional placeholder env vars, so basic use needs no credentials; per-platform guides for Claude Code, Claude Desktop, VS Code, and Cursor are referenced. Deductions: the `kastell-mcp` binary entry point cannot be verified from the supplied material; the example does not pin a version, so `npx -y` pulls latest and creates a reproducibility risk; full functionality depends on four optional tokens and degraded behavior without them is not documented. The static calibration caps setup at 15 without in-file execution evidence (real CI plus committed tests), and the evidence here is README/manifest only, hence 13.