- Reliability 9/20
- The manifest declares two transports (stdio via the OCI image docker.io/kubeshop/mcp-server:latest and a streamable-http remote) with controllable knobs (TK_MCP_TRANSPORT, SHTTP host/port, TK_DEBUG), and the required dependencies (org/environment IDs, access token, control-plane URL) are explicit, so the start path and diagnosability are at least defined. However, the supplied material contains only server.json, the README, and repository metadata — no committed tests and no CI workflow exercising the MCP init + tool-list handshake — so declared tools cannot be statically verified against real behavior, and no tool inventory, timeout, retry, or failure semantics are given. Static calibration caps reliability at 12; given the missing execution evidence, it drops to 9.
- Security and permissions 13/20
- Credential handling looks sound: TK_ACCESS_TOKEN is marked isSecret: true, the remote URL requires organization/environment variables, the default control-plane endpoint is HTTPS (api.testkube.io), and no real tokens or secrets appear in the README or examples. The scope (running tests, reading execution results and artifacts) matches the product's stated purpose, and no malware, covert exfiltration, or irreversible delete/pay/deploy defaults are visible. Deductions: the manifest does not state whether mutating operations (triggering or deleting executions, editing workflows) require human confirmation, nor does it recommend least-privilege token scopes; the remote endpoint's authentication scheme is undescribed; and TK_CONTROL_PLANE_URL / TK_DASHBOARD_URL are overridable to arbitrary hosts without data-flow disclosure. No red line is triggered, so this is not scored 0-4.
- Maintenance 15/20
- The project is active and not archived: 1,656 stars with only 49 open issues, manifest version 1.0.1 published against the official 2025-12-11 schema, plus in-repo governance files (CONTRIBUTING.md, ARCHITECTURE.md, DEVELOPMENT.md). Deductions: the repository license field is NOASSERTION while the manifest claims MIT, so the license must be human-verified (not absent, hence not 0); no evidence of dependency-update automation or a security-response channel (e.g. SECURITY.md); and no reviewable release cadence or changelog in the supplied material.
- Documentation 13/20
- Documentation resources are substantial: the manifest lists a homepage and dedicated MCP documentation page, and the README is layered, covering the two deployment models (OSS agent vs. commercial control plane), triggering options, and result/artifact aggregation, with links to focused docs. Deductions: MCP-specific installation, authentication, tool parameters, permission scopes, rate limits, cost, and troubleshooting are only linked, not described in the visible text, so they cannot be statically verified; no tool inventory or parameter schema is listed; enterprise/trial links carry marketing parameters that readers must weigh; and no minimal MCP usage example is included. Scored in the 10-13 band at 13.
- Setup experience 13/20
- The install paths are explicit and multiple: stdio via the OCI image docker.io/kubeshop/mcp-server:latest with environment variables (TK_ACCESS_TOKEN, TK_ORG_ID, TK_ENV_ID required; others optional with defaults), plus a hosted streamable-http remote, and the README points to Helm/CLI install and a Quickstart. Deductions: no client configuration examples for mainstream MCP clients (Claude Desktop, VS Code, etc.) appear in the supplied files; a Testkube org/environment and issued token must exist first (external prerequisite); the OCI package uses the mutable latest tag rather than a pinned digest, hurting reproducibility; and self-hosted shttp requires choosing host/port. Static calibration caps setup at 15; without committed verifiable execution evidence it drops to 13.