- Reliability 9/20
- Static evidence shows a plausible startup path: the only declared transport is a hosted streamable-http endpoint (https://api.cloud.vexa.ai/mcp) requiring just a bearer header, so the init + tool-list handshake is reasonably inferable with no local process or build. However, the source contains no CI workflows, no committed tests, and no reproducible execution evidence for the MCP endpoint itself, and the manifest declares no tools, resources, or prompts at all, making the core requirement — declared tools matching real behavior — unverifiable. The README self-discloses gaps: WebSocket transport not wired, PUT /bots/{…}/config and POST /bots/{…}/speak returning 404, POST /agent/meeting/{start,process} unavailable in the open-core stack, POST /bots returning 503 by default when STT is absent (GPU transcription unit or free token needed), and speaker attribution explicitly not guaranteed (empty speaker instead of guessing, ~4–7% of rows under heavy crosstalk). Dependency surface is broad and coupled (Docker, Postgres, Redis, MinIO, GPU STT, external API key), so diagnosability depends on the hosted side. Net: happy path plausible, boundaries and error handling thin, plus a documented capability gap between 'MCP server' claims and an empty MCP surface. Conservative score 9.
- Security and permissions 10/20
- Authentication is least-privilege in direction: the remote MCP uses a bearer token, the manifest marks that header isRequired and isSecret, the key comes from the user's account page, and examples use placeholders ($API_KEY, vxa_…) rather than real secrets — satisfying the no-leaked-credentials requirement. Deployment claims are data-sovereign (in-VPC, zero egress, air-gapped, own object storage, bring-your-own LLM endpoint); no covert exfiltration or malicious behavior is visible and no red line is triggered. Deductions are concrete: no MCP tool-level scope documentation, so the read/write/delete boundary is undefined at the manifest level; agents are described as writing to workspaces and producing git commits (side effects), yet the only safeguard described is a principle ('trusted chat writes, untrusted input propose-only') with no explicit confirmation mechanism for MCP-initiated actions; at-rest encryption for workspace/transcript/tokens is explicitly Planned, not implemented; the K8s/Docker agent runtime relies on substrate-enforced isolation rather than agent-enforced policy, and audit logging or rate limiting is not disclosed. Risks are visible but confirmation and scoping are incomplete — 10.
- Maintenance 13/20
- Maintenance signals are strong: explicit Apache-2.0 license, version 0.12.27, 2767 stars, not archived, plus SECURITY.md reporting channel, security-insights.yml (OpenSSF Security Insights), license-exceptions.json (dependency license gating), architecture.calm.json (FINOS CALM, claimed CI drift-gated), CONTRIBUTOR_RIGHTS.md with DCO, a deploy/helm chart, and public roadmap board and milestones. These point to sustained maintenance with clear ownership and an update path. Deductions: 544 open issues against 2767 stars suggests meaningful backlog, and no response-time or closure-rate evidence is provided; README self-reports several in-progress items (Helm/K8s status, Jitsi live validation, WebSocket, mid-call control); no concrete release notes or dependency-update evidence appears (README claims are untrusted evidence). Score 13.
- Documentation 14/20
- Documentation is layered and substantial: motivation, architecture diagram, quickstart for both hosted curl and self-host (make all/lite/dev), environment variables and runtime backends (docker/process/k8s), hosted vs self-host base URLs, endpoint tables for the Meetings API and Agent API, SSE frame types (message-delta, tool-call, commit, done, error), platform enum, STT/503 behavior, pricing ($0.30/hr, $5 free credit ≈ 16 hours), and an honest status/roadmap table with three states, including the 4–7% empty-speaker attribution disclosure, Jitsi pending live validation, and K8s backend status. Compliance artifacts are also cited (SECURITY.md, security-insights.yml, CALM, license-exceptions). Deductions: MCP-client documentation is delegated to docs.vexa.ai/vexa-mcp, so this repo lacks the tool list, parameters, return shapes, and client snippets; rate limits and quotas are unstated; the v0.12 404 behaviors are scattered between body text and a blockquote and are easy to miss; no troubleshooting section appears in the supplied source. Usable but with hidden assumptions and troubleshooting gaps — 14.
- Setup experience 11/20
- The hosted MCP path is very light: one streamable-http URL plus a bearer key from vexa.ai/account, no local runtime or build, few explicit steps — a clear plus. Self-hosting also documents concrete targets (make all/lite/dev/probe/down/help) and ports (Terminal 13000, Gateway 18056). Deductions are significant: the self-host bar is not low — Docker engine ≥ v26 and make, Linux (Ubuntu 24.04) as the production target, transcription as a separate GPU unit that is required by default (503 on POST /bots without it), and a nine-container stack (gateway, terminal, meeting-api, agent-api, runtime, admin-api, redis, postgres, minio); the agent plane is self-host only (absent from the hosted service); contributor builds want 8 vCPU / 16 GB; production needs a user-supplied TLS reverse proxy; and no mainstream MCP client config examples (Claude Desktop, Cursor, VS Code) appear — only curl samples. Steps are clear but component-heavy, platform-heavy, and missing client config samples; conservative score 11 under the static cap.