- Reliability 10/20
- Static evidence: the README claims tests/test_readme.py executes both code blocks on every commit and shows a CI badge pointing at actions/workflows/ci.yml; the project offers a free deterministic rehearsal (grapharc plan --scripted), trace-based replay (grapharc replay <trace> <run-id>), and per-node cost accounting, indicating deliberate design for reproducibility and diagnosability. However, no SERVER MANIFEST was cached, and the repository root supplies no verifiable MCP server declaration (no tool list, no schemas, no transport config, no stdio/HTTP launch command), so neither the MCP init/tool-list handshake nor declared-vs-real tool behavior can be confirmed. Deductions: the MCP entrypoint and tool contract are entirely absent; the self-reported test execution and CI are uncached self-claims and the README is untrusted evidence by rule; the Limits section itself admits the HTTP API does not yet use the durable session layer, that on the Claude CLI backend an agent node is 'delegated' rather than governed, and that the bypass tier has no checks at all — known unclosed reliability gaps. Structured rejection feedback and fail-closed budgets show error handling exists conceptually, but no verifiable error-code table or troubleshooting guide is present. Static calibration forbids >12 without committed CI workflows plus tests covering the server's key paths; scored 10.
- Security and permissions 10/20
- Static evidence: by design a deterministic admission gate (registry, policy, remaining budget, depth, acyclicity) sits between proposing and running, and unadmitted steps never execute; the Slack integration unconditionally appends --approve to --go so the run parks before its first node pending a human click, with the button bound to the plan fingerprint so a superseded proposal is refused rather than honoured; the tool plane uses DENY/ASK/ALLOW edge rules with an unmatched-default deny; budgets are worst-case priced before a run. No malware, credential theft, covert exfiltration, fabricated maintainer identity, or real tokens in install examples were found. Deductions, all self-disclosed in Limits: (1) the in-process sandbox is only defense in depth while ContainerExecutor is the real boundary, and run_command children are unconfined; (2) the MCP gate binds the MCP surface, not the host — an agent with its own file tools in the run directory could forge the approval decision, so the trust boundary is the working directory; (3) the Claude CLI bypass tier, explicitly opted into, has no checks at all. Credential handling is described only at the backend level (claude CLI / OpenRouter / OpenAI / Ollama), with no statement on secret storage, redaction, or log filtering; destructive kinds (e.g. deploy) rely on policy configuration rather than built-in hard constraints. These are disclosed but unclosed permission and boundary gaps; no red line (no unconfirmed irreversible deploy/pay/remote-command default) is present, so this is not scored 0-4. Scored 10.
- Maintenance 14/20
- Static evidence: MIT license is explicit; CHANGELOG.md and ROADMAP.md exist; version 0.1.7 is released on PyPI with a download badge; a GitHub Actions CI workflow exists; the repository is not archived; 69 stars and 13 open issues show external usage with publicly tracked problems; the release cadence read from the description indicates active maintenance. Deductions: issue response latency, dependency-update policy (e.g. Dependabot), and a security-response channel (SECURITY.md or private disclosure process) do not appear anywhere in the supplied material; governance and ownership are limited to an org name, with no maintainer list or release-authority statement; at 0.1.x the API stability is uncommitted. This fits the 'active but governance/versioning gaps' band, scored 14.
- Documentation 15/20
- Static evidence: docs are layered — the README covers install, quick start, a runnable direct graph-building example, the admission-gate example, the Slack supervised flow, and a comparison table against Claude Code/OpenClaw/raw LangGraph; separate docs/deep-dive.md, docs/cookbook/ (including 07-slack.md), docs/demo/ (stating per-recording what is and is not staged), bench/ with raw logs, CHANGELOG, ROADMAP, and a project website exist; the two README Python blocks are claimed to be executed by tests/test_readme.py on every commit to prevent drift (unverified); the Limits section lists limitations one by one with the most severe first; recordings (gif/mp4) are labeled for staging, giving reviewable evidence. Deductions: MCP-server-level documentation is missing entirely — no tool parameter descriptions, no MCP init/auth flow, no MCP client configuration example, no per-tool failure modes or troubleshooting section; cost and budgeting are conceptual with no concrete billing-example detail; the 'end to end implementation of Graph Engineering' claim lacks an alignment mapping to the referenced proposal. Usable but with hidden assumptions and troubleshooting gaps, scored 15.
- Setup experience 12/20
- Static evidence: the install path is short and explicit — pip install grapharc (Python >= 3.12), and grapharc demo stage0 costs nothing and needs no key; backends are split as extras (openrouter/openai/ollama/server/all); the default backend drives the claude CLI on PATH, usable with a subscription instead of an API key; grapharc start, grapharc init, and grapharc serve provide a guided tour, scaffolding, and a local browser view, with concrete copy-pasteable commands. Deductions: no configuration sample for any mainstream MCP client is provided (no claude_desktop_config.json or equivalent, no transport or launch-entrypoint description), and with no cached SERVER MANIFEST in the repository root there is no way to confirm how to attach this server to a client; the Python 3.12+ floor is high; the Slack flow needs additional workspace and bot configuration; backend choice and the presence of claude CLI on PATH are implicit prerequisites with no prerequisite check or troubleshooting given. Static calibration caps setup at 15 absent file-level execution evidence; scored 12.