- Reliability 10/20
- The static evidence is limited to the README, server.json, and repository metadata; no committed test files, CI workflow bodies, or source code are present. Verifiable-looking reliability evidence includes the declared 14 tools matching the README parameter tables, explicit degradation semantics (graph tools return 'graph backend unavailable' while the other 9 keep working), concept-graph failure never blocking core memory, a 1MB response limiter, an explicit recall_scan_truncated flag, a serialized write queue, and a described subprocess crash-respawn path. These are concrete and internally coherent design intent, but they are self-reported and cannot be checked statically. The README itself admits run-to-run variance and a concurrent gather/serial ratio of 0.63-0.86 that it labels measurement noise, so the performance numbers are not independently verifiable. The README is visibly truncated mid-sentence ('caught at s'). With no visible test coverage, fixtures, or reproducible verification steps, the static calibration rule forces a cap of 12 and requires deducting for uncertainty. Score 10: happy path is plausible, edge and failure paths are thinly evidenced.
- Security and permissions 11/20
- No red line is triggered: no malware, credential theft, covert exfiltration, or fabricated identity indicators; localhost-by-default binding and a mandatory MARM_API_KEY for Docker HTTP exposure are reasonable least-privilege stances. Deductions remain. SERVER_HOST defaults contradict each other between the manifest (0.0.0.0) and the README (localhost by default); taking the manifest at face value means listening on all interfaces by default, a potential exposure surface. The 'trusted' profile explicitly disables rate limiting and --expose-network binds to 0.0.0.0, neither of which has a built-in confirmation step, relying on the user to add a firewall and TLS proxy per the docs. Examples include 'MARM_API_KEY=test' and 'your-generated-key' placeholders, which are not real secrets so no leak occurs, but placing a throwaway-key pattern in copyable positions carries practical risk. Data-plane handling (zero cloud, everything under ~/.marm) is clearly disclosed, while permission granularity for console key import and cross-process locking is only partially described. Score 11: no red line, but exposure and permission controls are incomplete.
- Maintenance 13/20
- There is an Apache-2.0 license, clear attribution (Ryan Lyell / marm-memory), consistent semantic versioning at 2.48.0 across manifest, README, and Docker tags, plus a Discord community and GitHub Issues/Discussions channels. The README's change narrative (the v2.44.4 log-lane fix and the v2.33.1 to v2.44.3 LoCoMo progression) indicates an ongoing release cadence, and 350 stars with 12 open issues and an unarchived repository support continued activity. Deductions: no source-level dependency update policy, release signing, SECURITY.md, or security-response process is in evidence; the CodeQL and Publish badges point at CI whose contents are not provided; and one competitor comparison table self-describes its snapshot as 'Q3 2026' while admitting it comes from public docs rather than testing, which is a questionable time basis. Score 13: active with clear ownership, but governance and security-response channels are under-evidenced.
- Documentation 15/20
- Documentation is layered and substantial: quick start, six run configurations (pip HTTP/STDIO, Docker HTTP/STDIO, swarm profiles), client config examples for Claude Code, VS Code, Cursor, Codex, Gemini, Qwen, and Grok, parameter tables for all 14 tools, environment variable descriptions, internals (WAL, FTS5, chunked embeddings, write queue, isolated concept-graph DB), troubleshooting (401 vs 406 semantics), and backup/data-location guidance, plus docs-site and API-reference links. Deductions: (1) SERVER_HOST's default is directly contradictory between the manifest (0.0.0.0) and the README (localhost by default), a capability/config documentation conflict; (2) benchmarks openly admit run variance and the LoCoMo section reports ranges rather than point results, with some figures framed as 'shape not absolute values', limiting reviewability; (3) the README is truncated with an unfinished final sentence; (4) the cost model is described only in token terms with no concrete budget table. Score 15: usable layered docs with internal contradictions and unfinished content.
- Setup experience 14/20
- Installation paths are clear and varied: pip install followed by marm-memory init / start, or marm-mcp-stdio; Docker offers docker run, docker compose, docker stdio-command, and controlled flags such as --dry-run, --data-dir, and --repo, with key handling that writes to ~/.marm/.env instead of shell history, and examples for Windows/macOS/Linux including PowerShell syntax. Per the static calibration rule, scoring setup above 15 requires verifiable execution evidence in the files (real CI workflows plus committed tests covering key paths); the evidence shows only badge references to CI, with no workflow bodies or test files, so a cap of 15 applies. Deductions: default port 8001 plus the need to understand the HTTP/STDIO/Docker/multi-client matrix still leaves manual steps for newcomers; the manifest/README host-default conflict adds troubleshooting cost; and Docker code-graph indexing requires recreating the container with a repo mount, adding complexity. Score 14: reachable in a few steps with ample examples, but unverified and with ambiguous configuration semantics.