- Reliability 8/20
- Assessment is based solely on the static README and description. The README documents substantive internal mechanisms (a single FIFO runtime, paired globalSlots/dispatchSlots semaphores, a model-derived context budget formula, hybrid retrieval, four runaway guards, signal-triggered self-evolution) and cites concrete files such as bus.ts, context-budget.ts, workspace-state.ts, metacognition.ts, and actor-budgets.ts, which indicates deliberate design of key paths. However, no MCP server manifest is cached at the repository root, and there is no verifiable evidence of a clean startup or of the init + tool-list handshake. No CI workflows or committed tests covering the server's key paths are present, so the static calibration caps this dimension at 12; scored 8, deducted for the absence of reproducible execution evidence and error-path testing.
- Security and permissions 10/20
- The README states a local-first design: conversations, files, API keys, knowledge bases, and custom agents remain on disk, model calls go straight from the local machine to the provider and are never proxied or archived by Orkas, local models (Ollama/LM Studio) can be attached via an OpenAI-compatible endpoint, and embeddings are computed on-device by a bundled ONNX model (bge-small-zh-v1.5). That aligns with least privilege and clear data boundaries. Offsetting factors: the repository root offers no MCP manifest, so tool-level permission scopes and confirmation behavior for dangerous operations cannot be verified; the app drives external CLI coding agents (Claude Code, Codex, OpenCode, Cline) as local subprocesses, a potentially high-privilege side-effect surface whose confirmation and isolation policy is not described; and the packaged app prompts for sign-in on first launch. No malware, credential theft, covert exfiltration, or unconfirmed irreversible defaults were found, so no red line (0-4) is triggered, but given the missing tool-scope inventory and confirmation flow, a conservative 10 is assigned.
- Maintenance 11/20
- The MIT license is explicit, the README carries release/download badges and an update entry point, and the project reports 30 marketplace agents, 1852 stars, 13 open issues, and a non-archived state, indicating active maintenance with clear organizational ownership (Orkas-AI, author X @leochenpm). What cannot be seen from the supplied material is commit history, release cadence, dependency-update practice, and a security-response channel (e.g., SECURITY.md or a CVE process). Governance and versioning gaps remain, matching the 'active but governance/versioning gaps' anchor, so 11.
- Documentation 12/20
- Documentation is comparatively thorough: the install section separates packaged installers from source runs; lists Node 20+, macOS/Windows 10+/glibc 2.34+ Linux, and explicitly rules out Alpine/musl; explains that first launch fetches and verifies many pinned resources (uv, Node, ~95 MB embedding model, OfficeCLI, FFmpeg, whisper.cpp, multilingual speech model); documents the workspace path; notes on-demand installs for OCR and skill-specific Python packages; and provides an FAQ on local models, key storage, offline capability, account requirements, platform support, and pricing. Deductions: there are no concrete MCP client configuration examples, no detailed tool-parameter inventory or limits, thin troubleshooting content, and a notable share of the document is marketing comparison and outbound links that carry limited independent evidentiary weight.
- Setup experience 10/20
- The install path is broadly clear: packaged installers for macOS and Windows, source build for Linux via git clone plus ./run.sh or run.cmd, with first-launch preparation of the locked npm tree, Python/uv/Node, embedding model, OfficeCLI, FFmpeg, whisper.cpp, and speech model, plus an Electron-ABI check of platform-native modules. The bar is nonetheless high: Node 20+ is required, complex native dependencies may need Python 3 and a C/C++ toolchain, the first-launch download is large and of uncontrolled duration, and Linux has no installer and no musl support. More decisively, no MCP manifest is cached and there is no configuration example showing a working connection in a mainstream MCP client, so the final 'working connection' step cannot be statically verified; under the static calibration cap of 15, this scores 10.