- Reliability 10/20
- README shows a clear CLI/MCP architecture with `ao serve` for MCP and six tools. There is a contradiction: README claims 'npm + 2 dependencies' but actual project may have more. No tests or CI workflows were evident in the static review. Error handling and edge cases are thinly documented. Reliability is capped at 10 due to lack of verifiable execution evidence.
- Security and permissions 12/20
- API keys are handled via environment variables, no hardcoded keys. Several keyless providers (e.g., claude-code, gemini-cli) reduce key exposure, but invoking external LLMs via CLI sends code/text to third parties; data flow is disclosed but cross-network boundaries are not fully clarified. No malicious behavior found. Approval nodes (type: approval) require human confirmation, but MCP tools like run_workflow may execute workflows by default, including steps with external side effects. No explicit confirmation is evident. Security score 12, no red line issues.
- Maintenance 16/20
- Apache-2.0 license is clear, CI badge exists (though CI file not seen in static review), npm version suggests active releases, repository is active (README mentions v0.5, roadmap v0.6). No security response channel identified, but common for Apache projects. Dependency updates and issue response not directly confirmable. Maintenance score 16.
- Documentation 15/20
- README is extremely thorough: installation, configuration, 11 LLM providers, CLI commands, YAML schema, MCP config examples, built-in workflows, output structure. But lacks dedicated detailed documentation for MCP tools (parameters, error codes, limits). Cost is mentioned (e.g., DeepSeek top-up) but no concrete pricing. Limitations are disclosed (quality of local small models). Overall documentation good, but MCP tool docs thin, score 15.
- Setup experience 10/20
- Installation is straightforward: `npm install -g agency-orchestrator`, then `ao serve` for MCP. Configuration examples for Claude Code and Cursor are provided. However, dependency list is vague (claims 2 but actual may be more), and using a provider requires either a local CLI (like Claude Code) or API key setup, adding complexity. Docker deployment instructions exist. No execution verification possible, so actual setup may have surprises; score 10 (less than 15 cap due to no verifiable evidence).