- Reliability 7/20
- No test files, CI workflows, or source code are present in the evidence, so the MCP init/tool-list handshake cannot be verified as reproducible; per the static calibration rule reliability is capped at 12. More concerning, the manifest declares the npm package identifier as '@microsoft/azure-devops' while every install example in the README uses 'npx -y @azure-devops/mcp' / '@azure-devops/[email protected]' — a direct mismatch between the server.json manifest and the actual documented package, suggesting the manifest may be stale or inaccurate. The README also carries a prominent warning about a recent breaking 'tool consolidation and renaming', indicating recent instability. Combined, this warrants a low score of 7.
- Security and permissions 12/20
- The remote server authenticates via Microsoft account OAuth in-browser, and the local server supports azcli-based authentication; no hardcoded secrets or tokens appear in install examples, so no red line is triggered. However, the tool surface is broad (work-items, repositories, wiki, pipelines, advanced-security domains), includes write operations (creating/updating wiki pages), and the 'advanced-security' domain likely touches sensitive security-scan data — yet the documentation provides no explicit confirmation step, least-privilege scoping, or data-boundary disclosure for these write/sensitive paths; Domains only provide coarse tool-set filtering, not read/write separation. Main risks are visible but mitigations are incompletely documented, so 12 is appropriate.
- Maintenance 16/20
- The repository is maintained by Microsoft's official GitHub org under the MIT license, with 1942 stars and only 12 open issues, suggesting reasonably responsive issue handling. The README references nightly builds (@next) and version pinning (e.g., 2.8.1), indicating an active release cadence, and CONTRIBUTING.md plus a Code of Conduct establish governance. However, no explicit SECURITY.md or vulnerability-reporting channel is evidenced in the provided material, so it falls short of full marks: 16.
- Documentation 15/20
- Documentation is well-layered, spanning TOOLSET.md, GETTINGSTARTED.md, TROUBLESHOOTING.md, EXAMPLES.md, HOWTO.md, and FAQ.md, with multi-client configuration examples (VS Code, Visual Studio 2022, Codex, Claude Code, Cursor, etc.) and GIF walkthroughs. However, it does not disclose API rate limits, quota, or cost considerations, and the manifest's package name conflicts with the README's install command — an unresolved discrepancy that counts as a hidden inconsistency. This caps the score at 15.
- Setup experience 14/20
- The remote server requires only a single URL entry in mcp.json — near-zero installation friction — while the local server needs just one npx command plus an input parameter, with optional Domains and project/team defaults for finer control, all backed by clear step-by-step client configuration examples. Since this is a purely static review with no verifiable execution evidence (no CI or committed tests proving these steps actually work end-to-end), the static calibration rule caps setup at 15, so it is scored 14.