trustabl/agent-reliability-analyzer/trustabl-scan

Self-audit agent code with Trustabl

Trustabl finds and fixes reliability defects in agent-SDK code. It models the tools, agents, subagents, and MCP servers a repo declares and checks them against a rule catalog, reporting reliability and safety weaknesses with an explanation, a suggested fix, and a confidence score. Run it as a self-audit at generation time, upstream of CI, so issues are caught and fixed before the commit.

Trustabl is read-only: it detects and reports, and never writes to or modifies the scanned repo. You apply the fixes yourself, then re-scan to confirm.

When to use this

Invoke after you have just written or changed any of these, before committing:

• An agent definition (Agent(...), SandboxAgent(...), AgentDefinition,

new Agent({...}), ADK LlmAgent / SequentialAgent / etc., or a TS query(...) main-thread agent).

• A tool definition (a @function_tool / @tool / @claude_tool function, a

TS tool(...) factory call, an ADK FunctionTool wrapper, or a shell-invoking function).

• A subagent markdown file (.claude/agents/*.md), a skill, a slash command, or

a plugin manifest.

• An MCP server registration (@server.tool / @mcp.tool / createSdkMcpServer)

or an MCP config.

• Agent guardrails (@input_guardrail / @output_guardrail) or

.claude/settings.json permission settings.

Languages understood today: Python, TypeScript, and JavaScript (.ts / .tsx / .mts / .cts / .js / .jsx / .mjs / .cjs) for tools and agents, plus MCP-server tool discovery in Go, C#, PHP, and Rust. The dependency BOM (and --vuln-scan) spans all of those package ecosystems — pip, npm, Go modules, NuGet, Composer, and Cargo.

How to run the scan

Scanning runs through this plugin's bundled MCP server (trustabl), which exposes a scan tool. Call the tool `mcp__trustabl__scan` with the path to the repo you just edited:

• mcp__trustabl__scan with {"path": "."} — scan the current repo.
• Optionally {"path": ".", "rules_ref": "<branch-or-tag>"} to pin the

detection-rules ref.

• Optionally {"path": ".", "vuln_scan": true} to also match the repo's declared

dependencies against a pinned OSV database and report known CVEs as findings (plus a structured vulnerabilities[] array). Off by default; the first use downloads the OSV snapshot, then reuses the cache.

The tool returns the full scan result as JSON — the same ScanResult shape as the CLI's --format json output, with a findings array to reason over. There is no exit code: decide what is actionable by reading finding severities (medium and above are commit-blockers; see "How to read findings"). The server fetches and caches the agent-reliability-rules pack itself, falling back to its local cache when offline.

CLI fallback. If the mcp__trustabl__scan tool is not available this session (for example the MCP server failed to start), run the CLI directly with the resolved binary — "$TRUSTABL_BIN" when that variable is set, else trustabl on PATH:

"$TRUSTABL_BIN" scan . --format json          # JSON, same shape the tool returns
"$TRUSTABL_BIN" scan . --strict               # exit 1 on low+ findings (info/META never fail)
"$TRUSTABL_BIN" scan . --detectors claude_sdk # claude_sdk|openai_sdk|google_adk|openshell|mcp|langchain|crewai|pydantic_ai|vercel_ai|autogen|claude_skill
"$TRUSTABL_BIN" scan . --vuln-scan            # also match declared deps against OSV → CVE findings
"$TRUSTABL_BIN" scan . --bom-out bom.json     # write a CycloneDX SBOM (+ VEX vulnerabilities under --vuln-scan)
 
On Windows, give the output flags a native path. The binary does not resolve a
Git Bash style `/tmp/...` path: it writes nothing and reports no error, so the
missing file is the only symptom.

In the CLI fallback the exit code is the gate: 0 = no findings of medium severity or higher, 1 = at least one, 2 = scanner/I-O error or no usable rules (run "$TRUSTABL_BIN" rules pull once to pre-populate the rules cache).

Reading the inventory safely

Empty inventory fields are null in the JSON result, not []. subagents, skills and mcp_servers are all absent rather than empty when nothing of that kind was found, so take the length defensively:

count = len(result.get("mcp_servers") or [])

Treating them as lists unconditionally raises on a repository that happens to declare no subagents, which is most of them.

How to read findings

Each finding carries:

• severity (info / low / medium / high / critical) how bad it is.

Treat medium and above as commit-blockers; info / META never block. (In the CLI fallback these map to the exit code: medium+ → exit 1, or low+ under --strict.)

• confidence a heuristic score for how sure the rule is. It is not

LLM-judged and not yet calibrated, so treat findings as signal to investigate, not ground truth. Look at higher-confidence findings first.

• explanation why the pattern is a problem.
• suggested_fix the concrete remediation to apply.
• location where it fired, attributed to one of five scopes:
• tool a specific tool definition (file and line). Example: an HTTP call

with no timeout, untyped parameters, a missing docstring.

• agent a specific agent at its constructor call site, with its resolved

tool / handoff / guardrail edges. Example: an agent wired with shell tools but no input guardrails.

• subagent a .claude/agents/*.md declaration. Example: a subagent

granted Bash despite a read-only description.

• skill a SKILL.md declaration (a Claude Code skill). Example: a skill

that auto-approves unrestricted Bash, or runs a dynamic-context command that reads secrets before the model sees it.

• repo the whole repo, once per scan. Example: an SDK present with no

custom trace processor configured.

META findings (for example "this repo uses an SDK Trustabl does not currently audit", or "this dep is declared but unused") are honest coverage signals, not defects, and never fail the build. Two agents in the same repo can be in very different postures, so always fix the agent the finding names, not the repo as a whole.

Dependency vulnerabilities (only when the scan ran with --vuln-scan or the vuln_scan: true tool arg): each known CVE in a declared, concretely-pinned dependency is a finding whose rule_id is the advisory id (CVE / GHSA / PYSEC), located at the dependency's line in its manifest, with the fixed version in suggested_fix. They flow through the same severity gate as rule findings — fix by upgrading the dependency to the noted version.

Remediation loop

1. Call mcp__trustabl__scan with {"path": "."} and read the findings array.
2. For each finding of medium or higher, read its explanation and

suggested_fix and apply the fix to the code (Trustabl will not edit files for you).

1. Call the scan tool again and confirm the finding is gone from findings.
2. Repeat until only info / META signals remain. For a stricter bar before

committing, treat low findings as blockers too (or use the CLI fallback's --strict).

Installing the binary

Normally you do not need to. The plugin installs the pinned CLI itself: the bundled MCP launcher (scripts/trustabl-mcp.sh) and the SessionStart hook (scripts/check-trustabl.sh) share install logic (scripts/lib-trustabl.sh) that downloads the pinned version, verifies it against the release checksums.txt, and installs it into the plugin's private data directory. That one binary both backs the mcp__trustabl__scan server and is exposed as $TRUSTABL_BIN for direct CLI use. It installs only into the plugin's own data dir — never the user's system or their own trustabl — and is reversible.

Auto-install is skipped only when it cannot run: offline on the very first session, an unsupported platform (e.g. native Windows without bash), or missing curl / tar. It then falls back to whatever trustabl is on PATH. If you see a [trustabl] could not provide the trustabl CLI … notice, or the mcp__trustabl__scan tool is missing and no binary resolves, offer to install it system-wide, asking the user before you run any install command; never install silently:

# macOS / Linux (Homebrew)
brew install trustabl/tap/trustabl
 
# Windows (Scoop)
scoop bucket add trustabl https://github.com/trustabl/scoop-bucket
scoop install trustabl
 
# Docker (no local install; mount the repo at /repo)
docker run --rm -v "$PWD:/repo" ghcr.io/trustabl/trustabl:latest scan /repo

Prebuilt archives for each platform are on the GitHub Releases page (https://github.com/trustabl/agent-reliability-analyzer/releases). Confirm the install with trustabl version.