florianbruniaux/claude-code-plugins/eval-agents

Agent evaluator

Discover every custom agent in scope, validate it against the native schema of its host, score it, then review it with the user one agent at a time.

Agents are selected by the parent agent from their description. A vague or overlapping description makes delegation unpredictable, so overlap is a correctness defect, not a style issue. The goal is to leave every agent loadable, distinct, correctly modeled, and safe to delegate to.

When to use

• Auditing an agent fleet before wiring it into a delegation or orchestration workflow
• The parent agent keeps delegating to the wrong agent, or never delegates
• After copying agents from another project or installing a plugin
• A new agent was added and may collide with existing ones
• Periodic hygiene: do all these agents still do something distinct?

Not for: skills (eval-skills), hooks (eval-hooks), Claude Code rules (eval-rules), or a cross-host configuration health report (eval-agent-config).

Host reference

Claude Code: where agents load

Claude Code picks the definition from the highest-priority location when several share a name:

• .claude/agents/ and ~/.claude/agents/ are scanned recursively. Subfolders do not change identity.
• Project agents are discovered by walking up from the working directory; every .claude/agents/ up to the repository root is scanned, and the definition closest to the working directory wins.
• Directories added with --add-dir or /add-dir also contribute their .claude/agents/.
• Plugin subfolders become part of the scoped identifier: agents/review/security.md in plugin my-plugin registers as my-plugin:review:security.

Claude Code: files the runtime skips silently

• No name: treated as documentation kept beside the agents.
• Opening --- not on the first line: read as having no frontmatter.
• name starting with - or containing :: skipped, error in the debug log.
• name without description: skipped.
• YAML that does not parse: skipped. A plugin agent without name, or with unparseable frontmatter, still loads under its filename.

"Does not parse" means Claude Code's own parser, which accepts some frontmatter a strict YAML library rejects, such as an unquoted multi-line description whose lines contain : . Decide with claude plugin validate <agents directory> (v2.1.233 or later) or with the session's agent list (/agents), never with a strict YAML library alone. A file that only a strict parser rejects loads: report it as a portability warning (other tools may reject it) and suggest a > block scalar.

Two files in the same agents tree that declare the same name: only one loads, chosen by filesystem read order, not by a documented precedence. /doctor reports them.

Claude Code: frontmatter fields

Only name and description are required. Field names are camelCase and must match exactly; an unknown field is ignored without an error.

Plugin agents ignore hooks, mcpServers and permissionMode; initialPrompt is also ignored for plugin agents.

Flag as unrecognized (silently ignored by Claude Code): allowed-tools, disallowed-tools, context, argument-hint, and any other key not in the table. These are skill fields, not agent fields.

Claude Code: model resolution

1. The per-invocation model parameter passed by the parent
2. The agent's model frontmatter (inherit selects the main conversation's model)
3. CLAUDE_CODE_SUBAGENT_MODEL, when set to an alias or model ID
4. The main conversation's model

CLAUDE_CODE_SUBAGENT_MODEL_FORCE=1 makes every agent ignore its model field. A family alias resolves to the main conversation's exact model when that model belongs to the same family. A value blocked by the organization's availableModels is substituted.

Claude Code: tools a subagent never gets

The runtime removes these tools from every non-fork subagent, even when listed in tools: AskUserQuestion, EndConversation, EnterPlanMode, ExitPlanMode (unless permissionMode: plan), ScheduleWakeup, WaitForMcpServers, and Agent at the depth limit.

Codex: custom agents

A Markdown agent with YAML frontmatter, or a TOML file using prompt instead of developer_instructions, is a Claude-shaped definition and does not satisfy the Codex schema.

Scoring criteria

Claude Code agents (15 pts)

Codex agents (12 pts)

Thresholds: Good >= 80%, Needs work 60-79%, Fix < 60%.

Model tier signals

Flag both directions: an under-powered model for judgment work, and an over-powered model for mechanical work that runs many times. State cost from published prices instead of a fixed multiplier; for example the guide lists Opus 5.5 at $4/$20 and Haiku 5.5 at $0.10/$0.50 per million input/output tokens (prompts up to 100K tokens). Haiku 5.5 supports effort and defaults to medium; only the legacy Haiku 4.5 has no effort parameter, so an effort field on an agent pinned to Haiku 4.5 has no effect.

Human-in-the-loop anti-pattern

Subagents do not get AskUserQuestion or plan-mode tools, and by default they run in the background. Any system prompt instruction such as "ask the user", "confirm before deleting", or "wait for approval" cannot be carried out. Safe alternative: "If context is insufficient, return { "status": "needs_context", "missing": [...] } and stop."

Execution instructions

Step 1: Discovery

Resolve the scope from the argument, or default to the project plus user scope of each requested host.

# Claude Code: every .md file, recursively (-L follows symlinked directories)
find -L .claude/agents -name "*.md" 2>/dev/null
find -L ~/.claude/agents -name "*.md" 2>/dev/null   # user scope, when requested
 
# Codex: one TOML file per agent
find -L .codex/agents -name "*.toml" 2>/dev/null
find -L ~/.codex/agents -name "*.toml" 2>/dev/null  # user scope, when requested

Also list nested .claude/agents/ directories between the working directory and the repository root.

When a ctxharness doctor --format json report generated during this task is available, every agents-layer finding it reports must appear in the audit.

Done when: every candidate file is listed with its host and scope, or the report states that no agent directory exists and stops.

Step 2: Parse and check loadability