realsudarshan/sd-skill/gemini-github-behavioral-evals

Behavioral Evals

Overview

Behavioral evaluations (evals) are tests that validate the agent's decision-making (e.g., tool choice) rather than pure functionality. They are critical for verifying prompt changes, debugging steerability, and preventing regressions.

[!NOTE]

Single Source of Truth: For core concepts, policies, running tests, and general best practices, always refer to [evals/README.md](file:///Users/abhipatel/code/gemini-cli/docs/evals/README.md).

🔄 Workflow Decision Tree

1. Does a prompt/tool change need validation?

• No -> Normal integration tests.
• Yes -> Continue below.

1. Is it UI/Interaction heavy?

• Yes -> Use appEvalTest (AppRig). See [creating.md](references/creating.md).
• No -> Use evalTest (TestRig). See [creating.md](references/creating.md).

1. Is it a new test?

• Yes -> Set policy to USUALLY_PASSES.
• No -> ALWAYS_PASSES (locks in regression).

1. Are you fixing a failure or promoting a test?

• Fixing -> See [fixing.md](references/fixing.md).
• Promoting -> See [promoting.md](references/promoting.md).

📋 Quick Checklist

1. Setup Workspace

Seed the workspace with necessary files using the files object to simulate a realistic scenario (e.g., NodeJS project with package.json).

• *Details in [creating.md](references/creating.md)*

2. Write Assertions

Audit agent decisions using rig.setBreakpoint() (AppRig only) or index verification on rig.readToolLogs().

• *Details in [creating.md](references/creating.md)*

3. Verify

Run single tests locally with Vitest. Confirm stability locally before relying on CI workflows.

• *See [evals/README.md](file:///Users/abhipatel/code/gemini-cli/docs/evals/README.md) for running commands.*

📦 Bundled Resources

Detailed procedural guides:

• [creating.md](references/creating.md): Assertion strategies, Rig selection, Mock MCPs.
• [fixing.md](references/fixing.md): Step-by-step automated investigation, architecture diagnosis guidelines.
• [promoting.md](references/promoting.md): Candidate identification criteria and threshold guidelines.