ajshedivy/agno-cookbook/agentos-api-evals
Claude Code and agent skill by ajshedivy/agno-cookbook
- community
Checking that the SKILL.md file is still in the repository.
Interact with AgentOS Evals API endpoints. For standard operations (listing evals, running accuracy/performance evals, getting eval details), use the provided CLI script first. Only write custom Python when the script cannot handle the use case (e.g., advanced filtering, update/delete operations, chaining evals, agent-as-judge). Trigger when: running evaluations, listing eval runs, benchmarking agents, or asking things like "run an accuracy eval on my agent" or "show me the latest eval results."
Install
npx vskill@latest install ajshedivy/agno-cookbook/agentos-api-evalsSKILL.md
The same SKILL.md was published earlier as ajshedivy/agno-cookbook/agentos-api-evals.
AgentOS Evals API
Prerequisites
Start an AgentOS server with agents:
Default: Use the CLI Script
Always try the provided script first. It covers listing evals, running accuracy and performance evaluations, filtering by agent, and getting eval details — all from the command line with no custom code needed.
The script is at: scripts/run_evals.py
List all evaluation runs
Run an accuracy evaluation
Run a performance evaluation
Get details for a specific eval
Filter evals by agent
Use a different server
Full CLI reference
When to Write Custom Python
Only write ad-hoc Python when the CLI script cannot handle your use case:
- Advanced filtering (by team, workflow, model, eval type, pagination, sorting)
- Update or delete evaluation runs
- Agent-as-judge or reliability eval types (not supported by the CLI)
- Chaining multiple evals in a single script
- Custom error handling or retry logic
- Integration tests that assert on eval results programmatically
API Endpoints
| Method | Path | Description | |
|---|---|---|---|
| GET | /eval-runs | List evaluation runs (paginated, filterable) | |
| GET | /eval-runs/{eval_id} | Get evaluation run details | |
| POST | /eval-runs | Execute evaluation | |
| PATCH | /eval-runs/{eval_id} | Update evaluation run | |
| DELETE | /eval-runs | Delete evaluation runs |
Evaluation Types
| Type | Description | |
|---|---|---|
accuracy | Compare agent output against expected output | |
performance | Measure response time and throughput | |
agent_as_judge | Use another agent to evaluate quality | |
reliability | Test consistency across multiple runs |
Custom Python Examples
List with Advanced Filtering
The list endpoint supports filtering beyond what the CLI provides:
Supported filter parameters:
agent_id: Filter by agentteam_id: Filter by teamworkflow_id: Filter by workflowmodel_id: Filter by modeltype: Filter by component type (agent,team,workflow)eval_types: Comma-separated eval types (accuracy,performance,agent_as_judge,reliability)limit,page: Pagination (default: 20 per page)sort_by,sort_order: Sorting (default:created_atdesc)
Chained Evaluation Workflow
Run multiple eval types and aggregate results in a single script:
Anti-Patterns
- Don't write custom Python for basic operations — use the CLI script for listing, running accuracy/performance evals, and getting details
- Don't forget `await` — all client methods are async
- Don't hardcode agent IDs — discover them from
aget_config()or the CLI script - Don't skip error handling — eval endpoints may fail if agents aren't configured properly
- Don't use too many iterations — performance evals with high
num_iterationscan be slow and expensive - Don't ignore eval data —
eval_datacontains the actual evaluation results and metrics
Further Reading
For advanced eval API patterns, read references/api-patterns.md.
Security
- Security scanPassed, 90 of 100Pattern checks on the skill's files.
- Model reviewPassed
Quality evaluation
- Task evaluationNot runNo evaluation yet. Run one in Skill Studio.
Source
- SKILL.md in the repositoryCheckingplugins/agno-agentos-api/skills/agentos-api-evals/SKILL.md
Popularity
- 7-day trend score
- 0
Not refreshed yet.
Works with
Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.
- Claude Code
- Cursor
- GitHub Copilot
- Windsurf
- Codex
- Gemini CLI
- Cline
- Amp
- Roo Code
- Goose
- Continue
- Kilo Code
- Junie
- Trae
- OpenHands
- Qwen Code
- AI Agent
- Tabnine Chat
- Sourcegraph Cody
- Amazon Q
- Firebase Genkit
- Sweep
- Devin
- Phind
- Mentat
- Claude Desktop
- Aider
- GPT Pilot
- Plandex
- bolt.new
- v0
- Replit Agent
- Anse
- TypingMind
- BetterChatGPT
- LibreChat
- Warp AI
- Pieces
- Fig AI
Embed badge
[](https://verified-skill.com/skills/ajshedivy/agno-cookbook/agentos-api-evals)