yaalalabs/agent-kernel/ak-test

Testing & Debugging

Use this skill to set up testing for your Agent Kernel project or debug issues.

Instructions for the Agent

Setting Up Tests

1. Add Test Dependencies

Update pyproject.toml:

[dependency-groups]
dev = [
    "agentkernel[test]>=0.9.5",
    "black>=23.0.0",
    "isort>=5.0.0",
    "mypy>=1.0.0",
]

Run uv sync to install test dependencies.

2. Choose a Test Mode

Create test-config.yaml in the directory you run tests from — it is a separate, un-nested file (no top-level test: key), loaded only when the test harness runs. A test: section left over in config.yaml is ignored:

mode: score       # Options: score | llm | fallback (default: fallback)

For llm mode, configure the llm model:

mode: llm
llm:
  model: gpt-4o-mini
  provider: openai

Evaluator backend: evaluator selects the scoring backend used by both score and llm modes — deepeval (the default, pip install "agentkernel[test]"), opik (pip install "agentkernel[opik]", Opik by Comet, runs entirely locally) and jev (pip install "agentkernel[jev]", hosted TypeSafe JEV judge: mode: llm only, needs TYPESAFE_API_KEY, sends the comparison text to api.typesafe.ai) are the three built-ins. Set it to a dotted path (e.g. my_evaluator.MyEvaluator) to bring your own AKEvaluator subclass instead:

evaluator: opik   # switch to another built-in (jev also needs mode: llm)

mode: fallback
evaluator: my_evaluator.MyEvaluator   # resolves against my_evaluator.py next to your test file

2a. Bring Your Own Evaluator (optional)

Use this when none of the built-in evaluators' scoring fits your agent — e.g. deepeval's binary exact-match score mode is too strict and opik's graded LevenshteinRatio still doesn't capture what you need, or you want a judge call that doesn't depend on DeepEval/Opik/JEV at all, or a domain-specific rubric. No AK core change is required: any dotted path to an AKEvaluator subclass works as the evaluator: value, resolved the same way sandbox providers and session stores resolve their own bring-your-own backends.

1. Create a module next to your test file (e.g. my_evaluator.py) and subclass AKEvaluator,

importing the interface from agentkernel.test:

```python from agentkernel.test import ( AKEvaluationCase, AKEvaluationError, AKEvaluationResult, AKEvaluator, AKMissingInput, )

class MyEvaluator(AKEvaluator): def evaluate_by_score(self, case: AKEvaluationCase) -> AKEvaluationResult: if not case.expected: raise AKMissingInput("evaluate_by_score requires AKEvaluationCase.expected") score = ... # your deterministic, offline scoring logic return AKEvaluationResult( metric="my_metric", evaluator="my_evaluator", score=score, passed=score >= case.threshold, )

def evaluate_by_llm(self, case: AKEvaluationCase) -> AKEvaluationResult: if not case.expected: raise AKMissingInput("evaluate_by_llm requires AKEvaluationCase.expected") try: score = ... # your judge call (any LLM client — litellm, an SDK, a hosted judge) except Exception as exc: raise AKEvaluationError(f"judge call failed: {exc}") from exc return AKEvaluationResult( metric="my_llm_metric", evaluator="my_evaluator", score=score, passed=score >= case.threshold, ) ```

1. Both methods are synchronous and must set result.passed themselves — Test.compare decides

whether a failing passed is fatal (raises AssertionError) or, with return_metrics=True, returned to the caller; it never overrides passed.

1. Follow the same error contract every evaluator (built-in or custom) must honor: raise

AKMissingInput if a required AKEvaluationCase field (usually expected) is missing; raise AKEvaluationError if your backend fails (bad credentials, transport error, unparseable judge output) — never return a 0.0 to stand in for a failure, since 0.0 must only ever mean "scored zero". If your evaluator only supports one of the two modes (e.g. judge-only, no offline scoring), raise AKMetricNotSupported from the other and set mode to the supported one (e.g. mode: llm) — fallback does not catch it, so the error propagates and the test fails.

1. Point test-config.yaml at it by dotted path — module_name.ClassName, resolved against the

module's location (next to your test file, since that's what's on sys.path under pytest's default import mode):

``yaml evaluator: my_evaluator.MyEvaluator ``

1. No AK extra beyond agentkernel[test] is needed unless your evaluator's own dependencies

(an LLM client, a scoring library) require one — each built-in's import (deepeval, opik, jev) lives entirely inside its own resolution branch, so a custom evaluator never pulls any in.

See examples/cli/custom-evaluator/ for a complete worked example — a stdlib-only Jaccard token-overlap scorer plus a raw litellm judge call, no DeepEval dependency at all — and `docs/docs/testing/cli-testing.md` for the reference documentation.

3. Write CLI Agent Tests

For agents running via CLI (demo.py):

import pytest
import pytest_asyncio
from agentkernel.test import Test
pytestmark = pytest.mark.asyncio(loop_scope="session")
@pytest_asyncio.fixture(scope="session", loop_scope="session")
async def test_client():
    test = Test("demo.py")       # Path to your agent definition file
    await test.start()
    try:
        yield test
    finally:
        await test.stop()
@pytest.mark.order(1)
async def test_greeting(test_client):
    await test_client.send("Hello!")
    await test_client.expect(["Hello", "Hi", "Greetings"])
@pytest.mark.order(2)
async def test_specific_question(test_client):
    await test_client.send("What is the capital of France?")
    await test_client.expect(["Paris"])
@pytest.mark.order(3)
async def test_follow_up(test_client):
    # Follow-up questions work because session state is maintained
    await test_client.send("What is its population?")
    await test_client.expect(["2 million", "2.1 million", "approximately 2 million"])

Key patterns:

• Use @pytest.mark.order(n) for sequential tests where context matters
• Use scope="session" fixtures so the agent stays running across tests
• expect() takes a list of acceptable answer patterns
• The test framework uses the configured mode to compare responses
• Pass return_metrics=True to expect() (or Test.compare()) to get back an

AKEvaluationResult (score, evaluator, metric, reason) instead of raising AssertionError on a mismatch — useful for asserting on the score itself rather than just pass/fail

4. Write API Agent Tests

For agents running via REST API:

import asyncio
import os
import subprocess
import sys
import uuid
import httpx
import pytest
import pytest_asyncio
from agentkernel.test import Test
pytestmark = pytest.mark.asyncio(loop_scope="session")
class APITestClient:
    def __init__(self, url: str):
        self.url = url
        self.session_id = str(uuid.uuid4())
    async def send(self, prompt: str, agent: str = "triage") -> str:
        payload = {
            "prompt": prompt,
            "session_id": self.session_id,
            "agent": agent,
        }
        async with httpx.AsyncClient(timeout=30.0) as client:
            resp = await client.post(f"{self.url}/run", json=payload)
            resp.raise_for_status()
            return resp.json().get("result", "")
@pytest_asyncio.fixture(scope="session", loop_scope="session")
async def http_client():
    # Option A: Test against running server
    endpoint = os.getenv("AK_TEST_ENDPOINT", "http://localhost:8000")
    # Option B: Start server in fixture
    # proc = subprocess.Popen(["python3", "app.py"], stdout=sys.stdout, stderr=sys.stderr)
    # await asyncio.sleep(5)
    yield APITestClient(endpoint)
    # proc.terminate(); proc.wait()  # if using Option B
@pytest.mark.order(1)
async def test_basic_question(http_client):
    response = await http_client.send("What is 2+2?")
    Test.compare(response, ["4", "The answer is 4"])
@pytest.mark.order(2)
async def test_agent_routing(http_client):
    response = await http_client.send("Tell me about World War 2")
    Test.compare(response, ["World War II", "World War 2", "WWII"])

5. Run Tests

uv run pytest                          # All tests
uv run pytest demo_test.py             # Specific file
uv run pytest -k "test_greeting"       # By name pattern
uv run pytest -x                       # Stop on first failure
uv run pytest -v                       # Verbose output
uv run pytest --tb=long                # Full tracebacks

Debugging Common Issues

Issue: "No agents available"

Symptom: CLI shows "No agents available. Please load an agent module using !load."

Cause: The Module constructor was not called, so no agents are registered with Runtime.

Fix: Ensure your agent file calls the Module constructor:

# This line registers agents with the global Runtime
OpenAIModule([triage_agent, math_agent])

Issue: Session state not persisting

Symptom: Agent doesn't remember context from previous messages.

Causes & Fixes:

1. In-memory sessions (default): State is lost when the process restarts. Switch to Redis/DynamoDB/CosmosDB for persistence.
2. Different session IDs: Ensure you're using the same session_id across requests.
3. Lambda cold starts: Session state must be stored externally. Use Redis or DynamoDB.

Check session config:

session:
  type: redis
  redis:
    url: "redis://localhost:6379"
    prefix: "ak:myproject:"

Issue: "ToolContext not available"

Symptom: RuntimeError: ToolContext is not set inside a tool function.

Cause: The tool is being called outside of the agent execution context.

Fix: Ensure tool functions are bound via the framework's ToolBuilder and called within agent execution. Don't call tool functions directly outside of Runtime.run().

# Correct: bound via ToolBuilder
tools = OpenAIToolBuilder.bind([my_tool])
agent = Agent(name="test", tools=tools, instructions="...")
# Inside tool function:
def my_tool(query: str) -> str:
    context = ToolContext.get()  # Works during agent execution
    session = context.session
    return "result"

Issue: Guardrail blocks all requests

Symptom: Every request returns a guardrail violation message.

Fixes:

1. Check guardrail config — ensure config_path points to a valid JSON file
2. Review guardrail rules — thresholds may be too strict
3. Check the guardrail model — ensure model field is correct
4. Disable temporarily to isolate: set enabled: false in config

Issue: Import errors for framework packages

Symptom: ModuleNotFoundError: No module named 'agents' (or crewai, langgraph, etc.)