getzep/zep/zep-eval-harness

Zep Eval Harness

An end-to-end evaluation framework for testing Zep's memory retrieval and question-answering capabilities. The pipeline ingests user conversations, telemetry, and documents into Zep knowledge graphs, then evaluates retrieval quality by searching those graphs with test questions and grading the results against golden answers.

The pipeline has four steps:

1. Chunk documents — split documents and generate LLM-based summaries + contextualizations
2. Ingest users — create Zep users, add conversations and telemetry to user graphs
3. Ingest documents — send pre-chunked documents to a standalone Zep document graph
4. Evaluate — for each test case: search graphs → assess context completeness → generate LLM response → grade answer accuracy

Scope: Single-Shot Retrieval

The harness evaluates single-shot retrieval only. Every test case issues one retrieval from the raw test question through the context endpoint (config/evaluation_config/retrieval_strategy.py), then hands the resulting context block to the response model in a single turn — no second retrieval round, no query reformulation. This mirrors deterministic/programmatic retrieval, not the tool-based pattern where an agent is handed Zep search tools (e.g. search_graph from the Zep MCP server) and decides when and what to search.

That makes the harness a clean instrument for the ingestion and search configuration, but it says nothing about agent tool-use behavior. A tool-based agent may do better (several targeted searches, reformulating after a weak result) or worse (never searching, poorly phrased queries, running out of turns). When reporting results, scope conclusions to the config under test and never present them as a prediction of production agent performance.

Evaluation Metrics

Context Completeness (PRIMARY) — Did Zep retrieve sufficient information to answer the question?

• COMPLETE: All necessary information present in the retrieved context
• PARTIAL: Some relevant information, but incomplete
• INSUFFICIENT: Missing critical information

This is the metric that matters most. It directly measures Zep's retrieval quality — whether the knowledge graph and search surface the right facts, entities, and relationships.

When completeness is low, the key diagnostic question is: does the graph contain the right information but search failed to retrieve it, or is the information missing from the graph entirely? Use zep_graph_inspect.py to examine what's actually in the graph. If the information is there but not retrieved, the issue is retrieval strategy (context character budget, query phrasing). If the information is absent from the graph, the issue is upstream — ingestion, ontology, or custom instructions need adjustment.

Answer Accuracy (SECONDARY) — Did the LLM produce a correct answer from the retrieved context?

• CORRECT: Answer conveys the same key information as the golden answer
• WRONG: Answer is missing critical information or incorrect

This measures whether the response model uses the context well. It depends on the LLM and response prompt, not Zep. High completeness + low accuracy means retrieval is working but the response generation needs tuning (better model, better prompt).

Metrics are calculated in aggregate, per-category (based on test case category field), and per-user.

Context Block: the v4 context endpoint

The evaluation script retrieves context via build_context_block() in config/evaluation_config/retrieval_strategy.py. The default strategy calls graph.get_context with MAX_CHARACTERS = 10000, and prepends the user-node summary (fetched once per user via fetch_user_summary — the context endpoint does not include it).

The context endpoint packs relevant edges (facts), nodes (entities), observations, and thread summaries into a pre-assembled context string. It does not include episodes. The character budget controls the volume.

To change retrieval (for example, calls to the v4 search methods for each scope — graph.search_edges, graph.search_nodes, graph.search_episodes, graph.search_observations, and graph.search_thread_summaries — with a page_size and a reranker for each scope), edit build_context_block in that module. That function is the source of truth for the strategy.

All commands run from zep-eval-harness/ using uv run. Depending on the user's request, either run the scripts directly or provide the terminal commands for the user.

Data Folder (data/)

Configuration Folder (config/)

config/
├── constants.py                        # GEMINI_BASE_URL, POLL_INTERVAL, POLL_TIMEOUT
├── user_ingestion_config/
│   ├── ontology.py                     # User graph entity/edge types + set_custom_ontology()
│   ├── custom_instructions.py          # User custom instructions + set_custom_instructions()
│   └── user_summary_instructions.py    # User node summary instructions
├── document_ingestion_config/
│   ├── constants.py                    # DOCUMENTS_GRAPH_NAME
│   ├── ontology.py                     # Document graph entity/edge types
│   └── custom_instructions.py          # Document custom instructions
├── document_chunking_config/
│   └── constants.py                    # CHUNK_SIZE, CHUNK_OVERLAP, LLM_CONTEXTUALIZATION_MODEL
└── evaluation_config/
    ├── constants.py                    # LLM_RESPONSE_MODEL, LLM_JUDGE_MODEL
    ├── retrieval_strategy.py           # build_context_block() — retrieval + context assembly
    └── response_prompt.py              # get_response_system_prompt() — AI persona for eval

Scripts

1. Chunk Documents (zep_chunk_documents.py)

Splits documents into chunks and generates LLM summaries + contextualizations. Writes to runs/chunk_sets/.

uv run zep_chunk_documents.py [OPTIONS]

2. Ingest Users (zep_ingest_users.py)

Creates Zep users, adds conversations and telemetry, polls for processing completion. The Zep v4 API gives each user, thread, and graph a UUID. The script stores these UUIDs in the run manifest and uses them for all later calls.

uv run zep_ingest_users.py [OPTIONS]

3. Ingest Documents (zep_ingest_documents.py)

Reads a chunk set and sends chunks to a standalone Zep document graph.

uv run zep_ingest_documents.py [OPTIONS]

4. Evaluate (zep_evaluate.py)

Searches graphs, generates LLM responses, grades against golden answers. Calculates aggregate, per-category, and per-user metrics.

uv run zep_evaluate.py [OPTIONS]

5. Inspect Graph (zep_graph_inspect.py)

Print all entities and facts for any graph.

uv run zep_graph_inspect.py [OPTIONS]

Run Artifacts (runs/)

Each pipeline step creates a numbered, timestamped directory with a config snapshot for reproducibility.

Follow Mode

Start document ingestion while chunking is still running — the ingestion script tails the in-progress chunk set JSONL and ingests chunks as they appear:

# Terminal 1
uv run zep_chunk_documents.py --concurrency 10
 
# Terminal 2 (start immediately, even before chunking finishes)
uv run zep_ingest_documents.py --chunk-set 1 --custom-ontology --custom-instructions

The ingestion script detects "status": "in_progress" in meta.json and polls for new chunks.

Concurrency Guidance

Prefer higher concurrency values to maximize throughput:

• Chunking: --concurrency 10 to 20 (default 5 is conservative)
• Evaluation: --concurrency 30 or higher (default 15 is conservative)

All scripts include retry logic with exponential backoff (8 retries, max 5-minute delay) that handles rate limits automatically. If rate limit errors persist, reduce concurrency and retry.

Analyzing Results

Evaluation results live at runs/evaluations/{N}_{ts}/results.json. Each result references its parent user and document runs via the parent_runs field. When asked to analyze or compare runs:

1. Read the aggregate_scores, category_scores, and user_scores from each results file.
2. Present comparisons as a table — rows are runs/configs, columns are metrics.
3. Draw conclusions about which configurations perform better and with what level of certainty (e.g., small test sets = low confidence, large differences = higher signal).

Which Metrics Matter

Context completeness is the primary metric. It measures whether Zep retrieved sufficient information to answer the question — this directly evaluates Zep's retrieval quality. A COMPLETE rate of 100% means Zep always surfaces the right facts.