castform-ai/benchmax/generate-data

Generate or reference data

Data preparation is a Python workflow. Keep it in main.py's generate_data stage or a nearby project script so its inputs, transformations and outputs are reviewable. There is no separate data or corpus orchestration CLI.

BaseEnv JSONL path

The generated BaseEnv seed reads train.jsonl and eval.jsonl. Each line is a JSON object containing the fields the environment's row converter and reward actually use:

{"prompt": "What is 2 + 2?", "ground_truth": "4"}

• Keep train and eval disjoint.
• Build stable example identity from semantic payload content, not row position or

machine-local paths.

• Start small enough to inspect manually, then expand after validation shows a

meaningful signal.

• Keep large integer identifiers as strings when data will cross JSON/JavaScript

boundaries.

• Record provenance and make regeneration idempotent; never overwrite curated data

without an explicit force flag.

upload_assets accepts optional train and eval rows. The launch script decides what it uploads: omit a split when the environment resolves it at runtime, and do not upload unrelated preparation artifacts. None means “do not upload”; an empty list deliberately uploads an empty JSONL.

<!-- rag:start -->

Hosted corpus and RAG

Install castform[rag] in the project and use public modules under castform.rag from the data stage. Before implementing the workflow, inspect the matching maintained example:

• neon_rag: https://github.com/castform-ai/benchmax/tree/main/examples/neon_rag
• turbopuffer_rag: https://github.com/castform-ai/benchmax/tree/main/examples/turbopuffer_rag
• chroma_rag: https://github.com/castform-ai/benchmax/tree/main/examples/chroma_rag
• pinecone_rag: https://github.com/castform-ai/benchmax/tree/main/examples/pinecone_rag

Use its README.md, main.py, data.py, environment.py, and search.py as the reference for the provider. Typical data code composes:

• castform.rag.chunkers to turn source files into chunks;
• castform.rag.corpus.postgres.client.CorpusClient to create/find a corpus and

upload chunks;

• castform.rag.qa_generation to build grounded QA rows;
• the provider's example-local search adapter for runtime reads.

Read the concrete class signatures before wiring them; these are library components, not one magical pipeline command. Persist generated rows as ordinary project data and test at least one known retrieval query before validation.

Replace the generic seed environment and rows with the selected RAG example's structure. Confirm that each row contains question, answer, and reference_chunks, and that every reference chunk carries the source metadata expected by the citation reward. <!-- rag:end -->

Harbor-managed datasets

A Harbor dataset may be a local directory, Harbor package, registry reference or Git repository resolved by HarborEnv during runtime. Do not duplicate that data into JSONL merely to match the BaseEnv example. Only add an upload step when the chosen Harbor workflow genuinely needs a folder or artifact uploaded.

Traces

Normalize provider traces with the adapter for that provider and pass them through castform.traces.TracesPipeline, which ships with the base castform package. Keep the resulting train/eval split and detected prompt/tool assumptions visible in the project. Inspect for secrets, relayed tool output, duplicates and trivial examples before using traces as training data.

Verification

Before handing off to verify-environment:

1. validate the output schema with the environment's row converter;
2. check stable IDs and train/eval overlap;
3. inspect representative easy, hard and malformed rows;
4. confirm any corpus or Git reference is reachable in the rollout runtime;
5. run the deterministic reward against known answers where possible.