ai-evals-course

ai-evals-course/cartwheel-homeworks

Build a custom browser-based annotation interface tailored to your data for reviewing LLM traces and collecting structured feedbac...

Create diverse synthetic test inputs for LLM pipeline evaluation using dimension-based tuple generation. Use when bootstrapping an...

Calibrate an LLM judge against human labels using data splits, TPR/TNR, and bias correction. Use after writing a judge prompt (wri...

Design LLM-as-Judge evaluators for subjective criteria that code-based checks cannot handle. Use when a failure mode requires inte...

ai-evals-course/synthetic-scenarios-skill