opendcai/dataflow-webui/unified-bench-dataset-evaluator

Claude Code and agent skill by opendcai/dataflow-webui

Trust tier 3
  • community
  • ext:[{"type":"template","description":"Custom templates for code generation"}]
Source check pending

Checking that the SKILL.md file is still in the repository.

Reference documentation for the UnifiedBenchDatasetEvaluator operator. Use when: evaluating model answers on benchmark datasets.

Install

npx vskill@latest install opendcai/dataflow-webui/unified-bench-dataset-evaluator

Semi-extensibleMentions customization, but needs skill-specific knowledge. Read the extensible skills standard

SKILL.md

UnifiedBenchDatasetEvaluator Operator Reference

UnifiedBenchDatasetEvaluator supports 6 evaluation types (eval_type), scores generated answers, and writes 4 output columns.

1. Import

python
from dataflow.operators.core_text import UnifiedBenchDatasetEvaluator

2. Constructor

python
UnifiedBenchDatasetEvaluator(
    eval_type="key2_qa",
    llm_serving=None,
    prompt_template=None,
    eval_result_path=None,
    metric_type=None,
    use_semantic_judge=False,
    system_prompt="You are a helpful assistant specialized in evaluating answer correctness.",
)

IMPORTANT: Always pass prompt_template=None explicitly. The default value is AnswerJudgePrompt (the class itself), which triggers a TypeError.

ParameterRequiredDefaultDescription
eval_typeNo"key2_qa"Evaluation type
llm_servingConditionalNoneRequired when use_semantic_judge=True
prompt_templateNoAnswerJudgePromptPass None to use built-in fallback
eval_result_pathNoAuto-generatedStatistics JSON file path
metric_typeNoNoneEvaluation metric, auto-selected if not provided
use_semantic_judgeNoFalseUse LLM for semantic judgment
system_promptNo"You are a helpful assistant..."System prompt for LLM (used when use_semantic_judge=True)
eval_type and Required input_xxx_key
eval_typeRequired input_xxx_key
key1_text_scoreinput_text_key
key2_qainput_question_key, input_target_key
key2_q_mainput_question_key, input_targets_key
key3_q_choices_ainput_question_key, input_choices_key, input_label_key
key3_q_choices_asinput_question_key, input_choices_key, input_labels_key
key3_q_a_rejectedinput_question_key, input_better_key, input_rejected_key

3. run() Signature

python
op.run(
    storage=self.storage.step(),
    input_question_key="question",
    input_target_key="golden_answer",
    input_pred_key="generated_ans",
)
# returns: list of column names
ParameterRequiredDefaultDescription
storageYesNoneStorage step object
input_pred_keyNo"generated_ans"Model generated answer column
input_question_keyConditionalNoneQuestion column
input_target_keyConditionalNoneSingle target answer column
input_targets_keyConditionalNoneMultiple target answers column
input_choices_keyConditionalNoneChoices column
input_label_keyConditionalNoneSingle label column
input_labels_keyConditionalNoneMultiple labels column
input_better_keyConditionalNonePreferred answer column
input_rejected_keyConditionalNoneRejected answer column
input_context_keyNoNoneOptional context column for additional information

4. Output Columns (4 columns)

  • eval_valid: Boolean column indicating if evaluation is valid
  • eval_error: Error message column
  • eval_pred: Parsed prediction column
  • eval_score: Numeric score column

5. Usage Example

python
from dataflow.operators.core_text import UnifiedBenchDatasetEvaluator
from dataflow.serving import APILLMServing_request
from dataflow.utils.storage import FileStorage
class MyPipeline:
    def __init__(self):
        self.storage = FileStorage(
            first_entry_file_name="./data/bench.jsonl",
            cache_path="./cache",
            file_name_prefix="step",
            cache_type="jsonl"
        )
        self.llm_serving = APILLMServing_request(
            api_url="https://api.openai.com/v1/chat/completions",
            key_name_of_api_key="DF_API_KEY",
            model_name="gpt-4o",
            max_workers=10
        )
        self.evaluator = UnifiedBenchDatasetEvaluator(
            eval_type="key2_qa",
            llm_serving=self.llm_serving,
            prompt_template=None,
            use_semantic_judge=True
        )
    def forward(self):
        self.evaluator.run(
            storage=self.storage.step(),
            input_question_key="question",
            input_target_key="golden_answer",
            input_pred_key="generated_ans"
        )
if __name__ == "__main__":
    pipeline = MyPipeline()
    pipeline.forward()

6. Important Notes

  • eval_type must match BenchAnswerGenerator's eval_type
  • Statistics saved to eval_result_path JSON file
  • use_semantic_judge=True requires llm_serving

5. Runtime Logic

  1. Read DataFrame from storage.
  2. Validate required input_xxx_key columns exist based on eval_type.
  3. Create 4 output columns: eval_valid, eval_error, eval_pred, eval_score.
  4. For each row:
  • Extract prediction from input_pred_key
  • Extract ground truth based on eval_type
  • If use_semantic_judge=True: call LLM to judge correctness
  • If use_semantic_judge=False: use rule-based comparison
  • Parse and score the result
  1. Write results to 4 output columns.
  2. Save statistics to eval_result_path JSON file.
  3. Return list of column names.

6. prompt_template Usage

Important Notes
  • Default value is AnswerJudgePrompt (class, not instance) → causes TypeError
  • Always pass `prompt_template=None` to use built-in fallback
  • Only used when use_semantic_judge=True
Recommended Usage
python
# Recommended: pass None
evaluator = UnifiedBenchDatasetEvaluator(
    eval_type="key2_qa",
    llm_serving=llm_serving,
    prompt_template=None,
    use_semantic_judge=True
)
Custom Template Usage
python
from dataflow.prompts.core_text import AnswerJudgePrompt
# Pass an instance for custom prompts
custom_prompt = AnswerJudgePrompt()
evaluator = UnifiedBenchDatasetEvaluator(
    eval_type="key2_qa",
    llm_serving=llm_serving,
    prompt_template=custom_prompt,
    use_semantic_judge=True
)
Fields Passed to build_prompt(...)

When using custom AnswerJudgePrompt instance, the operator passes:

  • answer: The predicted answer
  • reference_answer: The ground truth answer

LLM response must contain:

json
{
  "judgement_result": true  // or false
}

Security

  • Security scanPassed, 90 of 100Pattern checks on the skill's files.
  • Model reviewPassed

See the full security report

Quality evaluation

  • Task evaluationNot runNo evaluation yet. Run one in Skill Studio.

See evaluation results

Source

Popularity

7-day trend score
0

Not refreshed yet.

Extension points

  • templateCustom templates for code generation

Learn the extensible skills standard

Works with

Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.

  • Claude Code
  • Cursor
  • GitHub Copilot
  • Windsurf
  • Codex
  • Gemini CLI
  • Cline
  • Amp
  • Roo Code
  • Goose
  • Continue
  • Kilo Code
  • Junie
  • Trae
  • OpenHands
  • Qwen Code
  • AI Agent
  • Tabnine Chat
  • Sourcegraph Cody
  • Amazon Q
  • Firebase Genkit
  • Sweep
  • Devin
  • Phind
  • Mentat
  • Claude Desktop
  • Aider
  • GPT Pilot
  • Plandex
  • bolt.new
  • v0
  • Replit Agent
  • Anse
  • TypingMind
  • BetterChatGPT
  • LibreChat
  • Warp AI
  • Pieces
  • Fig AI
Embed badge
vskill VERIFIED badge
[![vskill VERIFIED](https://verified-skill.com/api/v1/skills/opendcai/dataflow-webui/unified-bench-dataset-evaluator/badge)](https://verified-skill.com/skills/opendcai/dataflow-webui/unified-bench-dataset-evaluator)

Something wrong with this skill or its page? Tell us.