opendcai/dataflow-webui/bench-dataset-evaluator-question

Claude Code and agent skill by opendcai/dataflow-webui

Trust tier 3
  • community
Source check pending

Checking that the SKILL.md file is still in the repository.

Reference documentation for the BenchDatasetEvaluatorQuestion operator. Extended version of BenchDatasetEvaluator with question and subquestion support. Use when: evaluating answers with question context or multiple subquestions.

Install

npx vskill@latest install opendcai/dataflow-webui/bench-dataset-evaluator-question

SKILL.md

BenchDatasetEvaluatorQuestion Operator Reference

BenchDatasetEvaluatorQuestion extends BenchDatasetEvaluator with support for question context and subquestions.

1. Import

python
from dataflow.operators.core_text import BenchDatasetEvaluatorQuestion

2. Match Mode

Constructor
python
BenchDatasetEvaluatorQuestion(
    eval_result_path=None,
    compare_method="match",
)
ParameterRequiredDefaultDescription
eval_result_pathNoAuto-generatedPath to save evaluation statistics
compare_methodNo"match"Must be "match"
system_promptNo"You are a helpful assistant..."Not used in match mode
llm_servingNoNoneNot used in match mode
prompt_templateNoAnswerJudgePromptQuestionNot used in match mode
run() Signature
python
op.run(
    storage=self.storage.step(),
    input_question_key="question",
    input_test_answer_key="generated_cot",
    input_gt_answer_key="golden_answer",
)
# returns: list of column names
ParameterRequiredDefaultDescription
storageYesNoneStorage step object
input_question_keyNo"question"Question column
input_test_answer_keyNo"generated_cot"Predicted answer column
input_gt_answer_keyNo"golden_answer"Ground truth column
Usage Example
python
from dataflow.operators.core_text import BenchDatasetEvaluatorQuestion
from dataflow.utils.storage import FileStorage
class MyPipeline:
    def __init__(self):
        self.storage = FileStorage(
            first_entry_file_name="./data/bench.jsonl",
            cache_path="./cache",
            file_name_prefix="step",
            cache_type="jsonl"
        )
        self.evaluator = BenchDatasetEvaluatorQuestion(
            compare_method="match",
            eval_result_path="./results/match_eval.json"
        )
    def forward(self):
        self.evaluator.run(
            storage=self.storage.step(),
            input_question_key="question",
            input_test_answer_key="predicted_answer",
            input_gt_answer_key="ground_truth"
        )
if __name__ == "__main__":
    pipeline = MyPipeline()
    pipeline.forward()

3. Semantic Mode

Constructor
python
BenchDatasetEvaluatorQuestion(
    eval_result_path=None,
    compare_method="semantic",
    system_prompt="You are a helpful assistant specialized in evaluating answer correctness.",
    llm_serving=llm_serving,
    prompt_template=None,
    support_subquestions=False,
)
ParameterRequiredDefaultDescription
eval_result_pathNoAuto-generatedPath to save evaluation statistics
compare_methodYesNoneMust be "semantic"
system_promptNo"You are..."System prompt for LLM
llm_servingYesNoneLLM service object
prompt_templateNoAnswerJudgePromptQuestionPass None to use built-in fallback
support_subquestionsNoFalseEnable subquestion evaluation
run() Signature
python
op.run(
    storage=self.storage.step(),
    input_question_key="question",
    input_test_answer_key="generated_cot",
    input_gt_answer_key="golden_answer",
)
# returns: list of column names
ParameterRequiredDefaultDescription
storageYesNoneStorage step object
input_question_keyNo"question"Question column
input_test_answer_keyNo"generated_cot"Predicted answer column
input_gt_answer_keyNo"golden_answer"Ground truth column
Usage Example
python
from dataflow.operators.core_text import BenchDatasetEvaluatorQuestion
from dataflow.serving import APILLMServing_request
from dataflow.utils.storage import FileStorage
class MyPipeline:
    def __init__(self):
        self.storage = FileStorage(
            first_entry_file_name="./data/bench.jsonl",
            cache_path="./cache",
            file_name_prefix="step",
            cache_type="jsonl"
        )
        self.llm_serving = APILLMServing_request(
            api_url="https://api.openai.com/v1/chat/completions",
            key_name_of_api_key="DF_API_KEY",
            model_name="gpt-4o",
            max_workers=10
        )
        self.evaluator = BenchDatasetEvaluatorQuestion(
            compare_method="semantic",
            llm_serving=self.llm_serving,
            prompt_template=None,
            support_subquestions=False
        )
    def forward(self):
        self.evaluator.run(
            storage=self.storage.step(),
            input_question_key="question",
            input_test_answer_key="predicted_answer",
            input_gt_answer_key="ground_truth"
        )
if __name__ == "__main__":
    pipeline = MyPipeline()
    pipeline.forward()

4. Key Differences from BenchDatasetEvaluator

  1. Question context: Semantic mode includes question field in the prompt alongside answer and reference_answer.
  2. Subquestions support: When support_subquestions=True, evaluates multiple subquestions per row.
  3. Prompt templates: Uses AnswerJudgePromptQuestion (single question) or AnswerJudgeMultipleQuestionsPrompt (subquestions).

5. AnswerJudgePromptQuestion

AnswerJudgePromptQuestion is the default prompt template class for semantic mode.

Important Notes on prompt_template

Although the source code sets the default value to AnswerJudgePromptQuestion, this default is a class object, not an instance.

In normal usage, it's recommended to use one of these two approaches:

Option 1: Pass `None` (recommended)

python
prompt_template=None

This uses the built-in fallback logic.

Option 2: Pass an instance

python
from dataflow.prompts.core_text import AnswerJudgePromptQuestion
prompt_template=AnswerJudgePromptQuestion()
Fields Passed to build_prompt(...)

When prompt_template is an AnswerJudgePromptQuestion instance, the source code passes these fields:

  • question: The question being evaluated
  • answer: The predicted answer to evaluate
  • reference_answer: The ground truth answer
Expected LLM Response Format

LLM response must contain:

json
{
  "judgement_result": true  // or false
}

Security

  • Security scanPassed, 90 of 100Pattern checks on the skill's files.
  • Model reviewPassed

See the full security report

Quality evaluation

  • Task evaluationNot runNo evaluation yet. Run one in Skill Studio.

See evaluation results

Source

Popularity

7-day trend score
0

Not refreshed yet.

Works with

Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.

  • Claude Code
  • Cursor
  • GitHub Copilot
  • Windsurf
  • Codex
  • Gemini CLI
  • Cline
  • Amp
  • Roo Code
  • Goose
  • Continue
  • Kilo Code
  • Junie
  • Trae
  • OpenHands
  • Qwen Code
  • AI Agent
  • Tabnine Chat
  • Sourcegraph Cody
  • Amazon Q
  • Firebase Genkit
  • Sweep
  • Devin
  • Phind
  • Mentat
  • Claude Desktop
  • Aider
  • GPT Pilot
  • Plandex
  • bolt.new
  • v0
  • Replit Agent
  • Anse
  • TypingMind
  • BetterChatGPT
  • LibreChat
  • Warp AI
  • Pieces
  • Fig AI
Embed badge
vskill VERIFIED badge
[![vskill VERIFIED](https://verified-skill.com/api/v1/skills/opendcai/dataflow-webui/bench-dataset-evaluator-question/badge)](https://verified-skill.com/skills/opendcai/dataflow-webui/bench-dataset-evaluator-question)

Something wrong with this skill or its page? Tell us.