openvinotoolkit/openvino.genai/llm-bench-fail-analyzer

LLM Bench Fail Analyzer

Analyzes failed llm bench results for models, which were run with transformers, optimum-intel or GenAI backends; Provides fixes or insights for troubleshooting.

When to Use

• llm_bench fails during execution pipeline with the model

Inputs

The user must provide:

• model_dir: path to the directory with the model (e.g. /path/to/tencent_HY-MT1.5-1.8B)
• log_info: path to the folder containing the failed llm_bench log files or llm_bench log file or execution output

Code Structure Reference

When analyzing failures and implementing fixes, refer to the following key locations in the codebase:

Main benchmark script:

• tools/llm_bench/benchmark.py - Entry point for benchmarking, command-line argument parsing, and task orchestration

Model execution pipeline implementations:

• tools/llm_bench/task/text_generation.py - Text generation benchmarking for LLMs
• tools/llm_bench/task/image_generation.py - Image generation(text to image, image to image, inpainting) benchmarking
• tools/llm_bench/task/visual_language_generation.py - VLM benchmarking for multimodal models
• tools/llm_bench/task/video_generation.py - Video generation benchmarking
• tools/llm_bench/task/super_resolution_generation.py - Super resolution benchmarking
• tools/llm_bench/task/speech_to_text_generation.py - Speech-to-text (ASR) benchmarking
• tools/llm_bench/task/text_to_speech_generation.py - Text-to-speech (TTS) benchmarking
• tools/llm_bench/task/text_embeddings.py - Text embedding model benchmarking
• tools/llm_bench/task/text_reranker.py - Text reranking model benchmarking
• tools/llm_bench/task/pipeline_utils.py - Common pipeline utilities and base classes for all tasks

Core utilities:

• tools/llm_bench/llm_bench_utils/config_class.py - Configuration classes, model class definitions, attention backend settings
• tools/llm_bench/llm_bench_utils/model_utils.py - Model utility functions: parameter loading, config parsing, precision handling
• tools/llm_bench/llm_bench_utils/ov_utils.py - OpenVINO model creation and management (GenAI, optimum-intel)
• tools/llm_bench/llm_bench_utils/pt_utils.py - PyTorch model creation and torch.compile support
• tools/llm_bench/llm_bench_utils/ov_model_classes.py - Custom OpenVINO model classes (OVMPTModel, OVChatGLMModel, etc.)
• tools/llm_bench/llm_bench_utils/prompt_utils.py - Prompt loading, preprocessing for text/image/video inputs
• tools/llm_bench/llm_bench_utils/parse_json_data.py - JSON data parsing for prompts and configurations
• tools/llm_bench/llm_bench_utils/get_use_case.py - Use case detection and configuration

Model wrappers for performance measurement with transformers/optimum-intel:

• tools/llm_bench/llm_bench_utils/hook_forward.py - hooks for image generation, RAG and TTS pipelines
• tools/llm_bench/llm_bench_utils/hook_greedy_search.py - Greedy sampling hooks
• tools/llm_bench/llm_bench_utils/hook_beam_search.py - Beam search sampling hooks
• tools/llm_bench/llm_bench_utils/hook_common.py - determination of the required hook
• tools/llm_bench/llm_bench_utils/hook_forward_whisper.py - ASR forward hook
• tools/llm_bench/llm_bench_utils/llm_hook_sample/*.py - version-specific greedy hook implementations for different transformers versions (v4_43, v4_45, v4_51, v4_52, v4_55, v4_57, v5, v5_3)
• tools/llm_bench/llm_bench_utils/llm_hook_beam_search/*.py - version-specific beam search hook implementations for different transformers versions (v4_43, v4_45, v4_51, v4_52, v4_55, v4_57, v5, v5_3)

Output and reporting:

• tools/llm_bench/llm_bench_utils/metrics_print.py - Metrics printing and logging to console
• tools/llm_bench/llm_bench_utils/output_json.py - JSON output generation
• tools/llm_bench/llm_bench_utils/output_csv.py - CSV output generation
• tools/llm_bench/llm_bench_utils/output_file.py - construct file name and save output to file
• tools/llm_bench/llm_bench_utils/gen_output_data.py - convert output data to dict format

Use this reference throughout Steps 1-3 when analyzing logs and identifying where to implement fixes.

Procedure

Step 1: Analyze the logs

If the logs for llm_bench don't contain failures, proceed to Step 2. Otherwise, follow the next steps:

• Read the corresponding log for the full traceback and context.
• Analyze the failure root cause from the log.
• Define whether fail relates to llm_bench or backend/model.
• If it's a llm_bench bug/limitation, implement necessary fixes to llm_bench tool. Use the Code Structure Reference to locate the exact functions to modify. Follow OpenVINO GenAI coding guidelines from .github/copilot-instructions.md. Ensure changes don't break existing functionality. Add appropriate error messages and logging. Test changes by re-running llm_bench tool with corresponding cmd parameters.
• If it's a model issue or backend limitation, provide description in the report.

Step 2: Report Results

Add the results for each llm_bench run to the logs. Results format:

Model Information

• Model dir: <model_dir>
• Task: <task>
• Status: PASSED / FAILED
• Log: <log_dir or log_file if provided>
• Issue: <description if failed>
• Error message: <short and relevant error messages if failed>
• Possible fix: <description of fix if failed>
• LLM Bench modification: modified / nothing changed
• After fix status: <PASSED / FAILED if failed>

Security

• NEVER install any packages. Assume the environment is pre-configured.
• NEVER modify model_dir — pass it exactly as provided by the user.