llm-d/llm-d-benchmark/convert-guide

Convert llm-d Guide to Benchmark Files

Purpose

Convert llm-d deployment guides (Helm values files) into llm-d-benchmark scenario files (and optionally experiment files for testing). The converter:

• Extracts configuration from Helm values YAML or kustomize patches
• Maps values to LLMDBENCH_* environment variables
• Generates scenario files (shell scripts with exports) - always created
• Generates experiment files (YAML for parameter sweeps) - only when testing specs provided

Usage

/convert-guide <url-or-path>
/convert-guide <url-or-path> with <harness> <profile>
/convert-guide <url-or-path> varying <parameter> from <start> to <end>

Defaults

• Default Harness: inference-perf
• Default Profile: sanity_random.yaml

See references/harnesses.md for all available harnesses and profiles.

Examples

# Convert a guide
/convert-guide https://github.com/llm-d/llm-d/tree/main/guides/optimized-baseline
 
# Specify custom harness and profile
/convert-guide https://github.com/llm-d/llm-d/tree/main/guides/pd-disaggregation with inference-perf shared_prefix_synthetic
 
# Convert with accelerator variant
/convert-guide https://github.com/llm-d/llm-d/tree/main/guides/pd-disaggregation xpu
 
# Add parameter variations (creates scenario + experiment files)
/convert-guide ~/guides/config.yaml varying decode_replicas from 1 to 4

Workflow

Step 1: Parse the Input

Extract from the user's command:

• Source: URL or local file path
• Harness: Load generator (default: inference-perf)
• Profile: Workload profile (default: sanity_random.yaml)
• Testing Specifications: Check for variations, treatments, or test parameters

IMPORTANT: If NO testing specifications are provided, create ONLY a scenario file. Do NOT create an experiment file unless the user explicitly specifies what to test or vary.

Step 2: Read Reference Files

Read these reference files to understand the mapping rules and defaults:

1. Mapping Rules: references/mappings.md (bundled with this skill)

• Contains Helm path to LLMDBENCH variable mappings
• Notes on transformations and special handling

1. Default Values: setup/env.sh (codebase file - read at runtime)

• Current default values for all LLMDBENCH variables
• Only override values that differ from defaults

Note: Do NOT read files from scenarios/guides/ or experiments/ as examples - these are generated outputs.

Step 3: Fetch Guide Contents

Detect the Configuration Approach:

1. Look for kustomization.yaml files in the guide directory
2. If found, this is a kustomize-based guide
3. If only values.yaml files are found, this is a Helm values-based guide

TRACK SOURCE FILES: Record file paths AND line numbers for inclusion in scenario file documentation.

For Helm Values-Based Guides:

• Fetch ms-*/values.yaml (ModelService) and gaie-*/values.yaml (GAIE InferenceExtension)
• Fetch helmfile.yaml.gotmpl or helmfile.yaml if present - contains Helm chart versions
• For GitHub URLs, first fetch directory listing to identify subdirectories

For Kustomize-Based Guides:

• Read main kustomization.yaml and follow resources: references to find base files
• Extract configuration from patches (op: replace, op: add)
• Document the full kustomize hierarchy
• See references/templates.md for kustomize patch mappings

Step 4: Extract Configuration

Parse the YAML and extract key configuration sections. See references/mappings.md for complete field mappings.

From ModelService (`ms-*/values.yaml`):

• Model configuration (modelArtifacts.*)
• Container images (if explicitly specified)
• Decode/Prefill stage settings (replicas, parallelism, args, env, volumes)
• vLLM launch arguments

From GAIE (`gaie-*/values.yaml`):

• InferenceExtension settings (image, flags, plugins)
• CRITICAL: If pluginsCustomConfig exists, extract the ENTIRE YAML content
• InferencePool settings (ports, labels)

From Helmfile (`helmfile.yaml.gotmpl` or `helmfile.yaml`):

• Chart versions from releases[].version for each release:
• llm-d-infra release → LLMDBENCH_VLLM_INFRA_CHART_VERSION
• llm-d-modelservice release → LLMDBENCH_VLLM_MODELSERVICE_CHART_VERSION
• inferencepool (GAIE) release → LLMDBENCH_VLLM_GAIE_CHART_VERSION

Step 5: Map to LLMDBENCH Variables

Using references/mappings.md:

1. Find the corresponding LLMDBENCH_* variable for each extracted value
2. Compare against defaults from setup/env.sh
3. Only include variables that differ from defaults
4. Note any unmappable values in comments

Step 6: Apply llm-d-benchmark Standard Practices

CRITICAL: Always apply these practices regardless of what the guide specifies:

1. Always use custom model command:

``bash export LLMDBENCH_VLLM_MODELSERVICE_DECODE_MODEL_COMMAND=custom ``

1. Always include preprocess in EXTRA_ARGS:

```bash export LLMDBENCH_VLLM_COMMON_PREPROCESS="python3 /setup/preprocess/set_llmdbench_environment.py; source \$HOME/llmdbench_env.sh" export LLMDBENCH_VLLM_MODELSERVICE_DECODE_PREPROCESS=$LLMDBENCH_VLLM_COMMON_PREPROCESS

export LLMDBENCH_VLLM_MODELSERVICE_DECODE_EXTRA_ARGS=$(mktemp) cat << EOF > $LLMDBENCH_VLLM_MODELSERVICE_DECODE_EXTRA_ARGS REPLACE_ENV_LLMDBENCH_VLLM_MODELSERVICE_DECODE_PREPROCESS; \ vllm serve /model-cache/models/REPLACE_ENV_LLMDBENCH_DEPLOY_CURRENT_MODEL \ --host 0.0.0.0 \ --served-model-name REPLACE_ENV_LLMDBENCH_DEPLOY_CURRENT_MODEL \ --port REPLACE_ENV_LLMDBENCH_VLLM_COMMON_METRICS_PORT \ <...additional vllm flags from guide...> EOF ```

1. Always include preprocesses volume mount:

```bash

• name: preprocesses

mountPath: /setup/preprocess ```

1. Always include preprocesses configMap volume (FIRST in list):

```bash

• name: preprocesses

configMap: defaultMode: 0755 name: llm-d-benchmark-preprocesses ```

1. Always use REPLACE_ENV placeholders in heredoc blocks:

Inside EXTRA_ARGS, EXTRA_VOLUMES, and EXTRA_VOLUME_MOUNTS heredocs, any value that has a corresponding REPLACE_ENV_* placeholder (see references/mappings.md > Common REPLACE_ENV Placeholders) MUST use the placeholder, never a hardcoded literal. This enables experiment-level overrides and parameter sweeps.

Examples of values that MUST use placeholders:

• --port → REPLACE_ENV_LLMDBENCH_VLLM_COMMON_METRICS_PORT
• --max-model-len → REPLACE_ENV_LLMDBENCH_VLLM_COMMON_MAX_MODEL_LEN
• --block-size → REPLACE_ENV_LLMDBENCH_VLLM_COMMON_BLOCK_SIZE
• --gpu-memory-utilization → REPLACE_ENV_LLMDBENCH_VLLM_MODELSERVICE_{STAGE}_ACCELERATOR_MEM_UTIL (or REPLACE_ENV_LLMDBENCH_VLLM_COMMON_ACCELERATOR_MEM_UTIL when common)
• --tensor-parallel-size → REPLACE_ENV_LLMDBENCH_VLLM_MODELSERVICE_{STAGE}_TENSOR_PARALLELISM
• sizeLimit: in dshm volumes → REPLACE_ENV_LLMDBENCH_VLLM_COMMON_SHM_MEM (or stage-specific _DECODE_SHM_MEM/_PREFILL_SHM_MEM)

The only exception is values with NO corresponding LLMDBENCH variable (e.g., --kv-transfer-config JSON, --enable-prefix-caching boolean flags, custom env var names/values). These stay as literals.

See references/patterns.md for complete patterns including volumes, environment variables, GAIE plugins, and accelerator-specific configurations.

Step 7: Generate Output Files

Scenario File: Always created. Use template from references/templates.md.

Every environment variable MUST include source documentation:

# =============================================================================
# SOURCE: <relative-path-to-source-file>
# Lines <line-numbers>:
#   <original YAML/JSON content from source>
# =============================================================================
export LLMDBENCH_<VARIABLE_NAME>=<value>

Experiment File: Only created if user specified testing requirements. Use template from references/templates.md.

Step 8: Validate Standard Practices and Documentation

Before displaying or writing files, verify:

Standard Practices Checklist:

• LLMDBENCH_VLLM_MODELSERVICE_DECODE_MODEL_COMMAND=custom is set
• LLMDBENCH_VLLM_COMMON_PREPROCESS is defined
• LLMDBENCH_VLLM_MODELSERVICE_DECODE_PREPROCESS is set
• DECODE_EXTRA_ARGS starts with REPLACE_ENV_LLMDBENCH_VLLM_MODELSERVICE_DECODE_PREPROCESS; \
• DECODE_EXTRA_ARGS contains vllm serve /model-cache/models/REPLACE_ENV_LLMDBENCH_DEPLOY_CURRENT_MODEL \
• Volume mounts include preprocesses mount at /setup/preprocess
• Volumes include preprocesses configMap (should be first in list)
• If guide has inferenceExtension.pluginsCustomConfig, then LLMDBENCH_VLLM_MODELSERVICE_GAIE_CUSTOM_PLUGINS is defined
• If guide specifies explicit container image version, then LLMDBENCH_LLMD_IMAGE_TAG is set
• If guide uses LeaderWorkerSet (LWS), then LLMDBENCH_VLLM_MODELSERVICE_MULTINODE=true is set
• If helmfile specifies llm-d-infra version, then LLMDBENCH_VLLM_INFRA_CHART_VERSION is set
• If helmfile specifies llm-d-modelservice version, then LLMDBENCH_VLLM_MODELSERVICE_CHART_VERSION is set
• If helmfile specifies inferencepool (GAIE) version, then LLMDBENCH_VLLM_GAIE_CHART_VERSION is set
• Inside all heredoc blocks (EXTRA_ARGS, EXTRA_VOLUMES, EXTRA_VOLUME_MOUNTS), no literal values appear where a REPLACE_ENV_* placeholder exists in the references/mappings.md table

Completeness Checklist (MANDATORY - never omit source configuration):

• Environment variables: If guide defines env: in decode/prefill containers, ALL env vars MUST be captured in LLMDBENCH_VLLM_MODELSERVICE_DECODE_ENVVARS_TO_YAML or LLMDBENCH_VLLM_MODELSERVICE_PREFILL_ENVVARS_TO_YAML
• Volume mounts: If guide defines volumeMounts:, ALL mounts MUST be captured in EXTRA_VOLUME_MOUNTS
• Volumes: If guide defines volumes:, ALL volumes MUST be captured in EXTRA_VOLUMES
• vLLM args: If guide defines args:, ALL args MUST be captured in EXTRA_ARGS
• Container config: If guide defines resources, securityContext, or other container config, it MUST be captured

CRITICAL: The scenario file must be a complete representation of the guide. Configuration from the source guide should NEVER be silently dropped or omitted. If a value cannot be mapped, document it in a comment explaining why.

Source Documentation Checklist:

• Every environment variable has a # SOURCE: comment block
• SOURCE blocks include file path and line numbers
• SOURCE blocks include original content
• Framework additions marked as "Benchmark framework convention (not in guide)"

Step 9: Write Files and Report

CRITICAL: You MUST call the Write tool. Do NOT skip to success message without writing.

File Paths (use `ai.` prefix to indicate Claude-generated):

• Scenario files: scenarios/guides/ai.<guide_name>.sh
• Experiment files: experiments/ai.<guide_name>.yaml

ONLY after Write tool succeeds, report a summary including:

• File path(s) written
• Source configuration files used
• Key configuration highlights (model, replicas, notable vLLM flags, etc.)
• Usage instructions

Error Handling

1. Invalid URL: Report error, suggest checking the URL
2. Missing File: Report clearly if local path doesn't exist
3. Invalid YAML: Show error and problematic section
4. Unknown Helm Values: Log unmapped values as comments
5. Missing Required Values: Warn if critical values (like model name) are missing

Deriving Guide Name