contextjet-ai/awesome-llm-observability/validate-genai-spans

Validate GenAI spans

Broken instrumentation fails silently: the app works, the dashboards just quietly miss the model, the tokens, or the cost. This skill ships a validator that asserts your spans carry the required OpenTelemetry gen_ai.* fields, so you catch gaps in a test instead of in a half-empty dashboard.

Use the bundled script

`scripts/validate_span.py` is pure Python, no install needed:

from validate_span import validate_gen_ai_span, is_valid_gen_ai_span
problems = validate_gen_ai_span(span_attributes)          # [] means valid
validate_gen_ai_span(span_attributes, strict=True)        # also flag recommended fields
assert is_valid_gen_ai_span(span_attributes)              # use in an instrumentation test

It checks the required fields (gen_ai.provider.name, gen_ai.operation.name, gen_ai.request.model) and, in strict mode, the recommended usage fields (gen_ai.usage.input_tokens/output_tokens, gen_ai.response.model) that cost analysis depends on. Spans still using the pre-2025 gen_ai.system name pass but get a deprecation note, so you can migrate at your own pace.

How to apply it

1. Add an instrumentation test: capture the span your app emits for a sample call (most SDKs have an in-memory span exporter for tests) and assert is_valid_gen_ai_span(attrs).
2. Run it in CI so a change that breaks instrumentation fails the build, not production.
3. Use strict mode once basic spans pass, to push toward full cost/usage coverage.

Pairs with instrument-llm-observability (which gets the spans emitted in the first place).

Validation

Run the tests: pytest skills/validate-genai-spans/tests/. They cover a valid span, each missing required field, empty/wrong-typed values, the deprecated gen_ai.system alias, strict-mode recommended checks, and non-dict input.

Anti-patterns

• Assuming "traces show up" means they are complete (missing token fields break cost dashboards).
• Only checking spans by eye once, instead of asserting them in CI.
• Ignoring the response model id (you cannot break cost down by served model without it).