signalfx/obstudio/docs

Splunk Observability Studio Skills -- Product Requirements Document

Author: Tigran Najaryan, with AI skill specifications by Platform Engineering Status: Draft Version: 0.2

1. Executive Summary

Splunk Observability Studio is a developer tool that helps instrument (make observable) applications and services, verify the telemetry emitted by instrumentation, and iterate and improve instrumentation quickly.

The Studio is a free, open-source product, marketed through our OpenTelemetry channels. It is a no-strings-attached tool that can be used by developers using any Observability backend. On top of that, developers get additional capabilities when they use it with a supported observability backend.

Components

Business Goals

1. Brand image: Improve the project's reputation as a developer-friendly,

open-source-supporting company and increase brand awareness in the developer community.

1. Customer acquisition: Create a new funnel via optional Terraformer

functionality that leads users to O11y Cloud.

Brand image considerations always take precedence. The Studio must never feel like an ad platform.

Success KPIs

2. Architecture

Splunk Observability Studio follows a layered architecture where each layer is independently useful. See docs/architecture-proposal.md for the full design.

Layer 3: Distribution (VS Code extension, brew, npx, go install)
Layer 2: obstudio CLI (start, register, skill installer)
Layer 1: Core (Skills + Splunk Observability Studio)

Skills + Splunk Observability Studio Composition

Skills tell the agent what to do. Splunk Observability Studio tells the agent what happened. Together they form a closed loop:

1. Agent reads skill -- instruments the code
2. Developer runs the app
3. App sends OTLP to Splunk Observability Studio (localhost:4318)
4. Agent calls MCP tools -- inspects telemetry
5. Agent fixes issues -- go to step 2

Skill Folder Structure

skills/otel-observe/
  SKILL.md                      # Lean workflow: audit -> plan -> implement -> verify
  languages/
    node.md                     # Node.js-specific (loaded on-demand)
    python.md                   # Python-specific (loaded on-demand)
    go.md                       # Go-specific (loaded on-demand)
  references/                   # Shared reference material
    fault-domain-patterns.md
    signal-mapping-guide.md
    observability-template.md

Token-conscious design: the agent loads SKILL.md (~180 lines) plus only the one language file matching the detected codebase (~200 lines). Reference files are loaded only when the specific workflow step executes them.

MCP Tools (Splunk Observability Studio)

Endpoint: http://localhost:17900/mcp (Streamable HTTP, standard MCP).

3. Target Personas

Persona A: Application Developer

• Goal: Add observability without becoming an OTel expert.
• Typical prompt: "Add tracing to this Flask app", "Instrument my Express

API with OpenTelemetry".

• Success criteria: "It works on first run, I didn't have to read any docs."

Persona B: Platform Engineer

• Goal: Standardize instrumentation across 20+ services. Enforce semantic

conventions and metric naming.

• Typical prompt: "Instrument this service following our platform standards".
• Skill interaction: Configures via .observability.md to define expected

KPIs. Reviews output for compliance.

• Success criteria: "Every service produces the same signal names."

Persona C: SRE / On-Call Engineer

• Goal: Quickly add targeted instrumentation to diagnose a production issue.
• Typical prompt: "Add a span around the database retry logic".
• Success criteria: "I got the exact signal I needed in < 5 minutes."

4. Unified /otel-observe Skill

The Instrumenter is delivered as a single unified skill (skills/otel-observe/) that combines observability audit and OpenTelemetry implementation into one flow. The agent can run the full audit or skip directly to instrumentation.

4.1 Workflow

4.2 OOB vs Custom Decision Framework

flowchart TD
    A["Inspect codebase"] --> B{OTel SDK<br/>already initialized?}
    B -->|Yes| C["Extend existing setup"]
    B -->|No| D["Initialize SDK once"]
    C --> E{Auto-instrumentation<br/>library available?}
    D --> E
    E -->|Yes| F["Install OOB library"]
    E -->|No| G["Generate custom spans/metrics"]
    F --> H{OOB covers the SLI?}
    H -->|Yes| I["Done"]
    H -->|No| G
    G --> J["Apply semconv"] --> I

• OOB: use when a stable auto-instrumentation library exists (see

language files for per-language library maps)

• Custom: use when no OOB library exists, OOB coverage is too coarse,

business metrics are needed, or the signal is Category: Custom

4.3 Semantic Convention Enforcement

Span naming: {component}.{operation}, low-cardinality, stable semconv attribute keys.

Attribute naming: registered semconv attributes first; custom attributes follow {domain}.{noun}.{adjective} pattern; no high-cardinality values.

Metric naming: {namespace}.{noun}.{unit}; histograms for latency, counters for throughput, gauges for current state.

Error handling: set span status to ERROR, call recordException(), include error.type attribute on error metrics.

4.4 Trigger Rules

5. Non-Functional Requirements

5.1 Determinism

Structural elements must be identical across runs:

• SDK initialization location and pattern
• Auto-instrumentation library selection
• Semantic convention attribute keys
• OTLP exporter configuration

Stylistic variance is acceptable:

• Variable naming, import ordering
• Exact span placement within functions
• Async pattern choice

5.2 Evaluation Framework

Skills are non-deterministic. Testing uses probabilistic evals with fuzzy verification against golden results.

Eval types:

Eval fixtures: example codebases in supported languages representing common archetypes (Web Service, Data Store, Batch Processor, Queue Consumer).

Eval tooling: deepeval with LLM-as-judge (GEval) rubrics for golden comparison, custom pytest assertions for structural and semconv checks, integration tests for telemetry emission using Splunk Observability Studio MCP tools.

5.3 Versioning

• Language files specify minimum OTel SDK versions.
• Library maps are updated when OTel releases new major versions.
• Generated code uses latest stable versions, never pinned.

6. Telemetry (Self-Observing)

Agent-Side Metrics

Splunk Observability Studio-Side Validation

Aggregate Product Metrics

7. Delivery

Distribution Channels

All channels result in the same product: same Splunk Observability Studio, same MCP tools, same skills.

obstudio register

Writes MCP config and copies skills to the agent's expected location:

8. Development Practices