ollygarden/opentelemetry-agent-skills/otel-profiles

OpenTelemetry Profiles

The main OpenTelemetry producer is the eBPF profiler, shipped as the profiling receiver in the `otelcol-ebpf-profiler` Collector distribution. It profiles every process on a Linux host with no application changes.

Maturity: the signal entered public Alpha in March 2026, the OTLP profiles proto (v1development) is Development, the profiling receiver is Development, and the Collector gate service.profilesSupport is alpha. Expect breaking changes; pin the profiler, Collector, and backend versions together.

Collector-side component facts (OTLP receiver/exporter profile support, the fixed /v1development/profiles HTTP path, k8s_attributes, memory_limiter, filter profile_conditions) live in the otel-collector skill's component pages; OTTL profile/profilesample contexts live in otel-ottl.

otelcol-ebpf-profiler 0.164.0 bundles only: receivers profiling, nop; processors memory_limiter, k8s_attributes, resource_detection, resource, transform (plus batch, which does not support profiles); exporters otlp_grpc, otlp_http, debug, file, nop; extensions health_check, pprof, opamp, offcpu. For anything else (filter, the basicauth extension, vendor exporters), relay to a regular Collector or build a distribution (otel-collector-builder skill).

Version matrix (Verified 2026-11-05)

Evidence: Docker Hub tags, the `v0.163.0`, `v0.163.1`, `v0.163.2`, and `v0.164.0` distribution manifests; the `v0.164.0` changelog.

Read references/receiver.md when setting keys beyond samples_per_second, enabling off-CPU or OBI correlation, or pinning a release not in this table (keys follow the pinned ebpf-profiler, not the Collector version).

Workflow

1. Pick the image and pin. Use the latest release unless a needed key or component is missing from it (see the matrix). Alternative: an OCB build (otel-collector-builder skill) based on the distribution manifest.
2. Pick the environment.

• Linux host or plain Docker: Run on a host below.
• Kubernetes: references/kubernetes.md (DaemonSet, RBAC, Kubernetes metadata, memory sizing).
• Kubernetes nodes that are themselves containers (kind, k3d, Docker-in-Docker, minikube with the docker driver): also references/nested-containers.md.

1. Wire the pipeline. profiles pipeline: profiling receiver, memory_limiter first, enrichment processors, then otlp_grpc or otlp_http. Start the Collector with --feature-gates=+service.profilesSupport. Run validate --config=<file> --feature-gates=+service.profilesSupport with the target image; it catches wrong keys for the pinned version.
2. Export and verify. Add debug (verbosity: basic) first: each reporter_interval it logs "resource profiles": N, "sample records": M; no lines or zero counts mean the receiver is not producing (see troubleshooting). verbosity: detailed shows resource attributes. Then add the backend: references/backends.md for endpoints, auth, and per-backend label behavior.
3. Troubleshoot by exact error string: references/troubleshooting.md.

Run on a host

The profiler needs root or CAP_SYS_ADMIN/CAP_BPF/CAP_PERFMON, the host PID namespace, and a mounted tracefs. A container gets a fresh /sys without tracefs even when privileged, so bind-mount it:

docker run -d --name otel-ebpf-profiler --privileged --pid=host \
  -v /sys/kernel/tracing:/sys/kernel/tracing \
  -v "$PWD/config.yaml:/etc/otelcol/config.yaml:ro" \
  otel/opentelemetry-collector-ebpf-profiler:0.164.0 \
  --config=/etc/otelcol/config.yaml --feature-gates=+service.profilesSupport

receivers:
  profiling:
    samples_per_second: 20      # default
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 800
    spike_limit_mib: 200
exporters:
  debug:
    verbosity: basic
  otlp_grpc:
    endpoint: profiles-backend:4317   # backend-specific, see references/backends.md
    tls:
      insecure: true
service:
  pipelines:
    profiles:
      receivers: [profiling]
      processors: [memory_limiter]
      exporters: [otlp_grpc, debug]

The image is built FROM scratch: no shell, so debug from logs, not exec. Expected startup lines: eBPF tracer loaded, Attached sched monitor, Everything is ready.

Known limitations

• The receiver profiles every process; it has no process, container, or namespace selection, and upstream declined to add one (#358). To keep a subset, filter on enrichment attributes with filter profile_conditions; see "Keep only some workloads" in references/kubernetes.md. Sampling cost on the node stays the same; only exported volume drops.
• Linux amd64/arm64, kernel 5.10+ (checked at config validation; no_kernel_version_check bypasses it). Backends may lag the proto; check which OTLP profiles version they accept.
• Trace-to-profile correlation is partial: OTLP profiles can link samples to trace_id/span_id, but the profiler fills them only from a context producer (OBI via obi_process_ctx; the thread-context OTEP is accepted, but the receiver's thread_context support in v0.0.202640 is a stub). Go pprof labels are read as custom labels.