josefbacik/systing/systing-trace

Capturing a trace with systing

systing is a BPF-based Linux tracer. It captures scheduling events, IRQs, stack traces, network activity, memory events, syscalls, and more into a DuckDB database (or Perfetto trace) for later analysis.

Prerequisites

• Must run as root, or hold CAP_BPF + CAP_PERFMON (CAP_SYS_ADMIN also works). Use sudo.
• Output: prefer --output foo.duckdb for analysis with the systing-analyze MCP server. The default is `trace.pb` (Perfetto), so always pass --output explicitly when you want DuckDB.

Common invocations

Trace a command from start to finish

sudo systing --output trace.duckdb -- python3 myscript.py

Everything after -- is the command. Only the command, its threads, and children are traced.

Trace an existing PID for N seconds

sudo systing -p <PID> -d 10 --output trace.duckdb

Trace a cgroup

sudo systing -c /sys/fs/cgroup/my.slice -d 30 --output trace.duckdb

System-wide trace (no PID/cgroup filter)

sudo systing -d 5 --output trace.duckdb

-d 0 (the default) means "run until Ctrl-C, or until the -- command exits".

Recorders (what to capture)

List available recorders and their defaults:

systing --list-recorders

Enable extras with --add-recorder:

sudo systing --add-recorder network --add-recorder syscalls -d 10 --output trace.duckdb

Or restrict to only specific recorders with --only-recorder (disables every default first):

sudo systing --only-recorder sched --only-recorder network -d 10 --output trace.duckdb

Recorder dependencies and companions

• --add-recorder network also enables `network-packets` (the common investigation shape). --only-recorder network is state-only: connection/state tracking with no per-packet kprobes.
• network-packets and network-syscalls each require the base network recorder and enable it implicitly.
• memory-alloc requires and implicitly enables memory.
• Recorders that are on by default can also be turned off directly: --no-sched, --no-irq, --no-cpu-stack-traces, --no-sleep-stack-traces, --no-interruptible-stack-traces, -n/--no-stack-traces (all stacks).

Key options

Targeting and duration

Output

Sampling and counters

Symbolization

Memory recorder tuning

Custom events, markers, misc

TPU

TPU profiling

Systing can talk to the XLA/TPU runtime's gRPC services to correlate TPU activity with host-side scheduling and stacks in the same trace.

Two modes (use either or both):

• `--tpu-profile` — Full op-level profile. Connects to the XLA profiler service (port 8466), captures an XSpace profile for the trace duration, and records per-op timing, flops, and memory bytes into tpu_op / tpu_device. Heavier; only available while a workload is actively running, and requires a fixed --duration.
• `--tpu-metrics` — Lightweight polling. Connects to the RuntimeMetricService (port 8431), samples counters (duty cycle, HBM usage, latency distributions) at --tpu-metrics-interval into tpu_metric. Always available while the runtime is up.

Auto-discovery: By default systing scans /proc/net/tcp (across all network namespaces) for a single listener on the well-known port and connects to it, using setns if the service is in a container's netns. Zero listeners → error ("Is a TPU workload running?"); multiple listeners → disambiguate with --tpu-service-addr / --tpu-metrics-addr.

# Op-level TPU profile + host scheduling for 10s
sudo systing --tpu-profile -d 10 --output trace.duckdb
 
# Lightweight metrics at 500ms alongside network & sched
sudo systing --tpu-metrics --tpu-metrics-interval 500 \
    --add-recorder network -d 30 --output trace.duckdb
 
# Explicit address (container / multi-device host)
sudo systing --tpu-profile --tpu-service-addr 127.0.0.1:8466 -d 10 --output trace.duckdb

Output formats