edmundmiller/dotfiles/zbench

zsh-bench Integration

Proper benchmarking of interactive zsh using romkatv/zsh-bench. Measures real user-visible latency, NOT time zsh -lic exit (which is meaningless).

Commands

hey zbench              # Run + display with threshold indicators (auto-compares if baseline exists)
hey zbench-save         # Run + save as baseline + append history
hey zbench-compare      # Run + explicit diff against baseline
hey zbench-check        # Exit non-zero if over threshold (for git bisect)
hey zbench-baseline     # Show saved baseline (no run)
hey zbench-history      # Show TSV history

All commands accept extra zsh-bench args: hey zbench --iters 4 for quick runs. Run from the intended checkout: both installed hey and ./bin/hey execute that project's mise tasks. No host rebuild is needed for task edits. The Nushell entrypoints in bin/hey.d/zbench/ use def --wrapped ... [...args] so flags such as --iters pass through to zsh-bench.

Metrics & Thresholds

From romkatv's blind perception study — values at or below threshold are indistinguishable from zero:

Indicators: 🟢 ≤50% (headroom) · 🟡 ≤100% (imperceptible) · 🟠 ≤200% (noticeable) · 🔴 >200% (sluggish)

exit_time_ms is shown but not used for thresholds — it doesn't measure interactive performance.

Git Bisect Workflow

Find which commit made the shell slow:

git bisect start
git bisect bad HEAD
git bisect good <known-good-commit>
git bisect run hey zbench-check

zbench-check exits non-zero when any metric exceeds its threshold.

File Layout

benchmarks/zsh-bench/
├── <Host>.json              # Current baseline per host
└── history/
    └── <Host>.tsv           # Append-only history (timestamp, git_rev, metrics)
packages/zsh-bench/           # Nix package (romkatv/zsh-bench with internal/ helpers)
bin/hey.d/zbench/            # Project-local task entrypoints
bin/hey.d/.lib/zbench.nu      # Shared Nushell benchmark implementation
bin/zbench-report             # Python — parse, compare, format results

Baselines are per-host (MacTraitor-Pro.json, Seqeratop.json) because hardware varies.

Typical Workflow

# 1. Establish baseline on a clean build
hey zbench-save
 
# 2. Make zsh config changes
vim config/zsh/.zshrc
hey rebuild
 
# 3. Check for regressions
hey zbench                    # Shows comparison vs baseline
 
# 4. If satisfied, update baseline
hey zbench-save

Autoresearch Optimization Prompt

For a multi-iteration optimization loop, use pi-autoresearch to drive small zsh config experiments and zsh-bench to measure whether each experiment helped. Do not use autoresearch instead of zsh-bench; use it as the edit → benchmark → keep/revert loop around zsh-bench.

Run the local prompt template with pi-autoresearch:

pi -e npm:pi-autoresearch "$(cat .pi/prompts/zbench-autoresearch.md)"

If you prefer to install the extension persistently first:

pi install npm:pi-autoresearch
cat .pi/prompts/zbench-autoresearch.md

Canonical prompt file: .pi/prompts/zbench-autoresearch.md.

Debugging Slow Startup

Phase Timing Script

Don't guess — measure. Paste this into zsh -c '...' to time each phase of startup:

zsh -c '
zmodload zsh/datetime
export XDG_CONFIG_HOME="${XDG_CONFIG_HOME:-$HOME/.config}"
export XDG_CACHE_HOME="${XDG_CACHE_HOME:-$HOME/.cache}"
export ZDOTDIR="${ZDOTDIR:-$XDG_CONFIG_HOME/zsh}"
export ZSH_CACHE="${ZSH_CACHE:-$XDG_CACHE_HOME/zsh}"
function _source { [[ -f "$1" ]] && source "$1"; }
function _cache {
  local cache_dir="$XDG_CACHE_HOME/zsh"; local cache_file="$cache_dir/$1.zsh"
  if [[ ! -f "$cache_file" ]] || [[ "$commands[$1]" -nt "$cache_file" ]]; then
    mkdir -p "$cache_dir"; "$@" > "$cache_file"; fi
  source "$cache_file"
}
 
t0=$EPOCHREALTIME
source $ZDOTDIR/.zshenv 2>/dev/null; t1=$EPOCHREALTIME
source $ZDOTDIR/config.zsh; t2=$EPOCHREALTIME
# ... add phases matching your .zshrc ...
source $ZDOTDIR/completion.zsh 2>/dev/null; t3=$EPOCHREALTIME
_source $ZDOTDIR/extra.zshrc; t4=$EPOCHREALTIME
 
printf "zshenv:     %4.0fms\n" $(( (t1-t0)*1000 ))
printf "config:     %4.0fms\n" $(( (t2-t1)*1000 ))
printf "completion: %4.0fms\n" $(( (t3-t2)*1000 ))
printf "extra:      %4.0fms\n" $(( (t4-t3)*1000 ))
printf "TOTAL:      %4.0fms\n" $(( (t4-t0)*1000 ))
'

Adapt phases to match the actual .zshrc. The gap between this total and zsh-bench is overhead from /etc/zshrc (nix-darwin generated) and deferred plugin loading.

To drill into extra.zshrc, time each source line individually — one slow alias file can dominate.

Known Culprits (ranked by typical impact)

The _cache Pattern

Central to fast startup. Already defined in .zshrc:

function _cache {
  local cache_dir="$XDG_CACHE_HOME/zsh"
  local cache_file="$cache_dir/$1.zsh"
  if [[ ! -f "$cache_file" ]] || [[ "$commands[$1]" -nt "$cache_file" ]]; then
    mkdir -p "$cache_dir"
    "$@" > "$cache_file"
  fi
  source "$cache_file"
}
 
# Usage:
_cache zoxide init zsh        # instead of eval "$(zoxide init zsh)"
_cache direnv hook zsh        # instead of eval "$(direnv hook zsh)"
_cache entire completion zsh  # instead of source <(entire completion zsh)

Invalidates when the binary changes ($commands[$1] mtime check). Delete ~/.cache/zsh/*.zsh to force regeneration.

Replay Mode

Use zsh-bench --iters 1 --scratch-dir /tmp/zbench-debug then dbg/replay --scratch-dir /tmp/zbench-debug to watch what zsh-bench actually sees. For worktree experiments that should read the repo copy of config/zsh, run with ZDOTDIR=$PWD/config/zsh ZSH_CACHE=$HOME/.cache/zsh; otherwise zsh-bench may exercise the currently installed Home Manager generation.

Key Design Decisions

• Uses zsh-bench's non-raw output (median of 16 iterations) for stable numbers.
• `--raw` gives per-iteration arrays — useful for variance analysis but not default.
• Baselines stored as JSON for easy programmatic comparison.
• History stored as TSV for easy column -t, awk, or import into spreadsheets.
• Regression detection: flags changes > 20% or > 5ms (whichever is larger).