roger-luo/ion/dogfooding-ion

Dogfooding Ion on a real project

Overview

Unit tests prove the machinery works. Dogfooding proves Ion is pleasant and legible to drive — for both the human at the keyboard and the agent operating it programmatically. You use the real ion binary to accomplish real goals in a real repo, notice every place you (or an agent) hesitate, guess, or run an avoidable extra command, and then improve the CLI so the next person doesn't.

Judge three lenses at once. All three matter:

1. Human terminal UX — when a user runs a plain ion <cmd> (no --json),

is the output friendly to read? Scannable, correctly emphasized, not noisy, colors and spacing carrying meaning rather than decoration. A wall of undifferentiated text, or a bare Done! with no orientation, is a gap.

1. Progressive prompting — does each command's output *guide the reader to

the next step*, so neither a human nor an agent is ever left at a dead end? This spans both channels: the human text AND the --json envelope. After ion init you should know to run ion add; after ion add you should know the skill is live and how to use or commit it; when a command needs more input it should name the exact recovery command and flags. The brief: the printings teach the next move as the command runs.

1. Workflow ergonomics — does accomplishing one real goal take an awkward

sequence of commands? If "start using team skills in my repo" is init → search → add → edit .gitignore → commit, ask whether a step should be folded in (a sensible default, an auto-init, a combined command) or a new command should exist. Some gaps are fixed with a better sentence; others with a better command surface. Don't paper a missing command over with another hint.

The loop:

build the ion binary under test → pick a goal a real user has → drive it end to
  end on a scratch repo AND an established repo, reading output as a human would
   → at each step judge the three lenses: friendly? next step obvious (text +
     JSON)? too many commands?
   → log the rough edge (what you saw, what was missing, who stalls)
   → FIX it: a friendlier line, a next-step hint, a richer action_required
     envelope, OR a new/consolidated command — failing test FIRST
   → rebuild → re-drive that step → confirm it now lands
   → repeat across the command surface until the journeys feel obvious

When to use

• The user says "dogfood ion on <project>", "use ion for real and fix what's

clunky", "make ion guide the agent through the next steps", or asks to shake out the CLI's UX.

• After changing command output, flags, or the JSON envelope, to confirm the

end-to-end journeys still read well.

• To evaluate whether an agent could drive Ion from the output alone — the

north star is that the ion-cli skill becomes almost unnecessary because the commands teach their own next steps.

Do not use for changes verifiable by cargo nextest run alone (pure library logic, parsing, resolution). This skill is about the felt experience of using the tool, which unit tests can't see.

Setup

1. Build the binary under test. The point is to exercise your code, so

invoke the freshly built debug binary — not whatever ion is on PATH: ``bash cargo build # in the ion repo ION=$PWD/target/debug/ion # invoke this, not ion ` **Rebuild after every fix.** A stale binary is the #1 way to "fix" a gap that the run still shows. (If you'd rather test the exact thing users run, cargo install --path . --force puts it on PATH as ion` — but then you must reinstall after each change; the built-binary path is faster and has no stale trap.)

1. Prepare two target repos — you need both to see the whole journey:

• A scratch repo for the cold start — the first-run experience, where

next-step guidance matters most: ``bash mkdir /tmp/ion-dogfood-scratch && cd /tmp/ion-dogfood-scratch && git init ` Greenfield surfaces init → search → add → new → validate`, and the "how do I even begin" moment a new user actually hits.

• An established repo that already uses Ion (has Ion.toml/Ion.lock) —

e.g. a sibling project like autotune2. This surfaces the maintenance journeys: install-all, update, migrate, skill eject, remove. Work in a throwaway git worktree or a copy, never the user's live checkout — dogfooding mutates Ion.toml, .gitignore, and target dirs.

1. Read the intended agent contract. Skim .agents/skills/ion-cli/SKILL.md

— it documents the JSON interface an agent is supposed to rely on. Part of dogfooding is checking whether the CLI's actual output lets an agent succeed without that cheat sheet.

Drive the lifecycle

Walk real goals, not a checklist of commands. For each command in a journey, run it the way a human would (plain, no --json) and read the output as if you'd never seen it — then separately inspect the agent channel (--json) where the programmatic next step matters (anything that prompts, confirms, errors, or returns IDs the next call needs). Two representative journeys:

Journey A — cold start (scratch repo). "I want to start using skills here."

1. $ION init — choose targets. Does it confirm what was created *and point to

what's next* (ion search / ion add)? Or does it stop at a file list?

1. $ION search "code review" — do results tell you how to install one

(ion add <source>), or just list names you must know what to do with?

1. $ION add <source> — does it confirm the skill is installed, where, to which

targets, and what to do now (use it / add more / commit these files)? Or a bare Done!?

1. $ION new --path .agents/skills/my-skill — does it point to editing

SKILL.md / validating / where the skill lives?

1. $ION validate — when there's a warning or error, does the message name the

fix (which field, or --allow-warnings), not just the complaint?

1. $ION list — does empty read as "here's how to add one" rather than silence?

Journey B — established repo (real, in a worktree). "I inherited this repo."

1. $ION add (install-all from Ion.toml) — does the summary orient you

(installed / skipped / pinned) and say what to do next?

1. $ION update — do up-to-date vs updated vs skipped(pinned) read clearly, and

does it tell you how to act on each (e.g. how to update a pinned skill)?

1. $ION migrate, $ION skill eject <name>, $ION remove <name> — for each,

does the output explain what changed and what now? Does remove say how to undo, and does its confirmation flow (action_required → --yes) read well in both channels?

At every step, capture the actual output (copy it into your findings) — you're reviewing artifacts, not memories.

The rubric — judge each step

For every command's output, ask this short set of questions. They're the operational definition of the three lenses; a "no" is a finding.

Orientation (human): Reading only this output, do I know what just happened — what state changed, where files landed?

Readability (human): Is it scannable? Is emphasis (bold names, cyan paths, green success, yellow hints, dim secondary — see src/style.rs) carrying meaning, or is it a flat/noisy block? Does it degrade gracefully when piped (no color, no TTY)?

Next step (both channels): Does the output name the natural next action with the exact command to run? And is that guidance present in both the human text and the --json data — an agent reading JSON must not be told less than a human reading text. Inconsistency between channels is itself a gap.

Recovery (both channels): If the command needs input, failed, or hit a warning, does it name the exact recovery command and flags — action_required in JSON and a plain-text equivalent for the human? An error that says what broke but not what to do is only half-written.

Path length (ergonomics): How many commands did this goal take? If a common goal needs a rote sequence every time, that's a signal to fold a step in (a default, an auto-step) or add/consolidate a command — not to add a third hint. Note the sequence you actually ran.

Agent self-sufficiency (the north star): Could an agent with no ion-cli cheat sheet — only this output — construct the correct next call? If it would have to guess a flag or command name, the printing isn't teaching enough.

Log findings

Keep a running findings log (in the workspace, or inline for the user). One entry per rough edge so fixes are traceable:

FINDING: `ion add <source>` ends at "Done!" with no next step.
  Lens: progressive-prompting + human UX
  Saw (human):  "Done!"
  Saw (--json): {"success":true,"data":{"name":"code-review","installed_to":"…","targets":["claude"]}}
  Missing: human text doesn't confirm where it landed or what to do next; JSON
    data has the facts but no next-step signal. A new user doesn't know the
    skill is now live for Claude, or that these files should be committed.
  Who stalls: new human user; agent has enough in JSON to proceed but nothing
    tells it the install is "usable now".
  Fix: after the summary, print the target paths + a "the skill is now available
    to <targets>; commit .agents/ and Ion.lock" line; mirror a `next` field or
    equivalent in JSON.

Separate the small fixes (a missing line, a clearer error) from design findings (this goal needs a new/merged command). Design findings deserve a short proposal — what the command would be, what it replaces, why the current sequence is too long — before you build, since they change the CLI's surface.

Fix — test-first

Once you've reproduced a gap, fix it with a failing test first, so the improvement is pinned and can't silently regress. Ion gives you two testing tools; pick by what you're asserting.

• `scenario` crate (crates/scenario/) — purpose-built to test CLI output

under controlled terminal conditions: with/without a TTY, specific widths, color on/off, and interactive expect/send_line sessions. This is the right tool for human-channel assertions — "the output contains this next-step line", "the hint survives when piped (no color)", "the confirm prompt appears in a PTY". Write the scenario asserting the guidance you want, watch it fail against the current binary, then add the printing.

• Integration tests (tests/, invoking env!("CARGO_BIN_EXE_ion")) — for

the agent channel and exit codes: assert the --json envelope shape, the action_required string, exit code 2 on needs-input, and that a data field the next call needs is present. Assert on both channels when a fix touches both (it usually should — see the consistency rubric).

Where the printings live:

• Human text & hints — per-command println! in src/commands/<name>.rs,

styled via crate::style::Paint (bold/success/info/warn/dim). Reusable hints follow the model of print_no_targets_hint / print_codex_hint in src/commands/init.rs — factor a shared hint fn rather than copy-pasting a line into every command.

• Agent envelope — src/json.rs: print_success (exit 0),

print_action_required(action, data) (exit 2), print_error / print_failure (exit 1). Enriching the agent's next-step signal usually means adding to the data payload or introducing/strengthening an action_required case.

• New / consolidated commands — the clap Commands enum and dispatch in

src/main.rs, with the implementation in a new src/commands/<name>.rs. A consolidation may instead mean changing a default or chaining an existing step; prefer the smallest change that removes the extra command.