mpecan/tokf/tokf-filter

tokf Filter Authoring

You are an expert at writing tokf filter files. tokf is a config-driven CLI that compresses command output before it reaches an LLM context. Filters are TOML files that define how to process a command's output.

When the user asks you to create or modify a filter, follow this guide exactly. Produce valid, idiomatic TOML that matches the schema described below.

Section 1 — What a Filter File Is

A filter file is a TOML file that describes:

• Which command(s) it applies to (command)
• How to transform the raw output (steps, applied in a fixed order)
• What to emit on success vs. failure

Filters live in three places, searched in priority order:

1. .tokf/filters/ — project-local (repo-level overrides)
2. ~/.config/tokf/filters/ — user-level overrides
3. Built-in library (embedded in the tokf binary)

First match wins. Use tokf which "cargo test" to see which filter would activate for a given command.

Section 2 — Processing Order

Steps execute in this fixed order — do not rearrange them:

1. `match_output` — whole-output substring checks; if matched, short-circuits the entire pipeline and emits immediately
2. `[[replace]]` — per-line regex transforms applied to every line, in array order
3. `strip_ansi` / `trim_lines` — per-line cleanup (ANSI stripping, whitespace trimming)
4. `skip` / `keep` — line-level filtering (drop or retain lines by regex)
5. `dedup` / `dedup_window` — collapse duplicate consecutive lines
6. `lua_script` — Luau escape hatch; runs after dedup, before JSON/section/parse
7. `[json]` — JSON extraction via JSONPath; when configured, replaces section/parse/chunk
8. `[[section]]` OR `[parse]` — structured extraction (these are mutually exclusive; section is a state machine, parse is a declarative grouper). Skipped when [json] is configured.
9. `[[chunk]]` — block-based structured extraction with per-block aggregation, grouping, and tree output (runs on raw output, alongside sections). Skipped when [json] is configured.
10. Exit-code branch — [on_success] or [on_failure] depending on exit code
11. `[fallback]` — if neither on_success nor on_failure produced output
12. `strip_empty_lines` / `collapse_empty_lines` — post-processing cleanup on the final output

Within [on_success] and [on_failure], fields are processed as:

• head / tail → trim lines
• skip / extract → further filter
• aggregate → reduce collected sections
• output → final template render

Section 3 — Top-Level Fields Reference

Section 4 — Step Types

4.1 match_output — Whole-Output Short-Circuit

Check the entire raw output for a substring. If matched, emit a fixed string and stop — no further processing.

match_output = [
  { contains = "Everything up-to-date", output = "ok (up-to-date)" },
  { contains = "rejected", output = "✗ push rejected (try pulling first)" },
]

• contains: literal substring to search for (case-sensitive)
• output: string to emit if matched
• {line_containing} template variable: the first line that contains the substring

match_output = [
  { contains = "error", output = "Error on: {line_containing}" },
]

When to use: for well-known one-liner outcomes that make the rest of filtering irrelevant (e.g., "already up to date", "nothing to push", "authentication failed").

4.2 [[replace]] — Per-Line Regex Transforms

Applied to every line, in array order, before skip/keep. Use to reformat noisy lines.

[[replace]]
pattern = '^(\S+)\s+\S+\s+(\S+)\s+(\S+)'
output = "{1}: {2} → {3}"
 
[[replace]]
pattern = '^\s+Compiling (\S+) v(\S+)'
output = "compiling {1}@{2}"

• pattern: Rust regex (RE2 syntax, no lookaheads)
• output: template with {1}, {2}, … for capture groups; {0} is the full match
• If the pattern doesn't match a line, that line passes through unchanged
• Invalid patterns are silently skipped at runtime

When to use: when a line contains useful information but in a verbose format — reformat it rather than dropping it.

4.3 skip / keep — Line Filtering

skip drops lines matching any regex. keep retains only lines matching any regex. They compose:

skip = [
  "^\\s*Compiling ",
  "^\\s*Downloading ",
  "^\\s*$",
]
 
keep = ["^error", "^warning"]

• Both are arrays of regex strings
• Applied after [[replace]]
• skip is checked first, then keep
• A line must pass both: not skipped, and (if keep is non-empty) matching keep

When to use: skip for removing known noise patterns; keep for allow-listing (e.g., keep only lines that start with error or warning).

Also available inside [on_success] and [on_failure] for branch-level filtering.

4.4 dedup / dedup_window — Deduplication

dedup = true           # collapse consecutive identical lines
dedup_window = 10      # dedup within a 10-line sliding window

• dedup = true: removes consecutive duplicate lines (like uniq)
• dedup_window = N: deduplicates within a sliding window of N lines (catches near-consecutive repeats)
• They are independent; you can use both

When to use: for commands that emit repetitive progress lines (e.g., npm install printing the same package multiple times, spinner frames, repeated warnings).

4.5 lua_script — Luau Escape Hatch

For logic that pure TOML cannot express: numeric math, multi-line lookahead, conditional branching.

[lua_script]
lang = "luau"
source = '''
if exit_code == 0 then
    return "passed"
else
    local msg = output:match("Error: (.+)") or "unknown error"
    return "FAILED: " .. msg
end
'''

Or load the script from an external file:

[lua_script]
lang = "luau"
file = "scripts/my-filter.luau"

The file path resolves relative to the current working directory. Exactly one of source or file must be set.

Globals available:

• output (string): the full output after skip/keep/dedup
• exit_code (integer): the command's exit code
• args (table of strings): the arguments passed to the command

Return semantics:

• Return a string → replaces output, skips remaining TOML pipeline
• Return nil → fall through to [[section]] / [parse] / [on_success] / [on_failure]

Sandbox: io, os, and package are blocked. No filesystem or network access. Standard math/string/table libraries are available.

When to use: only when no TOML step can express the logic. Most filters do not need this. Consider it after exhausting match_output, skip/keep, [[replace]], [[section]], and [parse].

4.6 [json] — JSON Extraction via JSONPath

For commands that produce JSON output (e.g. kubectl get pods -o json, gh api, docker inspect). Extracts values using JSONPath (RFC 9535) queries and produces template variables and structured collections.

[json]
 
[[json.extract]]
path = "$.items[*]"
as = "pods"
 
[[json.extract.fields]]
field = "metadata.name"
as = "name"
 
[[json.extract.fields]]
field = "status.phase"
as = "phase"
 
[on_success]
output = "Pods ({pods_count}):\n{pods | each: \"  {name}: {phase}\" | join: \"\\n\"}"

`[[json.extract]]` fields:

`[[json.extract.fields]]` fields:

Result mapping:

• Single scalar → vars["as_name"] = string_value (no count, no chunk)
• Array → ChunkData::Flat collection + {as_name_count} variable
• Objects without `fields` → top-level scalars auto-flattened into chunk items
• Objects with `fields` → named fields extracted per item

Pipeline behavior: when [json] is configured, [[section]], [parse], and [[chunk]] are skipped. JSON replaces line-based structural processing. Extracted vars and chunks flow into [on_success]/[on_failure] template rendering.

Error handling: invalid JSON input → extraction skipped, pipeline falls back to raw output (templates are not rendered). Invalid JSONPath → rule silently skipped, other rules still run. Empty array with fields → emits {as_name_count} = "0".

When to use: when the command produces structured JSON output and you need to extract specific fields. Prefer this over [parse] + skip/keep for JSON-native commands.

4.7 [[section]] — State-Machine Section Collector

The most powerful step. Defines a state machine that collects lines into named variables as it scans top-to-bottom.

[[section]]
name = "failures"
enter = "^failures:$"      # regex: start collecting when this matches
exit = "^failures:$"       # regex: stop collecting when this matches (after start)
split_on = "^\\s*$"        # regex: split collected lines into blocks on blank lines
collect_as = "failure_blocks"
 
[[section]]
name = "summary"
match = "^test result:"    # regex: collect only lines matching this (no enter/exit)
collect_as = "summary_lines"