moonlight-lupin/agent-skills/input-token-overheads

Input Token Overheads

Audit every source of input token cost on a Hermes Agent instance, at two scopes. Measure each, rank by cost, act on the top consumers.

• Per-turn overhead audit — what the system prompt costs every turn: tool schemas, skills index, system fixed text, user profile, memory recall. Script: scripts/audit_overheads.py.
• Aggregate cost-accounting audit — total input token volume across sessions, cron fires, and tool traffic: where the tokens went, which consumers dominate, what config or prompt change reduces them. Script: scripts/audit.py. (Merged from the former input-token-analysis skill; v1.9.)

Per-turn context-window health is the overhead audit's job; provider billing disputes need the provider's usage dashboard — local telemetry cannot see provider-side caching.

When to Use

• User says "token overhead", "context too large", "why is input so expensive"
• Model output quality degrades from context dilution
• Cost optimization — fewer input tokens per turn means lower API spend
• After adding skills, plugins, or tools — verify the overhead delta
• Monthly/weekly input token bill is large and unexplained (e.g. "2.4B tokens in 30 days") — aggregate audit
• After applying token-saving changes — measure the delta against a baseline — aggregate audit
• A cron job is suspected of runaway token use — aggregate audit

The Overhead Map

Verified on a live build (2026-10), cli surface, 44 tools:

On-demand (not per-turn): full SKILL.md via skill_view, deferred tool schemas via tool_describe, reference files via skill_view(file_path=...).

Three-way measurement (v1.8), category by category:

1. Measured (ground truth): scripts/audit_overheads.py starts a REAL prompt (hermes -z "Reply with OK" --usage-file ...) and reads the provider's actual input_tokens for a fresh session. Providers report totals only.
2. Independent (per category): the script builds Hermes' own offline inspection agent, renders the real system prompt parts, and measures each category with the build's rough tokenizer — system fixed text, skills index, memory, user profile, builtin/MCP/subagent tool schemas. It re-derives categories from the rendered prompt; it does not read prompt-size output.
3. Budget: hermes prompt-size --json byte accounting (bytes/4 estimator), used only as the comparison column.

Plus the context-doctor port (jzOcb/context-doctor, MIT; concepts only, script native): injection-cap checks — TRUNCATED (over cap, silently cut every turn), MISSING (broken reference). Exit 1 when any injected file needs attention.

Health Ratio

The health metric is overhead ratio: measured (or estimated) overhead tokens divided by the model's context window. The absolute number matters for cost; the ratio matters for quality.

Why the ratio matters: Three studies confirm that input length degrades model performance independent of content quality:

1. Lost in the Middle (Liu et al., TACL 2023) — Models follow a U-shaped curve: best recall at the start and end of context, severe degradation in the middle. Overhead sits at the top of every turn, but it pushes conversation history into the degradation zone. arxiv.org/abs/2307.03172

1. Same Task, More Tokens (Levy et al., ACL 2024) — Reasoning performance degrades at input lengths far shorter than the model's stated maximum. The degradation appears even when the extra tokens are padding with no distracting content. The model's technical context window is not its effective context window. aclanthology.org/2024.acl-long.818

1. Context Length Alone Hurts (Du et al., EMNLP 2025) — Performance degrades 14-85% as input length increases, even when retrieval is perfect, irrelevant tokens are replaced with whitespace, or all tokens except relevant ones are masked. The sheer length of the input is itself a limitation. aclanthology.org/2025.findings-emnlp.1264

Cost compounding: Overhead is paid every turn. At 10k tokens over 100 turns, that is 1M input tokens spent on overhead alone. Reducing overhead by 2k tokens saves 200k tokens per 100-turn session.

Mitigations from the research:

Procedure

1. Measure each overhead source

python3 scripts/audit_overheads.py     # from the input-token-overheads skill dir

One run: starts the live one-shot prompt (one inference call), then prints the category-by-category comparison table:

category                     indep est    budget    delta
system fixed text                4,177     4,065      112
skills index                     5,379     5,357       22
memory (own facts)                   0         0        0
user profile                       820       816        4
tool schemas (all)              17,885    17,884        1
TOTAL                           28,261    28,124      137
provider measured                   25,077

• indep est — re-derived from the rendered prompt with the build's rough tokenizer (no prompt-size output involved)
• budget — hermes prompt-size bytes/4
• provider measured — the real fresh-session input_tokens; the honest denominator for the health ratio
• Per-category deltas over a few hundred tokens = drift between what you think ships and what ships — investigate that category
• Tool split (builtin/mcp/subagent) prints under the table

Flags: --platform cli|telegram|discord|... (inspection + budget surface), --skip-live (no inference call; measured row reads "unmeasured"), --json, --chart out.png, --ctx <window>, --live-timeout. Exit 1 = an injected file needs attention.

Also measure what neither estimate sees: memory-provider recall size per turn (in-session: read a fresh turn's injected recall blocks) — it rides on top of the measured fresh-session total and grows with facts.

Done: category table read; health ratio computed from the provider-measured total; every injected-file status checked.

2. Rank by cost

The table ranks. Typical ranking:

1. Tool schemas — largest block. Scales with enabled toolset count.
2. Skills index — always-on full index. Scales with indexed-skill count.
3. System fixed text — fixed stable tier.
4. User profile — capped, but usually near budget on long-lived agents.
5. Memory-provider recall — dynamic, per-turn, outside the budget; measure in-session, trim facts if it grows.

Done: top 3 identified — these are the optimization targets.

3. Act on top consumers

Tool schemas (largest block):

• Prefer deferred tools (loaded on demand) over always-on tools — each deferral replaces a full JSON schema with a ~150 B catalog row
• Disable unused toolsets; check requires_toolsets in every skill's frontmatter before removing a toolset a skill depends on
• Use platform_toolsets.<platform> in config.yaml to control per-surface access (a Telegram surface needs fewer tools than the CLI)

Truncated files (script exit 1):

• A TRUNCATED user profile is a correctness problem, not a cost problem — the tail is lost every turn. Trim under cap (batch-shorten or remove stale entries) and re-run until OK

Skills index:

• Disable unused skills in config.yaml under skills.disabled — each removal drops ~200-300 chars from the always-on index
• Trim descriptions over ~200 chars — they waste index budget without improving routing

Memory provider (if installed):

• Run consolidation to move working to episodic, reducing the recall budget per turn
• Invalidate stale facts; lower the recall limit if context is tight

Compression: load skill_view(name='hermes-compression-tuning') for threshold and target-ratio tuning.

Done: at least one optimization applied to each top-3 source, every truncated file trimmed under cap.

4. Verify the delta

Re-run the audit script (live). Compare each category's delta and the provider-measured total before/after; confirm all injected files read OK.

Done: before/after per-category delta reported. If no meaningful reduction, the remaining overhead is structural (tool schemas + skills index) and needs config changes, not content trims.

Aggregate Spend Audit (cost accounting)

Explain aggregate input token consumption. Answer three questions with numbers: where did the tokens go, which consumers dominate, what config or prompt change reduces them. This audit covers total volume across sessions, cron fires, and tool traffic; the per-turn procedure above covers system-prompt overhead.

Run it

python3 scripts/audit.py --days 30     # from the input-token-overheads skill dir
# other profiles: --db ~/.hermes/profiles/<name>/state.db
# custom cron ledger: --cron-audit /path/to/usage_audit.jsonl --jobs /path/to/jobs.json

Read-only, pure stdlib. The script prints: totals by task/model/provider, cache-read and cache-write columns, weighted per-call averages (long sessions and short-session floor), top sessions, cron offenders, active-only tool-result volume with the tool-share percentage, and oversized-result counts. Everything is derived from real DB rows — never estimate by hand when the script can measure.

Pre-Check Discipline

Verify before advising. Advice formed from memory is not acceptable.