moonlight-lupin/agent-skills/skill-retrieval

Skill Retrieval

This is a Hermes Agent plugin. It is not a Claude Code plugin and will not load in Claude Code — that runtime has no pre_llm_call event, no Python register() entry point, and reads .claude-plugin/plugin.json rather than plugin.yaml. Requires Hermes Agent >=0.21.1 (requires_hermes in plugin.yaml).

BM25-based progressive disclosure for Hermes Agent skills. Instead of dumping every skill description into the system prompt (~11.5K tokens), this plugin keeps a compact names-only index and injects only the top-K relevant descriptions per turn.

What it does

Two-phase progressive disclosure:

1. Phase 1 — System prompt compaction (an llm_request middleware):

Rewrites the outgoing request's <available_skills> block so it lists skill names only (descriptions stripped). All skills remain discoverable by name (~2K tokens instead of ~11.5K). The system prompt Hermes builds is never modified — prompt caching is unaffected.

1. Phase 2 — Per-turn BM25 retrieval (pre_llm_call hook): Tokenizes the

user message, ranks active skill descriptions with BM25 Okapi, and injects the top-K matches (~300 tokens) as context appended after the user message.

Architecture

Each turn (llm_request middleware)
    │
    ▼
Phase 1: rewrite <available_skills> → names only (~2K tokens, request only)
 
Each turn (pre_llm_call)
    │
    ▼
Phase 2: BM25Index.retrieve(user_message, top_k)
    └── inject "## Retrieved Skills ..." into user message (~300 tokens)

The BM25 index is built lazily on the first turn from standalone skills (~/.hermes/skills) and plugin-bundled skills (~/.hermes/plugins/*/skills), then cached per Hermes home, tool capability snapshot, session platform and resolved disabled-skill set (the same inputs Hermes keys its own skills prompt cache on). The cache is rebuilt without a restart when Hermes clears its skills prompt cache (skill_manage create/patch/delete, hub install, skill toggles) or when a cheap on-disk manifest changes (skill root and category dir mtimes, top-level SKILL.md files, config.yaml). Retrieval uses a pure-stdlib inverted index (term → posting list of precomputed BM25 weights) and is sub-millisecond for ~200 skills.

Token savings

Measured on a Hermes install with ~300 skills; savings scale with skill count.

Installation

Copy or symlink this directory into the Hermes plugins folder:

# From this repo
ln -s "$(pwd)/plugins/skill-retrieval" ~/.hermes/plugins/skill-retrieval
 
# Or copy
cp -r plugins/skill-retrieval ~/.hermes/plugins/skill-retrieval

User plugins are opt-in: Hermes discovers the directory but does not load it until it is enabled (this adds skill-retrieval to plugins.enabled in ~/.hermes/config.yaml):

hermes plugins enable skill-retrieval

Restart the agent session so register() runs — it registers the llm_request middleware and the pre_llm_call hook.

Dependencies (install into the Hermes Python env if missing):

pip install pyyaml

On CPython 3.14 prefer the host's hermes_yaml facade (ruamel.yaml) — the plugin resolves it automatically at runtime, falling back to PyYAML. If both are missing it logs one warning and keeps running with an empty index.

Configuration

export SKILL_RETRIEVAL_TOP_K=8
export SKILL_RETRIEVAL_INJECT_K=6
export SKILL_RETRIEVAL_COMPACT=0
export SKILL_RETRIEVAL_RERANK=jev
export SKILL_RETRIEVAL_RERANK_SESSION_CACHE=1

Jev rerank (optional, off by default)

When SKILL_RETRIEVAL_RERANK=jev, the BM25 top-N shortlist is reranked by the Jev system-one judgment model (POST https://api.typesafe.ai/v1/systemone, model jev-latest). Behavior:

• Fail-soft — 3.0 s timeout, no retries; on any error the request logs a

warning and BM25's original order stands. Retrieval never blocks on Jev.

• Env-only config — all settings come from os.environ (never a file

read): TYPESAFE_API_KEY (legacy TYPESAFE_KEY accepted), or OPENROUTER_API_KEY with SKILL_RETRIEVAL_RERANK_PROVIDER=openrouter (same contract, served through OpenRouter). Hermes loads the active profile's .env into environ at startup, so no cross-profile file is ever read.

• A/B log — every query records the timestamp, session id, a 200-char

query excerpt, both orders, probabilities, tokens and latency to the log path above; the log rotates past 5 MB; scripts/rerank_ab_report.py summarizes it locally.

• Outbound cap — at most RERANK_CANDIDATES (default 12) descriptions

are sent per call, even when the caller's shortlist is longer; the remainder keeps BM25 order at the tail and never leaves the machine.

Session rerank cache (optional, off by default)

This mode targets phase-continuous conversations. The first successful Jev rerank freezes the named session's BM25 candidate pool as a workset. Local BM25 still runs on every turn. When the candidates that would be injected are already in the workset or in the session's confirmed fallback set, the plugin uses the current BM25 order and skips the external rerank. A candidate outside those sets triggers Jev; it is added to the fallback set only when a successful Jev result selects it for injection.

The state is isolated by non-empty session id, bounded to 256 sessions, and cleared on skill-catalog mutation. Anonymous requests never reuse it. This optimization does not rewrite or mutate Hermes' system prompt.

Verify it's working

Phase 1 silently no-ops outside a full Hermes runtime, and the BM25 index can silently empty. After restart, check the Hermes logs.

Healthy start — look for these log lines:

• BM25 index built: N docs …
• Skill retrieval plugin registered (top_k=…, compact=true, rerank_session_cache=…)

Degraded — these warnings mean it's not working:

• Cannot locate prompt_builder — compaction skipped (Phase 1 failed; Phase 2

still runs for anonymous sessions, but named sessions skip injection because no capability snapshot is recorded)

• No active skills found for BM25 index (index is empty — zero retrieval injection)

How it works

• Tokenizer — lowercases text, strips punctuation, splits on whitespace.

Tokens with CJK-script characters are split into bigrams (for CJK-script runs; a 3+-character run is replaced by its bigrams): CJK Unified Ideographs incl. Extension A and B, Japanese kana (fullwidth and halfwidth), and hangul syllables — so Chinese, Japanese and Korean queries can match non-English corpus text, and a mixed token like 用python写 keeps the python term.

• Corpus — each skill becomes "name: description" plus its frontmatter

triggers: list and metadata.hermes.tags (when present) and — when the SKILL.md body contains CJK-script characters — the space-joined CJK runs of the body's first ~3000 characters, capped at 200 characters total. With the body in the corpus a CJK query can match skill content, not just the (usually English) description. Trigger phrases, tags and the CJK runs participate in retrieval scoring but only the description is injected into the prompt. Disabled skills from ~/.hermes/config.yaml are skipped.

• Index — BM25 Okapi TF saturation + Lucene IDF

log(1 + (N-df+0.5)/(df+0.5)) (always positive, so small corpora and common terms still score), stored as an inverted index: `dict[str, list[tuple[int, float]]]` mapping each term to a posting list of (doc_index, precomputed BM25 weight).

• Retrieve — for each unique query token present in the index, walk its

posting list and accumulate scores; sort by descending score (score > 0 only).

Performance

• Index built on first use and cached (~8 ms for 200 skills on a CPU-only

VM). Each later turn only stats the skill roots, their immediate subdirectories and config.yaml to validate the cache. A zero-skill install caches the empty result too (one warning, no rescans).

• Retrieval is sub-millisecond (~0.03 ms mean for 200 skills). The inverted

index touches only documents that share a query term — no full-corpus scan.

• No compiled dependencies. The plugin uses only the Python standard library

(plus pyyaml for config/frontmatter parsing). This removes a 154 MB numpy/scipy install and a ~573 ms import cost, which matters for subprocess spawning paths (e.g. a Claude Code UserPromptSubmit variant).

• Failures in the hook return None (no injection) so the agent keeps working.

Dependencies

• pyyaml — declared in plugin.yaml (pip_dependencies), so

hermes plugins install skill-retrieval installs it. On CPython 3.14 the host's hermes_yaml facade is used instead when PyYAML is unavailable.

Limitations