watcher-hermes/hermes-skills/attention-variant-picker
Claude Code and agent skill by watcher-hermes/hermes-skills
Trust tier 3
- community
Source check pending
Checking that the SKILL.md file is still in the repository.
Pick a full / sliding-window / sparse / differential attention topology for a new model given context length, retrieval demands, and compute profile.
Install
npx vskill@latest install watcher-hermes/hermes-skills/attention-variant-pickerSKILL.md
The same SKILL.md was published earlier as watcher-hermes/hermes-skills/attention-variant-picker.
Attention Variant Picker
Help a developer choose and justify an attention topology for a new transformer, or for an existing one they're extending to longer context.
Inputs to gather
- Target context length at training and at inference (often different — many models train at 16K and extend at inference).
- Retrieval demand on a 1–5 scale: 1 = pure chat, 5 = needle-in-haystack / RAG / code with long repository context.
- Inference memory budget per-request KV cache tolerance (bytes per token per layer is the right unit).
- Training cost tolerance — training SWA from scratch is cheap; retrofitting differential attention into a pretrained model is expensive.
- Hardware target — Hopper+ has full FlashAttention-3, Ada has FA2, older GPUs are mask-limited.
Decision rules
- Context ≤ 16K and retrieval ≤ 3: full attention with FlashAttention. Don't optimize prematurely.
- Context 16–128K and retrieval ≤ 3: mixed SWA + global at 5:1, window 1024 (Gemma 3 shape). Keeps retrieval workable while collapsing KV.
- Context > 128K: full SWA with a global layer every 4–6 layers, plus position interpolation / YaRN scaling (Lesson 04).
- Retrieval = 5 and training budget allows: consider differential attention in the top 4 layers only (half the KV doubling, most of the sink-cancellation win).
- You're shipping a public API: prefer stable patterns (full, SWA, Gemma-3 mix). Skip native-sparse / DIFF unless you have kernel engineers.
- You can't change the base model: SWA can be retrofitted at inference via masking; differential and sparse can't.
Always flag
- Pure-SWA models below 7B often lose measurably on reasoning benchmarks. Recommend against.
- Window size < 512 is almost never right. Go bigger or use a different topology.
- Differential attention reports in the paper are on small models (3–7B). Scale-up evidence is thin as of early 2026.
- Every variant interacts with RoPE / YaRN scaling (Lesson 04). State the position scheme explicitly.
Output format
Return:
- Recommendation — a single named topology (e.g. "Gemma-3 mix, W=1024, 5:1 SWA:global").
- Justification — map each input to the decision rule above.
- KV cache estimate — at target context, in bytes per token per layer and GB at batch 1.
- Migration path — if the base model is already trained, how to retrofit.
- Known risks — which benchmarks / workloads might regress.
Security
- Security scanPassed, 95 of 100Pattern checks on the skill's files.
- Model reviewPassed
Quality evaluation
- Task evaluationNot runNo evaluation yet. Run one in Skill Studio.
Source
- SKILL.md in the repositoryCheckingskills/attention-variant-picker/SKILL.md
Popularity
- 7-day trend score
- 5
Not refreshed yet.
Works with
Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.
- Claude Code
- Cursor
- GitHub Copilot
- Windsurf
- Codex
- Gemini CLI
- Cline
- Amp
- Roo Code
- Goose
- Continue
- Kilo Code
- Junie
- Trae
- OpenHands
- Qwen Code
- AI Agent
- Tabnine Chat
- Sourcegraph Cody
- Amazon Q
- Firebase Genkit
- Sweep
- Devin
- Phind
- Mentat
- Claude Desktop
- Aider
- GPT Pilot
- Plandex
- bolt.new
- v0
- Replit Agent
- Anse
- TypingMind
- BetterChatGPT
- LibreChat
- Warp AI
- Pieces
- Fig AI
Embed badge
[](https://verified-skill.com/skills/watcher-hermes/hermes-skills/attention-variant-picker)