shynlee04/hivemind-plugin/hf-l2-skill-synthesis

Overview

Synthesize new skills from GitHub repositories, existing codebases, or design patterns. Use when creating skills from repos, classifying skill patterns, building template libraries, generating eval frameworks, or creating eval-driven skill scaffolds. Produces complete skill packages with evals and trigger queries.

The Iron Law

NO SKILL WITHOUT EVALS

Every skill produced by this pipeline must include evaluation test cases (≥3) and trigger queries (≥20). A scaffold without evaluation coverage is incomplete. A skill without executed validation is untested.

skill-synthesis

On Load

1. Fetch the canonical spec — always fresh, never cached:

``bash webfetch https://agentskills.io/llms.txt `` Use this as the ground truth for frontmatter format, pattern definitions, and eval structure.

1. Check for planning files — if task_plan.md exists, read the Goal field. If missing, run:

``bash bash scripts/validate-gate.sh synthesize "<user-request>" <output-dir> `` This creates the planning scaffold and validates intent.

Pipeline Overview

INGEST → CLASSIFY → SCAFFOLD → VALIDATE

Phase 1: INGEST

Input: GitHub repo URL or owner/repo string Tools: repomix --remote, webfetch, websearch

# Pack remote repo (branch-targetable, no local clone)
repomix --remote <owner/repo> \
  --include "**/SKILL.md,**/skills/**/*.md" \
  --output /tmp/skill-ingest-<timestamp>.xml
 
# Fetch canonical spec (always fresh)
# Saves to /tmp/spec-<timestamp>.md
webfetch https://agentskills.io/llms.txt
 
# Search for real-world skill implementations across GitHub
# Results form a pattern corpus for classification
websearch "site:github.com SKILL.md agent skill frontmatter"

Output: Packed XML with all SKILL.md files, fresh spec, pattern corpus

Phase 2: CLASSIFY

For each SKILL.md found, classify along these axes:

Output: Classification report per skill, ranked by quality score

Phase 3: SCAFFOLD

1. Select top 3 examples matching target category
2. Extract common structural patterns (frontmatter shape, reference topics, script types)
3. Generate scaffold:

• SKILL.md with proper frontmatter (trigger phrases from decision tree analysis)
• Numbered reference files (150-300 lines each)
• Validation scripts (co-located with the skill's scripts/ directory)
• Evals: evals.json (3-8 test cases) + trigger-queries.json (20 queries, 60/40 train/val split)

1. Write to target directory

Output: Complete skill directory, ready for validation

Phase 4: VALIDATE

# Gate 0: Intent validation
bash scripts/validate-gate.sh synthesize "<user-request>" <output-dir>
 
# Structure validation
bash scripts/validate-skill.sh <output-dir>
 
# Overlap detection
bash scripts/check-overlaps.sh <output-dir>
 
# Trigger eval execution
bash scripts/run-trigger-evals.sh <output-dir>
 
# Quality scoring
bash scripts/grade-outputs.sh <output-dir>

If any step fails: Fix → re-run. Max 5 iterations. Output: Validated skill with passing evals and quality score ≥ 3.5

Decision Tree — Pick Your Path

User says...                                    → Load
─────────────────────────────────────────────────────────────────────
"create skills from GitHub" / "synthesize"      → references/01-github-ingestion.md
"find skill patterns" / "classify skills"       → references/02-pattern-classifier.md
"generate evals" / "test this skill"            → references/03-eval-framework.md
"score this skill" / "quality check"            → references/04-quality-matrix.md
"build a template" / "template library"         → references/05-template-library.md

Rule: Load only ONE reference file. Do not load all references.

Reference Map

Anti-Patterns

Error Handling

Non-Interactive Shell Compliance

All scripts must:

• Set CI=true, GIT_TERMINAL_PROMPT=0, GIT_PAGER=cat, PAGER=cat
• Use --yes / -y flags for all package managers
• Output structured JSON (not free-form text)
• Exit non-zero on any failure (set -euo pipefail)
• Include timeout 30 wrapper for any command that might hang
• Never use banned commands (vim, less, git add -p, etc.)

Integration with Existing Skills

Self-Correction

When the Task Keeps Failing

[Detection] repomix returns empty output or non-zero exit after 3 attempts. Classification is low-confidence across all axes. Validation loop hits 5-iteration max without passing. [Recovery] STOP the pipeline. Report what failed and where: which phase (INGEST/CLASSIFY/SCAFFOLD/VALIDATE), the specific error, and what was attempted. For repo ingestion failures: verify the repo exists, is public, and contains SKILL.md files. For validation failures: flag specific dimensions from grade-outputs.sh and suggest manual review.

When Unsure About the Next Step

[Detection] The user's request doesn't clearly map to a pipeline phase. Classification results are ambiguous (moderate confidence across conflicting axes). Unclear whether to scaffold from templates or synthesize from patterns. [Recovery] Run validate-gate.sh synthesize "<user-request>" <output-dir> to create a planning scaffold. The task_plan.md Goal field will clarify intent. If still unsure, ask the user up to 3 clarifying questions via the question tool — but default to the full pipeline (INGEST through VALIDATE) if ambiguity persists.

When the User Contradicts Skill Guidance

[Detection] User says "skip evals, just create the SKILL.md" or "I don't need validation, the output looks fine." User wants to bypass the "NO SKILL WITHOUT EVALS" iron law or validation gates. [Recovery] Acknowledge: "This skill's iron law states: NO SKILL WITHOUT EVALS. Skipping evals produces an untestable skill that may fail silently." If the user insists, proceed but document in the output SKILL.md as ## Known Limitations: Evals skipped by user request. Never claim the skill is "complete" without evals — mark it as "scaffold only."

When an Edge Case Is Encountered

[Detection] Repo has non-standard skill format (not agentskills.io spec). Skill content is in a language the classifier can't parse. Multiple repos provided simultaneously. Overlap with existing skill exceeds 80%. [Recovery] For non-standard formats: classify best-effort and flag as "non-standard — manual review recommended." For multiple repos: process sequentially, one pipeline per repo. For high overlap: report percentage to user with option to merge vs replace. Always surface edge cases in the classification report rather than silently handling them.