githubnext/gh-aw-workshop/markdown-chunker

Markdown Chunker Skill

Use this skill when you need to understand how the markdown-dedup workflow parsed markdown files into sections and what the chunk data means.

Parsing Strategy

The workflow uses a heading-based AST chunker that processes each file in two passes:

1. Primary (mistletoe AST): If mistletoe is importable, the file is parsed

to a full AST and sections are extracted from Heading nodes. This gives accurate text content free of raw markdown syntax characters.

1. Fallback (regex): If mistletoe is unavailable, a regex-based parser splits

on heading lines (^#{1,6}\s+) while tracking fenced code blocks to avoid false heading matches inside code examples.

Both approaches produce the same output schema.

Chunk Schema

Each chunk object in md-chunks.json has:

{
  "file": "workshop/07-your-first-workflow.md",
  "level": 2,
  "title": "Steps",
  "heading_path": ["Write Your First Workflow", "Steps"],
  "content": "...",
  "text": "Steps ...",
  "word_count": 142
}

Cluster Schema

Each cluster object in md-clusters.json has:

{
  "id": "cluster-42",
  "size": 3,
  "files": ["workshop/07-your-first-workflow.md", "workshop/11-build-daily-status.md"],
  "max_similarity": 0.82,
  "chunks": [ /* chunk objects as above */ ]
}

Similarity Algorithm

1. Tokenize each chunk's text (lowercase, alphabetical words ≥ 3 characters).
2. Compute IDF over all chunks: idf(t) = log(N / (1 + df(t))).
3. Build TF-IDF vectors per chunk.
4. Score all cross-file pairs; keep pairs with cosine similarity ≥ 0.55.
5. Union-Find groups overlapping pairs into clusters.
6. Discard clusters where all members belong to the same file.

Interpreting Similarity Scores

When max_similarity is in the lower band, read both sections in full before recommending consolidation.