tommy-yw/runbookhermes/huggingface-tokenizers
Claude Code and agent skill by tommy-yw/runbookhermes
- community
Checking that the SKILL.md file is still in the repository.
Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.
Install
npx vskill@latest install tommy-yw/runbookhermes/huggingface-tokenizersSKILL.md
The same SKILL.md was published earlier as tommy-yw/runbookhermes/huggingface-tokenizers.
HuggingFace Tokenizers - Fast Tokenization for NLP
Fast, production-ready tokenizers with Rust performance and Python ease-of-use.
When to use HuggingFace Tokenizers
Use HuggingFace Tokenizers when:
- Need extremely fast tokenization (<20s per GB of text)
- Training custom tokenizers from scratch
- Want alignment tracking (token → original text position)
- Building production NLP pipelines
- Need to tokenize large corpora efficiently
Performance:
- Speed: <20 seconds to tokenize 1GB on CPU
- Implementation: Rust core with Python/Node.js bindings
- Efficiency: 10-100× faster than pure Python implementations
Use alternatives instead:
- SentencePiece: Language-independent, used by T5/ALBERT
- tiktoken: OpenAI's BPE tokenizer for GPT models
- transformers AutoTokenizer: Loading pretrained only (uses this library internally)
Quick start
Installation
Load pretrained tokenizer
Train custom BPE tokenizer
Training time: ~1-2 minutes for 100MB corpus, ~10-20 minutes for 1GB
Batch encoding with padding
Tokenization algorithms
BPE (Byte-Pair Encoding)
How it works:
- Start with character-level vocabulary
- Find most frequent character pair
- Merge into new token, add to vocabulary
- Repeat until vocabulary size reached
Used by: GPT-2, GPT-3, RoBERTa, BART, DeBERTa
Advantages:
- Handles OOV words well (breaks into subwords)
- Flexible vocabulary size
- Good for morphologically rich languages
Trade-offs:
- Tokenization depends on merge order
- May split common words unexpectedly
WordPiece
How it works:
- Start with character vocabulary
- Score merge pairs:
frequency(pair) / (frequency(first) × frequency(second)) - Merge highest scoring pair
- Repeat until vocabulary size reached
Used by: BERT, DistilBERT, MobileBERT
Advantages:
- Prioritizes meaningful merges (high score = semantically related)
- Used successfully in BERT (state-of-the-art results)
Trade-offs:
- Unknown words become
[UNK]if no subword match - Saves vocabulary, not merge rules (larger files)
Unigram
How it works:
- Start with large vocabulary (all substrings)
- Compute loss for corpus with current vocabulary
- Remove tokens with minimal impact on loss
- Repeat until vocabulary size reached
Used by: ALBERT, T5, mBART, XLNet (via SentencePiece)
Advantages:
- Probabilistic (finds most likely tokenization)
- Works well for languages without word boundaries
- Handles diverse linguistic contexts
Trade-offs:
- Computationally expensive to train
- More hyperparameters to tune
Tokenization pipeline
Complete pipeline: Normalization → Pre-tokenization → Model → Post-processing
Normalization
Clean and standardize text:
Common normalizers:
NFD,NFC,NFKD,NFKC- Unicode normalization formsLowercase()- Convert to lowercaseStripAccents()- Remove accents (é → e)Strip()- Remove whitespaceReplace(pattern, content)- Regex replacement
Pre-tokenization
Split text into word-like units:
Common pre-tokenizers:
Whitespace()- Split on spaces, tabs, newlinesByteLevel()- GPT-2 style byte-level splittingPunctuation()- Isolate punctuationDigits(individual_digits=True)- Split digits individuallyMetaspace()- Replace spaces with ▁ (SentencePiece style)
Post-processing
Add special tokens for model input:
Common patterns:
Alignment tracking
Track token positions in original text:
Use cases:
- Named entity recognition (map predictions back to text)
- Question answering (extract answer spans)
- Token classification (align labels to original positions)
Integration with transformers
Load with AutoTokenizer
Convert custom tokenizer to transformers
Common patterns
Train from iterator (large datasets)
Performance: Processes 1GB in ~10-20 minutes
Enable truncation and padding
Multi-processing
Speedup: 5-8× with 8 cores
Performance benchmarks
Training speed
| Corpus Size | BPE (30k vocab) | WordPiece (30k) | Unigram (8k) | |
|---|---|---|---|---|
| 10 MB | 15 sec | 18 sec | 25 sec | |
| 100 MB | 1.5 min | 2 min | 4 min | |
| 1 GB | 15 min | 20 min | 40 min |
Hardware: 16-core CPU, tested on English Wikipedia
Tokenization speed
| Implementation | 1 GB corpus | Throughput | |
|---|---|---|---|
| Pure Python | ~20 minutes | ~50 MB/min | |
| HF Tokenizers | ~15 seconds | ~4 GB/min | |
| Speedup | 80× | 80× |
Test: English text, average sentence length 20 words
Memory usage
| Task | Memory | |
|---|---|---|
| Load tokenizer | ~10 MB | |
| Train BPE (30k vocab) | ~200 MB | |
| Encode 1M sentences | ~500 MB |
Supported models
Pre-trained tokenizers available via from_pretrained():
BERT family:
bert-base-uncased,bert-large-caseddistilbert-base-uncasedroberta-base,roberta-large
GPT family:
gpt2,gpt2-medium,gpt2-largedistilgpt2
This is the start of the file. Read the full SKILL.md on GitHub.
Security
- Security scanPassed, 95 of 100Pattern checks on the skill's files.
- Model reviewPassed
Quality evaluation
- Task evaluationNot runNo evaluation yet. Run one in Skill Studio.
Source
- SKILL.md in the repositoryCheckingoptional-skills/mlops/huggingface-tokenizers/SKILL.md
Popularity
- Repository stars
- 547
- Repository forks
- 41
- 7-day trend score
- 32
Stars and forks are for the whole GitHub repository, not this skill alone. Not refreshed yet.
Works with
Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.
- Claude Code
- Cursor
- GitHub Copilot
- Windsurf
- Codex
- Gemini CLI
- Cline
- Amp
- Roo Code
- Goose
- Continue
- Kilo Code
- Junie
- Trae
- OpenHands
- Qwen Code
- AI Agent
- Tabnine Chat
- Sourcegraph Cody
- Amazon Q
- Firebase Genkit
- Sweep
- Devin
- Phind
- Mentat
- Claude Desktop
- Aider
- GPT Pilot
- Plandex
- bolt.new
- v0
- Replit Agent
- Anse
- TypingMind
- BetterChatGPT
- LibreChat
- Warp AI
- Pieces
- Fig AI
Embed badge
[](https://verified-skill.com/skills/tommy-yw/runbookhermes/huggingface-tokenizers)