tommy-yw/runbookhermes/nemo-curator
Claude Code and agent skill by tommy-yw/runbookhermes
- community
Checking that the SKILL.md file is still in the repository.
GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora.
Install
npx vskill@latest install tommy-yw/runbookhermes/nemo-curatorSKILL.md
The same SKILL.md was published earlier as tommy-yw/runbookhermes/nemo-curator.
NeMo Curator - GPU-Accelerated Data Curation
NVIDIA's toolkit for preparing high-quality training data for LLMs.
When to use NeMo Curator
Use NeMo Curator when:
- Preparing LLM training data from web scrapes (Common Crawl)
- Need fast deduplication (16× faster than CPU)
- Curating multi-modal datasets (text, images, video, audio)
- Filtering low-quality or toxic content
- Scaling data processing across GPU cluster
Performance:
- 16× faster fuzzy deduplication (8TB RedPajama v2)
- 40% lower TCO vs CPU alternatives
- Near-linear scaling across GPU nodes
Use alternatives instead:
- datatrove: CPU-based, open-source data processing
- dolma: Allen AI's data toolkit
- Ray Data: General ML data processing (no curation focus)
Quick start
Installation
Basic text curation pipeline
Data curation pipeline
Stage 1: Quality filtering
Stage 2: Deduplication
Exact deduplication:
Fuzzy deduplication (16× faster on GPU):
Semantic deduplication:
Stage 3: PII redaction
Stage 4: Classifier filtering
GPU acceleration
GPU vs CPU performance
| Operation | CPU (16 cores) | GPU (A100) | Speedup | |
|---|---|---|---|---|
| Fuzzy dedup (8TB) | 120 hours | 7.5 hours | 16× | |
| Exact dedup (1TB) | 8 hours | 0.5 hours | 16× | |
| Quality filtering | 2 hours | 0.2 hours | 10× |
Multi-GPU scaling
Multi-modal curation
Image curation
Video curation
Audio curation
Common patterns
Web scrape curation (Common Crawl)
Distributed processing
Performance benchmarks
Fuzzy deduplication (8TB RedPajama v2)
- CPU (256 cores): 120 hours
- GPU (8× A100): 7.5 hours
- Speedup: 16×
Exact deduplication (1TB)
- CPU (64 cores): 8 hours
- GPU (4× A100): 0.5 hours
- Speedup: 16×
Quality filtering (100GB)
- CPU (32 cores): 2 hours
- GPU (2× A100): 0.2 hours
- Speedup: 10×
Cost comparison
CPU-based curation (AWS c5.18xlarge × 10):
- Cost: $3.60/hour × 10 = $36/hour
- Time for 8TB: 120 hours
- Total: $4,320
GPU-based curation (AWS p4d.24xlarge × 2):
- Cost: $32.77/hour × 2 = $65.54/hour
- Time for 8TB: 7.5 hours
- Total: $491.55
Savings: 89% reduction ($3,828 saved)
Supported data formats
- Input: Parquet, JSONL, CSV
- Output: Parquet (recommended), JSONL
- WebDataset: TAR archives for multi-modal
Use cases
Production deployments:
- NVIDIA used NeMo Curator to prepare Nemotron-4 training data
- Open-source datasets curated: RedPajama v2, The Pile
References
- [Filtering Guide](references/filtering.md) - 30+ quality filters, heuristics
- [Deduplication Guide](references/deduplication.md) - Exact, fuzzy, semantic methods
Resources
- GitHub: https://github.com/NVIDIA/NeMo-Curator ⭐ 500+
- Docs: https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/
- Version: 0.4.0+
- License: Apache 2.0
Security
- Security scanPassed, 90 of 100Pattern checks on the skill's files.
- Model reviewPassed
Quality evaluation
- Task evaluationNot runNo evaluation yet. Run one in Skill Studio.
Source
- SKILL.md in the repositoryCheckingoptional-skills/mlops/nemo-curator/SKILL.md
Popularity
- Repository stars
- 547
- Repository forks
- 41
- 7-day trend score
- 32
Stars and forks are for the whole GitHub repository, not this skill alone. Not refreshed yet.
Works with
Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.
- Claude Code
- Cursor
- GitHub Copilot
- Windsurf
- Codex
- Gemini CLI
- Cline
- Amp
- Roo Code
- Goose
- Continue
- Kilo Code
- Junie
- Trae
- OpenHands
- Qwen Code
- AI Agent
- Tabnine Chat
- Sourcegraph Cody
- Amazon Q
- Firebase Genkit
- Sweep
- Devin
- Phind
- Mentat
- Claude Desktop
- Aider
- GPT Pilot
- Plandex
- bolt.new
- v0
- Replit Agent
- Anse
- TypingMind
- BetterChatGPT
- LibreChat
- Warp AI
- Pieces
- Fig AI
Embed badge
[](https://verified-skill.com/skills/tommy-yw/runbookhermes/nemo-curator)