paishu-ai/pai-zong-nb-v5.8/huggingface-tokenizers

Claude Code and agent skill by paishu-ai/pai-zong-nb-v5.8

Trust tier 3
  • community
Source check pending

Checking that the SKILL.md file is still in the repository.

Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.

Install

npx vskill@latest install paishu-ai/pai-zong-nb-v5.8/huggingface-tokenizers

SKILL.md

The same SKILL.md was published earlier as paishu-ai/pai-zong-nb-v5.8/huggingface-tokenizers.

huggingface-tokenizers

Purpose

Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.

Use When

  • The user request matches huggingface-tokenizers, huggingface tokenizers, mlops or the skill description.
  • Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.
  • The task benefits from domain-specific workflow, checks, output structure, or safe fallback behavior.

Workflow

  1. Identify model/task, dataset, environment, compute, dependencies, and expected artifact.
  2. Prefer local inspection and minimal reproducible commands before expensive training or cloud use.
  3. Require confirmation for GPU spend, cloud resources, downloads, uploads, or long-running jobs.
  4. Report metrics, checkpoints, configs, and limitations clearly.

Codex Runtime Rules

  • Use only tools, plugins, MCP servers, files, commands, and credentials that are actually available in the current Codex session.
  • If an external write, install, publish, deploy, paid action, credential use, account operation, physical-device action, or production action is needed, ask for explicit confirmation first.
  • If the required tool is unavailable, return a portable fallback such as Markdown, CSV, prompt pack, implementation plan, config sketch, or verification checklist.
  • Do not claim files, media, issues, API calls, payments, deployments, transactions, messages, or integrations were created unless there is concrete local or tool evidence.
  • Treat source repository runtime details as historical source context, not active Codex capabilities.

Avoid

  • Do not start paid compute, large downloads, training jobs, or data uploads without confirmation.
  • Do not claim model quality without evaluation evidence.

Output

Return the useful artifact first, then evidence and verification status.

For non-trivial work, include:

markdown
Result: ...
Evidence: ...
Verification: PASS/FAIL/NOT RUN
Risks / Next Steps: ...

Security

  • Security scanPassed, 90 of 100Pattern checks on the skill's files.
  • Model reviewPassed

Quality evaluation

  • Task evaluationNot runNo evaluation yet. Run one in Skill Studio.

Source

Popularity

Repository stars
1
Repository forks
1
7-day trend score
8

Stars and forks are for the whole GitHub repository, not this skill alone. Not refreshed yet.

Works with

Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.

  • Claude Code
  • Cursor
  • GitHub Copilot
  • Windsurf
  • Codex
  • Gemini CLI
  • Cline
  • Amp
  • Roo Code
  • Goose
  • Continue
  • Kilo Code
  • Junie
  • Trae
  • OpenHands
  • Qwen Code
  • AI Agent
  • Tabnine Chat
  • Sourcegraph Cody
  • Amazon Q
  • Firebase Genkit
  • Sweep
  • Devin
  • Phind
  • Mentat
  • Claude Desktop
  • Aider
  • GPT Pilot
  • Plandex
  • bolt.new
  • v0
  • Replit Agent
  • Anse
  • TypingMind
  • BetterChatGPT
  • LibreChat
  • Warp AI
  • Pieces
  • Fig AI
Embed badge
vskill VERIFIED badge
[![vskill VERIFIED](https://verified-skill.com/api/v1/skills/paishu-ai/pai-zong-nb-v5.8/huggingface-tokenizers/badge)](https://verified-skill.com/skills/paishu-ai/pai-zong-nb-v5.8/huggingface-tokenizers)

Something wrong with this skill or its page? Tell us.