volcengine/searchcli/vs-crawler

Viking Content Crawler

When to Use

Use this skill when the user wants to crawl content from websites and import it into Viking AI Search to build a searchable knowledge base. This covers news sites, blogs, academic papers, GitHub repositories, product documentation, RSS feeds, and similar web content sources.

The agent writes crawler code tailored to the target sites, outputs data in a fixed JSONL schema, and then hands off to the vs-item-onboarding skill for dataset creation and import.

Do not use this skill when:

• The user already has a local file ready to import (use vs-item-onboarding directly).
• The user wants to import from a database (use vs-item-onboarding directly with MySQL).

Version Check

Before starting this skill workflow, run vs version check --json. Continue only when status is up-to-date. If status is update-available, stop and tell the user to update the cloned vs repository, then run git pull --ff-only, bash ./scripts/install.sh, and bash ./scripts/install-skills.sh all --target auto --force (PowerShell: scripts/install.ps1 and scripts/install-skills.ps1). If the status is unknown, stop and report that the CLI version could not be verified.

Fixed Schema

All crawled records MUST conform to this schema. Every record is a flat JSON object written as one line in a JSONL file.

Example Record

{
  "id": "viking-blog-introducing-viking-ai-search",
  "title": "Introducing Viking AI Search",
  "summary": "Viking AI Search is a new generation of hybrid search engine combining BM25 and vector search...",
  "content": "Full article text with HTML removed and paragraphs separated by newlines...",
  "category": "blog",
  "source": "Viking Blog",
  "author": "Jane Doe",
  "published_at": "2026-07-15T08:00:00Z",
  "tags": ["search", "vector database", "hybrid search"],
  "language": "en",
  "source_url": "https://viking.example.com/blog/introducing-viking-ai-search",
  "metadata": {
    "read_time": "8 min",
    "word_count": 2340,
    "views": 12580
  }
}

Standard Metadata Fields

Only these keys are allowed in `metadata`. Do not add custom keys — every crawled record, regardless of source or category, must use exactly these keys when the data is available, and omit keys whose data is unavailable. This guarantees schema consistency across all crawl sources so downstream consumers (schema inference, search relevance tuning) see a uniform shape.

Values must be flat scalars (string / number / boolean). No nested objects, no arrays. If a data point does not map to any standard key, omit it rather than inventing a new key.

Preconditions

• vs CLI >= 0.2.0 is installed and authenticated (vs auth status and vs doctor succeed).
• The crawl target is reachable from the execution environment.
• A suitable runtime is available (Python 3.8+ with requests and beautifulsoup4 recommended).

Commands

This skill delegates dataset creation and import to vs-item-onboarding. The crawler workflow itself uses:

Workflow

Run in strict order.

1. Confirm crawl mode — resolve whether the user wants one-time crawl or scheduled recurring crawl. Only skip the question when the request contains an explicit, unambiguous signal (apply detection to whatever language the user is writing in):

• Explicit one-time: phrases carrying "once", "one-time", "just this time", or equivalent single-crawl semantics.
• Explicit scheduled: phrases carrying "daily", "scheduled", "keep updated", "auto-crawl", "sync", "incremental", or equivalent recurring semantics.
• If the request is neutral — e.g. "crawl X", bare "crawl", mentions target sites but says nothing about scheduling/once — you MUST ask the user to choose. The bare crawl verb is NOT a one-time signal; it is ambiguous. Never silently default to one-time.

1. Identify crawl targets and write the crawler. Based on the user's target sites, write a crawler script. The crawler MUST:

• Output records conforming to the Fixed Schema as JSONL (one record per line).
• Write output to a stable path: /tmp/viking/crawler/<job-name>/items.jsonl.
• For scheduled mode: support incremental crawling — track the last crawl cursor (most recent published_at or last seen item IDs) in /tmp/viking/crawler/<job-name>/state.json so subsequent runs only fetch new content.
• Deduplicate by id within each run and against previous state.
• Strip HTML to plain text; never include raw HTML in content.
• When encountering PDF, DOC, or other document links, download the document and extract its text content directly into the content field. Use available libraries (e.g. PyPDF2/pypdf for PDF, python-docx for DOCX, beautifulsoup4 for HTML) to extract readable text. Do not store document links in records; put the extracted full text in content.
• Be polite: set a descriptive User-Agent, respect robots.txt, add 1-3 second delays between requests, retry transient errors with backoff.
• Prefer structured sources (RSS/Atom feeds > sitemap.xml > official APIs > HTML scraping).
• Log per-item errors and continue; do not abort on single-page failures.
• Strictly follow the Fixed Schema defined above — the same field names, types, and metadata key set, regardless of the source. Do not add source-specific top-level fields or metadata keys. Print a summary to stdout: crawled count, new count, output path.

1. Run the crawler to produce the initial JSONL file at /tmp/viking/crawler/<job-name>/items.jsonl.

1. Hand off to vs-item-onboarding. Invoke the vs-item-onboarding skill with the following context:

• Source type: JSONL file
• File path: /tmp/viking/crawler/<job-name>/items.jsonl
• Import mode: one-time import if the user chose one-time crawl; one-time import + ongoing incremental sync if the user chose scheduled crawl.
• App creation: required — the user wants both a dataset AND an application so the crawled content is immediately searchable. Tell vs-item-onboarding to run through app creation and dataset attachment (steps 12–13) rather than stopping after dataset creation.
• Schema confirmation: auto-confirm — the crawler outputs a fixed, well-defined schema (see Fixed Schema above). When vs-item-onboarding reaches the Schema Confirmation step (step 7), automatically reply yes to proceed without surfacing the confirmation prompt to the user. Only surface it if the backend returns warnings that indicate actual schema problems (e.g. missing PK BizAttr).
• Readiness: do NOT block waiting for Ready. After vs-item-onboarding prints its hand-off block with console links, the workflow is complete. Do not run vs app wait-ready, do not poll for readiness, do not add any extra waiting steps. The user will check the console themselves.
• Let vs-item-onboarding handle all subsequent steps (schema inference, confirmation, dataset creation, data write, app creation, dataset attach, optional sync start, console hand-off).
• Do NOT re-implement the onboarding steps yourself — defer entirely to vs-item-onboarding.

1. (Scheduled mode only) Set up recurring crawl + sync. After vs-item-onboarding completes successfully and the dataset is created:

• The JSONL connector sync (set up by vs-item-onboarding during step 4) already watches the JSONL file for new lines and imports them automatically. You do NOT need to separately configure vs connector init/run for the file — vs-item-onboarding handles this when it chooses the sync path.
• Create a wrapper script that:

1. Runs the crawler in incremental mode (using state.json to skip already-crawled content), appending new records to /tmp/viking/crawler/<job-name>/items.jsonl.
2. Exits cleanly if no new records are found.

• Schedule the wrapper script using the platform-appropriate mechanism:
• cron (macOS/Linux): add a crontab entry. Recommended interval: 30 minutes to a few hours depending on how frequently the source updates.
• launchd (macOS): create a LaunchAgent plist with StartInterval.
• Surface the schedule info, log file path, and how to stop/inspect the job in the hand-off.

Customer Environment Principle

• In customer environments, assume repository source code is unavailable.
• Execute tasks using only the installed skills, the packaged vs CLI surface (--help, command output, observed runtime behavior), and explicit user-provided information.
• If the installed CLI behavior conflicts with a skill, trust the installed CLI behavior first.

Constraints