assafkip/huntkit/osint

OSINT Skill v3.2

Systematic intelligence gathering on individuals. From a name or handle to a scored dossier with psychoprofile, career map, and entry points.

Phase Router

Determine entry point from context:

• New name/handle/URL, "пробей", "find out about" → Phase 0 (full cycle)
• "Add LinkedIn/Instagram data" to existing dossier → Phase 2 (extraction)
• "Build psychoprofile" from existing data → Phase 4
• "Rate completeness" of existing dossier → Phase 5
• "Reformat" or "present" findings → Phase 6

Default (full research request): Phase 0 → 1 → 1.5 → 2 → 3 → 4 → 5 → 6.

Environment

All API keys via environment variables. Never hardcode tokens.

• PERPLEXITY_API_KEY — Perplexity Sonar (fast answers + deep research)
• EXA_API_KEY — Exa AI (semantic search, company/people research, deep research)
• TAVILY_API_KEY — Tavily (agent-optimized search + extract, $0.005/req basic)
• APIFY_API_TOKEN — Apify scraping (LinkedIn, Instagram, Facebook)
• JINA_API_KEY — Jina reader/search/deepsearch
• PARALLEL_API_KEY — Parallel AI search
• BRIGHTDATA_MCP_URL — Bright Data MCP endpoint (full URL with token)
• MCPORTER_CONFIG — mcporter config path

Enrichment adapters (scripts/osint-tool.py, all optional; keyless paths work without them). python3 scripts/osint-tool.py --list shows what each adapter is missing.

• Infra / reputation: SHODAN_API_KEY (else keyless InternetDB), URLSCAN_API_KEY, OTX_API_KEY, GREYNOISE_API_KEY, ABUSEIPDB_API_KEY, VT_API_KEY (ioc-reputation.py), CERTSPOTTER_API_KEY
• Records / sanctions: OPENCORPORATES_API_KEY, OPENSANCTIONS_API_KEY or YENTE_URL (local yente), COURTLISTENER_API_TOKEN
• Crypto: ETHERSCAN_API_KEY, TRONGRID_API_KEY
• Identity: GITHUB_TOKEN, GITLAB_TOKEN, API_ID + API_HASH + PHONE_NUMBER (+ TELEGRAM_SESSION_DIR) for telegram_phone; OSINT_USERNAME_WORKERS, OSINT_USERNAME_SITE_TIMEOUT
• /q-corp (install-pp.sh): COMPANY_PP_CONTACT_EMAIL, COMPANIES_HOUSE_API_KEY, FEC_API_KEY_QUERY_AUTH, LDA_API_KEY
• Images: GOOGLE_VISION_API_KEY (reverse-image.sh, /q-face Vision backend), SEARCH4FACES_API_KEY, FACECHECK_API_KEY, SHADOWFINDER_CACHE
• Intake NER: GLINER_MODEL; LANGEXTRACT_API_KEY or GEMINI_API_KEY + LANGEXTRACT_MODEL (langextract SENDS TEXT TO GEMINI -- not for client documents without approval)
• Services (tools/docker-compose.yml): CRAWL4AI_URL (default http://localhost:11235), CRAWL4AI_API_TOKEN, CRAWL_MIN_CHARS, CHANGEDETECTION_URL + CHANGEDETECTION_API_KEY, ARCHIVEBOX_URL + ARCHIVEBOX_API_KEY, SPIDERFOOT_URL, AUTO_ARCHIVER_CONFIG, SOCIAL_TIMEOUT_SEC
• Endpoint overrides: OSINT_DOH_URL, OSINT_ETH_RPC_URL, OSINT_SOLANA_RPC_URL, OSINT_ENS_RESOLVER, OSINT_BLOCKSCOUT_URL, OSINT_CACHE_DIR, PD_HTTPX_BIN
• Telegram harvest: TGSPYDER_SRC. Opt-in hooks: Q_SCOPE_HOOK, Q_SCOPE_EXTRA, Q_BUDGET_HOOK, Q_BUDGET_FILE, Q_WORKSPACE (see scripts/hooks/README.md)

Install optional tools: bash scripts/setup-tools.sh --list (then --only <name>, --all, --docker <svc>).

Scripts

Run from skill dir: bash scripts/<name>.sh. Each script validates env vars, exits with descriptive error + URL to get the key.

Client-provided document intake (do this before any custom extraction script):

• ingest-client-document.sh <file> <slug> <document|screenshot_only> --case <case> — registers a client-provided file as an EV-NNNN evidence item (SHA-256 + chain-of-custody). See .claude/rules/evidence-capture-protocol.md.
• extract-intake.py --case <case> — deterministic verbatim extraction (no LLM, no network) for everything in investigation/intake/. Handlers: PDF, DOCX, DOC, CSV, PNG/JPG/TIFF, EML. No .md handler yet (markdown intake is already plain text — copy verbatim into evidence/extracted/<stem>/text.md by hand if the pipeline guard needs it satisfied).
• evidence-pipeline-guard.py (wired as a hook) blocks writing/running a custom script under a case's investigation/evidence/scripts/ until both of the above have run for that case's intake. A case-local extraction script should only exist for a genuinely new capability (e.g. platform-specific OCR, a non-portable OS-native OCR engine) — upstream anything generally useful into extract-intake.py instead of duplicating the scaffold per case (lesson from the 2026-07-21 extraction-dedup audit: 2 cases had reinvented Maltego-CSV classification, 5 more had reinvented file extraction).

Search & Research:

• diagnose.sh — run FIRST. Capability map of all tools.
• Perplexity (first-pass default): when operating through Claude, call the MCP tool mcp__perplexity-ask__perplexity_ask directly. It returns an AI answer with citations (equivalent to sonar mode). Use the shell script for search (ranked web results), reason (reconcile contradictions), and deep (long-form research) modes, or from bash pipelines.
• perplexity.sh — search <query> | sonar <query> (AI answer) | reason <query> (sonar-reasoning-pro, compare leads / reconcile contradictions) | deep <query> (deep research). Required by first-volley.sh and any bash-only pipeline.
• tavily.sh — search <query> (basic $0.005) | deep <query> (advanced) | extract <url>
• exa.sh — search <query> | company <name> | people <name> | crawl <url> | deep <prompt>
• first-volley.sh "Name" "context" — parallel search, all engines at once.
• merge-volley.sh <outdir> — deduplicate and merge first-volley results.

Scraping:

• apify.sh — linkedin <url> | instagram <handle> | run | results | store-search
• run-actor.sh — universal Apify runner (55+ actors). Embedded from apify/agent-skills.

Quick answer: bash scripts/run-actor.sh "actor/id" '{"input":"json"}' Export: bash scripts/run-actor.sh "actor/id" '{"input":"json"}' --output /tmp/out.csv

• jina.sh — read <url> | search <query> | deepsearch <query>
• parallel.sh — search <query> | extract <url>
• brightdata.sh — scrape <url> | scrape-batch | search | search-geo <cc> | search-yandex

Setup & diagnostics:

• diagnose.sh -- keys, binaries, capabilities, and the osint-tool.py adapter summary (ready/missing/import errors).
• setup-tools.sh -- --list | --only <name>[,<name>] | --all | --docker <svc>[,<svc>] [--dry-run]. pipx/pip/go/cargo installs; system packages are printed, never sudo'd.
• dead-hooks-lint.py -- flags hooks in settings files whose script is missing.
• mcp-client.py <mcp_url> --list-tools -- list an MCP server's tools.

Enrichment (adapters, every result carries its tier):

• osint-tool.py --list | osint-tool.py <slug> <query> [--mode M] [--json] [--case <case>|--save] -- 54 adapters (infra, records, sanctions, crypto, identity, reputation). Table: references/tools.md "Enrichment adapters". Same adapters via MCP osint-infra tools enrich / enrich_providers.
• identity-sweep.py <handle|email|phone> [--case <case>] [--only a,b] [--skip a,b] -- every ready identity adapter on one value, merged, T3. Shortcut: identity-lookup.sh sweep <value> [case].
• identity-lookup.sh keybase|github|mastodon|leakcheck <query> -- keyless single lookups.
• reddit-arctic.py user|sub|thread <x> [--case <case>] -- Reddit history: Arctic Shift, then PullPush, then RSS (reddit.com JSON 403s from servers).
• install-pp.sh list|install|run <name> --case C -- <args> -- printing-press public-records CLIs (SEC EDGAR, Form D, Companies House, FEC, LDA, FAA, EU tenders) for /q-corp.
• ioc-reputation.py extracted/<doc>/iocs.json [--sources ...] [--out rep.csv] -- reputation leads for extracted indicators (T3; exposes indicators to providers).

Crawl, archive, watch:

• crawl.sh fetch <url> [--case C] | crawl.sh map <url> [--depth N] -- crawl4ai, then Scrapling; fail-closed on empty/challenge pages. map uses katana.
• capture-evidence.sh ... [--social] [--archivebox] -- additive media download (yt-dlp / gallery-dl / auto-archiver) and ArchiveBox copy; never flips an item's status, never edits source.json.
• watch.py add|list|poll|remove -- page monitoring (local diff or changedetection.io) for /q-watch.

Images & geo:

• reverse-image.sh <image|url> [--case C] -- Google Vision web detection (GOOGLE_VISION_API_KEY).
• shadowfinder.sh find|sun ... [--case C] -- bellingcat ShadowFinder shadow-length geolocation.
• face-search.py -- /q-face pipeline (founder approval required; hits are T3).

Intake, analysis, exports:

• extract-intake.py --case <case> [--ner gliner|langextract|spacy] -- also writes iocs.json, entities.json, exif.json side files (T3 leads).
• resolve-entities.py <csv/json ...> --out same_as.csv [--ftm out.jsonl] -- entity resolution (splink or fallback).
• validate-citations.py <report.md> --case <case> -- link liveness (HEAD/GET) plus a check that the citing sentence's hard tokens appear in the captured EV item.
• export-intel.py stix|misp|csv|all --case <case> [--tlp ...] -- threat-intel exports.
• render-timeline.py --case <case> [--timesketch] -- HTML timeline (or Timesketch CSV).
• render-map.py --case <case> [--csv FILE] -- map of geotagged evidence.
• redact-pii.py <file.md> [--engine auto|presidio|regex] -- writes a redacted copy for /q-export --redact.
• render-report.py, resolve-deliverable.py, findings-verify-hook.py -- report rendering and pre-delivery checks (see .claude/rules/).

Related skills (siblings of this one under `skills/`):

• geo-sleuth/ -- geolocate / chronolocate a photo, outputs routed into EV-NNNN items.
• verification/ -- SIFT, C2PA provenance (c2patool), account authenticity, coordinated-behavior checks.
• telegram-harvest/ -- no-LLM Telegram channel harvest to XLSX (setup.sh once, then scripts/harvest.py).
• leak-record-classifier/, structured-analysis/ -- unchanged.

Research Escalation Flow

Принцип: от дешёвого к дорогому, от быстрого к глубокому.

Level 1: Quick Answers (секунды, ~$0.00)

Начни ВСЕГДА с этого. Получи быстрый контекст прежде чем копать. Запускай ВСЕ параллельно:

# Perplexity (default: MCP tool, not the shell script)
#   Claude call: mcp__perplexity-ask__perplexity_ask with a structured prompt
#   (see "Prompt Templates" section — use the Entity Profile template, not ad-hoc "Who is X")
# Shell fallback (bash pipelines or when you need sonar explicitly):
#   bash skills/osint/scripts/perplexity.sh sonar "<structured prompt from Entity Profile template>"
# Brave Search — классический поиск
web_search "<Name> <company> <role>"
# Tavily — agent-optimized search с AI answer
bash skills/osint/scripts/tavily.sh search "<Name> <context>"
# Exa — семантический поиск + company/people research
bash skills/osint/scripts/exa.sh search "<Name> <context>"
bash skills/osint/scripts/exa.sh people "<Name>"

→ Получаешь: быстрые факты, ссылки, контекст. → Решение: достаточно? → Phase 6. Нужно больше? → Level 1.5.

Level 1.2: Keyless records + identity sweep (free, run alongside Level 1)