eliasoulkadi/shokunin/research

research · 調査

調査 · ちょうさ — "investigation". Source verification and fact-checking for document generation.

Pre-loads facts before writing. Runs BEFORE humanize + kami + kagen in the document pipeline. Prevents hallucinated data, fake citations, unverified claims.

Based on: CIA Structured Analytic Techniques (Heuer & Pherson), US Government Tradecraft Primer, NPR Training verification guide, Princeton triangulation methodology, OSINT verification tiers (War Intel Hub), journalistic cross-verification research (Godler & Reich), AI hallucination benchmarks (Vectara HHEM 2026), and evidence hierarchy frameworks (NHMRC).

Core principle

Sources before phrasing. Do not write a claim without verifying it first. Every number, date, name, version, and citation must be traced to a primary or reputable secondary source.

Evidence hierarchy (adapted from NHMRC + intelligence community)

Use this hierarchy to determine the weight of each source:

Rule: a professional document only uses levels 1–4 for factual claims. Level 5 for context or direct quotes, marked as such. Level 6 is not published.

Source verification protocol (NPR + Princeton triangulation)

Step 1: The direct knowledge test (before citing)

Ask about each source:

• First-hand: Did the source witness the event, participate directly, or have documentation? → Strong, but requires corroboration
• Second-hand: Did the source hear it from someone else? → Useful for leads, insufficient to publish
• Third-hand+: Rumor, hearsay, speculation? → Not publishable as fact

Step 2: Triangulation (Princeton method)

Cross-check each claim against multiple independent sources:

Claim: "WordPress 6.7 has vulnerability X"
  → Source A: official WordPress advisory
  → Source B: entry in NVD/CVE
  → Source C: Wordfence or similar analysis
  → Do they agree? → VERIFIED
  → Only one source? → NOT VERIFIED

Step 3: Bias and agenda assessment (NPR)

For each source, identify:

• Financial interest: Does the source benefit financially from a particular narrative?
• Professional interest: Does their reputation/career depend on a certain outcome?
• Personal relationships: Friends, family, enemies of the subject?
• Funding: Who funds the source? What interests does that funder have?

Step 4: The five-minute background check (NPR)

When time is short:

1. Read the source's bio and "About" page
2. Identify who funds or backs them
3. Look for behavior patterns or prior claims
4. Verify if there are visible conflicts of interest

Verification tiers (adapted from OSINT + War Intel Hub)

Every claim in the document must have an assigned tier:

Rule: in professional documents, everything published without a mark must be VERIFIED or CORROBORATED. UNVERIFIED is omitted or declared as such.

Anti-hallucination checklist (based on Vectara HHEM benchmarks 2026)

LLMs hallucinate 15-20% of the time on factual queries (source: Prompt Guardrails, Vectara benchmark March 2026). Reasoning models hallucinate more than standard ones in summarization (DeepSeek-R1: 14.3% vs V3: 6.1%).

Before writing any claim, run it through this filter:

Common categories of AI-invented data (research flags)

Context adaptation by document type

Not all documents need the same verification level. Adjust by type:

Rule: if the document goes to an external client or has legal/security implications, always use maximum verification.

Document pipeline integration

• A specific software version → check official changelog
• A CVE or vulnerability → search NVD + official advisory
• A statistic or metric → find primary source or don't include
• A verbatim quote → confirm it exists
• An event date → verify in official source
• A person/tool/company name → verify it exists and is spelled correctly

Sources

• Heuer & Pherson, Structured Analytic Techniques for Intelligence Analysis (CIA Sherman Kent School)
• US Government, A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis (2009)
• NPR Training, "Don't just check the facts, check the source: a guide to verification" (March 2026)
• Princeton University Library, guide "Triangulation and Media Literacy" (2025)
• War Intel Hub, "OSINT Verification Methodology"
• Vectara HHEM hallucination benchmark leaderboard (March 2026)
• Prompt Guardrails, "AI Hallucination Detection and Prevention Guide" (2026)
• News Factory, "News Fact-Checking in 2026: Hallucination Benchmarks, RAG, and Verification Tools"
• Reuters / AP sourcing standards
• GlobalX Publications, "Fact-Checking, Triangulation, and Evidence Reliability in Research"
• NHMRC evidence hierarchy framework

Workflow

1. Define research scope — identify exactly what claims need verification: numbers, dates, versions, names, quotes, CVE IDs, statistics.
2. Search primary sources first — official changelogs, CVE entries (nvd.nist.gov), SEC filings, source code repositories, direct HTTP responses.
3. Triangulate across independent sources — cross-check each claim against 3+ unrelated sources. Do they independently agree? If only one source, mark as UNVERIFIED.
4. Assess source credibility — financial interest, professional stake, funding bias, personal relationships. Apply the five-minute background check from NPR methodology.
5. Assign verification tiers — VERIFIED (2+ primary sources), CORROBORATED (multiple sources, no primary), UNVERIFIED (single source). Tag every claim.
6. Run anti-hallucination filter — verify every number, date, version, and quote against the Vectara HHEM checklist before writing.
7. Adapt verification level to document type — maximum for pentest/security reports and CVs. Medium for white papers and exec summaries. Lower for personal letters.

Error Handling

Anti-Patterns