xorbitsai/xagent/evidence-based-rag

Evidence-Based RAG

Overview

evidence_based_rag is an evidence-first retrieval-augmented reasoning skill designed to retrieve, extract, and organize factual information from a specified knowledge base in support of decision-critical scenarios (e.g., due diligence, compliance review, research analysis).

The responsibility of this skill is to produce traceable, verifiable, and auditable findings, while explicitly surfacing conflicts and uncertainty signals. It does not perform business judgments or replace human decision-making.

Skill Responsibility

This skill is responsible for:

• Retrieving relevant information from a specified knowledge base
• Transforming raw text into verifiable evidence units
• Detecting conflicting claims related to the same factual question
• Evaluating answer sufficiency and confidence
• Explicitly surfacing uncertainty and risk signals, along with suggested follow-up actions

This skill is not responsible for:

• Arbitrating between conflicting claims
• Making investment, legal, or business decisions
• Controlling workflow execution (e.g., retries, looping, escalation)

Design Invariants

The following invariants must hold in any execution environment:

1. Evidence-bound

• All key conclusions must be supported by evidence from the knowledge base
• No external knowledge or common-sense assumptions may be introduced

1. Traceability

• Every material claim must be traceable to a specific location in the original source

(document name, page, section, paragraph, line, or clause)

1. Non-assumptive

• Insufficient or conflicting evidence must be explicitly marked as uncertain
• No gap-filling based on plausibility, convention, or experience

1. Conflict-aware

• Multiple conflicting claims about the same fact must be surfaced with their citations
• Conflicts are treated as risk signals, not automatically resolved

1. Agentic-compatible

• Outputs must expose structured signals usable by downstream agents or humans:

confidence, sufficient, conflicts, suggested_next_actions

1. Subject Existence (Hard Gate)

• A conclusion MUST NOT be produced unless at least one candidate subject that explicitly matches the query subject class is identified in the knowledge base
• If no such subject is found, the system MUST return sufficient = false and MUST NOT substitute semantically similar subjects
• "I don't know" is a valid capability in agent systems; "hard answering" under subject absence is a defect
• This invariant prevents semantic fallback under subject absence
• Applies universally to all subject types: entities, individuals, products, locations, time periods, jurisdictions, etc.

1. Entity Binding

• A candidate subject MUST have an explicit and verifiable binding to the parent entity specified in the query subject class
• Role or title similarity alone is insufficient
• Valid binding mechanisms: ownership, registration, contractual role, or explicit statement in source text
• If the parent entity relationship cannot be explicitly established from the knowledge base, the subject MUST NOT be considered a valid candidate
• This invariant prevents accepting subjects that "look like" matches but lack the required entity relationship

1. Subject-Evidence Integrity

• Any evidence used to support a conclusion MUST explicitly confirm that the subject of the evidence matches the subject of the query
• If subject identity or scope cannot be unambiguously established, the evidence MUST NOT be used to support the conclusion
• Subject drift is a critical error that invalidates the entire conclusion
• This invariant applies universally: companies, individuals, products, locations, time periods, jurisdictions, legal entities, or any other subject type

When to Use

Use this skill when the task requires:

• Knowledge base Q&A - Answering questions based on specific documents or knowledge bases with source attribution
• Evidence verification - Retrieving and verifying factual claims from documents
• Due diligence queries - Investigating facts from multiple sources with conflict detection
• Multi-source analysis - Synthesizing information from multiple documents with explicit citations
• Fact-checking - Verifying claims against source documents

Typical use cases include:

• Knowledge base queries requiring evidence traceability
• Document analysis with source attribution
• Compliance or legal document review
• Research analysis and fact verification

Not suitable for:

• Casual or conversational generation tasks
• Queries that do not require source attribution or traceability
• Creative writing or brainstorming tasks

Inputs & Outputs

This skill does not impose a fixed schema on inputs or outputs.

• Inputs are organized by the calling Agent or runtime (e.g., question, knowledge base scope, constraints).
• Outputs aim to be structured and machine-consumable, but concrete field shapes are determined by the integration context.

Core invariants enforced by the skill:

• All conclusions must be evidence-backed
• Conflicts, uncertainty, and risk signals must be explicitly surfaced

Execution State Machine

This skill implements a strict, irreversible state machine to eliminate non-deterministic execution paths.

States

stateDiagram-v2
    [*] --> INIT
    INIT --> SUBJECT_CLASS_IDENTIFIED
    SUBJECT_CLASS_IDENTIFIED --> CANDIDATE_SUBJECT_DISCOVERED
    CANDIDATE_SUBJECT_DISCOVERED --> SUBJECT_NOT_FOUND: No candidates
    CANDIDATE_SUBJECT_DISCOVERED --> EVIDENCE_RETRIEVAL: Candidates found
    SUBJECT_NOT_FOUND --> [*]: TERMINATE
    EVIDENCE_RETRIEVAL --> EVIDENCE_VERIFIED
    EVIDENCE_VERIFIED --> CONCLUDED
    CONCLUDED --> [*]: TERMINATE

State Definitions:

State Transition Rules (MANDATORY)

1. Linearity: State transitions MUST follow the defined order. No state may be skipped.

1. Irreversibility: Once a state is entered, the system MUST NOT transition back to a previous state.

1. Hard Gate Enforcement:

• CANDIDATE_SUBJECT_DISCOVERED is a HARD GATE
• If candidate_subjects == [], MUST transition to SUBJECT_NOT_FOUND
• SUBJECT_NOT_FOUND is a TERMINAL state - no further execution allowed

1. Forbidden Transitions:

• SUBJECT_NOT_FOUND → any state other than termination
• Any state → SUBJECT_CLASS_IDENTIFIED (re-planning forbidden)
• EVIDENCE_RETRIEVAL → SUBJECT_CLASS_IDENTIFIED (query re-interpretation forbidden)

1. Forbidden Behaviors:

• Skipping CANDIDATE_SUBJECT_DISCOVERED to proceed directly to retrieval
• Relaxing subject constraints after SUBJECT_NOT_FOUND
• Substituting semantically similar subjects when candidates are empty
• Re-interpreting the query to improve answer completeness

Execution Contract

Determinism Guarantee: The same input MUST always produce the same execution path through the state machine.

Termination Guarantee: Execution MUST terminate in one of two terminal states:

• SUBJECT_NOT_FOUND (when no candidate subjects exist)
• CONCLUDED (when evidence-based conclusion is generated)

No-Replay Guarantee: If execution terminates in SUBJECT_NOT_FOUND, the system MUST NOT:

• Retry with alternative search strategies
• Reformulate the query
• Relax subject constraints
• Attempt additional retrievals

Execution Semantics

State 0: INIT

Receive and validate the query.

State 1: Identify Query Subject Class

Entry Condition: State = INIT

Action:

• Analyze the query to identify the subject class and its defining characteristics
• Extract subject attributes: type, scope, constraints, relationships
• Example: "Entity A contractual obligations" → subject_class = {type: legal_entity, name: Entity A, document_type: contract}

Exit Condition: Subject class successfully extracted

Next State: SUBJECT_CLASS_IDENTIFIED → CANDIDATE_SUBJECT_DISCOVERED

State 2: Candidate Subject Discovery (HARD GATE)

Entry Condition: State = SUBJECT_CLASS_IDENTIFIED

Action:

• Search the knowledge base for subjects that match the query subject class
• Apply Candidate Subject Validity Rule (MANDATORY):

A subject S is a valid candidate IFF all conditions are satisfied:

1. S matches the subject type (e.g., individual, entity, product)
2. S matches the subject role (e.g., founder, shareholder, contractor)
3. S has an explicit parent_entity binding that matches the query

• Via ownership (e.g., "founder of Entity A")
• Via registration (e.g., "registered agent for Entity A")
• Via contractual role (e.g., "party to Master Services Agreement with Entity A")
• Via explicit statement in source text (e.g., "Entity A's founder is...")

If condition (3) is NOT satisfied, S MUST be rejected.

• INVALID Examples (role similarity without entity binding):
• Query: "Entity A's founder" → KB has "Company founder" but no binding to Entity A → REJECT
• Query: "Entity A's contractual obligations" → KB has "MSA obligations" but Entity A is not a party → REJECT
• Query: "Supplier X's subsidiaries" → KB has "subsidiaries" but not under Supplier X → REJECT

• Output format: candidate_subjects: [{subject_id, name, class_attributes, parent_entity_binding, source_location}]

Decision Point:

IF `candidate_subjects == []`:

• Set sufficient = false
• Set confidence = low
• Set risk_signal = subject_not_found
• Provide suggested_next_actions for sourcing missing subject information
• Next State: SUBJECT_NOT_FOUND → TERMINATE
• FORBIDDEN: Any further retrieval, query reformulation, or alternative interpretation

IF `candidate_subjects != []`:

• Proceed to evidence retrieval
• Next State: EVIDENCE_RETRIEVAL

State 3: Evidence Retrieval

Entry Condition: State = CANDIDATE_SUBJECT_DISCOVERED AND candidate_subjects != []

Action:

• Retrieve candidate document fragments from the specified knowledge base
• CONSTRAINT: All retrieval queries MUST include identifiers from candidate_subjects
• Filter retrieval to ONLY include documents that reference the identified candidate subjects
• Prefer multiple independent sources to enable cross-verification

Exit Condition: Retrieval completed

Next State: EVIDENCE_VERIFIED

State 4: Evidence Extraction

Entry Condition: State = EVIDENCE_RETRIEVAL

Transform retrieved text into locatable evidence units, ensuring each key claim can be traced back to its original source.

State 5: Subject-Evidence Integrity Verification

Entry Condition: State = EVIDENCE_EXTRACTION

Action:

• For each evidence unit, verify that the subject of the evidence matches the subject of the query
• Extract and compare subject identifiers: entity names, dates, locations, jurisdictions, roles, scopes
• Reject evidence where subject identity or scope cannot be unambiguously confirmed as matching

Next State: CONCLUDED

State 6: Concluded

Entry Condition: State = EVIDENCE_VERIFIED