dianyike/claude-code-insights/security-review-protocol

Security Review Protocol

Core business logic for the dual-verification security review workflow: cross-validation, conflict resolution, and confidence scoring.

Additional resources

• For MCP tool call patterns and parameters, see reference/mcp-tools.md
• For report output template, see templates/report-template.md

1. Cross-Validation Logic

1.1 Finding Normalization

Before comparing, normalize findings from both sources into a common format:

{
  id: "<unique-id>",
  source: "semgrep" | "codex",
  type: "<vulnerability-type>",     // e.g., "sql-injection", "xss", "hardcoded-secret"
  file: "<file-path>",
  line: <line-number>,
  severity: "CRITICAL" | "HIGH" | "MEDIUM" | "LOW",
  description: "<what the issue is>",
  evidence: "<specific code or pattern that triggered this>",
  trigger: "<how it can be exploited>",
  fix: "<remediation suggestion>"
}

1.2 Matching Algorithm

Two findings from different sources are considered matching when:

1. Same file (exact path match)
2. Overlapping location (line numbers within 5 lines of each other)
3. Same vulnerability category (e.g., both are injection-related, both are auth-related)

If criteria 1+3 match but lines differ by >5, flag as partial match — may be the same root cause manifesting at different points.

1.3 Classification

2. Conflict Resolution Protocol

2.1 Divergence Types

Type A: Codex says safe, Semgrep says vulnerable (most dangerous)

Conservative stance: treat as vulnerable until proven otherwise.

Resolution steps:

1. Read the flagged code and its surrounding context (50 lines)
2. Check if there are sanitization functions upstream
3. Run a targeted custom Semgrep rule for the specific pattern
4. If still ambiguous → report as finding with note "Codex disagrees — needs human review"

Type B: Codex says vulnerable, Semgrep says nothing (potential false positive)

Moderate stance: verify before reporting.

Resolution steps:

1. Use Grep to find the exact pattern Codex described
2. Use get_abstract_syntax_tree to verify the code structure
3. If Codex's reasoning is sound and code confirms → report as finding
4. If pattern not found or reasoning flawed → discard with note

Type C: Both found it but disagree on severity or fix

Resolution steps:

1. Prefer Semgrep's CWE/severity mapping (based on established rule databases)
2. Evaluate Codex's reasoning for severity — may have project-specific context
3. Report with the higher severity (conservative) and note the disagreement

2.2 Deep Verification Techniques

When conflicts arise, use these targeted verification methods:

3. Confidence Score Calculation

3.1 Base Weights

3.2 Score Formula

confidence = base_score + modifiers
 
Where:
  base_score:
    - Both agree:        max(60, 40) + 20 = 80%
    - Semgrep only:      60%
    - Codex only:        40%
 
  modifiers:
    - Deep verification confirms:           +15%
    - Semgrep rule has CWE reference:       +5%
    - Codex provided exploit scenario:      +5%
    - Historical finding exists (platform): +10%
    - Conflict unresolved:                  cap at 50%
 
  cap: 95% (never claim absolute certainty)

3.3 Escalation Threshold

• Confidence >= 70%: Report as confirmed finding
• Confidence 50-69%: Report with "needs verification" flag
• Confidence < 50%: Escalate to human review, do not auto-report as confirmed

4. Fix-Verify Loop

Boundary: This section is ONLY executed via /security:fix in the main conversation.

The security-reviewer subagent and all gate hooks MUST stop at Section 3.

Review (Steps 1-3) is read-only; remediation (Step 4) requires explicit user opt-in.

After findings are confirmed (confidence >= 50%), enter the fix-verify loop to remediate and validate fixes.

4.0 Mode Boundary

Sections 1-3 and Section 4 operate under fundamentally different structural guarantees.

• Sections 1-3 (diagnostic): dual-source cross-validation with weighted scoring. Both Semgrep and Codex contribute findings independently, and the 60/40 weights govern classification.
• Section 4 (remediation): each round must have exactly one strategy source. Usually that source is Codex; if 4.1 matches a canonical fix pattern, the source becomes the canonical pattern instead. In either case, Semgrep's role narrows to violation oracle — it can confirm "this rule still fires" or "this rule stopped firing," but it cannot confirm that the root cause is resolved or that the fix direction is correct.

The 60/40 diagnostic weights do NOT apply here. The constraints below exist to compensate for this single-proposer asymmetry — they are not bureaucracy.

4.1 Known Fix Gate

Before asking Codex for a fix strategy, check if the finding matches a canonical fix pattern:

If match found:

1. Apply the canonical fix
2. Still write the prediction block (see 4.2 Constraint 2) — the strategy source changes from Codex to canonical pattern, but the verification contract is identical
3. Open a read-only Codex session (mcp__codex__codex) with the code + applied canonical fix for thread continuity — this ensures codex-reply is available for 4.3 verification
4. Skip to 4.3

If no match → proceed to 4.2.

Rationale: For well-understood vulnerability classes, letting the model "be creative" only increases drift probability. The prediction block is not "for Codex" — it's the falsifiable target that makes verification meaningful.

4.2 Request Fix from Codex

For same-thread refinement, use codex-reply with the existing threadId to request a fix. If 4.4 requires a fresh strategy reset, open a new read-only mcp__codex__codex session using the fresh fix-strategy template in reference/mcp-tools.md. Two hard constraints apply either way:

Constraint 1 — Single Root-Cause Hypothesis

Each round targets exactly ONE root-cause cluster. A single root cause may manifest across multiple call sites — fixing them together in one round is correct.

What's prohibited: mixing unrelated fix directions in one round (e.g., fixing SQL injection AND XSS simultaneously). This isolates cause-effect so that Semgrep evidence is interpretable.

Constraint 2 — Prediction Block (MANDATORY before applying any fix)

Before applying the fix, write down the following prediction. This applies to BOTH the canonical path (4.1) and the Codex path (4.2):

prediction:
  root_cause_hypothesis: "<what you believe is the underlying cause>"
  minimal_change_scope: "<which files:lines will be modified>"
  expected_rules_to_clear: ["<semgrep-rule-id-1>", ...]
  expected_grep_patterns_to_disappear: ["<pattern>", ...]
  possible_new_findings: ["<rule-id or 'none'>"]
  disconfirming_evidence: "<what result would prove this hypothesis WRONG>"
  rollback_trigger: "<specific condition that means revert immediately>"

If the prediction cannot be stated concretely, the fix is too vague to apply. Stop and refine the hypothesis first.

For MCP prompt templates, see reference/mcp-tools.md.

4.3 Apply and Re-verify

After applying the fix:

1. Run Semgrep on modified files — record the actual output
2. Compare actual output against prediction block point-by-point:

• Did expected_rules_to_clear actually clear? (Y/N per rule)
• Did expected_grep_patterns_to_disappear actually disappear? (Y/N per pattern)
• Did any findings appear in possible_new_findings? Expected or unexpected?
• Did the disconfirming_evidence condition trigger?

1. Call Codex verify (using the verify prompt from reference/mcp-tools.md) — Codex must compare against the prediction block, not just re-review the code
2. Classify the round outcome:

• Prediction confirmed: all expected rules cleared, no unexpected findings → finding resolved
• Prediction partially confirmed: some rules cleared, others persist → record as hypothesis refinement evidence, proceed to next round. Constraint: partially confirmed only permits same root-cause family refinement. It does NOT permit switching to a new remediation direction in the same thread without recording the original hypothesis as failed or superseded
• Prediction falsified: disconfirming evidence triggered, or zero predictions matched → this is a hypothesis failure, not an implementation bug

Key framing: Semgrep checks "does the violation pattern still match?" — NOT "is the code now secure."

4.4 Iteration Protocol

Cap at 3 rounds. Each round escalates the strategy diversity requirement:

• Round 1 fails: Stay on same Codex threadId. Inject into the prompt: the failed prediction block, actual Semgrep output, and the specific mismatch. Ask Codex to revise its root-cause hypothesis, not just tweak the fix
• Round 2 fails: MANDATORY strategy switch:
• Abandon the current threadId — start a fresh Codex session
• New prompt MUST include: both failed hypotheses, their predictions, actual outputs, and the explicit instruction: "propose a fundamentally different approach — the previous two attempts targeted [X] and both failed because [Y]"
• Purpose: break narrative inertia from accumulated thread context
• Round 3 fails: Stop all automated repair. Escalate to human with a structured bundle:
• All 3 hypotheses with prediction blocks
• All 3 actual Semgrep outputs
• All 3 prediction-vs-actual comparisons
• Agent's assessment of why convergence failed

4.5 Rollback Rules

NOT a blanket "any new finding → revert." Apply tiered judgment:

4.6 Hypothesis Ledger