adrianco/retort/file-run-issues

File Run Issues

Overview

evaluate-run produces a findings.jsonl per run — one JSON object per observation. This skill aggregates those findings into assessment.json, a compact machine-readable summary used by FindingsScorer and downstream reporting. It does not create beads issues or GitHub issues.

Parameters

• run_dir (required): Same path evaluate-run used, e.g. experiment-1/runs/language=rust_model=opus_tooling=beads/rep2/
• min_severity (optional, default: info): Skip findings below this severity when computing counts. Options: critical, high, medium, low, info
• dry_run (optional, default: false): Print the assessment JSON without writing assessment.json

Steps

1. Load findings

test -f {run_dir}/findings.jsonl || { echo "no findings.jsonl — run evaluate-run first"; exit 1; }

Read the file line-by-line. Each line is one finding with at minimum these fields:

{"id": "R3", "kind": "requirement_missing", "severity": "high", "title": "...", "evidence": "...", "suggestion": "..."}

2. Filter by severity

Drop findings whose severity is below min_severity. The severity ordering from highest to lowest is: critical, high, medium, low, info.

3. Count severities

Count how many findings fall into each severity bucket:

{"critical": 0, "high": 2, "medium": 5, "low": 3, "info": 1}

4. Compute penalty_score

start = 1.0
subtract: critical * 0.25 + high * 0.10 + medium * 0.03 + low * 0.01
clamp result to [0.0, 1.0]

A run with no findings scores 1.0. A run with one critical finding scores 0.75. A run with four critical findings scores 0.0 (clamped).

5. Collect top findings

Select the top 5 findings by severity (critical first, then high, medium, low, info). Within the same severity level, preserve the original order from findings.jsonl. Include all fields from the original finding object.

6. Compute requirement_coverage

Count findings with kind in requirement_missing or requirement_partial — these represent requirements the agent did not fully implement. Estimate total requirements from R<N> IDs present in findings plus any implemented ones (inferred from evaluation.md if available, otherwise estimate from the highest R-number seen).

requirement_coverage = implemented_count / total_requirements

If total requirements cannot be determined, set requirement_coverage to null.

7. Read model from stack.json

cat {run_dir}/stack.json | jq -r '.model // .agent // "unknown"'

If stack.json is absent or has no model/agent field, use "unknown".

8. Write assessment.json

Write atomically (via .tmp rename):

{
  "severity_counts": {"critical": 0, "high": 2, "medium": 5, "low": 3, "info": 1},
  "penalty_score": 0.67,
  "top_findings": [...],
  "requirement_coverage": 0.75,
  "model": "haiku",
  "evaluated_at": "2026-04-18T21:00:00Z"
}

Constraints:

• You MUST write atomically — write to {run_dir}/assessment.json.tmp then rename to {run_dir}/assessment.json.
• evaluated_at MUST be an ISO 8601 UTC timestamp.
• penalty_score MUST be rounded to 4 decimal places.
• requirement_coverage MAY be null if total requirements cannot be determined.

9. Emit a summary

Print a terminal-readable summary:

Assessment written to {run_dir}/assessment.json
  Severity counts: critical=0 high=2 medium=5 low=3 info=1
  Penalty score:   0.6700  (1.0 = clean, 0.0 = critical failures)
  Req coverage:    75.0%
  Model:           haiku
  Top finding:     [high] No pagination support on GET /books

If --dry-run was specified, print the JSON to stdout and skip the file write.

Constraints Summary

• You MUST NOT create beads issues, GitHub issues, or any external tracker records.
• You MUST write assessment.json atomically.
• You MUST be safe to re-run repeatedly — re-running overwrites assessment.json with fresh aggregation.
• You MUST respect --dry-run by only printing what would be written.
• You MUST finish quickly — this is aggregation only, no LLM calls, no network calls.

Interaction with retort

• FindingsScorer in src/retort/scoring/scorers/findings.py reads {run_dir}/assessment.json and returns penalty_score directly.
• The retort CLI MAY invoke this skill automatically after evaluate-run completes.
• If assessment.json is absent, FindingsScorer returns 0.5 (neutral) rather than failing.

Troubleshooting

findings.jsonl is empty

• Write assessment.json with all-zero severity counts, penalty_score 1.0, empty top_findings.

stack.json is missing

• Use model: "unknown". Do not abort.

assessment.json.tmp rename fails (permissions)

• Fall back to direct write. Log a warning.