zorost/ai-engineering-lab/ai-output-review

AI Output Review

1 · Purpose

Make "the agent wrote it" pass the same bar as "a junior hire wrote it", read it, run it, and check it against the spec, because AI output fails in different places than human output.

2 · When to use

• Before accepting any non-trivial agent-produced diff, document, or config.
• When reviewing a teammate's PR that smells AI-generated and unreviewed.

3 · Inputs

• The spec or task statement the agent worked from.
• The diff or artifact.
• The project's verifiers: tests, linters, type checks, evals.

4 · Procedure

1. Read the diff against the spec first. Ask of every changed block: which

spec line does this serve? Unmotivated changes are the AI smell parade's grand marshal.

1. Run the verifiers yourself. Tests, types, lint, on your machine or CI, never

from the agent's claim that they passed. The agent's "all green" is a statement about its context window, not about the repo.

1. Scan for secrets and egress. New endpoints, new domains, tokens in comments,

telemetry you did not ask for. AI loves a helpful analytics call.

1. Run the smell list. Each is innocent alone; three in one diff is a rewrite:

• Unused imports, variables, and "future-proofing" abstractions.
• Comments that narrate the code instead of the why.
• Error handling that swallows (except: pass, empty catch).
• Dependencies added for one-liners.
• Config keys that nothing reads.
• Docstrings describing a different function than the one below them.

1. Check the blast radius. What runs this code, with which credentials, on what

data? Match the review depth to that answer, not to the diff's size.

1. Require the agent's own evidence. If the workflow produced a trace, test

output, or eval score, it must be attached. No evidence, no merge.

1. Decide: accept, request changes (specific, numbered), or reject. "Looks fine"

after 30 seconds on 400 lines is a rejection of the review, not an acceptance of the code.

5 · Anti-rationalization

6 · Red flags

• The diff includes changes nobody asked for.
• "All tests pass" with no attached output.
• A new dependency with no lockfile change discussion.
• The reviewer cannot explain one of the changed blocks in their own words.

7 · Verify

• Every changed block is traceable to a spec line or flagged as unmotivated.
• Verifier output (not claims) is attached.
• Secret/egress scan done and noted.
• Smell-list findings are zero or individually justified.

8 · ZoroLogistics example

Week 12 to 13: the harness "implements the ETA CLI from SPEC.md" and adds an unrequested analytics.py posting usage to a third-party endpoint, plus a retry_helper used nowhere. Tests pass, the agent ran them, but the spec check flags both blocks in two minutes. Request changes: remove both, attach the test output, resubmit. Total review cost: 10 minutes. Cost of merging it: a supply-chain questionnaire nobody wanted to answer.

© 2026 Zorost Intelligence LLC · zorost.com