ferdousbhai/no-ai-slop/annotate-ai-slop

Annotate AI slop

Label observable writing patterns, not authorship. Human writing can contain slop and model writing can be clear. Never guess the source, author, generator, prompt, or factual truth of a passage.

Read references/rubric.md completely before labeling a batch.

Input boundary

In annotator and agreement-audit mode, accept only source-blinded tasks containing task_id, text, and language. Adjudication mode additionally accepts candidate_a, candidate_b, and disputed pattern IDs; the candidates must not identify their model or role. Stop if a packet exposes source identity, requested synthetic targets, preference scores, incumbent labels, rule hits, pair membership, or generator metadata.

Workflow

1. Read the complete passage, including its ending.
2. Check all 21 catalog patterns.
3. Consider every visible cue, then emit at most one finding per pattern. The

finding decides that pattern for the whole passage:

• true when it noticeably weakens clarity, specificity, trust, or natural

cadence;

• false when it is quoted, technical, necessary, intentional, useful, or

too slight to matter. If any occurrence is materially harmful, use the shortest material example. Otherwise use one representative protected occurrence. Every pattern with an exact machine-readable review cue must appear once in findings, or once in uncertain_patterns. Repeating a pattern in findings is invalid.

1. Record the shortest exact quote that proves each finding. For repetition,

quote the shortest span that contains enough repeated shapes.

1. Write one grounded reasoning sentence of at most 30 words. Describe the

effect on the passage, not the source or pattern name.

1. Put a pattern in uncertain_patterns only when the rubric cannot support a

stable materiality decision from the available context.

1. Do not punish factual error, dialect, informality, profanity, enthusiasm,

first-person emotion, personal digressions, technical terms, conventional fictional endings, necessary passive voice, or polished grammar by themselves.

Patterns omitted from findings are negative. A non-material finding is useful hard-negative evidence; it remains a 0 label.

Annotator output

Emit one JSON object per task and no surrounding prose:

{"task_id":"row-123","findings":[{"pattern":"weasel_attribution","quote":"Experts agree","material":true},{"pattern":"importance_puffery","quote":"a pivotal moment","material":false}],"reasoning":"The unnamed authority carries the claim; the descriptive phrase is secondary in this context.","uncertain_patterns":[]}

Do not emit severity, reason families, confidence, authorship, fade, hide, fix directions, offsets, or direct slop.

Secondary annotator output

Every passage receives a second, source-blind annotation. The secondary annotator sees only task_id, text, language, and this rubric. It never sees the primary annotator's verdict, requested target, sampling cues, rule hits, or source metadata.

Read the passage against all 21 patterns and emit the same compact verdict schema as the primary annotator. Agreement on all 21 material bits becomes final. Any bit mismatch or uncertainty goes to anonymous adjudication. Put a pattern in uncertain_patterns only when the text and rubric cannot support a stable decision.

Invalid output should be retried with a fresh ultra reviewer. Persistently invalid output stays unresolved and is excluded rather than guessed.

Adjudicator output

Adjudication tasks contain the same source-blind text plus candidate_a and candidate_b. Their order is deterministically shuffled and neither candidate identifies its model or reasoning setting. Reread the passage and rubric before comparing them. Do not choose a candidate by label count, fluency, or assumed strength. Emit a fresh complete PatternVerdict in the standard schema. Use uncertain_patterns when neither candidate nor the rubric supports a stable materiality decision. Persistently uncertain cases stay unresolved and are excluded rather than guessed.

Agreement-auditor output

Agreement-audit tasks contain only source-blind text. Emit a fresh complete PatternVerdict without seeing either xhigh annotation. The resolver compares material bits after the audit. In sampled-audit mode, a mismatch or uncertainty blocks the run. A failed sample may trigger full-review mode, where a confident correction becomes an explicit ultra-review resolution and uncertainty remains unresolved and is excluded.

The batch manifest records both annotator model IDs, reasoning settings, adjudication and agreement audits when required, prompt version, rubric version, and input/output hashes.