qhjqhj00/research-skills-pool/ai-red-teaming-ctf-eval

ai-red-teaming-ctf-eval

Breaking Guardrails, Facing Walls: Insights on Adversarial AI for Defenders & Researchers — Bozdemir et al. (2025) (arXiv:2510.16005, 2025)

What this evaluates

Evaluates adversarial AI red-teaming capabilities by measuring participant success rates in bypassing LLM guardrails, manipulating model outputs, and extracting sensitive data through prompt injection and jailbreaking techniques.

Datasets

• AI Red Teaming CTF (CTF ID: 2604) — total 11; splits: test (11)

Metrics

• solve_rate (primary) — range: percent
• Calculated as the number of teams solving a specific challenge divided by the total number of teams (all registered or active), expressed as a percentage. The paper reports both overall and active-team completion rates.

Input / output format

Input: Adversarial challenge scenarios presented as CTF instances, requiring participants to craft prompts or multi-step attack sequences to bypass AI guardrails, manipulate outputs, or extract hidden data.

Output: A submitted flag string upon successfully completing a challenge instance.

Scoring recipe

def compute_solve_rate(challenges, teams, denominator='active'):
    total = 504 if denominator == 'all' else 217
    rates = {}
    for c in challenges:
        solves = sum(1 for t in teams if t.solved(c))
        rates[c] = (solves / total) * 100
    return rates

Common pitfalls

• Confusing 'active teams' (n=217) with 'all registered teams' (n=504) when interpreting completion percentages, leading to skewed difficulty assessments.
• Measuring time-to-first-solve from CTF start versus first instance launch yields drastically different distributions due to timezone and availability effects rather than pure problem-solving latency.
• Aggregating category success rates (e.g., Output Manipulation vs. Data Extraction) without accounting for the steep difficulty drop-off between early onboarding tasks and the final multi-step challenge.

Evidence (verbatim from paper)

Among active teams (n = 217), 170 (78.3%) launched the final challenge (Figure 6); 74 (34.1%) solved it; and 96 (44.2%) launched but did not solve.

Citation

@misc{bozdemir2025breaking,
  title={Breaking Guardrails, Facing Walls: Insights on Adversarial AI for Defenders & Researchers},
  author={Bozdemir et al. (2025)},
  year={2025},
  note={arXiv:2510.16005}
}

• arXiv: 2510.16005