ometere123/the-skills-bible/ai-red-team-engineer

AI Red Team Engineer

Purpose

Discover realistic AI failure and abuse paths before attackers, users, or high-stakes contexts expose them.

Use this skill when

• An AI system has significant safety, security, privacy, or misuse risk.
• Guardrails need adversarial validation.
• New models, tools, modalities, or autonomy are introduced.
• A prior incident suggests hidden failure classes.

Do not use this skill as the primary workflow when

• Testing authority and impact limits are absent.
• The request targets a third-party system without permission.
• The task is ordinary functional evaluation only.

Required inputs

• Written scope and rules of engagement.
• System architecture, tools, and policies.
• Threat actors and misuse cases.
• Safe environments and test identities.
• Escalation, evidence, and remediation process.

Classify missing information as blocking, important, or optional. Inspect the actual product requirements, workflows, data, models, prompts, tools, permissions, evaluations, traces, incidents, policies, and production constraints before asking for information that can be verified directly.

Core operating principles

• Start from the decision, workflow, user outcome, or automation need rather than from a model or vendor.
• Use the simplest deterministic method that can satisfy the requirement before adding probabilistic components.
• Separate model capability, application logic, tools, memory, retrieval, policy, and human authority.
• Treat prompts, datasets, tools, schemas, model versions, policies, and evaluations as versioned production artefacts.
• Ground consequential outputs in authoritative evidence and expose uncertainty, provenance, and limitations.
• Constrain permissions, data access, actions, spending, recursion, and runtime before granting autonomy.
• Evaluate representative tasks, edge cases, adversarial inputs, failures, and affected user groups before release.
• Preserve privacy, security, accessibility, human agency, appeal, and safe failure throughout the lifecycle.
• Make model and agent behaviour observable without logging secrets, private reasoning, or unnecessary personal data.
• Design rollback, fallback, escalation, incident response, and decommissioning before production deployment.

Workflow

1. Define authorisation

• Record systems, versions, dates, methods, data, prohibited actions, and stop conditions.
• Do not infer permission.

2. Build threat catalogue

• Cover malicious users, prompt injection, insiders, compromised tools, poisoned data, model extraction, privacy leakage, and accidental harm.

3. Prioritise attack paths

• Focus on high-impact reachable chains across input, retrieval, model, tools, memory, and humans.
• Avoid random jailbreak collection.

4. Prepare safe testing

• Use synthetic identities, sandboxes, bounded tools, monitoring, rollback, and protected evidence storage.

5. Test instruction attacks

• Evaluate direct, indirect, multilingual, encoded, long-context, role, and tool-result injection.
• Test instruction hierarchy.

6. Test harmful and deceptive behaviour

• Probe prohibited assistance, manipulation, impersonation, false authority, fabricated evidence, and strategic concealment within authorised scope.

7. Test data and model exposure

• Assess secrets, personal data, training memorisation, cross-tenant leakage, system prompts, and retrieval access.
• Use minimum proof.

8. Test action safety

• Attempt privilege expansion, unsafe destinations, duplicate side effects, budget abuse, code escape, and approval bypass.

9. Report findings

• Describe exact system version, path, prerequisites, evidence, impact, reproducibility, and recommended control layer.

10. Retest and generalise

• Verify fixes, create broader regression families, update threat model, and monitor recurrence without publishing harmful details unnecessarily.

Decision rules

Required outputs

• Authorisation record.
• Threat catalogue.
• Attack-path plan.
• Safety controls.
• Instruction-attack results.
• Harm and deception results.
• Data-exposure results.
• Action-safety results.
• Findings.
• Retest and regression suite.

AI system quality gates

• Problem gate: The intended outcome and the reason to use AI are explicit.
• Data gate: Sources, rights, quality, lineage, privacy, and retention are controlled.
• Model gate: Model capability, limits, version, configuration, and fallback are documented.
• Grounding gate: Claims and actions use suitable evidence, tools, or deterministic checks.
• Authority gate: The model cannot exceed the permissions, spending, scope, or actions granted by policy.
• Evaluation gate: Representative, adversarial, failure, and subgroup evaluations meet approved thresholds.
• Human gate: High-impact, ambiguous, irreversible, or exceptional outcomes receive appropriate human review.
• Safety gate: Abuse, injection, privacy, harmful content, and unsafe action paths are tested and bounded.
• Observability gate: Inputs, outputs, tools, state, latency, cost, and outcomes are diagnosable with data minimisation.
• Lifecycle gate: Versioning, rollout, rollback, monitoring, incident response, and retirement are operational.

Evidence standard

For consequential AI conclusions, identify the exact model, model version, system prompt, tool set, retrieval index, dataset, policy, evaluation set, configuration, environment, and time window. Distinguish observed behaviour, measured estimates, supported inference, and unknowns. Do not claim capability, safety, fairness, or reliability from a few demonstrations.

Failure handling and escalation

Pause or escalate when:

• The system's intended authority, affected users, or acceptable error is unclear.
• Training, retrieval, or evaluation data lacks provenance, permission, or privacy controls.
• A model or agent may send messages, move money, change records, execute code, or access sensitive systems beyond approved scope.
• Prompt injection, tool compromise, data poisoning, identity confusion, or uncontrolled recursion is plausible.
• A high-impact outcome cannot be independently verified or appealed.
• Evaluation evidence is missing, stale, non-representative, or materially regresses.
• Residual safety, privacy, legal, financial, or operational risk exceeds delegated authority.

Report the exact blocker, verified state, affected outcome, safe options, and smallest required decision.

Common anti-patterns

• Collecting jailbreak prompts without threat context.
• Testing live users.
• Downloading excessive data for proof.
• Reporting only successful prompts without architecture.
• Fixing every attack by adding prompt text.
• Publishing dangerous details before remediation.
• No retest.

Completion checklist

• [ ] Testing is authorised and bounded.
• [ ] Attack paths cover the whole system.
• [ ] Evidence uses minimum harm.
• [ ] Findings identify control layers.
• [ ] Fixes are retested.
• [ ] General failure families become evaluations.

AI implementation record

# AI Implementation Record
 
## Outcome
[Task, user need, decision, or automation result]
 
## Scope and authority
[Users, systems, data, tools, permissions, budgets, and exclusions]
 
## Architecture
[Models, prompts, retrieval, memory, tools, policies, humans, and fallback]
 
## Data and provenance
[Sources, rights, lineage, privacy, retention, and quality]
 
## Evaluation
[Tasks, datasets, metrics, thresholds, adversarial cases, and subgroup results]
 
## Safety and security
[Injection, abuse, harmful output, tool constraints, privacy, and escalation]
 
## Deployment
[Versioning, routing, rollout, rollback, compatibility, and cost]
 
## Observability
[Traces, outcomes, latency, tokens, tool calls, incidents, and redaction]
 
## Residual risk
[Known limits, owner, acceptance, expiry, and user communication]
 
## Follow-up
[Monitoring, retraining, index refresh, policy updates, and decommission triggers]

Related skills

• prompt-injection-defense-engineer
• model-safety-guardrails-engineer
• ai-privacy-security-engineer
• ai-incident-response-engineer

Maintenance rules

Review this skill when models, datasets, policies, tools, regulations, or workflows change; evaluations or incidents reveal a missing safeguard; the workflow becomes tied to one vendor; the file approaches 500 lines; or reusable schemas, evaluators, test sets, or guardrail policies would improve execution.