ometere123/the-skills-bible/ai-reliability-engineer

AI Reliability Engineer

Purpose

Keep AI-assisted services useful and safe when probabilistic components fail, degrade, disagree, or become unavailable.

Use this skill when

• AI provider or model failures affect critical workflows.
• Invalid outputs, timeouts, and degraded retrieval need handling.
• An agent must resume or fail safely.
• Service objectives and fallback need design.

Do not use this skill as the primary workflow when

• The main issue is general infrastructure reliability.
• The AI feature is non-critical and can simply disappear.
• No task-level success definition exists.

Required inputs

• Critical AI tasks and service objectives.
• Models, providers, retrieval, tools, and dependencies.
• Failure and incident history.
• Fallback and human-support options.
• Latency, budget, and consistency constraints.

Classify missing information as blocking, important, or optional. Inspect the actual product requirements, workflows, data, models, prompts, tools, permissions, evaluations, traces, incidents, policies, and production constraints before asking for information that can be verified directly.

Core operating principles

• Start from the decision, workflow, user outcome, or automation need rather than from a model or vendor.
• Use the simplest deterministic method that can satisfy the requirement before adding probabilistic components.
• Separate model capability, application logic, tools, memory, retrieval, policy, and human authority.
• Treat prompts, datasets, tools, schemas, model versions, policies, and evaluations as versioned production artefacts.
• Ground consequential outputs in authoritative evidence and expose uncertainty, provenance, and limitations.
• Constrain permissions, data access, actions, spending, recursion, and runtime before granting autonomy.
• Evaluate representative tasks, edge cases, adversarial inputs, failures, and affected user groups before release.
• Preserve privacy, security, accessibility, human agency, appeal, and safe failure throughout the lifecycle.
• Make model and agent behaviour observable without logging secrets, private reasoning, or unnecessary personal data.
• Design rollback, fallback, escalation, incident response, and decommissioning before production deployment.

Workflow

1. Define reliability outcomes

• Specify task success, correct abstention, bounded latency, action integrity, availability, and recovery by workflow.

2. Map failure modes

• Cover provider outage, model regression, invalid output, truncation, stale retrieval, missing source, tool failure, quota, timeout, and policy outage.

3. Allocate deadlines

• Set end-to-end and component timeouts for retrieval, models, tools, approvals, and retries.
• Propagate cancellation.

4. Design retries

• Retry only transient safe operations with backoff, jitter, limits, and idempotency.
• Avoid multiplying retries across layers.

5. Design fallbacks

• Use smaller model, alternate provider, deterministic rule, cached evidence, reduced capability, queued work, or human review with compatibility checks.

6. Design degraded modes

• Make unavailable features, stale evidence, lower confidence, and manual completion explicit to users and operators.

7. Protect actions

• Reconcile unknown tool outcomes, checkpoint workflows, prevent duplicates, and keep authority valid across failover.

8. Design isolation

• Use quotas, circuit breakers, bulkheads, tenant limits, and priority so one task or provider cannot exhaust all capacity.

9. Test resilience

• Inject model errors, malformed output, slow retrieval, tool timeout, provider outage, quota exhaustion, and recovery.

10. Operate reliability

• Monitor task success, fallback rate, retries, queues, provider health, recovery, and quality under degraded paths.

Decision rules

Required outputs

• Reliability outcomes.
• Failure map.
• Deadline budget.
• Retry policy.
• Fallback design.
• Degraded modes.
• Action integrity.
• Isolation controls.
• Resilience tests.
• Reliability telemetry.

AI system quality gates

• Problem gate: The intended outcome and the reason to use AI are explicit.
• Data gate: Sources, rights, quality, lineage, privacy, and retention are controlled.
• Model gate: Model capability, limits, version, configuration, and fallback are documented.
• Grounding gate: Claims and actions use suitable evidence, tools, or deterministic checks.
• Authority gate: The model cannot exceed the permissions, spending, scope, or actions granted by policy.
• Evaluation gate: Representative, adversarial, failure, and subgroup evaluations meet approved thresholds.
• Human gate: High-impact, ambiguous, irreversible, or exceptional outcomes receive appropriate human review.
• Safety gate: Abuse, injection, privacy, harmful content, and unsafe action paths are tested and bounded.
• Observability gate: Inputs, outputs, tools, state, latency, cost, and outcomes are diagnosable with data minimisation.
• Lifecycle gate: Versioning, rollout, rollback, monitoring, incident response, and retirement are operational.

Evidence standard

For consequential AI conclusions, identify the exact model, model version, system prompt, tool set, retrieval index, dataset, policy, evaluation set, configuration, environment, and time window. Distinguish observed behaviour, measured estimates, supported inference, and unknowns. Do not claim capability, safety, fairness, or reliability from a few demonstrations.

Failure handling and escalation

Pause or escalate when:

• The system's intended authority, affected users, or acceptable error is unclear.
• Training, retrieval, or evaluation data lacks provenance, permission, or privacy controls.
• A model or agent may send messages, move money, change records, execute code, or access sensitive systems beyond approved scope.
• Prompt injection, tool compromise, data poisoning, identity confusion, or uncontrolled recursion is plausible.
• A high-impact outcome cannot be independently verified or appealed.
• Evaluation evidence is missing, stale, non-representative, or materially regresses.
• Residual safety, privacy, legal, financial, or operational risk exceeds delegated authority.

Report the exact blocker, verified state, affected outcome, safe options, and smallest required decision.

Common anti-patterns

• Retrying every model failure.
• Fallback model with different output semantics.
• Serving stale retrieval without disclosure.
• Agent restart loses completed side effects.
• Provider availability used as task-success metric.
• No manual fallback.
• Circuit breaker shared across unrelated tasks.

Completion checklist

• [ ] Task-level reliability is defined.
• [ ] Retries and deadlines are bounded.
• [ ] Fallbacks are quality-tested.
• [ ] Degraded behaviour is explicit.
• [ ] Actions remain idempotent and reconcilable.
• [ ] Failure injection validates recovery.

AI implementation record

# AI Implementation Record
 
## Outcome
[Task, user need, decision, or automation result]
 
## Scope and authority
[Users, systems, data, tools, permissions, budgets, and exclusions]
 
## Architecture
[Models, prompts, retrieval, memory, tools, policies, humans, and fallback]
 
## Data and provenance
[Sources, rights, lineage, privacy, retention, and quality]
 
## Evaluation
[Tasks, datasets, metrics, thresholds, adversarial cases, and subgroup results]
 
## Safety and security
[Injection, abuse, harmful output, tool constraints, privacy, and escalation]
 
## Deployment
[Versioning, routing, rollout, rollback, compatibility, and cost]
 
## Observability
[Traces, outcomes, latency, tokens, tool calls, incidents, and redaction]
 
## Residual risk
[Known limits, owner, acceptance, expiry, and user communication]
 
## Follow-up
[Monitoring, retraining, index refresh, policy updates, and decommission triggers]

Related skills

• model-serving-inference-engineer
• agent-orchestration-engineer
• workflow-automation-engineer
• ai-observability-engineer

Maintenance rules

Review this skill when models, datasets, policies, tools, regulations, or workflows change; evaluations or incidents reveal a missing safeguard; the workflow becomes tied to one vendor; the file approaches 500 lines; or reusable schemas, evaluators, test sets, or guardrail policies would improve execution.