ometere123/the-skills-bible/agent-orchestration-engineer

Agent Orchestration Engineer

Purpose

Coordinate uncertain model decisions within a deterministic operational envelope that remains observable, resumable, and bounded.

Use this skill when

• An AI workflow requires several reasoning and tool steps.
• Tasks may pause for external events or human input.
• Agent runs need checkpoints and recovery.
• Plans must adapt without uncontrolled loops.

Do not use this skill as the primary workflow when

• A fixed workflow can solve the task directly.
• The agent has no bounded completion condition.
• Tools cannot be permissioned or reversed safely.

Required inputs

• Task and completion definition.
• Available tools and permissions.
• Workflow state and external dependencies.
• Budgets, deadlines, and human approval points.
• Failure, rollback, and evaluation requirements.

Classify missing information as blocking, important, or optional. Inspect the actual product requirements, workflows, data, models, prompts, tools, permissions, evaluations, traces, incidents, policies, and production constraints before asking for information that can be verified directly.

Core operating principles

• Start from the decision, workflow, user outcome, or automation need rather than from a model or vendor.
• Use the simplest deterministic method that can satisfy the requirement before adding probabilistic components.
• Separate model capability, application logic, tools, memory, retrieval, policy, and human authority.
• Treat prompts, datasets, tools, schemas, model versions, policies, and evaluations as versioned production artefacts.
• Ground consequential outputs in authoritative evidence and expose uncertainty, provenance, and limitations.
• Constrain permissions, data access, actions, spending, recursion, and runtime before granting autonomy.
• Evaluate representative tasks, edge cases, adversarial inputs, failures, and affected user groups before release.
• Preserve privacy, security, accessibility, human agency, appeal, and safe failure throughout the lifecycle.
• Make model and agent behaviour observable without logging secrets, private reasoning, or unnecessary personal data.
• Design rollback, fallback, escalation, incident response, and decommissioning before production deployment.

Workflow

1. Define task contract

• State objective, inputs, outputs, constraints, authority, deadline, and completion proof.
• Avoid vague solve this missions.

2. Choose orchestration pattern

• Use fixed graph, dynamic planner, state machine, durable workflow, or hybrid according to uncertainty.
• Keep deterministic structure where possible.

3. Design state

• Persist task, plan, steps, observations, tool outcomes, approvals, costs, versions, and checkpoints.
• Separate generated notes from authority.

4. Design planning

• Limit plan depth, require explicit next action, verify prerequisites, and replan only on defined conditions.
• Avoid endless reflection.

5. Enforce budgets

• Bound model calls, tokens, tools, runtime, retries, spend, data, and sub-agents.
• Stop predictably.

6. Coordinate tools

• Validate calls, dependencies, idempotency, concurrency, ordering, and unknown outcomes.
• Protect side effects.

7. Add human gates

• Pause before consequential decisions, exceptions, spending, publication, or irreversible actions.
• Provide evidence and preview.

8. Handle interruption

• Checkpoint before side effects, resume idempotently, cancel safely, compensate where possible, and expire abandoned runs.

9. Define completion

• Use deterministic acceptance checks, required artefacts, source support, and unresolved-item reporting.
• Do not rely on model saying done.

10. Evaluate runs

• Test long tasks, loops, missing tools, conflicting evidence, approval delays, partial failure, cancellation, and recovery.

Decision rules

Required outputs

• Task contract.
• Orchestration pattern.
• Durable state.
• Planning policy.
• Budgets.
• Tool coordination.
• Human gates.
• Interruption handling.
• Completion proof.
• Agent-run evaluation.

AI system quality gates

• Problem gate: The intended outcome and the reason to use AI are explicit.
• Data gate: Sources, rights, quality, lineage, privacy, and retention are controlled.
• Model gate: Model capability, limits, version, configuration, and fallback are documented.
• Grounding gate: Claims and actions use suitable evidence, tools, or deterministic checks.
• Authority gate: The model cannot exceed the permissions, spending, scope, or actions granted by policy.
• Evaluation gate: Representative, adversarial, failure, and subgroup evaluations meet approved thresholds.
• Human gate: High-impact, ambiguous, irreversible, or exceptional outcomes receive appropriate human review.
• Safety gate: Abuse, injection, privacy, harmful content, and unsafe action paths are tested and bounded.
• Observability gate: Inputs, outputs, tools, state, latency, cost, and outcomes are diagnosable with data minimisation.
• Lifecycle gate: Versioning, rollout, rollback, monitoring, incident response, and retirement are operational.

Evidence standard

For consequential AI conclusions, identify the exact model, model version, system prompt, tool set, retrieval index, dataset, policy, evaluation set, configuration, environment, and time window. Distinguish observed behaviour, measured estimates, supported inference, and unknowns. Do not claim capability, safety, fairness, or reliability from a few demonstrations.

Failure handling and escalation

Pause or escalate when:

• The system's intended authority, affected users, or acceptable error is unclear.
• Training, retrieval, or evaluation data lacks provenance, permission, or privacy controls.
• A model or agent may send messages, move money, change records, execute code, or access sensitive systems beyond approved scope.
• Prompt injection, tool compromise, data poisoning, identity confusion, or uncontrolled recursion is plausible.
• A high-impact outcome cannot be independently verified or appealed.
• Evaluation evidence is missing, stale, non-representative, or materially regresses.
• Residual safety, privacy, legal, financial, or operational risk exceeds delegated authority.

Report the exact blocker, verified state, affected outcome, safe options, and smallest required decision.

Common anti-patterns

• Unbounded while loop.
• Model decides its own permissions.
• Completion based on fluent summary.
• State stored only in prompt history.
• Retrying side effects without reconciliation.
• No cancellation path.
• Sub-agents spawned without budget inheritance.

Completion checklist

• [ ] Task and completion are measurable.
• [ ] State survives interruption.
• [ ] Budgets and permissions are enforced externally.
• [ ] Human gates precede impact.
• [ ] Runs cancel and resume safely.
• [ ] Loop and failure cases are evaluated.

AI implementation record

# AI Implementation Record
 
## Outcome
[Task, user need, decision, or automation result]
 
## Scope and authority
[Users, systems, data, tools, permissions, budgets, and exclusions]
 
## Architecture
[Models, prompts, retrieval, memory, tools, policies, humans, and fallback]
 
## Data and provenance
[Sources, rights, lineage, privacy, retention, and quality]
 
## Evaluation
[Tasks, datasets, metrics, thresholds, adversarial cases, and subgroup results]
 
## Safety and security
[Injection, abuse, harmful output, tool constraints, privacy, and escalation]
 
## Deployment
[Versioning, routing, rollout, rollback, compatibility, and cost]
 
## Observability
[Traces, outcomes, latency, tokens, tool calls, incidents, and redaction]
 
## Residual risk
[Known limits, owner, acceptance, expiry, and user communication]
 
## Follow-up
[Monitoring, retraining, index refresh, policy updates, and decommission triggers]

Related skills

• tool-use-agent-engineer
• workflow-automation-engineer
• human-in-the-loop-systems-engineer
• agent-evaluation-engineer

Maintenance rules

Review this skill when models, datasets, policies, tools, regulations, or workflows change; evaluations or incidents reveal a missing safeguard; the workflow becomes tied to one vendor; the file approaches 500 lines; or reusable schemas, evaluators, test sets, or guardrail policies would improve execution.