ometere123/the-skills-bible/ai-systems-architect

AI Systems Architect

Purpose

Create coherent AI architecture that satisfies real workflow needs without granting uncontrolled authority or hiding critical dependencies.

Use this skill when

• A new AI product or platform is being designed.
• Several models, tools, data sources, and humans must work together.
• An existing prototype needs production architecture.
• AI responsibilities and trust boundaries are unclear.

Do not use this skill as the primary workflow when

• The task is one prompt revision.
• A deterministic rules engine can fully solve the problem.
• The user needs only general product strategy.

Required inputs

• User outcomes and workflow.
• Data sources, tools, and actions.
• Risk, privacy, security, and regulatory constraints.
• Latency, scale, cost, and availability targets.
• Evaluation, deployment, and ownership model.

Classify missing information as blocking, important, or optional. Inspect the actual product requirements, workflows, data, models, prompts, tools, permissions, evaluations, traces, incidents, policies, and production constraints before asking for information that can be verified directly.

Core operating principles

• Start from the decision, workflow, user outcome, or automation need rather than from a model or vendor.
• Use the simplest deterministic method that can satisfy the requirement before adding probabilistic components.
• Separate model capability, application logic, tools, memory, retrieval, policy, and human authority.
• Treat prompts, datasets, tools, schemas, model versions, policies, and evaluations as versioned production artefacts.
• Ground consequential outputs in authoritative evidence and expose uncertainty, provenance, and limitations.
• Constrain permissions, data access, actions, spending, recursion, and runtime before granting autonomy.
• Evaluate representative tasks, edge cases, adversarial inputs, failures, and affected user groups before release.
• Preserve privacy, security, accessibility, human agency, appeal, and safe failure throughout the lifecycle.
• Make model and agent behaviour observable without logging secrets, private reasoning, or unnecessary personal data.
• Design rollback, fallback, escalation, incident response, and decommissioning before production deployment.

Workflow

1. Define AI role

• State what the system should infer, generate, retrieve, classify, recommend, or execute.
• Separate assistance from authority.

2. Map workflow

• Trace users, inputs, decisions, tools, data, approvals, side effects, and exceptions.
• Identify deterministic steps.

3. Choose system pattern

• Compare direct model call, retrieval, tool use, workflow automation, human review, and agentic control.
• Use the least autonomous suitable pattern.

4. Design model layer

• Select model classes, routing, fallbacks, configuration, and context budgets.
• Avoid one model for every task automatically.

5. Design knowledge layer

• Define retrieval sources, freshness, provenance, indexing, and citation needs.
• Keep authoritative systems separate from generated text.

6. Design action layer

• Specify tools, schemas, permissions, idempotency, budgets, confirmation, and rollback.
• Constrain irreversible actions.

7. Design state and memory

• Separate session context, durable user preferences, workflow state, and derived memory.
• Set retention and deletion.

8. Design safety and human control

• Add policy, input and output checks, injection resistance, escalation, appeal, and emergency disablement.

9. Design evaluation and operations

• Define task suites, traces, metrics, monitoring, incident response, and model or prompt versioning.

10. Validate architecture

• Prototype high-risk paths, dependency failure, model degradation, tool misuse, and rollback with representative tasks.

Decision rules

Required outputs

• AI role definition.
• Workflow and trust map.
• Architecture pattern.
• Model architecture.
• Knowledge architecture.
• Tool and action architecture.
• State and memory model.
• Safety and human controls.
• Evaluation and operations plan.
• Validation evidence.

AI system quality gates

• Problem gate: The intended outcome and the reason to use AI are explicit.
• Data gate: Sources, rights, quality, lineage, privacy, and retention are controlled.
• Model gate: Model capability, limits, version, configuration, and fallback are documented.
• Grounding gate: Claims and actions use suitable evidence, tools, or deterministic checks.
• Authority gate: The model cannot exceed the permissions, spending, scope, or actions granted by policy.
• Evaluation gate: Representative, adversarial, failure, and subgroup evaluations meet approved thresholds.
• Human gate: High-impact, ambiguous, irreversible, or exceptional outcomes receive appropriate human review.
• Safety gate: Abuse, injection, privacy, harmful content, and unsafe action paths are tested and bounded.
• Observability gate: Inputs, outputs, tools, state, latency, cost, and outcomes are diagnosable with data minimisation.
• Lifecycle gate: Versioning, rollout, rollback, monitoring, incident response, and retirement are operational.

Evidence standard

For consequential AI conclusions, identify the exact model, model version, system prompt, tool set, retrieval index, dataset, policy, evaluation set, configuration, environment, and time window. Distinguish observed behaviour, measured estimates, supported inference, and unknowns. Do not claim capability, safety, fairness, or reliability from a few demonstrations.

Failure handling and escalation

Pause or escalate when:

• The system's intended authority, affected users, or acceptable error is unclear.
• Training, retrieval, or evaluation data lacks provenance, permission, or privacy controls.
• A model or agent may send messages, move money, change records, execute code, or access sensitive systems beyond approved scope.
• Prompt injection, tool compromise, data poisoning, identity confusion, or uncontrolled recursion is plausible.
• A high-impact outcome cannot be independently verified or appealed.
• Evaluation evidence is missing, stale, non-representative, or materially regresses.
• Residual safety, privacy, legal, financial, or operational risk exceeds delegated authority.

Report the exact blocker, verified state, affected outcome, safe options, and smallest required decision.

Common anti-patterns

• Model-first architecture.
• Agent for every workflow.
• Generated text treated as source of truth.
• Tools with broad credentials.
• Memory storing all conversations indefinitely.
• No fallback when model or provider fails.
• Evaluation added after launch.

Completion checklist

• [ ] AI use is justified.
• [ ] Deterministic and probabilistic responsibilities are separated.
• [ ] Tools and data have bounded authority.
• [ ] Human control matches impact.
• [ ] Evaluation and rollback are designed.
• [ ] High-risk paths are validated.

AI implementation record

# AI Implementation Record
 
## Outcome
[Task, user need, decision, or automation result]
 
## Scope and authority
[Users, systems, data, tools, permissions, budgets, and exclusions]
 
## Architecture
[Models, prompts, retrieval, memory, tools, policies, humans, and fallback]
 
## Data and provenance
[Sources, rights, lineage, privacy, retention, and quality]
 
## Evaluation
[Tasks, datasets, metrics, thresholds, adversarial cases, and subgroup results]
 
## Safety and security
[Injection, abuse, harmful output, tool constraints, privacy, and escalation]
 
## Deployment
[Versioning, routing, rollout, rollback, compatibility, and cost]
 
## Observability
[Traces, outcomes, latency, tokens, tool calls, incidents, and redaction]
 
## Residual risk
[Known limits, owner, acceptance, expiry, and user communication]
 
## Follow-up
[Monitoring, retraining, index refresh, policy updates, and decommission triggers]

Related skills

• ml-problem-framing-specialist
• llm-application-engineer
• tool-use-agent-engineer
• ai-evaluation-architect
• model-lifecycle-governance-engineer

Maintenance rules

Review this skill when models, datasets, policies, tools, regulations, or workflows change; evaluations or incidents reveal a missing safeguard; the workflow becomes tied to one vendor; the file approaches 500 lines; or reusable schemas, evaluators, test sets, or guardrail policies would improve execution.