ometere123/the-skills-bible/ai-performance-cost-optimizer

AI Performance and Cost Optimiser

Purpose

Meet user outcomes and service objectives at sustainable cost without hiding quality or safety regressions.

Use this skill when

• AI latency or cost is too high.
• Context and token use are uncontrolled.
• Several models or serving options can be routed.
• Inference capacity and quality trade-offs need evidence.

Do not use this skill as the primary workflow when

• The system's task quality is not evaluated.
• The bottleneck is ordinary application code only.
• Cost reduction would remove required safeguards.

Required inputs

• Task quality thresholds.
• Latency and throughput objectives.
• Request, token, context, and tool distributions.
• Models, providers, hardware, and pricing.
• Serving, retrieval, and cache telemetry.

Classify missing information as blocking, important, or optional. Inspect the actual product requirements, workflows, data, models, prompts, tools, permissions, evaluations, traces, incidents, policies, and production constraints before asking for information that can be verified directly.

Core operating principles

• Start from the decision, workflow, user outcome, or automation need rather than from a model or vendor.
• Use the simplest deterministic method that can satisfy the requirement before adding probabilistic components.
• Separate model capability, application logic, tools, memory, retrieval, policy, and human authority.
• Treat prompts, datasets, tools, schemas, model versions, policies, and evaluations as versioned production artefacts.
• Ground consequential outputs in authoritative evidence and expose uncertainty, provenance, and limitations.
• Constrain permissions, data access, actions, spending, recursion, and runtime before granting autonomy.
• Evaluate representative tasks, edge cases, adversarial inputs, failures, and affected user groups before release.
• Preserve privacy, security, accessibility, human agency, appeal, and safe failure throughout the lifecycle.
• Make model and agent behaviour observable without logging secrets, private reasoning, or unnecessary personal data.
• Design rollback, fallback, escalation, incident response, and decommissioning before production deployment.

Workflow

1. Define optimisation objective

• Set quality, safety, latency percentile, throughput, cost per task, and availability constraints.
• Avoid cost-only optimisation.

2. Establish baseline

• Measure end-to-end time and cost across model, queue, retrieval, tools, network, and post-processing on representative tasks.

3. Reduce unnecessary calls

• Use deterministic routing, prechecks, combined requests, better task decomposition, and removal of duplicate reflection or agent work.

4. Optimise context

• Retrieve selectively, compress safely, cache stable instructions, remove duplicate text, and protect critical evidence from truncation.

5. Route models

• Use task complexity, modality, latency, privacy, and quality to choose models with verified escalation and fallback.

6. Optimise generation

• Control output length, structured decoding, stop conditions, sampling, speculative methods, and repeated retries.

7. Optimise serving

• Use batching, quantisation, compilation, hardware selection, caching, and autoscaling while measuring quality changes.

8. Optimise retrieval and tools

• Tune top-k, reranking, index, result size, parallelism, and tool selection without reducing grounding.

9. Measure economics

• Track cost per successful task, user, workflow, tenant, model, tool, and fallback rather than tokens alone.

10. Validate trade-offs

• Run paired evaluation for quality, safety, latency, and cost and monitor production after rollout.

Decision rules

Required outputs

• Optimisation objective.
• Cost and latency baseline.
• Call reduction.
• Context optimisation.
• Model routing.
• Generation tuning.
• Serving optimisation.
• Retrieval and tool optimisation.
• Unit economics.
• Trade-off evaluation.

AI system quality gates

• Problem gate: The intended outcome and the reason to use AI are explicit.
• Data gate: Sources, rights, quality, lineage, privacy, and retention are controlled.
• Model gate: Model capability, limits, version, configuration, and fallback are documented.
• Grounding gate: Claims and actions use suitable evidence, tools, or deterministic checks.
• Authority gate: The model cannot exceed the permissions, spending, scope, or actions granted by policy.
• Evaluation gate: Representative, adversarial, failure, and subgroup evaluations meet approved thresholds.
• Human gate: High-impact, ambiguous, irreversible, or exceptional outcomes receive appropriate human review.
• Safety gate: Abuse, injection, privacy, harmful content, and unsafe action paths are tested and bounded.
• Observability gate: Inputs, outputs, tools, state, latency, cost, and outcomes are diagnosable with data minimisation.
• Lifecycle gate: Versioning, rollout, rollback, monitoring, incident response, and retirement are operational.

Evidence standard

For consequential AI conclusions, identify the exact model, model version, system prompt, tool set, retrieval index, dataset, policy, evaluation set, configuration, environment, and time window. Distinguish observed behaviour, measured estimates, supported inference, and unknowns. Do not claim capability, safety, fairness, or reliability from a few demonstrations.

Failure handling and escalation

Pause or escalate when:

• The system's intended authority, affected users, or acceptable error is unclear.
• Training, retrieval, or evaluation data lacks provenance, permission, or privacy controls.
• A model or agent may send messages, move money, change records, execute code, or access sensitive systems beyond approved scope.
• Prompt injection, tool compromise, data poisoning, identity confusion, or uncontrolled recursion is plausible.
• A high-impact outcome cannot be independently verified or appealed.
• Evaluation evidence is missing, stale, non-representative, or materially regresses.
• Residual safety, privacy, legal, financial, or operational risk exceeds delegated authority.

Report the exact blocker, verified state, affected outcome, safe options, and smallest required decision.

Common anti-patterns

• Optimising tokens before measuring task success.
• One cheap model for every task.
• Caching personalised outputs across users.
• Removing citations to save tokens.
• Quantisation without quality evaluation.
• Agent loops counted as useful work.
• Average latency only.

Completion checklist

• [ ] Quality and safety constraints lead optimisation.
• [ ] End-to-end cost is decomposed.
• [ ] Routing and context changes are evaluated.
• [ ] Cache isolation is safe.
• [ ] Cost per successful outcome is known.
• [ ] Production trade-offs are monitored.

AI implementation record

# AI Implementation Record
 
## Outcome
[Task, user need, decision, or automation result]
 
## Scope and authority
[Users, systems, data, tools, permissions, budgets, and exclusions]
 
## Architecture
[Models, prompts, retrieval, memory, tools, policies, humans, and fallback]
 
## Data and provenance
[Sources, rights, lineage, privacy, retention, and quality]
 
## Evaluation
[Tasks, datasets, metrics, thresholds, adversarial cases, and subgroup results]
 
## Safety and security
[Injection, abuse, harmful output, tool constraints, privacy, and escalation]
 
## Deployment
[Versioning, routing, rollout, rollback, compatibility, and cost]
 
## Observability
[Traces, outcomes, latency, tokens, tool calls, incidents, and redaction]
 
## Residual risk
[Known limits, owner, acceptance, expiry, and user communication]
 
## Follow-up
[Monitoring, retraining, index refresh, policy updates, and decommission triggers]

Related skills

• model-serving-inference-engineer
• embedding-vector-search-engineer
• multi-agent-systems-engineer
• ai-observability-engineer

Maintenance rules

Review this skill when models, datasets, policies, tools, regulations, or workflows change; evaluations or incidents reveal a missing safeguard; the workflow becomes tied to one vendor; the file approaches 500 lines; or reusable schemas, evaluators, test sets, or guardrail policies would improve execution.