magnus919/agent-skills/ai-operating-economics

AI Operating Economics

Overview

AI initiatives are operating interventions, not merely model purchases or ROI spreadsheets. Their value depends on what work changes, who benefits, what quality or risk changes with it, what the complete intervention costs, and whether the organization can observe and govern those changes.

This skill provides the cross-domain decision spine for evaluating an AI-enabled workflow. It does not replace financial modeling, product measurement, statistical inference, agent evaluation, runtime operations, or AI governance. It makes those inputs meet in one accountable decision record.

The core question is not “Did the model make people faster?” It is: “What changed in this workflow, for whom, at what full cost, with what outcome and countermetric evidence, and what authority should the organization grant next?”

Entry Points

When to Use

Load this skill when the user needs to:

• Build an evidence-backed business case for an AI use case or agentic workflow.
• Decide whether an AI pilot should scale, remain bounded, be redesigned, or stop.
• Review claimed AI productivity, savings, adoption, or transformation results.
• Design an AI value-realization or post-launch outcome review.
• Connect model and tool spend to workflow outcomes and worker or customer effects.
• Compare AI options while accounting for measurement uncertainty and non-comparable evidence.
• Prepare an executive, product, portfolio, or lifecycle decision about an AI-enabled intervention.

When Not to Use

Non-Negotiable Reasoning Rules

1. Workflow evidence beats model evidence. A benchmark, demo, or vendor claim does not establish value in the target workflow.
2. Speed is not value. Time saved can be spent on lower-value work, offset by review and exception handling, or enable higher-value work. Measure the business or user outcome directly.
3. Averages are not enough. Inspect worker, user, task, geography, tenure, risk, and quality slices. An aggregate gain can hide a subgroup loss.
4. Every benefit metric needs a countermetric. Pair throughput or cost with quality, safety, customer, worker, privacy, or reliability measures appropriate to the workflow.
5. Token cost is not total cost. Include model calls, tools, retrieval, storage, networking, observability, engineering, human review, change management, governance, and unused committed capacity when material.
6. Evidence classes must stay separate. Label observed results, causal estimates, inferences, vendor-reported findings, stakeholder assertions, and normative requirements distinctly.
7. Missing evidence is a decision input. Do not turn an unknown into a favorable assumption. Record the gap, owner, consequence, and next evidence needed.
8. Authority follows evidence. A positive pilot does not justify unrestricted autonomy. Scale capability and authority in bounded slices with explicit reversal conditions.
9. Do not manufacture precision. Use ranges, scenarios, sensitivity, and confidence where inputs are uncertain. Do not rank non-comparable studies or vendors.
10. The decision is reversible only if the artifact says how. Record the stop trigger, rollback or containment path, decision owner, and review date.

Core Workflow

Use this sequence for an AI initiative review. Load the detailed method and the evidence-record template when the task requires a durable artifact.

Quick Start by Need

Choose Review Depth

1. Define the intervention and decision

Name the workflow, population, task boundary, intervention mode, baseline, decision sought, and decision owner. State whether the AI assists, recommends, routes, executes, or replaces/removes work. Define what remains human-controlled.

Do not begin with the model name or a claimed percentage. Begin with the work that changes and the decision the evidence must support.

2. State the value hypothesis

Write a falsifiable hypothesis:

For [population] doing [workflow], [intervention] will change [outcome] by [direction/range] without exceeding [countermetric boundary], at [full operating cost boundary], compared with [baseline], over [period].

If the proposed outcome is only “productivity,” decompose it into the actual customer, employee, operational, financial, or mission outcome. If the outcome cannot be observed or credibly proxied, mark the initiative measurement-incomplete rather than inventing a proxy.

3. Build the outcome and countermetric map

Define:

• Primary outcome: the result the initiative exists to improve.
• Leading indicators: early evidence that the mechanism is operating.
• Countermetrics: quality, safety, customer, worker, privacy, reliability, or equity measures that could worsen.
• Adoption and substitution measures: who uses the system, what work changes, and what work is displaced or added.
• Guardrail thresholds: contextual limits with an owner and response.

Route metric definitions and instrumentation plans to product analytics. Route statistical or causal design to data science. This skill owns the connection between the evidence and the decision, not the detailed statistical method.

4. Establish the full economic boundary

Record both:

• Marginal economics: what changes when one more task, user, or workflow unit is served.
• Fully loaded economics: the costs required to make the intervention available and govern it.

At minimum consider inference, tool use, retrieval, storage, data transfer, observability, engineering, evaluation, human review, training, support, change management, governance, security, and committed capacity. Separate fixed, variable, step-function, and avoided costs. Define the denominator precisely: task, resolved case, completed workflow, active user, customer outcome, or another meaningful unit.

Route the detailed model to capacity-and-cost-engineering or financial-modeling. Never divide total spend by an undifferentiated request count when requests have materially different resource or outcome profiles.

5. Design the evidence comparison

Choose the strongest feasible comparison before interpreting results:

• Randomized or staggered rollout when feasible.
• Matched or difference-in-differences comparison when appropriate.
• Within-workflow baseline with explicit pre-period and seasonality limits.
• Controlled pilot with a documented task and population boundary.
• Descriptive before/after evidence only when stronger designs are infeasible, labeled accordingly.

Record selection effects, learning effects, concurrent initiatives, task-mix changes, worker self-selection, quality measurement gaps, and changes in pay or incentives. If the comparison cannot support the requested claim, narrow the claim rather than upgrading the method rhetorically.

6. Segment before aggregating

Report the overall result and inspect slices that could change the decision:

• Worker experience, skill, role, and training status.
• Task complexity, risk, volume, and exception rate.
• Customer or user segment.
• Geography, language, accessibility, and relevant demographic groups when lawful and appropriate.
• Human-review burden and escalation path.
• Quality, safety, and error severity.

Treat heterogeneous effects as a finding, not noise to average away. A tool that helps novices while harming expert quality may need differentiated assistance modes, not universal rollout.

7. Classify the evidence

For every material claim, label it: