muvon/octomind-tap/ai-cost-finops

Overview

LLM cost is an engineering surface, not a line on the card statement. The levers that move it most in 2026, in order: prompt caching (cache reads cost a tenth of input or less), Batch/Flex tiers (50% off), effort or reasoning-depth tuning, then model routing. This skill holds the dated price book, the discount mechanics, and the optimization order with primary-source citations.

Use it when projecting cost for a new feature, hunting waste in an existing pipeline, picking a model or effort level, or setting up per-feature attribution. Skip it for prototypes spending under ~$100/month.

Mental model

Every call costs uncached_input × $in + cache_writes × $write + cache_reads × $read + output × $out (thinking tokens bill as output), plus retrieval, rerank, and storage for RAG. Judge cost per completed task, not per request: a cheaper call that needs more turns or retries is not cheaper.

Work the levers in order. Free wins first, because they don't trade quality: caching, input hygiene, output hygiene, batch. Then quality trades, measured against an eval: effort level, then model choice. The cheapest call is the one not made; then the cached one; then the batched one; then the one at lower effort or on a smaller model.

Output tokens cost 5× input on most current models, so output-length discipline (no markdown for machine consumers, no preambles) is the next lever after caching.

Rules

1. Price book (2026: verified 2026-10-12, per 1M tokens, standard tier, input/output)

Prices change quarterly. Re-check the pricing pages before any budget.

Sources: Anthropic pricing, OpenAI pricing, Gemini pricing. Older generations stay listed at their own prices; Claude Opus 4.7 and later use a tokenizer that emits ~1–1.35× the tokens of older Claude models, so re-baseline with token counting when migrating. Regional data-residency endpoints carry uplifts on some providers (OpenAI: +10%).

2. Prompt caching

Anthropic (docs):

• Writes cost 1.25× input (5-minute TTL, the default) or 2× (1-hour TTL, set "ttl": "1h"). Reads cost ~0.1× input; 0.05× on Opus 5.5 and 0.025× on Fable 5.1.
• Break-even: two requests on the 5-minute TTL, three on the 1-hour. A cache read refreshes the TTL, so traffic that arrives less than 5 minutes apart never needs the 1-hour TTL.
• Prefix match in render order tools → system → messages; any byte change invalidates everything after it. Max 4 breakpoints; minimum cacheable prefix is model-dependent (512–4096 tokens) and shorter prefixes silently don't cache.
• Top-level automatic caching moves the breakpoint forward as a conversation grows; add an explicit breakpoint on the static system prefix for agent loops.
• Caches are model-scoped, and changing top-level effort mid-conversation invalidates the message cache.
• Verify with usage.cache_read_input_tokens; zero across repeated requests means a silent invalidator (timestamps in the system prompt, unsorted JSON, a varying tool list).

OpenAI: automatic prefix caching, no code change; cached input is billed at 5–10% of input on the GPT-6 family (pricing page). Gemini: implicit caching plus explicit context caches billed at ~10% of input plus hourly storage.

Cache system prompts, tool and skill definitions, few-shot examples, and long static documents — anything long, stable, and re-sent.

3. Batch and Flex (50% off anything nobody waits on)

• Anthropic Message Batches — 50% off every token, including cache reads and writes; results within 24 hours, in any order (key by custom_id). No mid-batch tool loop. docs
• OpenAI Batch and Flex — both 50% below standard on current models. pricing
• Gemini Batch and Flex — 50% off; Priority costs ~1.8×. pricing

Use for evals, backfills, embeddings, dataset enrichment, and scheduled jobs.

4. Effort before routing

Current frontier models expose a reasoning-depth control: Claude output_config.effort (low → max, with adaptive thinking), OpenAI reasoning effort, Gemini thinking level. Effort is the first quality-trading lever: it trades thoroughness for tokens inside one model and keeps one cache namespace.

• Effort names don't mean the same thinking budget across models; re-run the sweep per model. Anthropic reports Opus 5.5 at medium matching or beating Opus 5 at high on coding and knowledge work (prompting guide).
• Chat, classification, and high-volume routes often hold quality at low; coding and long-horizon agents repay high/xhigh. Tune per route, not globally.
• Before building a multi-model cascade, measure the strongest model at lower effort on the same tasks.

5. Model routing

Pattern: small model by default (Haiku, Flash-Lite, Luna), escalate to mid-tier on low confidence or failed validation, reserve the top tier for hard-reasoning paths — with an eval gate per tier. Every model switch forfeits cache reuse.

6. Output and input hygiene

• Structured outputs (Anthropic output_config.format and strict tools, OpenAI Structured Outputs) kill preambles and parse-retry loops.
• No markdown when the consumer is a machine; ban preambles in the system prompt.
• Don't ask current reasoning models to write their reasoning into the answer: it costs output tokens, and on the newest Claude models it can be declined (reasoning_extraction). Lower effort instead.
• Trim tool results to high-signal fields; cap or compact long histories.
• Set max_tokens with room for thinking — a truncated answer you retry costs twice.

7. FinOps observability and attribution

Check current plan pricing on each vendor's site. Tag every call with feature, team, customer_tier, experiment_arm; use per-project API keys or workspaces so usage reports are attributable. Without attribution you can't optimize.

8. Waste patterns

9. RAG cost stack

• Embeddings are usually a rounding error next to generation; upgrade the embedding model only when retrieval evals show lift.
• Rerankers bill per search (Cohere: one query + up to 100 documents; Rerank 4 shipped Apr 2026) — cost scales with candidates, so rerank the top 50, not the top 500.
• Managed vector DBs (Pinecone, Weaviate, Qdrant, Turbopuffer) price on storage, reads, and writes; self-hosting wins at very large scale. Price your own volume on the vendor pages.
• A stable corpus under ~200K tokens can skip RAG entirely: put it in the prompt and cache it (Anthropic Contextual Retrieval).

Examples

Bad: "Use Opus for everything; it's the best." No per-task cost, no eval, no effort setting.

Good: "Support triage runs 40K requests/day with a 6K-token static prefix. Caching the prefix cuts that input to ~0.1× after the first write; the nightly re-classification moves to Batch at 50% off; an effort sweep shows low holds the 94% routing accuracy of high. Projected per-ticket cost drops from $X to $Y (prices from the pricing page, 2026-10-12)."

Case studies (Klarna, Intercom Fin) are useful for direction only; budget from your own attribution data, not vendor blog numbers.

Checklist

• [ ] Prices quoted with the date checked and the source URL
• [ ] Cache hit rate measured from usage fields; static prefix byte-stable; TTL chosen from request gaps
• [ ] Every non-real-time path on Batch or Flex
• [ ] Effort swept per route against an eval before any model routing
• [ ] Routing tiers each gated by an eval; cache-scope loss accounted for
• [ ] Structured outputs on machine-consumed paths; no preambles; max_tokens leaves room for thinking
• [ ] RAG: rerank depth bounded; long-context-with-cache considered for small corpora
• [ ] Every call tagged for per-feature attribution; per-project keys or workspaces
• [ ] Cost reported per completed task, not per request

Composition / References

Within-domain pairings:

• Pairs with the sibling AI skill on RAG patterns (embeddings, rerank, vector DB costs).
• Pairs with the sibling AI skill on agent design (multi-agent and tool-loop cost).
• Pairs with the sibling AI skill on evals (every effort or routing change needs an eval gate; eval runs belong on Batch).

Primary sources:

• Anthropic pricing
• Anthropic prompt caching
• Anthropic batch processing
• Anthropic effort
• OpenAI pricing
• Gemini pricing
• Langfuse token and cost tracking
• RouterArena, arXiv:2510.00202