lukasa1993/tanstack-skills-extracted/tanstack-ai-core-middleware

Middleware

Dependency note: This skill builds on ai-core. Read it first for critical rules.

Setup — Analytics Tracking Middleware

import { chat, toServerSentEventsResponse } from '@tanstack/ai'
import { openaiText } from '@tanstack/ai-openai'
import { trackAnalytics, reportError } from './analytics'
export async function POST(request: Request) {
  const { messages, threadId } = await request.json()
  const stream = chat({
    adapter: openaiText('gpt-5.5'),
    messages,
    threadId,
    middleware: [
      {
        onStart: (ctx) => {
          console.log('Chat started:', ctx.model)
        },
        onFinish: (ctx, info) => {
          trackAnalytics({ model: ctx.model, tokens: info.usage?.totalTokens })
        },
        onError: (ctx, info) => {
          reportError(info.error)
        },
      },
    ],
  })
  return toServerSentEventsResponse(stream)
}

Hooks Reference

Every hook receives a ChatMiddlewareContext as its first argument, which provides requestId, streamId, phase, iteration, chunkIndex, callMetadata, model, provider, signal, abort(), defer(), and more. parentRunId names the run this one continues. subagentRunId is set only inside a subagent and names its card.

Terminal hooks (onFinish, onAbort, onError) are mutually exclusive -- exactly one fires per chat() invocation.

Sampling in `onConfig`: temperature, topP, and maxTokens are not

first-class fields on ChatMiddlewareConfig. To adjust sampling from

middleware, return a partial that mutates config.modelOptions using the

provider's native key (e.g. OpenAI temperature / max_output_tokens,

Anthropic max_tokens, Ollama nested options.num_predict). Returning a

top-level temperature/maxTokens has no effect.

Phase values

ctx.phase is one of:

Separate-finalization path (adapters without native-combined support):

• onStructuredOutputConfig fires before onConfig at the structured-output boundary.
• onConfig re-fires at the same boundary with ctx.phase === 'structuredOutput', receiving the post-onStructuredOutputConfig view of the config (minus outputSchema).
• onChunk and onUsage fire for every chunk and usage event emitted by the structured-output call, with ctx.phase === 'structuredOutput'.
• onIteration does not fire for finalization — it is agent-loop-only.
• Terminal `info` and structured-output: info.usage / info.finishReason / info.content reflect the agent loop's terminal state, NOT the finalization step. Finalization state is intentionally segregated to keep agent-loop semantics clean. For a tools-less chat({ outputSchema }) run, info.usage is undefined and info.finishReason is null (no agent-loop iteration produced RUN_FINISHED). To capture finalization tokens, use onUsage — it fires for both agent-loop iterations and the final call. For the structured-output result itself, observe the structured-output.complete CUSTOM event in onChunk.

Native-combined output:

• The schema-constrained JSON is produced by a normal agent-loop iteration. onStructuredOutputConfig does not fire, ctx.phase remains 'modelStream', and onIteration fires for that iteration.
• info.content includes the structured JSON because it is agent-loop text. Middleware observes the structured-output.complete event in onChunk during the same phase.

Both paths:

• On successful completion, onFinish fires once after the structured result completes. Terminal-hook exclusivity still holds.
• By onFinish, ctx.messages includes the completed terminal assistant messages. Native-combined output keeps the structured result on its terminal assistant message. The separate-finalization path can preserve the agent loop's plain-text message followed by a distinct structured-output message.

onStructuredOutputConfig

A dedicated config hook that fires only at the separate-finalization boundary. Use it to transform the JSON Schema sent to the provider (inject $defs, strip vendor-incompatible keywords) or to apply structured-output- specific config changes that should not affect the agent-loop adapter calls.

Signature:

import type {
  ChatMiddlewareContext,
  StructuredOutputMiddlewareConfig,
} from '@tanstack/ai'
// Excerpt of the `ChatMiddleware` interface exported by '@tanstack/ai'
interface ChatMiddleware {
  onStructuredOutputConfig?: (
    ctx: ChatMiddlewareContext,
    config: StructuredOutputMiddlewareConfig,
  ) =>
    | void
    | null
    | Partial<StructuredOutputMiddlewareConfig>
    | Promise<void | null | Partial<StructuredOutputMiddlewareConfig>>
}

`StructuredOutputMiddlewareConfig` shape:

import type { ChatMiddlewareConfig, JSONSchema } from '@tanstack/ai'
// As exported by '@tanstack/ai'
interface StructuredOutputMiddlewareConfig extends Omit<
  ChatMiddlewareConfig,
  'tools'
> {
  outputSchema: JSONSchema // The JSON Schema being sent to the provider
}

Note the Omit<…, 'tools'>: there is no `config.tools` on this hook. The structured-output call is the final, tool-free call, so reading or returning tools here is a compile error, not a no-op. Transform tools in onConfig instead.

Ordering rule:

• onStructuredOutputConfig fires before onConfig at the structured-output boundary.
• onConfig re-fires at the same boundary with ctx.phase === 'structuredOutput', receiving the post-onStructuredOutputConfig view of the config (minus outputSchema).
• Use onConfig for general-purpose transforms that apply to every adapter call (agent-loop iterations and the final structured-output call).
• Use onStructuredOutputConfig when you need to transform the JSON Schema or apply structured-output-specific behavior.

Core Patterns

Pattern 1: Analytics and Logging Middleware

Use onStart, onFinish, onUsage, and onError for comprehensive observability. Use ctx.defer() for non-blocking async side effects that should not block the stream.

import {
  chat,
  toServerSentEventsResponse,
  type ChatMiddleware,
} from '@tanstack/ai'
import { openaiText } from '@tanstack/ai-openai'
const analytics: ChatMiddleware = {
  name: 'analytics',
  onStart: (ctx) => {
    console.log(`[${ctx.requestId}] Chat started — model: ${ctx.model}`)
  },
  onUsage: (ctx, usage) => {
    console.log(`[${ctx.requestId}] Tokens: ${usage.totalTokens}`)
  },
  onFinish: (ctx, info) => {
    ctx.defer(
      fetch('/api/analytics', {
        method: 'POST',
        body: JSON.stringify({
          requestId: ctx.requestId,
          model: ctx.model,
          duration: info.duration,
          tokens: info.usage?.totalTokens,
          finishReason: info.finishReason,
        }),
      }),
    )
  },
  onError: (ctx, info) => {
    ctx.defer(
      fetch('/api/errors', {
        method: 'POST',
        body: JSON.stringify({
          requestId: ctx.requestId,
          error: String(info.error),
          duration: info.duration,
        }),
      }),
    )
  },
}
export async function POST(request: Request) {
  const { messages, threadId } = await request.json()
  const stream = chat({
    adapter: openaiText('gpt-5.5'),
    messages,
    threadId,
    middleware: [analytics],
  })
  return toServerSentEventsResponse(stream)
}

Pattern 2: Tool Interception Middleware