peterfriese/system-one-foundation-models/clef-foundation-models

Cloudflare Clef Foundation Models Bridge

[!TIP]

Multi-Backend Architecture: For projects integrating on-device Core ML (LayaOnDevice), self-hosted HTTP daemons (LayaFoundationModels), and TypeSafe Jev cloud alongside Cloudflare Clef, refer to the unified system-one-foundation-models skill and package target. This skill focuses specifically on Cloudflare Clef & Clef-Flash multimodal decision evaluation, AVFoundation camera integration, and Cloudflare edge gateway infrastructure.

This skill guides the design, implementation, and testing of applications integrating Cloudflare Clef (27B) and Cloudflare Clef-Flash (9B) multimodal decision models via Apple's Foundation Models framework (FoundationModels) in Swift 6.

1. Core Mental Model: Multimodal Decisions, Not Autoregressive Chat

Cloudflare Clef and Clef-Flash are multimodal System One decision models, not conversational vision-language models (VLMs). Rather than autoregressively generating textual captions or chat descriptions, Clef evaluates application text and visual attachments in a single forward pass, returning calibrated decision primitives in sub-100ms with zero output tokens (output_tokens: 0).

• Multimodal State: Contextual text plus up to 4 image attachments passed inside Prompt { ... }.
• `@Generable` Type $\to$ Decision Questions: Strongly-typed Swift structs and enums define the questions evaluated over that visual scene.
• Dual Signal Output:

1. The Typed Content: What the model judged (response.content containing enums, booleans, and rubric scores).
2. Calibrated Epistemic Uncertainty: Whether to automate the action (response.judgement(...), response.decision(...), response.scoreValue(...)).

Model Family Comparison

2. Pluggable Endpoint Topologies (ClefEndpoint)

Configure the target endpoint via the ClefEndpoint enum. ClefLanguageModel connects to global Cloudflare edge nodes or local inference servers:

import FoundationModels
import ClefFoundationModels
 
// 1. Cloudflare Workers AI Edge (Direct Edge REST API)
let workersEndpoint = ClefEndpoint.workersAI(
    accountID: "cloudflare-account-id",
    model: .clefFlash // or .clef
)
let model = ClefLanguageModel(endpoint: workersEndpoint, apiToken: "cloudflare-api-token")
let session = LanguageModelSession(model: model)
 
// 2. Cloudflare AI Gateway (Centralized caching, analytics, rate limiting)
let gatewayEndpoint = ClefEndpoint.gateway(
    accountID: "cloudflare-account-id",
    gatewayID: "mobile-production",
    model: .clefFlash
)
let gatewayModel = ClefLanguageModel(endpoint: gatewayEndpoint, apiToken: "cloudflare-api-token")
 
// 3. Local Native Runner (Docker Model Runner, vLLM, Cog, MLX on localhost)
let localEndpoint = ClefEndpoint.local(port: 8000, model: .clefFlash)
let localModel = ClefLanguageModel(endpoint: localEndpoint)
 
// 4. Custom Fully-Qualified URL
let customEndpoint = ClefEndpoint.custom(
    URL(string: "https://ai.internal.mycompany.com/v1/evaluate")!,
    model: .clef
)
let customModel = ClefLanguageModel(endpoint: customEndpoint, apiToken: "internal-bearer-token")

3. Multimodal Inputs with Apple Foundation Models (Attachment)

Passing Image Attachments via Prompt

Visual inputs are attached directly to standard Apple Foundation Models Prompt builder blocks using Attachment(cgImage) or Attachment(data:type:):

import FoundationModels
import ClefFoundationModels
import CoreGraphics
 
let session = LanguageModelSession(
    model: ClefLanguageModel(
        endpoint: .workersAI(accountID: accountID, model: .clefFlash),
        apiToken: apiToken
    )
)
 
// Construct multimodal prompt
let prompt = Prompt {
    "Examine this manufacturing component for surface defects and dimensional integrity."
    Attachment(capturedCGImage)
    // Or pass raw Data: Attachment(data: jpegData, type: .jpeg)
}
 
let response = try await session.respond(
    to: prompt,
    generating: DefectInspectionDecision.self
)
 
let decision: DefectInspectionDecision = response.content

Apple Framework Capability Gating: LanguageModelCapabilities.Capability.vision

Apple's FoundationModels framework performs client-side capability validation before passing generation requests to an underlying LanguageModelExecutor.

If a caller submits a Prompt containing visual attachments:

• The session runtime verifies model.capabilities.
• The capability set must include `.vision` alongside .guidedGeneration.
• If .vision is missing, LanguageModelSession traps immediately with LanguageModelSession.GenerationError.unsupportedCapability: "The selected model does not support image input. Consider trying again with a different model."
• ClefLanguageModel explicitly advertises both capabilities:

``swift public var capabilities: LanguageModelCapabilities { LanguageModelCapabilities([.guidedGeneration, .vision]) } ``

• Supported MIME/UTTypes: PNG (.png), JPEG (.jpeg), and WebP (.webP).

Image Preprocessing Best Practice: 1024px Downsampling & 0.8 JPEG Compression

Modern Apple mobile hardware captures photos at resolutions up to 48MP. Uploading raw, uncompressed frames or large PNGs leads to severe failure modes (see Tech Note 0015):

1. BPE Tokenizer Blowout: Cloudflare's edge gateway serializes images under the images array as RFC 2397 Data URLs (data:image/jpeg;base64,...). Unrecognized schemas or oversized text blobs fall back to standard Byte-Pair Encoding (BPE) text tokenizers, calculating 700,000+ input tokens and instantly exceeding the 64k context limit (HTTP 413).
2. Payload & Patch Inflation: Uncompressed frames consume 10–30 MB of bandwidth and generate excessive vision transformer patches, degrading latency.

Production Best Practice: Always downsample camera frames to a maximum dimension of 1024px and compress to JPEG at 0.8 quality:

// Utility available in SystemOneCore / CameraManager
if let downsampledData = TranscriptAttachmentExtractor.convertCGImageToJPEG(
    rawCGImage,
    maxDimension: 1024,
    quality: 0.8
) {
    let prompt = Prompt {
        "Verify item condition"
        Attachment(data: downsampledData, type: .jpeg)
    }
}

This reduces upload payloads to 100–300 KB, maps cleanly to ~1,000 vision patch tokens, and guarantees sub-100ms edge processing.

Multimodal Guardrails

SystemOneImage.Guardrails enforces strict validation prior to wire serialization:

• Maximum Images: Up to 4 images per evaluation (Guardrails.maxImageCount = 4).
• Maximum Resolution: Up to 16 megapixels per image (Guardrails.maxMegapixels = 16.0).
• Maximum Payload Size: 13 MiB total payload cap (Guardrails.maxPayloadBytes = 13 * 1024 * 1024).

4. @Generable Schema Modeling for Visual Decisions

SchemaTranslator maps Apple @Generable types directly to System One decision primitives:

Negative Constraints (Strictly Forbidden)

• ❌ No unconstrained `String` properties: Clef models do not generate free-form text. A String property without enum choices throws SystemOneError.invalidSchema.
• ❌ No unstructured collections: Arrays of open-ended values ([String]) or dictionaries are unsupported.
• ❌ No requests without schemas: Calling session.respond(to: prompt) without a generating: schema throws SystemOneError.structuredOutputRequired.
• ❌ No tool calling: LanguageModelCapabilities only includes [.guidedGeneration, .vision].

Production Schema Example

import FoundationModels
import ClefFoundationModels
 
@Generable
public enum ItemCategory: String, Sendable, CaseIterable, Codable {
    case snack
    case beverage
    case electronics
    case document
    case household
    case clothing
    case unknown
}
 
@Generable
public struct VisualInspectionDecision: Sendable, Codable, Equatable {
    @Guide(description: "Is a physical item or subject clearly recognized in the camera view?")
    public var isRecognized: Bool
 
    @Guide(description: "Primary category classification of the recognized item")
    public var itemCategory: ItemCategory
 
    @Guide(description: "Physical condition rubric: 0 (unusable/damaged), 1 (worn), 2 (good), 3 (pristine)", .range(0...3))
    public var conditionScore: Int
 
    @Guide(description: "Safety approval: does the item satisfy safety standards without visible hazards or contamination?")
    public var safetyApproval: Bool
}

5. Visual Confidence Routing with RoutingPolicy

Never write naive if prob > 0.5 checks:

• `0.50` indicates maximum epistemic uncertainty, not "half true".
• The undecided band ($0.35 \dots 0.65$) indicates the model is genuinely undecided (answer == nil).
• A probability of 0.05 is a confident "no" ($\text{decisiveness} = \max(p, 1 - p) = 0.95$), which routes to .auto with answer: false.

Use RoutingPolicy to triage visual judgments into operational actions:

let policy = RoutingPolicy(
    escalateBelow: 0.60,
    autoAtOrAbove: 0.85,
    undecidedBand: 0.35...0.65
)
 
// 1. Boolean (Noul) Visual Verification
let recognizedJudgement = response.judgement(for: "isRecognized", policy: policy)
switch recognizedJudgement.decision {
case .auto:
    if recognizedJudgement.answer == true {
        print("Item recognized with high confidence:", recognizedJudgement.confidence)
    } else {
        print("Confident rejection: No valid item in viewfinder frame")
    }
case .confirm:
    promptUserToHoldSteady() // Leaning
case .escalate:
    requestManualOverride()  // In undecided band
}
 
// 2. Categorical Classification (Choice)
let categoryAction = response.decision(for: "itemCategory", policy: policy)
switch categoryAction {
case .auto:     routeToCatalog(decision.itemCategory)
case .confirm:  askUserToConfirmCategory(decision.itemCategory)
case .escalate: routeToUncategorizedQueue()
}
 
// 3. Rubric Score Telemetry (ScoreValue)
if let conditionScore = response.scoreValue(for: "conditionScore") {
    print("Continuous Weighted Score:", conditionScore.value) // e.g. 2.72
    print("Discrete Rounded Level:", conditionScore.rounded)   // 3 (Pristine)
    print("Score Confidence:", conditionScore.confidence)      // 0.91
}

6. AVFoundation Camera Integration Pipeline

For interactive iOS, iPadOS, and macOS apps, capture live camera frames, downsample using CIContext, and evaluate via LanguageModelSession:

import SwiftUI
@preconcurrency import AVFoundation
import CoreImage
import CoreGraphics
import Observation
import FoundationModels
import ClefFoundationModels