google-ai-edge/litert-samples/on-device-verification

On-device verification

A device result is done when three things hold:

1. the model compiles and runs on the accelerator you claim it runs on,
2. the output matches the source model numerically and on a task-level

gate,

1. the record names the device, the runtime version, and the residency

line. A number without those is not reproducible and not a result.

Host-side checks (the CompiledModel checker used in gpu-clean-conversion) exercise the host GPU. The device has its own shader compiler, its own precision behavior, and its own memory ceiling — every failure mode in the table below was hit by a model that had already passed on the host. The Python API below runs on the host only (ai-edge-litert ships wheels for Linux, macOS and Windows); on a phone the same inputs go through the app's inference class or a push-run-pull binary (last bullet of "Watch for").

Loop

1. Dump references from the source model, once. Fixed inputs — one real sample plus fixed-seed random — saved as .npy next to the recipe (dump_*_ref.py). These are ground truth for every later step; regenerate them only when the source model changes.

2. Run the same inputs on the host runtime, CPU then GPU, then on the device. One argument switches the accelerator (ai-edge-litert 2.2.0, 2.3.0):

from ai_edge_litert.compiled_model import CompiledModel
from ai_edge_litert.hardware_accelerator import HardwareAccelerator
model = CompiledModel.from_file(
    "model.tflite", hardware_accel=HardwareAccelerator.GPU)  # or .CPU

The device-CPU run is the control. If it already diverges from the source dump, the problem is the conversion, not the GPU — go back to gpu-clean-conversion. A full worked example of the A/B lives in this repo at samples/litert/speech_recognition/convert/verify_tflite.py (its ai_edge_torch.compiled_model import no longer resolves with litert-torch 0.9.4; read it as ai_edge_litert.compiled_model).

Ask for the strict accelerator. Compiling with HardwareAccelerator.CPU | HardwareAccelerator.GPU permits partial delegation and hides fallback; use the combined mode only to discover which ops fell back after a strict GPU compile fails. Strict is per graph, not per op: when the GPU cannot compile the graph, from_file raises (Some ops are not accelerated. Add kLiteRtHwAcceleratorCpu … in 2.3.0); when the delegate declines one node while building (TopK requires src tensor C dimension to be divisible by 4), that node runs on the CPU with only the log line, the call returns, and is_fully_accelerated() still reads true (measured with the 2.2.0 and 2.3.0 wheels on a Mac). The log is the residency signal.

3. Read the delegate log before reading any numbers. The CompiledModel runtime prints, per op it declines and then as a total:

Following operations are not supported by GPU delegate:
TRANSPOSE: Tensor "..." has bad input dims size: 6.
N operations will run on the GPU, and the remaining M operations will run on the CPU.

Record it as N/T (N on the GPU of T nodes; benchmark_model prints the same as Replacing N out of T node(s) with delegate plus a partition count). Any op left on the CPU means part of the graph runs there — decide whether that is acceptable before quoting any accuracy or latency number.

4. Gate the output three ways.

• Numeric: correlation and max abs diff against the source dump.

Expect genuine GPU fp16 to drift in the last digits — exact equality is a symptom, not a pass (see below).

• Task: argmax match, IoU, token-for-token greedy decode — whatever

the model is for.

• Artifact, for generative models: the decoded text / audio / image.

An artifact gate catches argmax ties that numeric tolerance misses.

5. Record it: device, runtime version, N/T (and the partition count where benchmark_model gave one), correlation, max abs diff, task result — one row per device in the recipe README.

The silent CPU fallback

A recurring false positive: everything runs and the numbers match perfectly. If the GPU output is bit-identical to the device CPU fp32 output, re-read the residency line before believing the GPU ran — a full N/T decides, not the equality; fp16 GPU runs usually drift in the last digits, and an fp32-forced or integer graph may not.

When the comparison is between two bundles of the same weights (a re-pack against its source, external against inline weights), byte-compare the text on single-thread CPU (--cpu-thread-count 1) or on the GPU: at the default thread count the CPU greedy text of one file moves between fresh processes, so a byte difference there is not a file difference. Long replies in several fresh processes per file are the test; short answers agree even when long ones do not.

Device-only failures

Watch for