claude-dev-suite/claude-dev-suite/edge-inference

Edge / On-Device Inference

Why on-device (the decision drivers)

• Latency: on-device token/inference in ~tens of ms vs 200–500ms cloud

round-trip — decisive for voice, AR, control loops.

• Privacy / offline / cost: data never leaves the device; works without

connectivity; no per-call cloud cost.

• Cost of doing it: tight memory/energy/thermal budgets; smaller models;

more engineering.

The hardware tiers (match model to silicon)

Architectural levers

• Quantization is the key enabler: FP16 → INT8 → INT4 trades accuracy for

memory/throughput/energy. Most edge LLMs run INT4/INT8. Validate accuracy loss.

• Model choice: small instruct/distilled models (e.g. 3–8B class) and vision

models sized to the NPU's memory bandwidth, not just TOPS.

• Runtime: TFLite/LiteRT, ONNX Runtime, ExecuTorch, llama.cpp/Ollama,

vendor SDKs (TensorRT for Jetson). Pick what targets the accelerator.

• Budgets: state the TOPS, RAM, and energy-per-inference budget up front —

they bound everything. Batch=1, KV-cache memory dominates LLM RAM.

When to recommend edge vs cloud vs hybrid

• Hard latency / offline / privacy / per-unit cost → on-device.
• Large model / variable load / centralized updates → cloud serving.
• Both needs → hybrid (small local model + cloud escalation): see

hybrid-edge-cloud.