denisovav/flutter_edge_ai/flutter-edge-ai-inference

Running a model with flutter_edge_ai

Rules

1. Depend on flutter_edge_ai and an engine package, and import both. Engine packages do not re-export core.
2. Register the engine in FlutterEdgeAi.initialize(inferenceEngines: [...]). Core ships none.
3. Declare fileType on installModel. It defaults to ModelFileType.task, and the declaration — never the file name — picks the engine.
4. maxTokens is the context window. Cap the reply with maxOutputTokens on the session or chat.
5. Pass isUser: true on every user Message.
6. Close a session or chat when its conversation ends. Keep the model while the feature is in use, and close it when the app no longer needs it.
7. Keep Hugging Face tokens out of source: read them with String.fromEnvironment. That keeps a token out of git, not out of the app — it is compiled into the binary, and on web into main.dart.js. A shipped app should download from a repo that needs no token.
8. On Android, set minSdk 30 for anything built on .litertlm — inference, embeddings, speech.
9. Read references/platform-setup.md before the first build on a platform: without those entries the model fails to load or the app is killed for memory.

Setup — the recommended engine (.litertlm)

flutter pub add flutter_edge_ai flutter_edge_ai_litertlm

import 'package:flutter/widgets.dart';
import 'package:flutter_edge_ai/flutter_edge_ai.dart';
import 'package:flutter_edge_ai_litertlm/flutter_edge_ai_litertlm.dart';
 
// In main(), before runApp: initialize() needs the Flutter binding.
WidgetsFlutterBinding.ensureInitialized();
await FlutterEdgeAi.initialize(inferenceEngines: [LiteRtLmEngine()]);
 
await FlutterEdgeAi.installModel(
  modelType: ModelType.gemma4,
  fileType: ModelFileType.litertlm,
)
    .fromNetwork(
      'https://huggingface.co/litert-community/gemma-4-E2B-it-litert-lm/resolve/main/gemma-4-E2B-it.litertlm',
    )
    .withProgress((int percent) => print('downloading: $percent%'))
    .install();
 
final InferenceModel model = await FlutterEdgeAi.getActiveModel(maxTokens: 1024);

Gemma 4 E2B is 2.6 GB and needs no token. On web use gemma-4-E2B-it-web.litertlm from the same repo (2.0 GB).

Supported models, their file types and engines: https://flutteredge.ai/docs/models.

install() skips the download when the file is already installed from the same source, so calling it at every launch is safe. The latest install becomes the model getActiveModel loads.

A gated repo needs a token, given once:

const hfToken = String.fromEnvironment('HUGGINGFACE_TOKEN');
 
await FlutterEdgeAi.initialize(
  inferenceEngines: [LiteRtLmEngine()],
  huggingFaceToken: hfToken.isEmpty ? null : hfToken,
);

Build with --dart-define=HUGGINGFACE_TOKEN=hf_....

When a Hugging Face repo publishes a deployment manifest, one call picks the variant and its tested runtime settings. The engine carries its own resolver, so registering LiteRtLmEngine is enough:

final install = await FlutterEdgeAi.installModel(
  modelType: ModelType.general,
  fileType: ModelFileType.litertlm,
).fromHuggingFace('litert-community/LFM2.5-230M').install();
 
final model = await FlutterEdgeAi.getActiveModel(defaults: install.runtime);

Other sources on the same builder: .fromAsset(path) for a model bundled in the app, .fromFile(path) for one already on disk, .fromBundled(name) for a platform-bundled resource. A fromFile file is used in place and stays the app's: uninstalling leaves it, and it is active again after a restart. Installing the same file name again from another source replaces the install.

modelType tells flutter_edge_ai how the model writes tool calls and reasoning, and on some engines it also picks the prompt format. Gemma 3 and Gemma 3n are ModelType.gemmaIt — there is no gemma3. The full set: ModelType.general, ModelType.gemmaIt, ModelType.gemma4, ModelType.deepSeek, ModelType.qwen, ModelType.qwen3, ModelType.qwen35, ModelType.llama, ModelType.hammer, ModelType.functionGemma, ModelType.phi. A wrong type still generates text; tool calls and reasoning then arrive as raw text.

Traps

Core not imported

• Symptom: Undefined name 'FlutterEdgeAi', Undefined class 'InferenceModel', with only the engine package imported.
• Fix: import 'package:flutter_edge_ai/flutter_edge_ai.dart'; as well.

flutter_gemma names after upgrading to 2.0

• Symptom: Undefined name 'FlutterGemma', Undefined name 'GemmaLogLevel', or a package:flutter_gemma/... import that does not resolve.
• Fix: the flutter_gemma names are gone in 2.0. Rename the dependencies and imports (flutter_gemma* → flutter_edge_ai*; flutter_gemma_rag_sqlite and flutter_gemma_rag_qdrant become flutter_edge_ai_sqlite and flutter_edge_ai_qdrant), then run dart fix --apply, which renames the old class names. Delete the old flutter-gemma-* skill directories and run dart run skills@ get --all again — they still teach the old names. RAG also left core in 2.0: see the flutter-edge-ai-rag skill.

Deep imports after upgrading to 2.3

• Symptom: Target of URI doesn't exist: 'package:flutter_edge_ai/core/model.dart' — or any other package:flutter_edge_ai/core/…, /mobile/…, /web/…, /desktop/… or flutter_edge_ai_interface.dart import.
• Fix: since flutter_edge_ai 2.3.0 core's internals are private under lib/src. Apps import package:flutter_edge_ai/flutter_edge_ai.dart; an engine or tokenizer package imports package:flutter_edge_ai/engine.dart. The deprecated members went in the same release — setModelPath, installModelFromAsset, initialize(enableWebCache:) and the rest; core's MIGRATION.md lists each with its replacement.

After upgrading to 2.5

• Symptom: The method 'fromLegacyUrl' isn't defined for the type 'InferenceModelSpec', generateEmbedding/generateEmbeddings/getDimension undefined, or a deprecation warning on FlutterEdgeAiPlugin.instance.createModel.
• Fix: build the spec as InferenceModelSpec(name:, modelSource: ModelSource.network(url) /.asset/.bundled/.file, modelType:); fromLegacyUrl defaulted to ModelType.general, so pass a model type explicitly. Replace createModel(...) (removed in 2.6.0) with FlutterEdgeAi.getActiveModel(...) — on Android, iOS and desktop it ignored modelType and fileType and returned the active model anyway — or with FlutterEdgeAi.openModel for several models. Embeddings: embed(EmbeddingInput.text(t), taskType: TaskType.retrievalQuery), embedAll, dimension (see the flutter-edge-ai-rag skill and core's MIGRATION.md).

No engine registered

• Symptom: StateError: No inference engine can handle this model (ModelFileType.litertlm). Add the engine package to pubspec.yaml and pass it in inferenceEngines: of FlutterEdgeAi.initialize(...)
• Fix: add the engine package and register its provider — or fix fileType if the wrong engine is registered.
• A .gguf model needs flutter_edge_ai_llama (llama.cpp through llamadart): register LlamaEngineProvider() and install with fileType: ModelFileType.gguf. Its build setup — the hooks: user_defines: llamadart: block, the Apple companion package with SwiftPM, iOS 16.4 / macOS 14, Android arm64-v8a and x86_64 only — is in that package's README.

`maxTokens` used as a reply length

// WRONG — asks for a 100-token context, not a 100-token reply
final model = await FlutterEdgeAi.getActiveModel(maxTokens: 100);

• Symptom: replies are as long as ever. On native .litertlm the value is raised to 1024, the smallest context those models support, and only a debug-mode log says so. The web .litertlm engine does not take the value at all; on MediaPipe it is the real limit.
• Fix:

final model = await FlutterEdgeAi.getActiveModel(maxTokens: 1024);
final session = await model.createSession(maxOutputTokens: 100);

Use 4096 or more with images or audio — one image costs hundreds of tokens.

`isUser` left out

• Symptom: an empty response, no error. Message.isUser defaults to false, so the prompt is read as the model's own turn.
• Fix: Message(text: prompt, isUser: true).

Same reply every time

• Symptom: identical output for identical input.
• Causes: topK: 1 is passed somewhere (greedy; it was the default up to flutter_edge_ai 2.1); the model's family decodes greedily by default (ModelType.phi, ModelType.hammer); or the seed is fixed — an unset randomSeed is 1 (or the .litertlm bundle's own), so a freshly loaded model repeats its first sampled answer.
• Fix: leave temperature, topK and topP unset to get the model's own sampler or its family's defaults (SamplingParams.forModelType), or pass a temperature — with no topK, a greedy family or bundle then contributes no topK: 1. Set them on the first session after getActiveModel: on .litertlm the sampler of the first generation stays in effect for later sessions (a one-time warning says so when a later session asks for other values), and its random state carries on, so a new chat already gets a new answer. MediaPipe starts each session from its seed: vary randomSeed there for a different answer to the same prompt.
• Exception: decoding stays greedy whatever you pass on .litertlm running on the NPU (activeBackend == PreferredBackend.npu); a one-time warning says so when you set any of them. ONNX follows the model's genai_config.json, which is greedy unless it sets do_sample; values you pass override it.

`Session is closed`

• Symptom: StateError: Session is closed from a session or chat that is still in use.
• Cause: createSession and createChat fill one slot per model; creating another closes the one before.
• Fix: one conversation at a time, or openSession / openChat for several (below). On the web .litertlm engine a second createSession hands back the session that is already open, history and all, rather than a fresh one — close the current chat before creating the next.

Generate

// Sampling left unset: the model's own sampler, else its family's defaults.
final InferenceModelSession session = await model.createSession(
  maxOutputTokens: 256,
);
try {
  await session.addQueryChunk(Message(text: prompt, isUser: true));
  final String reply = await session.getResponse();
} finally {
  await session.close();
}

Streaming:

final reply = StringBuffer();
await session.addQueryChunk(Message(text: prompt, isUser: true));
await for (final token in session.getResponseAsync()) {
  reply.write(token); // update the UI here
}

To stop early, call await session.stopGeneration() — chat.stopGeneration() on a chat. Cancelling the stream subscription detaches Dart but does not stop native decoding on every engine. On .litertlm on Android, iOS and desktop the chat keeps working after a stop — whether the app called stopGeneration() or cancelled its subscription. With flutter_edge_ai_litertlm 1.15.0 and later it is rewound to before the stopped turn, which is put back with its partial reply, so the model still sees images and audio from earlier turns. With 1.14.0 and earlier, and on the NPU, it is rebuilt from text instead, and those are gone — re-send an image there if the next question is about it. The legacy flutter_gemma_litertlm package needs 1.8.1 or later for a stopped chat to answer again at all.

Multi-turn chat

final InferenceChat chat = await model.createChat(
  systemInstruction: 'You are a concise assistant.',
  maxOutputTokens: 512,
);
try {
  await chat.addQueryChunk(Message(text: prompt, isUser: true));
  final reply = StringBuffer();
  await for (final r in chat.generateChatResponseAsync()) {
    switch (r) {
      case TextResponse(:final token):
        reply.write(token);
      case ThinkingResponse() || FunctionCallResponse() || ParallelFunctionCallResponse():
        break;
    }
  }
} finally {
  await chat.close();
}