denisovav/flutter_edge_ai/flutter-edge-ai-rag

On-device RAG with Flutter Edge AI

Non-negotiable architecture

1. RAG is not initialized by FlutterEdgeAi.initialize(). Use the independent

flutter_edge_ai_rag package and an app-owned FlutterEdgeAiRag instance.

1. Storage packages are providers: flutter_edge_ai_sqlite registers

SqliteVectorStoreProvider; flutter_edge_ai_qdrant registers QdrantVectorStoreProvider. Application code uses RagIndex.

1. One persistent location belongs to one stable EmbeddingProfile and one

document modality (VectorStoreSpec.documents: text, image or audio). The profile id versions weights, tokenizer, pooling, normalization, and document/query prefixes; a mutable URL or file path is not an identity. Profile ids may not contain control characters.

1. Keep one live RagIndex per location, make opening single-flight, and share

it across widgets. Web SQLite enforces this with an exclusive Web Lock.

1. Dispose indexes before FlutterEdgeAi.dispose() or before disposing a

custom embedder. An index owns its vector store but only borrows its embedder.

1. Declare every filter field in VectorStoreSpec.filterSchema before the

SQLite index is created. Changing SQLite's physical vec0 schema requires a new schema-versioned location and re-index.

1. location is an absolute path on native — a database file for SQLite, a

directory for Qdrant; build it from getApplicationDocumentsDirectory() — and a plain name on Web.

Packages

flutter pub add flutter_edge_ai flutter_edge_ai_rag flutter_edge_ai_sqlite path path_provider

For the default core embedder also add its runtime/tokenizer packages, usually flutter_edge_ai_litertlm and flutter_edge_ai_embeddings. Replace SQLite with flutter_edge_ai_qdrant for qdrant-edge on native platforms. SQLite runs on Android, iOS, Web, macOS, Windows, and Linux; qdrant-edge has no Web arm.

flutter_edge_ai_sqlite needs Flutter 3.47 or newer. On older Flutter use flutter_edge_ai_qdrant (native only) or upgrade Flutter.

flutter_edge_ai_rag re-exports EmbeddingInput, EmbeddingModality, TaskType and the EmbeddingException family, so RAG code needs no core import for them.

Default active embedder

Initialize only embedding runtime pieces in core and install the embedder before opening an index: open() resolves the embedder's profile, which loads the model. Pin model downloads to an immutable revision.

FlutterEdgeAiActiveEmbedder.forActiveModel(namespace:) derives the profile id from the installed file names ('<namespace>@<files joined by +>', with %, @ and + inside a name percent-encoded), so installing a model under different file names changes the id and the old location refuses to open instead of mixing two embedding spaces. EmbeddingGemma 1 and 2 are both 768-dimensional: the id is the only thing that tells them apart. The id follows file NAMES, not bytes: install with revision-qualified filename:s, as below, so new weights under the same download name get a new id too.

import 'package:flutter/foundation.dart';
import 'package:flutter_edge_ai/flutter_edge_ai.dart';
import 'package:flutter_edge_ai_embeddings/flutter_edge_ai_embeddings.dart';
import 'package:flutter_edge_ai_litertlm/flutter_edge_ai_litertlm.dart';
import 'package:flutter_edge_ai_rag/flutter_edge_ai_rag.dart';
import 'package:flutter_edge_ai_sqlite/flutter_edge_ai_sqlite.dart';
import 'package:path/path.dart' as p;
import 'package:path_provider/path_provider.dart';
 
const revision = '29888fcee3216acadc7e844906e5fe0d79a61875';
const modelBase =
    'https://huggingface.co/litert-community/embeddinggemma-300m/resolve/$revision';
// EmbeddingGemma is gated: build with --dart-define=HUGGINGFACE_TOKEN=hf_...
const hfToken = String.fromEnvironment('HUGGINGFACE_TOKEN');
 
await FlutterEdgeAi.initialize(
  embeddingBackends: const [LiteRtEmbeddingBackend()],
  embeddingTokenizers: const [GemmaEmbeddingTokenizers()],
  huggingFaceToken: hfToken.isEmpty ? null : hfToken,
);
await FlutterEdgeAi.installEmbedder()
    .modelFromNetwork(
      '$modelBase/embeddinggemma-300M_seq256_mixed-precision.tflite',
      filename: 'embeddinggemma-300m-seq256-mp__rev-29888fcee321.tflite',
    )
    .tokenizerFromNetwork(
      '$modelBase/sentencepiece.model',
      filename: 'embeddinggemma-300m__sentencepiece__rev-29888fcee321.model',
    )
    .install();
 
final rag = FlutterEdgeAiRag(
  providers: const [SqliteVectorStoreProvider()],
);
// Native: an absolute path to the database file. Web: a plain name.
const databaseName = 'knowledge-v1.db';
final databasePath = kIsWeb
    ? databaseName
    : p.join((await getApplicationDocumentsDirectory()).path, databaseName);
final index = await rag.open(
  spec: VectorStoreSpec(
    providerId: SqliteVectorStoreProvider.providerId,
    location: databasePath,
    filterSchema: FilterSchema(fields: [
      FilterField(name: 'lang', type: FilterFieldType.string),
      FilterField(name: 'year', type: FilterFieldType.number),
    ]),
  ),
  embedder: FlutterEdgeAiActiveEmbedder.forActiveModel(namespace: 'knowledge'),
);

The EmbeddingGemma repo is gated: the Hugging Face account behind the token must accept the Gemma license on the model page, or the download is refused. Keep the token out of source — it is compiled into the app, so a shipped app should download from a repo that needs none.

An explicit id is the alternative when one store must outlive a file rename, or serve several bundles known to produce identical vectors. Name everything that shapes the vectors:

import 'package:flutter_edge_ai_rag/flutter_edge_ai_rag.dart';
 
final embedder = FlutterEdgeAiActiveEmbedder(
  profileId:
      'embeddinggemma-300m-seq256-mp-rev-29888fcee321-'
      'retrieval-prefix-meanpool-l2-v1',
);

The adapter pins the embedding model that is active when it is created and resolves the model again on every operation, so it follows core's embedder cache. After switching embedders it throws StateError: create a new embedder and open a new index at a new location. A request that failed because core replaced the model before it ran (EmbedderClosedException) is retried once. Queries over text documents use TaskType.retrievalQuery; pass queryTask: for another task, e.g. TaskType.questionAnswering.

GGUF embeddings (flutter_edge_ai_llama 0.0.3)

LlamaEmbeddingBackend in flutter_edge_ai_llama 0.0.3 (core 2.5.0, llamadart 0.15.1) runs EmbeddingGemma 2 GGUF text embeddings natively: measured on macOS CPU/Metal and Linux/Windows CPU; Android (CPU only) and iOS are not qualified, and only the Q8_0 file below is numerically qualified. A .gguf carries its own tokenizer: a tokenizerFrom…() makes install() throw ArgumentError.

App setup (that package's README, Add it to your app, and doc/platforms.md): hooks: user_defines: llamadart: llamadart_native_runtimes: [llama_cpp] in the app's pubspec, or the workspace root's; on iOS/macOS also llamadart_llama_cpp_flutter: 0.15.0 in the app's own dependencies, flutter config --enable-swift-package-manager, and iOS 16.4 / macOS 14 deployment targets; on Android arm64-v8a and x86_64 only. From flutter_edge_ai_litertlm 1.19.0 LiteRT-LM keeps Swift Package Manager on too, and both native libraries load in one macOS app; running both engines' models in one app is not qualified.

import 'package:flutter_edge_ai/flutter_edge_ai.dart';
import 'package:flutter_edge_ai_llama/flutter_edge_ai_llama.dart';
 
await FlutterEdgeAi.initialize(
  embeddingBackends: const [LlamaEmbeddingBackend()],
);
await FlutterEdgeAi.installEmbedder()
    .modelFromNetwork(
      'https://huggingface.co/ggml-org/embeddinggemma-2-GGUF/resolve/'
      'bfcd298762cc34d0357ece5ebdd31791a3a374d8/embeddinggemma-2-Q8_0.gguf',
    ) // no tokenizer
    .install();
final embedder = await FlutterEdgeAi.getActiveEmbedder();

• Text only: supportImage, supportAudio and PreferredBackend.npu throw

EmbeddingLoadException. On Android (every device) and iOS simulators an explicit gpu falls back to CPU with a warning, or throws with LlamaEmbeddingBackend(backendFallback: false); activeBackend reports what loaded.

• Only a mean-pooled gemma-embedding2 GGUF loads; any other architecture or

pooling throws EmbeddingLoadException.

• maxInputTokens is 1024, task prefix and special tokens included. A longer

input throws EmbeddingInputTooLongException and is never truncated.

• A URL that does not end in .gguf needs

modelFromNetwork(url, filename: '<name>.gguf'); without it install() throws StateError asking for a tokenizer.

• On Web it claims no file: with it the only backend registered, installing a

.gguf throws UnsupportedError before anything is downloaded.

• Keep GGUF and LiteRT under separate profile ids, even at 768 dimensions.

FlutterEdgeAiActiveEmbedder.forActiveModel(namespace:) does it by itself, because the file names differ; with an explicit profileId, name the format and quantization (e.g. embeddinggemma2-gguf-q8_0). A store bound to the LiteRT id refuses to open with the GGUF embedder.

Custom embedding providers (core 2.5.0)

Engine authors import the SPI:

import 'package:flutter_edge_ai/engine.dart';

Core accepts self-contained .gguf embedding files without a separate tokenizer (the app recipe is the GGUF embeddings section above). When backends are registered, installation requires one that can run the file; with none registered, installation can proceed. flutter_edge_ai_embeddings contains tokenizer implementations only; runtime passes belong to their engine packages.

Providers declare explicit accelerator capabilities with honouredBackends(spec); core computes the cache key. Automatic requests reuse only a non-NPU embedder for the same installed model, while NPU requests always retain a separate key. Repeated explicit CPU/GPU requests reuse their originating fallback until a rebuild is needed or the model is closed. GPU-capable providers must report their actual loaded accelerator through EmbeddingPassInfo.activeBackend; legacy fixed descriptors remain compatible. Reporting providers print one release-visible WARNING per new fallback load, including an NPU request; reuse does not repeat it. EmbeddingModel.maxInputTokens is the effective loaded context/single-pass batch capacity, including task prefixes and special tokens. A GGUF profile configured with a 1024-token micro-batch cannot embed a longer chunk even if the file names a larger context. Core hosts, caches and closes the pass.

EmbeddingGemma 2 (one .litertlm file)

EmbeddingGemma 2 ships as one .litertlm bundle that carries its own tokenizer: install the model alone — no tokenizerFrom…() call, and no flutter_edge_ai_embeddings or embeddingTokenizers: needed. The repo is not gated:

const eg2Revision = '9be6e8b90982095dc05c2bd162e4b954ee4dbac7';
// One id for the 270m, 440m and 740m bundles: their text towers are identical.
const eg2ProfileId = 'embeddinggemma2-litertlm-text-4410a540ce75-prefix-v1';
 
await FlutterEdgeAi.initialize(
  embeddingBackends: const [LiteRtEmbeddingBackend()],
);
await FlutterEdgeAi.installEmbedder()
    .modelFromNetwork(
      'https://huggingface.co/litert-community/'
      'embeddinggemma-2-text-270m-litert-lm/resolve/$eg2Revision/'
      'embeddinggemma-2-text-270m.litertlm',
    )
    .install();
final embedder = await FlutterEdgeAi.getActiveEmbedder();
final limit = embedder.maxInputTokens; // 1024
 
final rag = FlutterEdgeAiRag(
  providers: const [SqliteVectorStoreProvider()],
);
final eg2Index = await rag.open(
  spec: VectorStoreSpec(
    providerId: SqliteVectorStoreProvider.providerId,
    location: p.join(
      (await getApplicationDocumentsDirectory()).path,
      'knowledge-embeddinggemma2-text-v1.db',
    ),
  ),
  embedder: FlutterEdgeAiActiveEmbedder(profileId: eg2ProfileId),
);