ag-ui-protocol/ag-ui/agui-dotnet-multimodal

AG-UI .NET — multimodal messages

Goal: include an image (or other binary content) in a turn so a multimodal model can describe or reason about it.

A multimodal message is an ordinary Microsoft.Extensions.AI ChatMessage whose Contents mixes a TextContent with one or more binary parts. AGUIChatClient carries those parts across the AG-UI wire; the server streams the model's reply as usual.

Install

dotnet add package AGUI.Client

Microsoft.Extensions.AI supplies ChatMessage, TextContent, DataContent, and UriContent.

Attach inline bytes

Use DataContent(bytes, mediaType) to embed the data directly in the message:

using AGUI.Client;
using Microsoft.Extensions.AI;
 
byte[] imageBytes = await File.ReadAllBytesAsync("photo.png");
 
var messages = new List<ChatMessage>
{
    new(ChatRole.User,
    [
        new TextContent("Describe this image"),
        new DataContent(imageBytes, "image/png"),
    ]),
};
 
await foreach (var update in client.GetStreamingResponseAsync(messages))
{
    Console.Write(update.Text);
}

The media type must match the bytes (image/png, image/jpeg, audio/wav, application/pdf, …) so the model decodes them correctly.

Reference a hosted URL

When the asset already lives at a URL, use UriContent(uri, mediaType) instead — the bytes never pass through your process:

new ChatMessage(ChatRole.User,
[
    new TextContent("What's in this picture?"),
    new UriContent("https://example.com/cat.jpg", "image/jpeg"),
]);

Prefer UriContent to reference an asset already hosted at a URL; use DataContent for local bytes the model host can't reach by URL.

Anti-patterns

• Sending a large asset as inline `DataContent`. Inline binary is base64-encoded into the request, inflating it ~33% and forcing the whole payload through memory on both ends. For anything sizable that is reachable by URL, send UriContent and let the model host fetch it.
• Targeting a model that isn't multimodal. The content parts cross the wire fine, but a text-only deployment ignores or rejects the image. Point the server's IChatClient at a vision-capable model/deployment.

Verify

1. The assistant's streamed reply describes the actual image content (not a generic "I can't see images" message).
2. The request carries the binary part: the user message on the wire has both a text part and an image/data part with the media type you set.