frankiethull/kuzco/kuzco-extract-text

kuzco-extract-text

Extract text visible in an image (OCR) using a local vision language model, exposed as an ellmer tool by the kuzco R package.

When to use

The user wants the text content of an image: screenshots, scanned documents, photos of signs or labels, memes, charts with annotations. If the task is about the image as a whole rather than its text, use kuzco-classification instead.

Setup

library(kuzco)
library(ellmer)
 
chat <- chat_ollama(model = "qwen2.5vl")
chat$register_tool(tool_llm_image_extract_text())

Tool arguments (LLM-controlled)

• image (required): local file path to a jpeg, jpg, or png image.
• additional_prompt (optional): extra instructions, e.g. "only transcribe the headline".
• language (optional): output language, e.g. "English".

Result

Compact JSON text with the extracted text and a confidence score. Reproduce the transcribed text faithfully; flag low confidence to the user rather than silently "fixing" unclear passages.

Example

User: "What does the sign say in img/sign.jpg?"

1. Call kuzco_image_extract_text with image = "img/sign.jpg".

2. Quote the extracted text and note the confidence.