Image Understanding Skill
This skill provides specialized image understanding functionality using the z-ai-web-dev-sdk package, enabling AI models to analyze, describe, and extract information from static images.
Skills Path
Skill Location: {project_path}/skills/image-understand
this skill is located at above path in your project.
Reference Scripts: Example test scripts are available in the {Skill Location}/scripts/ directory for quick testing and reference. See {Skill Location}/scripts/image-understand.ts for a working example.
Overview
Image Understanding focuses specifically on static image analysis, providing capabilities for:
- Image description and scene understanding
- Object detection and recognition
- OCR (Optical Character Recognition) and text extraction
- Image classification and categorization
- Visual content analysis
- Quality assessment
- Accessibility (alt text generation)
IMPORTANT: z-ai-web-dev-sdk MUST be used in backend code only. Never use it in client-side code.
Prerequisites
The z-ai-web-dev-sdk package is already installed. Import it as shown in the examples below.
CLI Usage (For Simple Tasks)
For quick image analysis tasks, you can use the z-ai CLI instead of writing code. This is ideal for simple image descriptions, testing, or automation.
Basic Image Analysis
bash
# Describe an image from URL
z-ai vision --prompt "What's in this image?" --image "https://example.com/photo.jpg"
# Using short options
z-ai vision -p "Describe this image" -i "https://example.com/image.png"
Analyze Local Images
bash
# Analyze a local image file
z-ai vision -p "What objects are in this photo?" -i "./photo.jpg"
# Save response to file
z-ai vision -p "Describe the scene" -i "./landscape.png" -o description.json
Multiple Images Comparison
bash
# Compare multiple images
z-ai vision \
-p "Compare these two images and highlight the differences" \
-i "./photo1.jpg" \
-i "./photo2.jpg" \
-o comparison.json
# Analyze a series of images
z-ai vision \
--prompt "What patterns do you see across these images?" \
--image "https://example.com/img1.jpg" \
--image "https://example.com/img2.jpg" \
--image "https://example.com/img3.jpg"
Advanced Analysis with Thinking
bash
# Enable chain-of-thought reasoning for complex tasks
z-ai vision \
-p "Count all people in this image and describe what each person is doing" \
-i "./crowd.jpg" \
--thinking \
-o analysis.json
# Complex object detection with reasoning
z-ai vision \
-p "Identify all safety hazards in this workplace image" \
-i "./workplace.jpg" \
--thinking
Streaming Output
bash
# Stream the analysis in real-time
z-ai vision -p "Provide a detailed description" -i "./photo.jpg" --stream
CLI Parameters
--prompt, -p <text>: Required - Question or instruction about the image(s)--image, -i <URL or path>: Optional - Image URL or local file path (can be used multiple times)--thinking, -t: Optional - Enable chain-of-thought reasoning (default: disabled)--output, -o <path>: Optional - Output file path (JSON format)--stream: Optional - Stream the response in real-time
Supported Image Formats
- PNG (.png) - Best for diagrams, screenshots, graphics with transparency
- JPEG (.jpg, .jpeg) - Best for photos and complex images
- GIF (.gif) - Supports both static and animated images
- WebP (.webp) - Modern format with good compression
- BMP (.bmp) - Uncompressed bitmap format
When to Use CLI vs SDK
Use CLI for:
- Quick image analysis or descriptions
- One-off OCR tasks
- Testing image understanding capabilities
- Simple batch processing scripts
- Generating alt text for accessibility
Use SDK for:
- Multi-turn conversations about images
- Complex image processing pipelines
- Production applications with error handling
- Custom integration with your application logic
- Batch processing with custom business logic
Recommended Approach
For better performance and reliability, use base64 encoding to pass images to the model instead of image URLs.
Basic Image Understanding Implementation
Single Image Analysis
javascript
import ZAI from 'z-ai-web-dev-sdk';
async function analyzeImage(imageUrl, prompt) {
const zai = await ZAI.create();
const response = await zai.chat.completions.createVision({
messages: [
{
role: 'user',
content: [
{
type: 'text',
text: prompt
},
{
type: 'image_url',
image_url: {
url: imageUrl
}
}
]
}
],
thinking: { type: 'disabled' }
});
return response.choices[0]?.message?.content;
}
// Usage examples
const description = await analyzeImage(
'https://example.com/landscape.jpg',
'Describe this landscape in detail, including colors, lighting, and mood'
);
const objectDetection = await analyzeImage(
'https://example.com/room.jpg',
'List all objects visible in this room'
);
Multiple Images Comparison
javascript
import ZAI from 'z-ai-web-dev-sdk';
async function compareImages(imageUrls, question) {
const zai = await ZAI.create();
const content = [
{
type: 'text',
text: question
},
...imageUrls.map(url => ({
type: 'image_url',
image_url: { url }
}))
];
const response = await zai.chat.completions.createVision({
messages: [
{
role: 'user',
content: content
}
],
thinking: { type: 'disabled' }
});
return response.choices[0]?.message?.content;
}
// Usage
const comparison = await compareImages(
[
'https://example.com/before.jpg',
'https://example.com/after.jpg'
],
'What are the key differences between these before and after images?'
);
Base64 Image Support (Recommended)
javascript
import ZAI from 'z-ai-web-dev-sdk';
import fs from 'fs';
import path from 'path';
async function analyzeLocalImage(imagePath, prompt) {
const zai = await ZAI.create();
// Read image file and convert to base64
const imageBuffer = fs.readFileSync(imagePath);
const base64Image = imageBuffer.toString('base64');
// Determine MIME type based on file extension
const ext = path.extname(imagePath).toLowerCase();
const mimeTypes = {
'.png': 'image/png',
'.jpg': 'image/jpeg',
'.jpeg': 'image/jpeg',
'.gif': 'image/gif',
'.webp': 'image/webp',
'.bmp': 'image/bmp'
};
const mimeType = mimeTypes[ext] || 'image/jpeg';
const response = await zai.chat.completions.createVision({
messages: [
{
role: 'user',
content: [
{
type: 'text',
text: prompt
},
{
type: 'image_url',
image_url: {
url: `data:${mimeType};base64,${base64Image}`
}
}
]
}
],
thinking: { type: 'disabled' }
});
return response.choices[0]?.message?.content;
}
// Usage
const result = await analyzeLocalImage(
'./product-photo.jpg',
'Analyze this product image for e-commerce listing'
);
Advanced Use Cases
OCR and Text Extraction
javascript
import ZAI from 'z-ai-web-dev-sdk';
async function extractText(imageUrl, options = {}) {
const zai = await ZAI.create();
const prompt = options.preserveLayout
? 'Extract all text from this image. Preserve the exact layout, formatting, and structure.'
: 'Extract all visible text from this image.';
const response = await zai.chat.completions.createVision({
messages: [
{
role: 'user',
content: [
{ type: 'text', text: prompt },
{ type: 'image_url', image_url: { url: imageUrl } }
]
}
],
thinking: { type: 'disabled' }
});
return response.choices[0]?.message?.content;
}
// Usage examples
const receiptText = await extractText(
'https://example.com/receipt.jpg',
{ preserveLayout: true }
);
const businessCardInfo = await extractText(
'https://example.com/business-card.jpg'
);
Object Detection and Counting
javascript
import ZAI from 'z-ai-web-dev-sdk';
async function detectObjects(imageUrl, objectType) {
const zai = await ZAI.create();
const prompt = objectType
? `Count and locate all ${objectType} in this image. Provide their positions and describe each one.`
: 'Detect and list all objects in this image with their approximate locations.';
const response = await zai.chat.completions.createVision({
messages: [
{
role: 'user',
content: [
{ type: 'text', text: prompt },
{ type: 'image_url', image_url: { url: imageUrl } }
]
}
],
thinking: { type: 'enabled' } // Enable thinking for complex counting
});
return response.choices[0]?.message?.content;
}
// Usage
const peopleCount = await detectObjects(
'https://example.com/crowd.jpg',
'people'
);
const allObjects = await detectObjects(
'https://example.com/room.jpg'
);
Image Classification and Tagging
javascript
import ZAI from 'z-ai-web-dev-sdk';
async function classifyAndTag(imageUrl) {
const zai = await ZAI.create();
const prompt = `Analyze this image and provide a comprehensive classification:
1. Primary category (e.g., nature, urban, portrait, product)
2. Subject matter (main focus of the image)
3. Style or mood (e.g., professional, casual, artistic, vintage)
4. Color palette description
5. Suggested tags (10-15 keywords, comma-separated)
Format your response as structured JSON.`;
const response = await zai.chat.completions.createVision({
messages: [
{
role: 'user',
content: [
{ type: 'text', text: prompt },
{ type: 'image_url', image_url: { url: imageUrl } }
]
}
],
thinking: { type: 'disabled' }
});
const content = response.choices[0]?.message?.content;
try {
return JSON.parse(content);
} catch (e) {
return { rawResponse: content };
}
}
// Usage
const classification = await classifyAndTag(
'https://example.com/photo.jpg'
);
console.log('Tags:', classification.tags);
Quality Assessment
javascript
import ZAI from 'z-ai-web-dev-sdk';
async function assessImageQuality(imageUrl) {
const zai = await ZAI.create();
const prompt = `Assess the technical quality of this image:
1. Sharpness and focus (1-10)
2. Exposure and brightness (1-10)
3. Color balance (1-10)
4. Composition (1-10)
5. Any technical issues (blur, noise, artifacts, etc.)
6. Overall quality rating (1-10)
7. Suggestions for improvement
Provide specific feedback for each criterion.`;
const response = await zai.chat.completions.createVision({
messages: [
{
role: 'user',
content: [
{ type: 'text', text: prompt },
{ type: 'image_url', image_url: { url: imageUrl } }
]
}
],
thinking: { type: 'disabled' }
});
return response.choices[0]?.message?.content;
}
Accessibility - Alt Text Generation
javascript
import ZAI from 'z-ai-web-dev-sdk';
async function generateAltText(imageUrl, context = '') {
const zai = await ZAI.create();
const prompt = context
? `Generate concise, descriptive alt text for this image. Context: ${context}. Focus on the most important visual elements that convey the image's purpose.`
: 'Generate concise, descriptive alt text for this image suitable for screen readers. Focus on key visual elements.';
const response = await zai.chat.completions.createVision({
messages: [
{
role: 'user',
content: [
{ type: 'text', text: prompt },
{ type: 'image_url', image_url: { url: imageUrl } }
]
}
],
thinking: { type: 'disabled' }
});
return response.choices[0]?.message?.content;
}