<< All versions
Skill v1.0.0
currentAutomated scan100/100zju-real/clawgui/gui-mobile
──Details
PublishedApril 29, 2026 at 10:06 PM
Content Hashsha256:80092ac6c5b502a9...
Git SHAadf33a2882c6
──Files
Files (1 file, 4.4 KB)
SKILL.md4.4 KBactive
SKILL.md · 109 lines · 4.4 KB
version: "1.0.0" name: gui-mobile description: Control a connected mobile device (Android/HarmonyOS) via natural language using the gui_execute tool. metadata: {"nanobot":{"emoji":"📱","always":true,"requires":{"bins":["adb"]}}}
GUI Mobile Use
Use the gui_execute tool to control a connected Android or HarmonyOS phone through natural language instructions. The tool captures the phone screen, reasons about it with a vision-language model, and performs actions (tap, swipe, type, home, back, etc.) in a loop until the task completes.
Prerequisites
- ADB installed and on PATH (
adbfor Android,hdcfor HarmonyOS) - Phone connected via USB with USB debugging enabled
- ADB Keyboard installed and enabled on the device
- Download: https://github.com/senzhk/ADBKeyBoard
- Install:
adb install ADBKeyboard.apk - Enable: Settings → Languages & Input → Virtual Keyboard → ADB Keyboard
Usage
Basic
gui_execute(task="Open WeChat and send 'I will be late' to Zhang San")
With step limit
gui_execute(task="Search for Bluetooth headphones on Taobao and add to cart", max_steps=80)
Examples
| User says | Tool call | |
|---|---|---|
| 帮我打开微信给张三发消息说我晚点到 | gui_execute(task="打开微信给张三发一条消息说我晚点到") | |
| Turn on Bluetooth on my phone | gui_execute(task="Open Settings and turn on Bluetooth") | |
| 帮我看看手机上有什么新通知 | gui_execute(task="下拉通知栏查看最新通知并汇报内容") | |
| Open YouTube and search for cooking videos | gui_execute(task="Open YouTube app and search for cooking videos") | |
| 截个屏看看手机现在显示什么 | gui_execute(task="截取当前屏幕截图并描述屏幕内容") | |
| 帮我在淘宝搜蓝牙耳机加入购物车 | gui_execute(task="打开淘宝搜索蓝牙耳机,选择第一个商品加入购物车", max_steps=80) |
Parameters
| Parameter | Type | Required | Default | Description | |
|---|---|---|---|---|---|
task | string | ✅ | — | Natural language description of what to do on the phone | |
max_steps | integer | ❌ | 50 | Max number of GUI action steps (1–200) |
Tips
- Be specific in task descriptions — include app names, contact names, exact text to type.
- Set higher max_steps for multi-step flows (e.g. navigating through several screens).
- The tool will automatically check that ADB is installed, the device is connected, and ADB Keyboard is available before attempting any action.
- If an error is returned about missing ADB Keyboard, guide the user to install it.
Configuration
The GUI tool is configured in ~/.nanobot/config.json under tools.gui:
json
{"tools": {"gui": {"enable": true,"deviceType": "adb","deviceId": null,"maxSteps": 50,"useExternalModel": true,"guiBaseUrl": "https://open.bigmodel.cn/api/paas/v4/","guiApiKey": "YOUR_API_KEY","guiModelName": "autoglm-phone","promptTemplateLang": "cn","promptTemplateStyle": "autoglm","traceEnabled": false,"traceDir": "gui_trace"}}}
| Config Key | Description | |
|---|---|---|
enable | Enable/disable the GUI tool | |
deviceType | adb (Android) or hdc (HarmonyOS) | |
deviceId | Specific device serial (auto-detect if null) | |
maxSteps | Default max steps per task | |
useExternalModel | Use external GUI VLM (true) or nanobot model (false) | |
guiBaseUrl | API base URL for the external GUI model | |
guiApiKey | API key for the external GUI model | |
guiModelName | Model name (e.g. autoglm-phone) | |
promptTemplateLang | Prompt language: cn or en | |
promptTemplateStyle | Prompt template: autoglm / uitars / qwenvl / maiui / guiowl | |
traceEnabled | Save execution trace to traceDir | |
traceDir | Directory for episode folders and screenshots |
Troubleshooting
| Error | Solution | |
|---|---|---|
| ADB not found | Install: brew install android-platform-tools (macOS) or apt install android-tools-adb (Linux) | |
| No devices connected | Connect phone via USB, enable USB debugging in Developer Options | |
| ADB Keyboard not installed | Install APK from https://github.com/senzhk/ADBKeyBoard | |
| External model API key missing | Set guiApiKey in config.json | |
| Task timeout / too many steps | Increase max_steps or simplify the task description |