evilfreelancer/screenshotting-gui/screenshotting-gui

Screenshotting a GUI app

Produce documentation-grade screenshots of a running GUI application without a window manager and without touching the operator's screen. The app runs on a throwaway Xvfb display; you drive it to the exact state you want with synthetic X input, capture the frame, and annotate it.

Everything the model already knows (what X11, ImageMagick or a clipboard are) is omitted on purpose - this file is only the parts that are easy to get wrong.

The one rule that keeps it safe

Run on display `:99` (Xvfb), never `:0`. :0 is the operator's live desktop - clicking, typing and screenshotting it is a privacy and correctness disaster.

The subtle trap: xctl.py / xpaste.py default their own target to :99, but any GUI app you launch reads `$DISPLAY`. If $DISPLAY is still :0, the app opens on the real screen while your captures read :99 - winlist comes back empty and you capture a black frame. Always `export DISPLAY=:99` for the app launch, not just XCTL_DISPLAY.

Toolchain

Deterministic pieces live in scripts/ (run any with --help):

Requirements: Xvfb, python3 + python-Xlib, ImageMagick (import, convert). Capture default is ImageMagick import (no extra dependency, no max-size ceiling). Fallback if ImageMagick is unavailable: python-Xlib root.get_image + Pillow (~2x faster, but needs Pillow and must tile a root above ~4.2 MP; pixel order is BGRX, not RGB).

Workflow

DISP=:99
export XCTL_DISPLAY=$DISP
S=scripts   # path to this skill's scripts
 
# 1. Isolated display
Xvfb $DISP -screen 0 1920x1080x24 -ac >/tmp/xvfb.log 2>&1 &
sleep 1; DISPLAY=$DISP xdpyinfo >/dev/null || { echo "Xvfb down"; exit 1; }
 
# 2. Launch the app ON :99 (export DISPLAY, not only XCTL_DISPLAY)
DISPLAY=$DISP <app> &            # e.g. an IDE, a browser, a native app
sleep 2                          # heavy Electron/JVM apps need 3-8 s; poll with winlist
python3 $S/xctl.py winlist       # confirm the window is on :99, note its geometry
 
# 3. Drive it to the exact state you want (see command table below)
python3 $S/xctl.py click 1490 1003        # open a menu, toggle a panel, etc.
sleep 0.5
 
# 4. Capture (full, or crop to the region)
python3 $S/xctl.py shot /tmp/raw.png --crop 1270 560 650 520
 
# 5. Annotate and save
python3 $S/annotate.py box /tmp/raw.png out.png 1300 600 1720 680

Find pixel coordinates from a winlist geometry plus a full shot you eyeball; without a window manager, screen coordinate = window origin + widget offset.

Driving the app - xctl.py

Focus follows X input focus: click into the target widget before `type` or `ctrl+v`, or the keys go nowhere.

Unicode / non-Latin text - xpaste.py

type is ASCII-only by design. For Cyrillic or any symbol-heavy string (@, {}), never type it - paste it:

python3 $S/xpaste.py "любой текст с @ и {скобками}" 15 &   # own CLIPBOARD ~15 s
sleep 0.6                                                   # let it acquire first
python3 $S/xctl.py click 1580 200                          # focus the input
python3 $S/xctl.py key ctrl+v

Annotating - annotate.py

python3 $S/annotate.py box  in.png out.png X1 Y1 X2 Y2   # rounded #ff6b35 outline
python3 $S/annotate.py crop in.png out.png X Y W H       # crop a region
python3 $S/annotate.py arrow in.png out.png X1 Y1 X2 Y2  # line + arrowhead at the tip
python3 $S/annotate.py vstack out.png top.png bottom.png # stack two honest crops

Default accent #ff6b35, width 5, radius 14.

Gotchas

• `$DISPLAY` vs `$XCTL_DISPLAY`. The app reads $DISPLAY; the scripts default to

:99. A mismatch silently launches the app on :0 while you capture :99 (empty winlist, black frames, and your keystrokes leaking to the real desktop). Export DISPLAY=:99 for the launch.

• No window manager is the simpler default. Without a WM there is no reparenting,

so screen coordinate = window origin + widget offset (clean math for place). Apps that need a WM for focus/stacking may want openbox/twm; try without first.

• Focus before keys. XTEST keystrokes and ctrl+v route to the focused window -

click the widget first.

• `scroll` ignores coordinates - it wheels at the current pointer; move first.
• Timing. Give the app ~1-2 s (heavy Electron/JVM apps 3-8 s) before winlist;

give xpaste.py ~0.5-1 s to acquire the selection before ctrl+v, or you paste stale or empty clipboard.

• `type` skips non-ASCII (with a stderr warning) - use xpaste.py. Shifted ASCII

(_ { } @ =) is handled.

• Blank-frame check. A real capture has non-zero pixel stddev; a near-black frame

means the app is not mapped or you shot the wrong display.

• `pkill -f xpaste.py` footgun - it also matches the killing shell's own command

line. Kill background helpers by numeric PID.

• UI coordinates drift between app versions and window sizes - never hardcode from

memory; winlist + a fresh full shot first.

• Reset app state between shots. For repo-mutating apps (an IDE agent editing

files), snapshot and restore - e.g. git -C <project> checkout . && git clean -fd.

Quick reference

export XCTL_DISPLAY=:99
Xvfb :99 -screen 0 1920x1080x24 -ac & sleep 1
DISPLAY=:99 <app> & sleep 3
python3 scripts/xctl.py winlist
python3 scripts/xpaste.py "текст" 15 & sleep 0.6; python3 scripts/xctl.py key ctrl+v
python3 scripts/xctl.py shot raw.png --crop X Y W H
python3 scripts/annotate.py box raw.png final.png X1 Y1 X2 Y2
pkill -x Xvfb   # or kill the specific PID when done