sybil-solutions/glm-flash-lite/glm53-offload-setup

glm-flash-lite: set up GLM-5.3-Flash on the user's machine

You set up glm-flash-lite on the machine in front of you, measure it, tune it, and report how it compares with the reference host. Work step by step. Show the user what you found before anything slow or destructive.

To learn how the tiers work, read docs/how-it-works.md in the repo. For error messages, read docs/troubleshooting.md.

Ground rules

• Never touch the user's other GPU work. Before starting, list running containers and GPU processes

(docker ps, nvidia-smi). Do not stop, restart or reconfigure any of them without asking. Pick a GPU that is idle, or ask which one to use.

• Ask first before anything that needs root or is hard to undo:
• rebooting;
• BIOS changes (PCIe bifurcation, PCIe generation);
• creating a RAID (mdadm) or a filesystem (mkfs), which erases the drives;
• changing mounts, /etc/fstab, or Docker and NVIDIA runtime configuration.

Never run mkfs, mdadm --create or wipefs on a device until the user has confirmed that exact device name.

• Never cap output length. Never set max_tokens or max_new_tokens on smoke tests or benchmarks. Let every answer

finish naturally. bench/sweep.py already does this.

• Bound every command that can hang (docker, ssh, network, anything reading a socket). Use timeout 20 ..., and

treat no answer as "unavailable".

• Keep the API on 127.0.0.1. The server has no authentication.
• Pin everything. Use the image digest and model revision below, never latest or a branch.
• Report only what you measured. If a cell was not run, write "not run".

Pins

Step 1: detect the hardware

Run these and keep the answers. Each one says what it is for.

# GPU: name, VRAM, driver, PCIe link, and what is already running on it
timeout 20 nvidia-smi --query-gpu=index,name,memory.total,memory.used,driver_version,pcie.link.gen.max,pcie.link.gen.current,pcie.link.width.current --format=csv
timeout 20 nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
# CPU: logical CPUs, cores, AVX2 / FMA / F16C
nproc; lscpu | grep -E 'Model name|^CPU\(s\)|Thread|Core|Socket|NUMA node\(s\)'
grep -o -w -E 'avx2|fma|f16c' /proc/cpuinfo | sort -u
# RAM: total and available, swap/zram, and any cgroup limit on this shell
free -g; swapon --show; cat /sys/fs/cgroup/memory.max 2>/dev/null
# DRAM channels (optional, needs root): populated DIMMs
sudo -n dmidecode -t memory 2>/dev/null | grep -E '^\s+Size: [0-9]' | wc -l
# Storage: NVMe drives, filesystems, RAID, free space
lsblk -o NAME,MODEL,SIZE,TYPE,FSTYPE,MOUNTPOINTS,ROTA,TRAN
cat /proc/mdstat; findmnt -t xfs,ext4,btrfs,zfs -o TARGET,SOURCE,FSTYPE,OPTIONS
df -h
for d in /sys/class/nvme/nvme*; do echo "$d $(cat $d/model) link $(cat $d/device/current_link_speed 2>/dev/null) x$(cat $d/device/current_link_width 2>/dev/null)"; done
# Docker and the NVIDIA runtime
timeout 10 docker info --format '{{.ServerVersion}} {{.Runtimes}}' ; nvidia-ctk --version
ls -l /etc/cdi/ /var/run/cdi/ 2>/dev/null; timeout 10 nvidia-ctk cdi list 2>/dev/null
command -v hf || command -v huggingface-cli

Then answer these questions:

• GPU. Is it 24 GB or more, with a driver that supports CUDA 13.0 (580 or newer)? The image is built for sm_86

(RTX 3090 / A5000); sm_89 (RTX 4090) runs the same binaries, but not measured. Blackwell (RTX 5090, sm_120) needs the image rebuilt with --build-arg TORCH_CUDA_ARCH_LIST="8.6;12.0", also not measured. Below 23,000 MiB the entrypoint refuses to start.

• GPU link. The reference host runs PCIe 4.0 x16. A Gen3 or x8 link slows the admission copies, which are 28 % of

a decode token on the reference.

• CPU. Does it have AVX2, FMA and F16C (needed by the CPU lane in nvme and fast)? How many physical cores and

logical CPUs? The measured layout needs 40 logical CPUs (cpus 2-39).

• RAM. How much is available for this server, after what the user needs for everything else? Is swap or zram on?

It is fine to leave it on, as long as the container sets --memory-swap equal to --memory.

• NVMe. Which drive or array is fastest, and is it local NVMe with xfs or ext4? Is there a RAID0? Does it have

117.3 GB free for the store? The 125.3 GB of weights can go on any disk; only their non-expert part is read, once at load.

• Docker. Does Docker answer? Is the NVIDIA container toolkit installed, and is the CDI spec current (see

Troubleshooting)?

Step 2: choose the mode

RAM below means the memory you will give the container (--memory). Leave the user's normal working set free.

Notes:

• With 120-240 GB free, nvme with a cap of 55g or more beats exact (17 vs 12.6 tok/s) unless outputs must be

bit-exact. Caps above 58 GiB are not measured; 58 vs 55 GiB showed no difference.

• nvme decode is near-exact: paired decode KL 0.0047-0.0051, top-1 agreement 0.983-0.986. Prefill is exact in every

mode.

Reference host: 1x RTX 3090 (PCIe 4.0 x16), EPYC 7443P (24 cores, 48 threads), 8-channel DDR4, and 4x Samsung 9100 PRO 1 TB in md RAID0 (512k chunk, xfs) on a PCIe 4.0 x16 card (~26 GB/s).

What to expect on weaker hardware. None of this is measured. Say so to the user, and measure.

• One NVMe drive instead of a RAID0. A Gen4 drive peaks around 7 GB/s. On the reference, capping the array at

8 GB/s cost 8 % of C1, 33 % of C4 and 27 % of prefill at 55 GiB. Expect at least that.

• At 16 GiB, decode read ~20 GB/s on the reference, so one drive will be much slower there. Give it as much RAM as

you can.

• A Gen3 drive (~3.5 GB/s) is worse again.
• Fewer cores. The CPU lane runs one thread per physical core (22 on the reference), and it already finishes last

in 54-86 % of layers. Fewer cores means a slower lane. Measure nvme against nvme-exact: with very few cores the lane may not pay off.

• 2-channel DDR. At C1 the reference moves ~48 GB/s through DRAM, 35 % of its ~138 GB/s. A 2-channel desktop has a

much lower peak, so DRAM may become a limit.

• GPU on PCIe Gen3 or x8. Admission copies run at ~14.5 GB/s on the reference's Gen4 x16. On a slower link they

take longer.

• More VRAM (32 GB, 48 GB). The cache sizes itself from free VRAM, so you get more slots and more hits. Drop the

GLM53_EC_MAX_SLOTS cap, or raise it.

Tell the user which mode you chose and why, and what to expect. Then continue.

Step 3: prepare the store location (only if needed)

The store needs a local NVMe filesystem that supports O_DIRECT: plain xfs or ext4 on the drive or on an md RAID0.

• Not suitable: a loop file on btrfs, network filesystems, FUSE, or tmpfs. Some LUKS setups refuse O_DIRECT too.
• Check it with the image's own probe. It needs no GPU:

``bash IMG=ghcr.io/sybil-solutions/glm53-flash-offload@sha256:aa74200f16c86588b101be2315aed2f643d148559179eef52a02599016b56e69 timeout 900 docker pull "$IMG" mkdir -p /path/on/nvme/glm53 && docker run --rm -v /path/on/nvme/glm53:/nvx "$IMG" python3 /opt/glm53/docker/preflight.py odirect /nvx && echo O_DIRECT_OK ``

• Several empty NVMe drives that the user agrees to dedicate: a RAID0 helps most at small RAM caps. Propose

mdadm --create /dev/md/glm53 --level=0 --chunk=512K --raid-devices=N <devices>, then mkfs.xfs, then a mount. This erases the drives, so run it only after the user confirms the exact device names.

• A quad-M.2 PCIe card needs the slot bifurcated x4x4x4x4 in the BIOS, and the right PCIe generation. If

lsblk shows fewer drives than are installed, tell the user. Do not reboot into the BIOS yourself.

Step 4: download the weights (pinned)

W=/path/to/GLM-5.3-Flash-exl3-3.05bpw     # 125.3 GB, any local disk
hf download turboderp/GLM-5.3-Flash-exl3 --revision 332ab457b709b7ba30dd9a448be5de03b80a7ac9 --local-dir "$W"
test -f "$W/config.json" && test -f "$W/quantization_config.json" && du -sh "$W"

If the hf CLI is missing: pip install -U huggingface_hub. The download takes a while. Run it in the background and check progress with du -sh.

Step 5: pack and verify the store

S=/path/on/nvme/glm53
docker run --rm -v "$W":/models:ro -v "$S":/nvx "$IMG" pack-store
docker run --rm -v "$W":/models:ro -v "$S":/nvx:ro "$IMG" verify-store
ls -l "$S"    # glm53_flash_exl3_3.05bpw_experts.bin (117,326,610,432 B) + .json

pack-store does three things: it writes every record, re-reads each one with O_DIRECT and checks its sha256, and byte-compares 300 sampled records against the checkpoint. It runs on the CPU only. verify-store repeats the sha256 pass; it took 11 s on the reference array.

• Any bad count above 0 is a failure. Do not serve from that store.
• The verify-store speed is a first read on the drive's speed: 6-11 GB/s on the reference. A much lower number

predicts slow decode.

Step 6: run

docker run -d --name glm53 --gpus '"device=0"' \
  --memory 55g --memory-swap 55g --shm-size 1g --ulimit memlock=-1 \
  -e GLM53_MODE=nvme -e GLM53_EC_MAX_SLOTS=1376 \
  -p 127.0.0.1:30000:30000 -v "$W":/models:ro -v "$S":/nvx:ro "$IMG"
timeout 600 sh -c 'until curl -sf localhost:30000/health >/dev/null; do sleep 5; done' && echo READY
docker logs glm53 2>&1 | grep -E '^\[glm53\]|nv2: rcs|RAM tier|nv2 CPU tier|startup verify|loaded in'

Adjust the command:

• Device. device=N is the idle GPU from step 1.
• Memory. --memory and --memory-swap take the same value: your cap from step 2.
• Mode. GLM53_MODE is nvme or nvme-exact.
• Expert cache cap. Keep GLM53_EC_MAX_SLOTS=1376 when a desktop runs on the same GPU. On a headless GPU you can

leave it out.

• CPU pinning. With 40 or more logical CPUs, add nothing, or add --cpuset-cpus 2-39 to fence the server's

threads off the rest of the system as the reference did. With fewer CPUs, add nothing; the layout is derived.

• All-RAM modes. Use the all-RAM image with --shm-size 16g --ulimit memlock=-1, no --memory, no /nvx, and

-e GLM53_MODE=exact when needed.

Check the startup lines. Write down these values for the report:

• the RAM tier size (-> RAM tier N slots)
• the expert-cache slots (curl -s localhost:30000/stats, expert_cache[0].slots)
• the CPU layout line (measured layout or derived)
• the startup verify result: the nv2 startup verify line should show 'vram_bad': 0 and 'ram_bad': 0

Step 7: smoke test