drawthingsai/draw-things-community/cpuscript

Draw Things CPU Proxy Server Setup

Use this workflow from Scripts/ServerManagement/CPUScript.

Files

• InitCPUServer.sh: install/update Docker, install Tailscale, run tailscale up --ssh, lock the root password, and pull drawthingsai/draw-things-proxy-server-cli:v20260928.2 plus Envoy 1.39.1 pinned to its Linux amd64 image digest.
• InitCertificate.sh: request a Let's Encrypt cert for compute.drawthings.ai with standalone certbot on port 80, set the permissions required by the non-root proxy container, and install the renewal deploy hook.
• restart-drawthings-tls.sh: Certbot deploy hook that repairs renewed certificate permissions and restarts proxy_service and envoy_grpc_web_proxy so both processes load the renewed certificate.
• LaunchCPUServer.sh: start proxy_service on public 443 and Tailscale control port TS_IP:50002.
• LaunchEnvoyServer.sh: copy the provided Envoy config into root-owned /etc/drawthings/envoy, validate it, and start envoy_grpc_web_proxy with host networking. Preserve the previous container for rollback and restore it automatically if replacement startup or readiness fails.
• model-list: mounted through its containing directory as /app/Documents/<filename> so host-side edits and control-panel writes target the same file.
• envoy-config.yaml: copied to a separate file per deployment and mounted read-only as /etc/envoy/envoy.yaml; uses typed CORS and upstream HTTP/2 configuration.

Inputs

Have these ready before launch:

• New CPU server SSH target, usually root@<tailscale-ip>.
• Datadog API key. Treat it as a secret; do not commit or echo it in summaries.
• Let's Encrypt notification email if issuing a new cert.
• Optional existing cert source server, if DNS is not routed to the new VM yet.
• Updated model-list.
• A known-good envoy-config.yaml; the admin socket must include a port, typically 127.0.0.1:9901.

Initial State Check

Check SSH, OS, Docker, Tailscale, certs, containers, and ports:

ssh root@<new-ts-ip> 'hostname; uname -a; id; command -v apt || true; command -v docker || true; command -v tailscale || true; tailscale ip -4 2>/dev/null || true'
ssh root@<new-ts-ip> 'docker ps -a --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}" 2>/dev/null || true; ss -ltnp | grep -E ":(80|443|8443|50002)" || true; ls -la /etc/letsencrypt/live/compute.drawthings.ai 2>/dev/null || true'

Upload the Script Bundle

Install the CPUScript files on the new server:

ssh root@<new-ts-ip> 'mkdir -p /root/CPUScript'
scp InitCertificate.sh InitCPUServer.sh LaunchCPUServer.sh LaunchEnvoyServer.sh restart-drawthings-tls.sh envoy-config.yaml model-list root@<new-ts-ip>:/root/CPUScript/
ssh root@<new-ts-ip> 'chmod +x /root/CPUScript/*.sh; wc -l /root/CPUScript/model-list; ls -la /root/CPUScript'

If the model list changes, recopy only that file and verify checksum:

scp model-list root@<new-ts-ip>:/root/CPUScript/model-list
shasum -a 256 model-list
ssh root@<new-ts-ip> 'shasum -a 256 /root/CPUScript/model-list; wc -l /root/CPUScript/model-list'

If the local Envoy config is stale, copy the known-good config from the current CPU server, then upload it:

scp root@<current-cpu-ts-ip>:/root/envoy/envoy-config.yaml ./envoy-config.yaml
rg -n "port_value|address:|filename:|sni:|admin:" envoy-config.yaml
scp envoy-config.yaml root@<new-ts-ip>:/root/CPUScript/envoy-config.yaml

Run CPU Server Init

Run the dependency/image setup:

ssh root@<new-ts-ip> 'cd /root/CPUScript && ./InitCPUServer.sh'

Expect apt output, Docker install/update, Tailscale install/update, root password lock, and Docker image pulls. A pending kernel upgrade warning is not a launch blocker.

Certificate Strategy

The scripts expect:

/etc/letsencrypt/live/compute.drawthings.ai/fullchain.pem
/etc/letsencrypt/live/compute.drawthings.ai/privkey.pem

If compute.drawthings.ai already routes to the new VM and public port 80 reaches it, run:

ssh root@<new-ts-ip> 'cd /root/CPUScript && printf "%s\n" "<email>" | ./InitCertificate.sh'

If DNS is not routed to the new VM yet, certbot standalone will fail. In that case, copy only the active cert and key from the current CPU server. Do not copy the whole /etc/letsencrypt unless deliberately migrating Certbot renewal ownership; that also copies ACME account keys, renewal configs, and archived cert material.

ssh root@<current-cpu-ts-ip> 'ls -l /etc/letsencrypt/live/compute.drawthings.ai/fullchain.pem /etc/letsencrypt/live/compute.drawthings.ai/privkey.pem; openssl x509 -in /etc/letsencrypt/live/compute.drawthings.ai/fullchain.pem -noout -subject -issuer -dates'
 
mkdir -p /private/tmp/cpu-cert-transfer
chmod 700 /private/tmp/cpu-cert-transfer
scp root@<current-cpu-ts-ip>:/etc/letsencrypt/live/compute.drawthings.ai/fullchain.pem root@<current-cpu-ts-ip>:/etc/letsencrypt/live/compute.drawthings.ai/privkey.pem /private/tmp/cpu-cert-transfer/
chmod 644 /private/tmp/cpu-cert-transfer/fullchain.pem
chmod 600 /private/tmp/cpu-cert-transfer/privkey.pem
 
ssh root@<new-ts-ip> 'mkdir -p /etc/letsencrypt/live/compute.drawthings.ai && chmod 755 /etc/letsencrypt /etc/letsencrypt/live /etc/letsencrypt/live/compute.drawthings.ai'
scp /private/tmp/cpu-cert-transfer/fullchain.pem /private/tmp/cpu-cert-transfer/privkey.pem root@<new-ts-ip>:/etc/letsencrypt/live/compute.drawthings.ai/
ssh root@<new-ts-ip> 'chmod 644 /etc/letsencrypt/live/compute.drawthings.ai/fullchain.pem; chmod 644 /etc/letsencrypt/live/compute.drawthings.ai/privkey.pem; openssl x509 -in /etc/letsencrypt/live/compute.drawthings.ai/fullchain.pem -noout -subject -issuer -dates'
rm -f /private/tmp/cpu-cert-transfer/fullchain.pem /private/tmp/cpu-cert-transfer/privkey.pem

Use 0644 for privkey.pem with the current Docker launch because the proxy process inside the container may not run as root. If proxy_service logs Permission denied opening privkey.pem, fix permissions and recreate the container.

Automatic Renewal and TLS Reload

Copying only fullchain.pem and privkey.pem creates usable TLS files but does not migrate Certbot renewal ownership. A managed certificate must appear in certbot certificates and have all of:

/etc/letsencrypt/renewal/compute.drawthings.ai.conf
/etc/letsencrypt/archive/compute.drawthings.ai/
/etc/letsencrypt/live/compute.drawthings.ai/ -> ../../archive/compute.drawthings.ai/

InitCertificate.sh installs this executable deploy hook:

/etc/letsencrypt/renewal-hooks/deploy/restart-drawthings-tls.sh

Certbot normally creates /etc/letsencrypt/archive as root-only and creates each renewed private key as mode 0600. The proxy runs as appuser, so following the live symlink fails unless the archive directories are searchable and the private key is readable. The hook reapplies:

chmod 0711 /etc/letsencrypt/archive
chmod 0755 /etc/letsencrypt/archive/compute.drawthings.ai
chmod 0644 /etc/letsencrypt/live/compute.drawthings.ai/privkey.pem

The proxy and the statically configured Envoy listener load the certificate into memory at process startup. After a successful renewal, the deploy hook restarts both containers so they load the new certificate:

/usr/bin/docker restart proxy_service
/usr/bin/docker restart envoy_grpc_web_proxy

Verify the snap timer and the complete renewal-to-reload path:

systemctl is-enabled snap.certbot.renew.timer
systemctl is-active snap.certbot.renew.timer
certbot renew --dry-run --run-deploy-hooks
docker ps --filter name=proxy_service
docker ps --filter name=envoy_grpc_web_proxy

Expected results are enabled, active, Congratulations, all simulated renewals succeeded, and both containers returning to Up. --run-deploy-hooks is needed only to make a dry run execute deploy hooks; a real successful renewal automatically executes every executable in /etc/letsencrypt/renewal-hooks/deploy.

Confirm that ports 443 and 8443 serve the same renewed certificate:

openssl s_client -connect 127.0.0.1:443 -servername compute.drawthings.ai </dev/null 2>/dev/null | openssl x509 -noout -dates -serial
openssl s_client -connect 127.0.0.1:8443 -servername compute.drawthings.ai </dev/null 2>/dev/null | openssl x509 -noout -dates -serial

Launch Services

Launch the CPU proxy first. Any extra arguments after the model-list path are passed through to ProxyServiceCLI, including -g GPU worker ranges:

ssh root@<new-ts-ip> 'cd /root/CPUScript && ./LaunchCPUServer.sh "<DATADOG_API_KEY>" /root/CPUScript/model-list -g 100.70.225.70:40001-40008:1 -g 100.115.89.107:40001-40008:1 -g 100.102.100.71:40001-40008:1 -g 100.102.159.74:40001-40008:1 -g 100.121.197.14:40001-40008:1 -g 100.110.198.32:40001-40008:1'

Verify each host's current Tailscale address before executing this example. The rebuilt dfw-242-009 and dfw-243-010 replaced the retired 100.83.253.70 and 100.64.66.84 addresses. LaunchCPUServer.sh stops and recreates the central proxy container, which can interrupt in-flight generation requests; use a planned maintenance window. A plain restart of an existing container retains its previously saved -g arguments.

LaunchCPUServer.sh mounts the directory containing the model list as /app/Documents and passes --model-list-path /app/Documents/<filename>. It also makes that directory and file writable by the container's non-root appuser; this is required because update-model-list writes atomically by creating a temporary file in the same directory.

Do not use the old single-file read-only mount:

-v /root/CPUScript/model-list:/app/model-list:ro

That mount lets the server read the file, but it prevents the control-panel update-model-list RPC from writing back. The server currently uses try? around the file write, so the CLI can appear to succeed even when the file was not updated.

Then launch Envoy:

ssh root@<new-ts-ip> 'cd /root/CPUScript && ./LaunchEnvoyServer.sh /root/CPUScript/envoy-config.yaml'

Expected ports:

• proxy_service: 0.0.0.0:443->8080/tcp and <tailscale-ip>:50002->50000/tcp.
• envoy_grpc_web_proxy: host network, listening on 0.0.0.0:8443.
• Envoy admin: 127.0.0.1:9901.

Verification

Verify containers, listeners, logs, TLS, and Envoy readiness:

ssh root@<new-ts-ip> 'docker ps -a --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}"'
ssh root@<new-ts-ip> 'docker inspect --format "{{range .Mounts}}{{println .Source \"->\" .Destination \"RW=\" .RW}}{{end}}" proxy_service'
ssh root@<new-ts-ip> 'docker inspect --format "{{range .Args}}{{println .}}{{end}}" proxy_service | grep -E "^100\\..*:40001-40008:1$" || true'
ssh root@<new-ts-ip> 'ss -ltnp | grep -E ":(443|8443|50002|9901)" || true'
ssh root@<new-ts-ip> 'docker logs --tail 50 proxy_service 2>&1; docker logs --tail 80 envoy_grpc_web_proxy 2>&1'
ssh root@<new-ts-ip> 'openssl s_client -connect 127.0.0.1:443 -servername compute.drawthings.ai </dev/null 2>/dev/null | openssl x509 -noout -subject -dates'
ssh root@<new-ts-ip> 'openssl s_client -connect 127.0.0.1:8443 -servername compute.drawthings.ai </dev/null 2>/dev/null | openssl x509 -noout -subject -dates'
ssh root@<new-ts-ip> 'curl -fsS --max-time 5 http://127.0.0.1:9901/ready'

Verify external GCP access:

curl -vk --connect-timeout 10 https://<external-ip>:443/ImageGenerationService/Echo
openssl s_client -connect <external-ip>:443 -servername compute.drawthings.ai </dev/null 2>/dev/null | openssl x509 -noout -subject -dates -ext subjectAltName
curl -vk --resolve compute.drawthings.ai:443:<external-ip> --connect-timeout 10 https://compute.drawthings.ai:443/ImageGenerationService/Echo

An HTTP 415 from a raw GET to a gRPC endpoint is acceptable; it means the request reached the service but was not a valid gRPC call.

Model List Updates