gm3dmo/the-power/scale-environment

Scale Environment

Bulk-create GitHub resources to populate a test environment. Three performance tiers are available depending on how fast you need to go.

Working directory

All scaling scripts must be run from the root of a the-power repository clone (the directory containing .gh-api-examples.conf).

Find an existing clone:

find ~ -maxdepth 5 -name "create-many-repos.sh" -path "*/the-power/*" 2>/dev/null

Safety first

Start small. Create 10–50 resources to verify your config before scaling to hundreds or thousands. Bulk creation is not easily reversible.

Creating users is GHES-only. The create-many-users.sh and python-create-many-users-* scripts call the site admin API, which is only available on GitHub Enterprise Server — not github.com.

Configuration

All bulk scripts read defaults from .gh-api-examples.conf:

number_of_repos=3
number_of_orgs=3
number_of_teams=3
number_of_branches=10
number_of_users_to_create_on_ghes=3
 
repo_prefix="gestrepo"
org_prefix="fruitbat"
user_prefix="testusertoad"
team_prefix="testteam"
branch_prefix="testbranch"
file_prefix="testfile"

The Python scripts also accept CLI overrides:

python3 python-create-many-repos-connection-reuse.py --repos 1000 --prefix mtst1 --org acme

Performance tiers

Tier 1 — Shell scripts (simplest)

Plain curl-in-a-loop. Easy to read and modify. Each request opens a new connection.

Tier 2 — Python with connection reuse (~2× faster)

Reuses a single http.client.HTTPSConnection for all requests, cutting out TCP/TLS handshake overhead per call.

Tier 3 — Python with connection pooling (~4× faster)

Uses urllib3 connection pooling for maximum throughput. Requires:

pip install urllib3

Choosing a tier

Benchmarks

Measured against a GHES test instance:

30 repos

1000 repos

1000 orgs

Example workflow

# 1. Configure for the target instance
cat .gh-api-examples.conf | grep -E '^(hostname|org)='
 
# 2. Verify with a small run
number_of_repos=5 ./create-many-repos.sh
 
# 3. Scale up with Python pooling
pip install urllib3
python3 python-create-many-repos-connection-reuse-pool.py \
  --repos 1000 --prefix load --org acme

Error handling

• 422 "name already exists" — a resource with that prefix already exists.

Change the prefix or bump the starting number.

• 403 rate limit — you are hitting secondary rate limits. Reduce batch size

or add a delay between requests.

• 401 — token is invalid or expired.
• 404 — the target org does not exist. Create it first (see

create-test-org skill).

Boundaries

• Do not bulk-create on github.com without understanding the rate limits.

Dotcom has stricter secondary rate limits than GHES.

• Do not create users on github.com. The admin user API is GHES-only.
• Always confirm the target instance and count with the user before running

bulk scripts.

• Do not display the GITHUB_TOKEN. Show only the first 8 characters.