blushneybusiness-glitch/roblox-omniskills/rbx-abtest

rbx-abtest

Laws

1. One variable per experiment. Price OR tutorial length OR button copy — never two.

Two simultaneous changes make both arms unreadable, and so does shipping an unrelated update mid-window; freeze the surrounding surface for the test's duration.

1. Assignment is a pure function of UserId + experiment salt. Same player, same arm,

every server, every session, no coordination. Random-per-session assignment shows players both arms and measures nothing.

1. No exposure event, no data point. A player counts only from the moment they actually

met the variant (opened the shop with the new price, hit the tutorial step). Counting all assigned players dilutes the effect with players who never saw the change.

1. The read is scheduled before the test starts. Metric, direction, window, and minimum

sample are written down pre-launch. Peeking daily and stopping on the first green day is how a coin flip becomes a "12% winner" that regresses after shipping.

1. The arm persists in save data. Mid-experiment salt or bucket-boundary changes

reshuffle players across arms and poison every prior day; the stored arm wins over the recomputed one for the experiment's lifetime.

Passes

1. Register — name the experiment (ShopPrice_2026w34), the single variable, the arms

(control first), the target metric + expected direction, the window (7 days minimum — one full weekday/weekend cycle), and the minimum sample (≥ 1,000 exposed players per arm for conversion-rate deltas; small effects on small games are unreadable — say so and test bigger swings instead, 2× price not +10%).

1. Assign — server module:

``luau local function bucket(userId: number, salt: string): number local h = 0 for _, c in string.split(salt .. userId, "") do h = (h * 31 + string.byte(c)) % 100 end return h -- 0–99 end -- arm = bucket < 50 and "B" or "A" (50/50; shift the threshold for 90/10 ramps) `` The per-experiment salt de-correlates experiments — without it, every test's "B" group is the same players and cross-contamination compounds.

1. Persist — write { experiment = arm } into the player's profile

(rbx-data-profilestore) on first assignment; read the stored arm first thereafter.

1. Expose — at the moment the variant is met, log once per player via the

rbx-analytics-events wrapper: LogCustomEvent(player, "ABExposure", 1, CustomField01 = experiment, CustomField02 = arm). Debounce with a per-profile flag — re-logging every session inflates the denominator.

1. Serve — one branch point server-side; the client receives the resolved value (the

price, the flag), never the assignment logic. Exploiters reading a LocalScript must not learn the experiment map.

1. Read — at window end, not before: exposed players per arm, metric per arm, relative

delta. Verdict rules: sample below minimum → "underpowered, no verdict" is the honest output; delta under ~5% relative on a 1,000-per-arm test → treat as a tie and keep control (control wins ties: it's free). Check the guardrail metric (D1 for monetization tests, revenue for retention tests) — a "winner" that bleeds the guardrail loses.

1. Close — ship the winner to 100%, delete the branch point within one update, log the

result in memory. Zombie experiments that keep splitting traffic forever are the most common end state; closing is part of the test.

Performance budget

Assignment is pure math — zero DataStore calls after the first-session write. Exposure is one AnalyticsService call per player per experiment (rides the shared 120 + 20 × CCU/min budget). Never more than 2 experiments live at once on a game under 10k DAU; traffic split four ways is four underpowered tests.

Load with

rbx-analytics-events — exposure logging or the test is noise; don't ask, include it. rbx-analytics-reading (guardrail benchmarks), rbx-data-profilestore (arm persistence), rbx-mtx-pricing when the variable is a price. Verdict interpretation for retention metrics: rbx-analytics-reading.

AI-tells

• ☐ math.random() assignment — arm changes every session
• ☐ No salt: every experiment splits on the same UserId parity
• ☐ Exposure logged at assignment, not at first sight of the variant
• ☐ Test read daily and stopped the first day B led
• ☐ Two variables changed between arms "to save time"
• ☐ Assignment logic or experiment map shipped in a LocalScript
• ☐ Declared winner without reporting per-arm sample sizes
• ☐ Experiment branch still in the code three updates after the verdict

Memory

Read: the game's projects.md block for live and past experiments — never reuse a salt, and never re-test a question that already has a verdict without saying so. Write: each experiment's registration line (name, variable, arms, window, minimum sample) at launch, and the verdict + per-arm numbers at close; a live test goes into manual steps pending: with its scheduled read date so the next session picks the read back up.