madappgang/claude-code/ab-testing-patterns

plugin: instantly updated: 2026-01-20

A/B Testing Patterns

Testing Fundamentals

One Variable at a Time

CRITICAL: Only change one element per test for clear attribution.

Sample Size Requirements

Formula:

sample_size = (Z^2 * p * (1-p)) / E^2
 
Where:
  Z = 1.96 for 95% confidence
  p = expected conversion rate (use 0.5 if unknown)
  E = margin of error (typically 0.05)

Subject Line Testing

Test Categories

Best Practices

1. Test 2-3 variants maximum - More variants require more sample
2. Run for minimum 3 days - Account for daily patterns
3. Test during stable periods - Avoid holidays, major events
4. Document everything - Record hypothesis, results, learnings

Body Copy Testing

Elements to Test

Copy Frameworks to Test

PAS vs AIDA:

• PAS: Problem-Agitate-Solution (emotional)
• AIDA: Attention-Interest-Desire-Action (logical)

Test Hypothesis: PAS performs better for pain-point-heavy ICPs, AIDA for solution-seekers.

Timing Tests

Variables to Test

Default Schedule (Starting Point)

Optimal Sending Windows:
  Primary: Tuesday-Thursday, 9-11am local time
  Secondary: Tuesday-Thursday, 2-4pm local time
  Avoid: Monday morning, Friday afternoon

Statistical Significance

Quick Significance Check

Decision Framework

IF lift >= 15% AND sample >= 100/variant:
  Declare winner with medium confidence
 
IF lift >= 10% AND sample >= 250/variant:
  Declare winner with high confidence
 
IF lift < 10% OR sample < 100/variant:
  Continue test or call it inconclusive

Implementing A/B Tests in Instantly

Method 1: Split Leads

1. Export lead list
2. Randomly split into Variant A and Variant B groups
3. Create two identical campaigns with one variable different
4. Use move_leads_to_campaign to assign leads

Method 2: Sequential Testing

1. Run Control for X days, collect metrics
2. Update campaign with Variant (update_campaign_sequence)
3. Run Variant for X days, collect metrics
4. Compare (less rigorous, use only if lead volume is limited)

Tracking Results

## A/B Test Log
 
**Test ID**: {uuid}
**Campaign**: {campaign_name}
**Variable**: {what_was_tested}
**Hypothesis**: {expected_outcome}
 
**Control**:
-Version: {control_description}
-Sample: {n}
-Open Rate: {x}%
-Reply Rate: {y}%
 
**Variant**:
-Version: {variant_description}
-Sample: {n}
-Open Rate: {x}%
-Reply Rate: {y}%
 
**Result**: {Winner|Inconclusive}
**Lift**: {z}%
**Confidence**: {confidence}%
**Learning**: {what_we_learned}