mejbaurbahar/awesome-qa-agent-skills/ab-test-analysis

Claude Code and agent skill by mejbaurbahar/awesome-qa-agent-skills

Trust tier 3
  • community
Source check pending

Checking that the SKILL.md file is still in the repository.

Analyze A/B test results with statistical significance, sample size validation, confidence intervals, and ship/extend/stop recommendations. Use when evaluating experiment results, checking if a test reached significance, interpreting split test data, or deciding whether to ship a variant.

Install

npx vskill@latest install mejbaurbahar/awesome-qa-agent-skills/ab-test-analysis

SKILL.md

The same SKILL.md was published earlier as nvlabs/skill2env/ab-test-analysis.

A/B Test Analysis

Evaluate A/B test results with statistical rigor and translate findings into clear product decisions.

Context

You are analyzing A/B test results for $ARGUMENTS.

If the user provides data files (CSV, Excel, or analytics exports), read and analyze them directly. Generate Python scripts for statistical calculations when needed.

Instructions
  1. Understand the experiment:
  • What was the hypothesis?
  • What was changed (the variant)?
  • What is the primary metric? Any guardrail metrics?
  • How long did the test run?
  • What is the traffic split?
  1. Validate the test setup:
  • Sample size: Is the sample large enough for the expected effect size?
  • Use the formula: n = (Z²α/2 × 2 × p × (1-p)) / MDE²
  • Flag if the test is underpowered (<80% power)
  • Duration: Did the test run for at least 1-2 full business cycles?
  • Randomization: Any evidence of sample ratio mismatch (SRM)?
  • Novelty/primacy effects: Was there enough time to wash out initial behavior changes?
  1. Calculate statistical significance:
  • Conversion rate for control and variant
  • Relative lift: (variant - control) / control × 100
  • p-value: Using a two-tailed z-test or chi-squared test
  • Confidence interval: 95% CI for the difference
  • Statistical significance: Is p < 0.05?
  • Practical significance: Is the lift meaningful for the business?

If the user provides raw data, generate and run a Python script to calculate these.

  1. Check guardrail metrics:
  • Did any guardrail metrics (revenue, engagement, page load time) degrade?
  • A winning primary metric with degraded guardrails may not be a true win
  1. Interpret results:
OutcomeRecommendation
Significant positive lift, no guardrail issuesShip it — roll out to 100%
Significant positive lift, guardrail concernsInvestigate — understand trade-offs before shipping
Not significant, positive trendExtend the test — need more data or larger effect
Not significant, flatStop the test — no meaningful difference detected
Significant negative liftDon't ship — revert to control, analyze why
  1. Provide the analysis summary:

``` ## A/B Test Results: [Test Name]

Hypothesis: [What we expected] Duration: [X days] | Sample: [N control / M variant]

MetricControlVariantLiftp-valueSignificant?
[Primary]X%Y%+Z%0.0XYes/No
[Guardrail]...............

Recommendation: [Ship / Extend / Stop / Investigate] Reasoning: [Why] Next steps: [What to do] ```

Think step by step. Save as markdown. Generate Python scripts for calculations if raw data is provided.

Further Reading

Security

  • Security scanPassed, 90 of 100Pattern checks on the skill's files.
  • Model reviewPassed

See the full security report

Quality evaluation

  • Task evaluationNot runNo evaluation yet. Run one in Skill Studio.

See evaluation results

Source

Popularity

7-day trend score
0

Not refreshed yet.

Works with

Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.

  • Claude Code
  • Cursor
  • GitHub Copilot
  • Windsurf
  • Codex
  • Gemini CLI
  • Cline
  • Amp
  • Roo Code
  • Goose
  • Continue
  • Kilo Code
  • Junie
  • Trae
  • OpenHands
  • Qwen Code
  • AI Agent
  • Tabnine Chat
  • Sourcegraph Cody
  • Amazon Q
  • Firebase Genkit
  • Sweep
  • Devin
  • Phind
  • Mentat
  • Claude Desktop
  • Aider
  • GPT Pilot
  • Plandex
  • bolt.new
  • v0
  • Replit Agent
  • Anse
  • TypingMind
  • BetterChatGPT
  • LibreChat
  • Warp AI
  • Pieces
  • Fig AI
Embed badge
vskill VERIFIED badge
[![vskill VERIFIED](https://verified-skill.com/api/v1/skills/mejbaurbahar/awesome-qa-agent-skills/ab-test-analysis/badge)](https://verified-skill.com/skills/mejbaurbahar/awesome-qa-agent-skills/ab-test-analysis)

Something wrong with this skill or its page? Tell us.