itseffi/productize/productize-ab-test-analysis
Claude Code and agent skill by itseffi/productize
- community
Checking that the SKILL.md file is still in the repository.
A/B Test Analysis. Use when evaluating A/B test results, checking statistical significance, validating sample size, reading confidence intervals, or deciding whether to ship, extend, or stop an experiment.
Install
npx vskill@latest install itseffi/productize/productize-ab-test-analysisSKILL.md
The same SKILL.md was published earlier as itseffi/productize/productize-ab-test-analysis.
A/B Test Analysis
Productize Preamble
Before producing the artifact, implementation step, or code change, classify the work:
- User/persona: founder, product leader, AI PM, AI builder, stakeholder, or mixed/unknown.
- Product stage: idea, validation, PMF search, growth, scale, pivot, or unknown.
- Artifact mode: strategy memo, PRD, research plan, positioning, experiment, deck, roadmap, execution brief, diagnostic, or decision record.
- Artifact format: Markdown for short, repo-native, diff-sensitive artifacts; self-contained HTML for long, visual, shareable, interactive, or explicitly requested artifacts.
- Evidence standard: what is known, assumed, missing, and risky.
- Decision mode: recommend, ask for a blocking input, or proceed with explicit assumptions.
Operating rules:
- Do not produce generic product strategy filler. Tie every recommendation to the user's context, evidence, stage, and decision pressure.
- Separate facts from assumptions. Convert high-risk assumptions into tests, research prompts, or instrumentation.
- Search existing context, attached docs, repo files, or prior Productize artifacts before inventing new structure when those sources are available.
- Ask only for inputs that materially change the output. If the next step is obvious, proceed and state assumptions.
- If another Productize skill is a better fit, name it and explain the routing in one sentence before continuing.
- Enforce the output contract for this skill. Produce the artifact, implementation step, verification evidence, or code change; do not stop at a meta-explanation of the method.
- End with the concrete next action, decision owner, validation step, metric, or artifact handoff when applicable.
- Use the user's saved Productize preferences when available, but do not let stale preferences override explicit instructions in the current request.
- If you must ask a blocking question, use this compact format:
AskUserQuestion: <one question> Why it matters: <decision it changes>. - Record completion status for durable work as
completed,blocked,deferred, orneeds-reviewwhen runtime hooks are available.
Artifact format policy:
- Default to Markdown for short notes, repo-native documentation, changelog fragments, or artifacts where clean source diffs matter.
- Use a self-contained HTML file when the user asks for HTML, the artifact is likely to exceed about 100 lines, the reader needs diagrams/tables/comparisons/screenshots, the artifact is meant to be shared, or interaction/export controls would help the user stay in the loop.
- Route by product job first, then choose the format. Do not create a generic HTML workflow when a Productize playbook, gate, or routed skill owns the work.
- HTML artifacts must be local-first and portable: one file, embedded CSS/JS, no remote dependencies unless explicitly requested, readable without a dev server, responsive, accessible, and easy to skim.
- Prefer semantic sections, tables, SVG diagrams, annotated code blocks, status chips, collapsible detail, and copy/export buttons when they improve review speed. Avoid decorative complexity that hides the decision.
- For implementation work with ambiguities, use
$implementation-noteswhen the user asks for a running notes file, especiallyimplementation-notes.htmlorimplementation-notes.md.
Runtime hooks, if available:
- Use
productize-workflow start "<user request>"at the beginning of durable product work; it restores context, routes the request, records the session, and returns the required artifact contract. - Use
productize-workflow complete --id <id> --status completed|blocked|deferred|needs-review --artifact-type <type> --summary <summary>before ending durable product work; it records the artifact, saves context, and logs completion status. - Use
productize-update-check --strictat the start of maintenance, setup, release, or generated-output work. - Use
productize-configto read or write user/team preferences such as persona, artifact mode, evidence threshold, or update-check policy. - Use
productize-session-logto record important workflow decisions. - Use
productize-artifact-logwhen a durable artifact is produced. - Use
productize-context-restorebefore restarting long-running product work from scratch. - Use
productize-context-saveafter producing a durable strategy, research, spec, or stakeholder artifact. - Use
productize-registry-searchorproductize-skill-routerwhen routing is ambiguous. - Use
productize-completion-statusto log whether the workflow completed, blocked, deferred, or needs review.
Telemetry standard:
- Keep telemetry local by default in
.productize/orPRODUCTIZE_STATE_DIR. - Log artifact type, routing decision, evidence gaps, and completion status; do not log secrets or private customer data unless the user explicitly asks for it.
Current skill metadata:
- Skill:
ab-test-analysis - Lifecycle: Measure
- Category: Analytics
- Primary artifact: A/B test decision memo with statistical readout and ship, extend, or stop recommendation
Use when evaluating A/B test results, checking statistical significance, validating sample size, reading confidence intervals, or deciding whether to ship, extend, or stop an experiment.
Productize Contract
- Primary lifecycle: Measure
- Supporting lifecycle: Launch & Learn
- Primary artifact: A/B test decision memo with statistical readout and ship, extend, or stop recommendation
- Source method: pm-skills-main/pm-data-analytics/skills/ab-test-analysis/SKILL.md
Method
A/B Test Analysis
Evaluate A/B test results with statistical rigor and translate findings into clear product decisions.
Context
You are analyzing A/B test results for $ARGUMENTS.
If the user provides data files (CSV, Excel, or analytics exports), read and analyze them directly. Generate Python scripts for statistical calculations when needed.
Instructions
- Understand the experiment:
- What was the hypothesis?
- What was changed (the variant)?
- What is the primary metric? Any guardrail metrics?
- How long did the test run?
- What is the traffic split?
- Validate the test setup:
- Sample size: Is the sample large enough for the expected effect size?
- Use the formula: n = (Z2/2 x 2 x p x (1-p)) / MDE2
- Flag if the test is underpowered (<80% power)
- Duration: Did the test run for at least 1-2 full business cycles?
- Randomization: Any evidence of sample ratio mismatch (SRM)?
- Novelty/primacy effects: Was there enough time to wash out initial behavior changes?
- Calculate statistical significance:
- Conversion rate for control and variant
- Relative lift: (variant - control) / control x 100
- p-value: Using a two-tailed z-test or chi-squared test
- Confidence interval: 95% CI for the difference
- Statistical significance: Is p < 0.05?
- Practical significance: Is the lift meaningful for the business?
If the user provides raw data, generate and run a Python script to calculate these.
- Check guardrail metrics:
- Did any guardrail metrics (revenue, engagement, page load time) degrade?
- A winning primary metric with degraded guardrails may not be a true win
- Interpret results:
| Outcome | Recommendation | |
|---|---|---|
| Significant positive lift, no guardrail issues | Ship it -- roll out to 100% | |
| Significant positive lift, guardrail concerns | Investigate -- understand trade-offs before shipping | |
| Not significant, positive trend | Extend the test -- need more data or larger effect | |
| Not significant, flat | Stop the test -- no meaningful difference detected | |
| Significant negative lift | Don't ship -- revert to control, analyze why |
- Provide the analysis summary:
``` ## A/B Test Results: [Test Name]
Hypothesis: [What we expected] Duration: [X days] | Sample: [N control / M variant]
| Metric | Control | Variant | Lift | p-value | Significant? | |
|---|---|---|---|---|---|---|
| [Primary] | X% | Y% | +Z% | 0.0X | Yes/No | |
| [Guardrail] | ... | ... | ... | ... | ... |
Recommendation: [Ship / Extend / Stop / Investigate] Reasoning: [Why] Next steps: [What to do] ```
Think step by step. Save as markdown. Generate Python scripts for calculations if raw data is provided.
Further Reading
- A/B Testing 101 + Examples
- Testing Product Ideas: The Ultimate Validation Experiments Library
- Are You Tracking the Right Metrics?
Productize Output Rules
- Produce the artifact named in the Productize contract, not a generic framework summary.
- Separate known facts, assumptions, missing evidence, and risky leaps before recommending.
- Convert uncertain claims into validation steps, metrics, owner decisions, or launch/readout checks.
- If this reference method conflicts with Productize evidence standards, keep the Productize standard.
Security
- Security scanPassed, 95 of 100Pattern checks on the skill's files.
- Model reviewPassed
Quality evaluation
- Task evaluationNot runNo evaluation yet. Run one in Skill Studio.
Source
- SKILL.md in the repositoryCheckingskills/productize-ab-test-analysis/SKILL.md
Popularity
- 7-day trend score
- 5
Not refreshed yet.
Works with
Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.
- Claude Code
- Cursor
- GitHub Copilot
- Windsurf
- Codex
- Gemini CLI
- Cline
- Amp
- Roo Code
- Goose
- Continue
- Kilo Code
- Junie
- Trae
- OpenHands
- Qwen Code
- AI Agent
- Tabnine Chat
- Sourcegraph Cody
- Amazon Q
- Firebase Genkit
- Sweep
- Devin
- Phind
- Mentat
- Claude Desktop
- Aider
- GPT Pilot
- Plandex
- bolt.new
- v0
- Replit Agent
- Anse
- TypingMind
- BetterChatGPT
- LibreChat
- Warp AI
- Pieces
- Fig AI
Embed badge
[](https://verified-skill.com/skills/itseffi/productize/productize-ab-test-analysis)