itseffi/productize/productize-ab-test-analysis

Claude Code and agent skill by itseffi/productize

Trust tier 3
  • community
Source check pending

Checking that the SKILL.md file is still in the repository.

A/B Test Analysis. Use when evaluating A/B test results, checking statistical significance, validating sample size, reading confidence intervals, or deciding whether to ship, extend, or stop an experiment.

Install

npx vskill@latest install itseffi/productize/productize-ab-test-analysis

SKILL.md

The same SKILL.md was published earlier as itseffi/productize/productize-ab-test-analysis.

A/B Test Analysis

Productize Preamble

Before producing the artifact, implementation step, or code change, classify the work:

  • User/persona: founder, product leader, AI PM, AI builder, stakeholder, or mixed/unknown.
  • Product stage: idea, validation, PMF search, growth, scale, pivot, or unknown.
  • Artifact mode: strategy memo, PRD, research plan, positioning, experiment, deck, roadmap, execution brief, diagnostic, or decision record.
  • Artifact format: Markdown for short, repo-native, diff-sensitive artifacts; self-contained HTML for long, visual, shareable, interactive, or explicitly requested artifacts.
  • Evidence standard: what is known, assumed, missing, and risky.
  • Decision mode: recommend, ask for a blocking input, or proceed with explicit assumptions.

Operating rules:

  1. Do not produce generic product strategy filler. Tie every recommendation to the user's context, evidence, stage, and decision pressure.
  2. Separate facts from assumptions. Convert high-risk assumptions into tests, research prompts, or instrumentation.
  3. Search existing context, attached docs, repo files, or prior Productize artifacts before inventing new structure when those sources are available.
  4. Ask only for inputs that materially change the output. If the next step is obvious, proceed and state assumptions.
  5. If another Productize skill is a better fit, name it and explain the routing in one sentence before continuing.
  6. Enforce the output contract for this skill. Produce the artifact, implementation step, verification evidence, or code change; do not stop at a meta-explanation of the method.
  7. End with the concrete next action, decision owner, validation step, metric, or artifact handoff when applicable.
  8. Use the user's saved Productize preferences when available, but do not let stale preferences override explicit instructions in the current request.
  9. If you must ask a blocking question, use this compact format: AskUserQuestion: <one question> Why it matters: <decision it changes>.
  10. Record completion status for durable work as completed, blocked, deferred, or needs-review when runtime hooks are available.

Artifact format policy:

  • Default to Markdown for short notes, repo-native documentation, changelog fragments, or artifacts where clean source diffs matter.
  • Use a self-contained HTML file when the user asks for HTML, the artifact is likely to exceed about 100 lines, the reader needs diagrams/tables/comparisons/screenshots, the artifact is meant to be shared, or interaction/export controls would help the user stay in the loop.
  • Route by product job first, then choose the format. Do not create a generic HTML workflow when a Productize playbook, gate, or routed skill owns the work.
  • HTML artifacts must be local-first and portable: one file, embedded CSS/JS, no remote dependencies unless explicitly requested, readable without a dev server, responsive, accessible, and easy to skim.
  • Prefer semantic sections, tables, SVG diagrams, annotated code blocks, status chips, collapsible detail, and copy/export buttons when they improve review speed. Avoid decorative complexity that hides the decision.
  • For implementation work with ambiguities, use $implementation-notes when the user asks for a running notes file, especially implementation-notes.html or implementation-notes.md.

Runtime hooks, if available:

  • Use productize-workflow start "<user request>" at the beginning of durable product work; it restores context, routes the request, records the session, and returns the required artifact contract.
  • Use productize-workflow complete --id <id> --status completed|blocked|deferred|needs-review --artifact-type <type> --summary <summary> before ending durable product work; it records the artifact, saves context, and logs completion status.
  • Use productize-update-check --strict at the start of maintenance, setup, release, or generated-output work.
  • Use productize-config to read or write user/team preferences such as persona, artifact mode, evidence threshold, or update-check policy.
  • Use productize-session-log to record important workflow decisions.
  • Use productize-artifact-log when a durable artifact is produced.
  • Use productize-context-restore before restarting long-running product work from scratch.
  • Use productize-context-save after producing a durable strategy, research, spec, or stakeholder artifact.
  • Use productize-registry-search or productize-skill-router when routing is ambiguous.
  • Use productize-completion-status to log whether the workflow completed, blocked, deferred, or needs review.

Telemetry standard:

  • Keep telemetry local by default in .productize/ or PRODUCTIZE_STATE_DIR.
  • Log artifact type, routing decision, evidence gaps, and completion status; do not log secrets or private customer data unless the user explicitly asks for it.

Current skill metadata:

  • Skill: ab-test-analysis
  • Lifecycle: Measure
  • Category: Analytics
  • Primary artifact: A/B test decision memo with statistical readout and ship, extend, or stop recommendation

Use when evaluating A/B test results, checking statistical significance, validating sample size, reading confidence intervals, or deciding whether to ship, extend, or stop an experiment.

Productize Contract

  • Primary lifecycle: Measure
  • Supporting lifecycle: Launch & Learn
  • Primary artifact: A/B test decision memo with statistical readout and ship, extend, or stop recommendation
  • Source method: pm-skills-main/pm-data-analytics/skills/ab-test-analysis/SKILL.md

Method

A/B Test Analysis

Evaluate A/B test results with statistical rigor and translate findings into clear product decisions.

Context

You are analyzing A/B test results for $ARGUMENTS.

If the user provides data files (CSV, Excel, or analytics exports), read and analyze them directly. Generate Python scripts for statistical calculations when needed.

Instructions
  1. Understand the experiment:
  • What was the hypothesis?
  • What was changed (the variant)?
  • What is the primary metric? Any guardrail metrics?
  • How long did the test run?
  • What is the traffic split?
  1. Validate the test setup:
  • Sample size: Is the sample large enough for the expected effect size?
  • Use the formula: n = (Z2/2 x 2 x p x (1-p)) / MDE2
  • Flag if the test is underpowered (<80% power)
  • Duration: Did the test run for at least 1-2 full business cycles?
  • Randomization: Any evidence of sample ratio mismatch (SRM)?
  • Novelty/primacy effects: Was there enough time to wash out initial behavior changes?
  1. Calculate statistical significance:
  • Conversion rate for control and variant
  • Relative lift: (variant - control) / control x 100
  • p-value: Using a two-tailed z-test or chi-squared test
  • Confidence interval: 95% CI for the difference
  • Statistical significance: Is p < 0.05?
  • Practical significance: Is the lift meaningful for the business?

If the user provides raw data, generate and run a Python script to calculate these.

  1. Check guardrail metrics:
  • Did any guardrail metrics (revenue, engagement, page load time) degrade?
  • A winning primary metric with degraded guardrails may not be a true win
  1. Interpret results:
OutcomeRecommendation
Significant positive lift, no guardrail issuesShip it -- roll out to 100%
Significant positive lift, guardrail concernsInvestigate -- understand trade-offs before shipping
Not significant, positive trendExtend the test -- need more data or larger effect
Not significant, flatStop the test -- no meaningful difference detected
Significant negative liftDon't ship -- revert to control, analyze why
  1. Provide the analysis summary:

``` ## A/B Test Results: [Test Name]

Hypothesis: [What we expected] Duration: [X days] | Sample: [N control / M variant]

MetricControlVariantLiftp-valueSignificant?
[Primary]X%Y%+Z%0.0XYes/No
[Guardrail]...............

Recommendation: [Ship / Extend / Stop / Investigate] Reasoning: [Why] Next steps: [What to do] ```

Think step by step. Save as markdown. Generate Python scripts for calculations if raw data is provided.


Further Reading

Productize Output Rules

  • Produce the artifact named in the Productize contract, not a generic framework summary.
  • Separate known facts, assumptions, missing evidence, and risky leaps before recommending.
  • Convert uncertain claims into validation steps, metrics, owner decisions, or launch/readout checks.
  • If this reference method conflicts with Productize evidence standards, keep the Productize standard.

Security

  • Security scanPassed, 95 of 100Pattern checks on the skill's files.
  • Model reviewPassed

Quality evaluation

  • Task evaluationNot runNo evaluation yet. Run one in Skill Studio.

Source

Popularity

7-day trend score
5

Not refreshed yet.

Works with

Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.

  • Claude Code
  • Cursor
  • GitHub Copilot
  • Windsurf
  • Codex
  • Gemini CLI
  • Cline
  • Amp
  • Roo Code
  • Goose
  • Continue
  • Kilo Code
  • Junie
  • Trae
  • OpenHands
  • Qwen Code
  • AI Agent
  • Tabnine Chat
  • Sourcegraph Cody
  • Amazon Q
  • Firebase Genkit
  • Sweep
  • Devin
  • Phind
  • Mentat
  • Claude Desktop
  • Aider
  • GPT Pilot
  • Plandex
  • bolt.new
  • v0
  • Replit Agent
  • Anse
  • TypingMind
  • BetterChatGPT
  • LibreChat
  • Warp AI
  • Pieces
  • Fig AI
Embed badge
vskill VERIFIED badge
[![vskill VERIFIED](https://verified-skill.com/api/v1/skills/itseffi/productize/productize-ab-test-analysis/badge)](https://verified-skill.com/skills/itseffi/productize/productize-ab-test-analysis)

Something wrong with this skill or its page? Tell us.