A/B Test Results Analyzer

Analyze the following A/B test results and provide a recommendation:

**Experiment Name**: [EXPERIMENT NAME]
**Hypothesis**: [WHAT YOU EXPECTED, e.g., "Changing the CTA button from blue to green will increase click-through rate"]
**Test Duration**: [HOW LONG THE TEST RAN, e.g., "14 days, March 1-14, 2026"]
**Traffic Split**: [HOW TRAFFIC WAS SPLIT, e.g., "50/50"]

**Primary Metric**: [YOUR PRIMARY SUCCESS METRIC, e.g., "Click-through rate (CTR)"]
**Results**:
```
Control (A):
- Sample size: [NUMBER]
- Conversions (or metric value): [NUMBER]
- Rate: [PERCENTAGE OR AVERAGE]

Variant (B):
- Sample size: [NUMBER]
- Conversions (or metric value): [NUMBER]
- Rate: [PERCENTAGE OR AVERAGE]
```

**Secondary Metrics** (optional):
```
[LIST ANY SECONDARY METRICS AND THEIR VALUES FOR BOTH GROUPS]
```

**Segment Data** (optional):
```
[IF AVAILABLE: results broken down by device, country, user type, etc.]
```

Perform this analysis:

### 1. Statistical Significance Test
- State the null and alternative hypotheses
- Choose the appropriate test (z-test for proportions, t-test for means, chi-square, etc.) and explain why
- Report: p-value, confidence interval (95%), and whether to reject the null hypothesis
- If the test is underpowered, calculate the minimum sample size needed and how many more days to run

### 2. Effect Size
- Calculate the relative and absolute effect size
- Assess practical significance: is the effect large enough to matter to the business?
- If applicable, estimate the annualized revenue/engagement impact

### 3. Validity Checks
- Sample Ratio Mismatch (SRM): is the actual split close to the intended split?
- Novelty/primacy effect risk: based on the test duration, could the result be temporary?
- Multiple testing concern: if testing multiple metrics, apply Bonferroni correction

### 4. Segment Analysis (if segment data provided)
- Are there segments where the variant performs significantly differently?
- Flag any Simpson's paradox risks

### 5. Recommendation
- Clear verdict: SHIP, DO NOT SHIP, or EXTEND THE TEST
- If SHIP: what to monitor post-launch
- If DO NOT SHIP: what to test next
- If EXTEND: how many more days/users needed