A/B TESTING WORKSHEET
Space Office | 2026-10-13
Companion guide: https://www.spaceoffice.ai/blog/a-b-testing

Use this planning aid for one decision. It does not replace a statistical
design review. Mark unknowns rather than inventing data or permissions.

1. DECISION AND EVIDENCE
Experiment name / version:
Human decision owner:
Implementation owner:
Analyst / analysis reviewer:
Current customer problem:
Approved evidence, source dates, audience, and limitations:
Decision the experiment should inform:
Hypothesis: For [eligible audience], changing [experience] will change
[defined outcome], because [reason grounded in evidence].
What would make us reject or revise this hypothesis?

2. AUDIENCE AND VERSIONS
Eligibility and exclusions (set before launch):
Assignment unit (person, account, or other justified unit):
Stable identifier and repeat-visit behavior:
Allocation to control A / variation B:
Control A, screenshot / version reference:
Variation B, screenshot / version reference:
Exact difference between versions:
Exposure event and timestamp:
Other experiments or releases affecting this audience:

3. MEASUREMENT
One primary outcome:
Numerator:
Denominator:
Observation window per assigned unit:
Deduplication / bot / test-account rules:
Data source and event definitions:
Guardrails, harm thresholds, and response owner:
Diagnostic metrics (separate from the primary decision):
Multiple-comparison handling, if applicable:

4. SAMPLE AND ANALYSIS PLAN
Historical baseline and comparable period:
Business improvement needed to justify the change:
Minimum detectable effect (MDE), with units:
Relative effect or absolute percentage points? State explicitly:
Analysis method and platform configuration:
Power / significance / confidence settings, as applicable:
Required sample per version and calculation record:
Expected new eligible units per week:
Estimated enrollment duration:
Relevant business cycles:
Additional time for final participants' outcomes to mature:
Planned stopping rule and final analysis point:
Safety / instrumentation stop rules (separate from declaring success):
Feasibility decision and analyst approval:

5. LAUNCH CHECKS
[ ] Current offer facts and proposed wording approved
[ ] Both versions work across relevant devices and accessibility paths
[ ] Eligibility, random allocation, and stable assignment checked
[ ] Exposure and conversion events verified in both versions
[ ] Duplicate, failed, and delayed events checked
[ ] Required access and permitted account actions confirmed
[ ] Sample ratio mismatch check and owner identified
[ ] Guardrail monitoring and rollback path ready
[ ] Experiment plan, screenshots, and configuration saved
[ ] Human launch approval recorded

6. RESULTS (AFTER THE PLANNED ANALYSIS POINT)
Analysis date / complete observation window:
Assignment and instrumentation quality findings:
Material incidents, outages, or other changes:
A eligible units:
A primary outcomes:
A rate = A outcomes / A eligible units:
B eligible units:
B primary outcomes:
B rate = B outcomes / B eligible units:
Absolute difference = B rate - A rate:
Relative change = (B rate - A rate) / A rate:
If A rate is zero, relative change is undefined. Do not divide by zero.
Uncertainty interval, units, method, and adjustments:
Guardrail outcomes:
Prespecified conclusion:
Exploratory observations (label separately):
Important limitations:

7. HUMAN DECISION
Roll out / revise / keep current version / investigate data / other:
What does the evidence support?
What remains uncertain?
Practical benefits, costs, and possible harm:
Reason for the decision:
Human approver and date:
Implementation / rollback owner:
Follow-up date and monitoring:
Next research question, if needed:

FICTIONAL ARITHMETIC EXAMPLE
A: 500 requests from 10,000 people = 5.0%.
B: 550 requests from 10,000 people = 5.5%.
Absolute difference: +0.5 percentage points.
Relative difference: +10%.
Simple unadjusted normal-approximation 95% interval for the absolute
difference: about -0.12 to +1.12 percentage points. This includes zero;
the example does not establish a positive effect at that threshold.
All example numbers are fictional, not customer results or a forecast.
Different analysis methods and adjustments can produce different intervals.