Comparing Signup Pages with A/B Testing is a practical data-analysis project for learners who can write basic Python functions and select rows in pandas. A community workshop has two signup-page versions. Your task is to compare their assigned visitors fairly, then report what a small randomized experiment does and does not support.
Three Guided Labs build the workflow: begin with six visitor cards, turn raw events into one seven-day outcome per person, explore fixed-size random reassignments, and interpret a complete exact allocation distribution. One independent Challenge applies the same rules to another experiment with different identities, group sizes and event order. Every unit starts with its own prepared data and files.
What You Will Learn
After completing this course, you will be able to:
- retain every assigned visitor, including people without events, and match records by stable identity;
- retain only signup events inside each person's UTC window and count each assigned person once;
- compare signup rates as B minus A in percentage points;
- preserve both group sizes while generating reproducible exploratory reassignments;
- compute a bounded exact permutation test with an explicitly defined inclusive two-sided p value;
- apply a preselected threshold and the observed direction without forcing a winner or claiming that inconclusive evidence proves equality;
- save the actual person table, complete analysis and a readable count chart with a visible observed comparison.
Who This Course Is For
This course is for Python and data-analysis learners who want to understand the unit of analysis, denominators and evidence behind an A/B comparison.
Prerequisites: Basic terminal navigation, Python functions, introductory pandas selection, basic NumPy arrays and introductory Matplotlib. Joins, UTC windows, fixed-size reassignments and the required p-value definition are explained in the course. Prior statistical inference knowledge is not required.
Learning environment: Each unit provides an independent browser-based WebIDE with Python, pandas, NumPy, SciPy and Matplotlib prepared. You work with local synthetic CSV records and Python files; no local installation or previous unit's VM files are required.
Frequently Asked Questions
Will I deploy a real A/B testing service?
No. The records are original synthetic teaching data representing randomized visitor allocation. You analyze a small fixed experiment; you do not track real visitors, deploy a website or establish a real business effect.
Do I need to know p values before starting?
No. Visitor cards and a hand exchange establish the person-level comparison first. The second Guided Lab explores 399 randomized reassignments without grading a sampled p value. The third then defines the exact test and its interpretation step by step.
Must page B win or the result be significant?
No. A correct result follows the complete assigned population, stated test and preselected threshold. Evidence may support either direction or remain inconclusive. A p value is not the probability that there is no effect, and an inconclusive result does not prove the pages equal.
How does the Challenge extend the Guided Labs?
You independently combine the same taught functions and statistical convention on another allocation. The changes require checking identities, windows and denominators again; they do not introduce a new statistical method.
How long should I allow?
The four units total 215 minutes in the author's estimates, including reading and ordinary corrections. This estimate has not been measured through beginner user testing.
