Create a lightweight A/B testing tool that lets you split traffic, track conversion metrics, and visualize results — the kind of tool that a startup needs to make data-driven decisions.
Welcome to another official UVF IT build guide. Today, we are going to build your own A/B testing tracker inspired by PostHog. Instead of relying on a heavy SaaS platform, we will create a minimal, self-contained system that handles traffic splitting and event tracking using Python and SQLite.
This project is designed for beginners who want to understand the core mechanics of experimentation. By the end of this 45-minute session, you will have a working local server that can assign users to variants, record their actions, and calculate basic conversion rates. It is a perfect foundation for learning how product analytics tools work under the hood.
pip to install dependencies, so ensure your environment is set up correctly.First, we set up our Python environment and install the necessary libraries. We will use FastAPI for the web framework because it is fast and easy to use, and Uvicorn to run the server. We also install SQLite3 (built into Python) for our database.
Create a new directory for your project and initialize a virtual environment to keep your dependencies isolated.
mkdir uvf_ab_tracker && cd uvf_ab_tracker
python3 -m venv venv
source venv/bin/activate # On Windows use: venv\Scripts\activate
pip install fastapi uvicornNext, we define our database schema. We need two main tables: experiments to store the test configurations (name, variants, split ratio) and events to log user actions (user_id, experiment_id, variant, converted).
We will write a Python script to initialize the SQLite database and create these tables if they don't already exist. This ensures our data structure is ready before we start handling requests.
database.py
import sqlite3
from contextlib import contextmanager
DB_NAME = "ab_testing.db"
@contextmanager
def get_db():
"""Context manager to handle database connections."""
conn = sqlite3.connect(DB_NAME)
conn.row_factory = sqlite3.Row
try:
yield conn
conn.commit()
except Exception as e:
conn.rollback()
raise e
finally:
conn.close()
def init_db():
"""Initialize the database and create tables if they don't exist."""
with get_db() as conn:
cursor = conn.cursor()
# Table for experiment configurations
cursor.execute('''
CREATE TABLE IF NOT EXISTS experiments (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)
''')
# Table for tracking user events and conversions
cursor.execute('''
CREATE TABLE IF NOT EXISTS events (
id INTEGER PRIMARY KEY AUTOINCREMENT,
experiment_id INTEGER NOT NULL,
user_id TEXT NOT NULL,
variant TEXT NOT NULL,
converted INTEGER DEFAULT 0,
timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (experiment_id) REFERENCES experiments(id)
)
''')
# Index for faster lookups by experiment
cursor.execute('CREATE INDEX IF NOT EXISTS idx_events_exp ON events(experiment_id)')
print(f"Database initialized: {DB_NAME}")The heart of any A/B testing tool is the traffic splitter. We need a function that takes a user_id and an experiment_id and deterministically assigns the user to either Variant A or Variant B.
We will use a simple hash function on the user_id to ensure the same user always gets the same variant. This prevents users from flickering between experiences. We'll use a 50/50 split for simplicity.
logic.py
import hashlib
def assign_variant(user_id: str, experiment_id: int) -> str:
"""
Deterministically assign a user to a variant.
Uses MD5 hashing of the combined user_id and experiment_id.
This ensures:
1. Consistency: Same user + experiment always gets the same variant.
2. Distribution: Hashes are evenly distributed, approximating 50/50 split.
"""
# Combine user and experiment to ensure isolation between tests
key = f"{user_id}_{experiment_id}"
# Generate a hash
hash_obj = hashlib.md5(key.encode()).hexdigest()
# Take the last character of the hash (0-9, a-f)
# Map to 0-15 range
last_char = hash_obj[-1]
hash_value = int(last_char, 16)
# 50/50 Split: 0-7 -> A, 8-15 -> B
if hash_value < 8:
return "A"
else:
return "B"Now we connect the logic to the web interface using FastAPI. This file, main.py, acts as the entry point for our application. We import our database helpers and the variant assignment logic to wire everything together. We define three main routes: one to create experiments, one to assign users, and one to track events.
The GET /assign/{experiment_id}/{user_id} endpoint is crucial. It calls our assign_variant function to determine which group the user belongs to and returns that variant in the JSON response. The POST /track endpoint accepts a payload with the user, experiment, and conversion status, then writes this data to our SQLite database via the track_event function.
main.py
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import database
import logic
app = FastAPI()
class ExperimentCreate(BaseModel):
name: str
variants: list[str]
split_ratio: int = 50
class TrackEvent(BaseModel):
user_id: str
experiment_id: int
variant: str
converted: bool
@app.on_event("startup")
def startup_event():
database.init_db()
@app.post("/experiments")
def create_experiment(experiment: ExperimentCreate):
return database.create_experiment(experiment.name, experiment.variants, experiment.split_ratio)
@app.get("/assign/{experiment_id}/{user_id}")
def assign_variant(experiment_id: int, user_id: str):
experiment = database.get_experiment(experiment_id)
if not experiment:
raise HTTPException(status_code=404, detail="Experiment not found")
variant = logic.assign_variant(user_id, experiment['variants'], experiment['split_ratio'])
return {"user_id": user_id, "experiment_id": experiment_id, "variant": variant}
@app.post("/track")
def track_event(event: TrackEvent):
database.track_event(event.user_id, event.experiment_id, event.variant, event.converted)
return {"status": "tracked"}
@app.get("/results/{experiment_id}")
def get_results(experiment_id: int):
return database.get_results(experiment_id)Finally, we run the server and test our endpoints using curl. We will create an experiment, assign a few users, track some conversions, and check the results. This verifies that the entire flow from data creation to metric calculation works correctly.
Start the server in your terminal. Then, open a new terminal to run the curl commands below to simulate user behavior. Pay attention to the JSON responses to ensure the data is being saved and retrieved as expected.
# Terminal 1: Start the server
uvicorn main:app --reload
# Terminal 2: Test the endpoints
# 1. Create an experiment
# This returns a JSON object with an 'id', which we will use in subsequent steps
# For this example, let's assume the returned id is 1
# curl -X POST http://127.0.0.1:8000/experiments -H "Content-Type: application/json" -d '{"name": "Landing Page Test", "variants": ["A", "B"], "split_ratio": 50}'
# 2. Assign a user to a variant
# Check the response to see which variant (A or B) user123 is assigned to
# curl http://127.0.0.1:8000/assign/1/user123
# 3. Track a conversion for that user
# If user123 was assigned to 'A', we track a conversion for 'A'
# curl -X POST http://127.0.0.1:8000/track -H "Content-Type: application/json" -d '{"user_id": "user123", "experiment_id": 1, "variant": "A", "converted": true}'
# 4. Check the results
# This returns the conversion rates for both variants
# curl http://127.0.0.1:8000/results/1| Check | Command / Why it matters |
|---|---|
| Server starts without errors | Run uvicorn main:app --reload and look for 'Application startup complete' in the terminal. |
| Experiment creation works | Run the curl command to create an experiment and verify it returns a JSON object with an id. |
| Variant assignment is consistent | Call /assign/user123/{id} twice. The returned variant should be the same both times. |
| Results calculation is correct | After tracking events, call /results/{id} and verify the rate is a percentage between 0 and 100. |
You now have a working A/B testing tracker! To extend this, you could add a simple HTML dashboard to visualize the results, implement statistical significance testing (p-values), or add authentication to manage experiments securely. Happy building!