Counting, Analytics & ML Serving

A/B Testing / Experimentation Platform

Assign users to variants and measure the effect: stateless deterministic bucketing, feature-flag config for ramp and kill, an event pipeline that joins exposures to outcomes, and statistics that survive peeking and sample-ratio bugs.

~30 min · intermediate

Problem & Requirements

Split traffic between variants of a product, then decide which one is better. A unit — usually a user — is assigned to a variant, the application behaves accordingly, and the platform measures whether the variant moved a metric. Assignment happens on the hot path of every request and must be consistent: the same user sees the same variant across pages, sessions, and devices, or the experience breaks and the measurement is contaminated. Measurement happens far from assignment, asynchronously, and turns a pile of events into a decision with a known error rate.

The decision that shapes the platform is that assignment is a pure function, not a stored record. Bucketing a unit is a hash of its id and the experiment's salt mapped into an allocation, so it is computed in-process in microseconds, needs no lookup or write, and is reproducible offline for analysis. This is deterministic hashing, and it keeps assignment stateless and consistent everywhere it runs. The experiment's allocation, targeting, and on/off state live in feature flags so traffic can ramp or be killed without a deploy. Exposures and outcomes flow through an event pipeline that joins who-saw-what to what-they-did, and statistics turn that join into an effect estimate with error control. The spine throughout is that assignment is recomputed from a hash on demand and never persisted, while measurement is a separate pipeline that catches up later.

Functional

get_variant(experiment, unit_id, attrs) deterministically assigns a variant, honoring targeting and rollout.
Emit an exposure event on assignment and outcome events from the app; join them for analysis.
Compute per-variant metric estimates and a significance decision.

Non-functional (back-of-envelope)

Quantity	Target	What it forces
Assignment latency	< 1 ms, in-process	Stateless hash; no lookup or write per assignment
Consistency	same unit → same variant, always	Deterministic hashing on a stable unit id, not randomness
Concurrent experiments	hundreds, overlapping	Layered salts so experiments bucket orthogonally
Config change	ramp, target, kill without deploy	A feature-flag control plane pushed to SDKs
Events	10⁵–10⁶/s exposures and outcomes	Async pipeline; assignment never blocks on logging
Analysis	per-metric estimate, trustworthy	Statistics with peeking and sample-ratio controls, not raw counts
False-positive control	held across many metrics and peeks	Sequential tests or fixed-horizon discipline; multiple-comparison control

The dangerous row is false-positive control, not assignment. Assigning variants and counting events is the easy part. The subtle failure is a result that looks decisive and is wrong: analysts peek at the dashboard daily and stop the moment it's significant, which inflates the false-positive rate far past the nominal 5%; or a bucketing bug splits traffic 51/49 instead of 50/50 (a sample ratio mismatch) and silently invalidates every conclusion. The platform happily prints a p-value either way. Making that number trustworthy is build step six, and it is the difference between an experimentation platform and a machine for shipping noise with confidence.

Design

Six components, each tied to the principle it applies:

Experiment config — the definition: variants, traffic allocation, targeting rules, and a salt, edited in a control plane separate from code.
Deterministic bucketing — hash(salt + unit_id) mapped into the allocation ranges, stateless and reproducible. This is deterministic hashing.
Flag evaluation — assignment is delivered as a flag the app reads, gated by a rollout fraction, targeting, and a kill switch, all changeable without a deploy. This is feature flags.
Exposure logging — an exposure event emitted at assignment time, which is the join key that ties a unit to the variant it actually saw.
Event pipeline — exposures and outcome events stream into a warehouse where they're deduplicated, joined, and aggregated per experiment, variant, and metric. This is event pipelines.
Statistical analysis — per-variant estimates with variance, a significance test, guardrails, and a decision. This is statistics.

The named systems differ on where bucketing runs and how analysis is done. Optimizely ships SDKs that bucket client- or server-side with a MurmurHash over the unit and experiment, serves flags and experiments from a config the SDK streams, and analyzes with its Stats Engine, which uses sequential, always-valid inference so results can be monitored continuously without the peeking penalty. The warehouse-native open-source and internal platforms — GrowthBook, Eppo, Statsig — bucket in the SDK the same way but push metric computation down to the company's data warehouse as SQL, and often add variance reduction (CUPED) on top. The large internal platforms documented by Microsoft (Kohavi's ExP) and Google emphasize overlapping experiment layers, standing sample-ratio-mismatch monitoring, and A/A validation. The shared core is stateless hash bucketing plus a decoupled measurement pipeline; they diverge on client-versus-server assignment and on whether analysis is a bespoke pipeline or warehouse SQL. The classic frequentist t-test on a fixed sample is the counterpoint to the sequential approach step six builds toward.

Build it

Start with the obvious version: pick a variant at random when asked. It splits traffic correctly in aggregate and is trivial. The failures are immediate — a user gets a different variant on every request, page, and device, so the experience flickers and is incoherent, and because nothing records the choice deterministically, the assignment can't be reproduced for analysis. The next step makes assignment a stable function of the unit.

Replace randomness with a hash of the unit id and the experiment's salt, mapped into a fixed space and then into the allocation ranges. The same unit always lands in the same bucket and the same variant, with no stored state and no lookup, and the same computation reproduces offline during analysis. This is deterministic hashing — production SDKs use MurmurHash for speed and uniformity; the digest here is for clarity. The failure it exposes is operational: the experiment is hardcoded, so changing the split, targeting a segment, ramping traffic, or killing a bad variant all require a code deploy.

Move the experiment definition out of code and into a config the SDK reads and hot-reloads, so allocation, targeting, and on/off state change without a deploy. Evaluation now checks a kill switch, applies targeting rules, and gates entry with a second hash so only a chosen fraction of traffic enters the experiment at all — the rollout ramp. This is feature flags, the same mechanism that powers gradual rollouts and instant kills. The failure left over is that the platform assigns variants but measures nothing: there's no record linking who saw what to what they did.

Record what each unit actually saw, and let outcomes flow in separately. On a real assignment, emit an exposure event; elsewhere the app emits outcome events as users act. Both land on a stream that feeds a warehouse, where the exposure is the join key that attributes later outcomes to the variant the unit was exposed to. This is an event pipeline, and routing logging off the hot path means assignment never blocks on it. Logging only fires when a variant is genuinely assigned, so the analysis counts only exposed units. The failure that remains is interpretation: raw per-variant counts don't say whether a 2% gap is a real effect or noise.

Turn the joined data into a decision. Join each exposed unit to its outcomes once, compute the per-variant metric mean and variance, and run a two-sample test for the difference, reporting the lift, a confidence interval, and a p-value rather than bare counts. This is statistics — the estimate quantifies the effect and the test quantifies how surprised you should be under the null. The failure this sets up is the trustworthiness row from the requirements: a fixed-horizon p-value is only valid if you look once, peeking at it daily inflates false positives, and a sample ratio mismatch can invalidate the whole comparison before any test runs.

Guard the decision against the two ways it lies. Before trusting anything, run a sample-ratio-mismatch check — if the observed split deviates from the intended one beyond chance, a bucketing or logging bug is present and the experiment is invalid, full stop. Then replace the fixed-horizon p-value with an always-valid (sequential) bound so that monitoring the result continuously no longer inflates the false-positive rate, and require guardrail metrics to hold before declaring a winner. This is the trustworthiness step the requirements flagged, and it sits on top of the estimate from step five.

import random

def assign(variants):
    return random.choice(variants)       # new variant every call —
                                         # inconsistent across requests,
                                         # and impossible to reproduce later

Tradeoffs

Decision	What it buys	What it costs
Deterministic hashing	Stateless, consistent, reproducible assignment	Changing a salt or allocation reshuffles users; the hash must be uniform
Feature flags	Ramp, target, and kill without a deploy	A config plane to operate, propagation lag, and flag debt over time
Event pipeline	Assignment never blocks on logging; measurement scales	Results lag; biased exposure logging silently skews the analysis
Statistics	Counts become a decision with a known error rate	Rests on assumptions; naive use (peeking, many metrics) produces false wins
Sequential tests + SRM guard	Continuous monitoring is safe; invalid tests are caught	Somewhat less power than a perfectly-run fixed test; more machinery

Scaling it up

Hundreds of experiments can't each consume all the traffic, so assignment grows layers. Overlapping experiment infrastructure, as Google described it, partitions experiments into orthogonal layers with independent salts, so many experiments run on the same users simultaneously without their bucketing correlating, with explicit mutual exclusion only where two experiments would interfere. The single salt of the prototype becomes a layer-and-experiment salt scheme, and the platform tracks which experiments may coexist.

Sensitivity is where mature platforms invest, because a faster significant result means more experiments per quarter. CUPED reduces variance by regressing out a pre-experiment covariate (a user's prior metric value), and stratification and the delta method for ratio metrics correct variance that the naive two-sample test gets wrong. These don't change the assignment at all; they change the analysis so the same traffic yields a tighter estimate, which is the bulk of the statistical engineering in systems like Microsoft's ExP, Netflix's, and Statsig's.

Trustworthiness extends well past the SRM check. A/A tests validate that the pipeline reports no effect when there is none, SRM monitoring runs as a standing guardrail on every experiment, and the analysis has to reckon with novelty and primacy effects, with interference between units when the product is a marketplace or network (which breaks the independence the t-test assumes and forces switchback or cluster-randomized designs), and with the multiple-comparisons problem when an experiment is scored on dozens of metrics, where false-discovery-rate control replaces a single threshold. Kohavi's practical guidance is largely a catalog of these traps.

Where assignment runs shapes consistency. Client-side bucketing causes a visible flicker as the page corrects itself and can be tampered with; server-side or edge assignment is consistent and tamper-resistant but needs the experiment config delivered to the edge with low latency, which is what the streaming SDKs of LaunchDarkly and Optimizely provide. Identity stitching matters too — a logged-out user bucketed on a device id must keep their variant after login, which means sticky bucketing that maps the stable identity forward.

Analysis is increasingly warehouse-native and governed. Rather than a bespoke pipeline, GrowthBook, Eppo, and Statsig compute metrics as SQL against the company's existing warehouse, with curated metric definitions, holdout and guardrail metrics, and an experiment review process so teams don't each reinvent the statistics. From here the concrete follow-ons are an overlapping-layers assignment prototype that buckets many experiments orthogonally with mutual exclusion, a variance-reduction analysis prototype that adds CUPED and correct ratio-metric variance, and a warehouse-native metrics prototype that pushes the join and aggregation down to SQL with governed metric definitions. Each extends this hash-plus-pipeline core without rebuilding it.

References

Kohavi, Tang & Xu, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing (2020) — SRM, guardrails, and experiment pitfalls: https://experimentguide.com/
Tang et al., Overlapping Experiment Infrastructure: More, Better, Faster Experimentation (Google, KDD 2010) — layered/orthogonal experiments: https://research.google/pubs/pub36500/
Deng, Xu, Kohavi & Walker, Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data (CUPED, WSDM 2013): https://dl.acm.org/doi/10.1145/2433396.2433413
Johari, Pekelis, Walsh & Koomen, Always Valid Inference: Continuous Monitoring of A/B Tests — sequential testing behind Optimizely's Stats Engine: https://arxiv.org/abs/1512.04922
GrowthBook — open-source, warehouse-native experimentation (SDK bucketing + SQL analysis): https://docs.growthbook.io/
Statsig and LaunchDarkly — feature-flag delivery, deterministic bucketing, and gradual rollout: https://docs.statsig.com/ · https://docs.launchdarkly.com/
MurmurHash — the non-cryptographic hash production SDKs use for bucketing: https://github.com/aappleby/smhasher
Foundations referenced inline: deterministic hashing · feature flags · event pipelines · statistical inference