The setup: route 50% of users to variant A (current system) and 50% to variant B (proposed change). Collect metrics for both: response quality ratings, task completion rates, user retention, time-on-task, and business metrics (conversion, revenue). Run until you have statistical significance (typically 95% confidence). If B wins, roll it out to 100%. If A wins, discard B.
A/B testing AI systems has unique challenges. Response quality is subjective and hard to measure automatically. Users might rate responses differently based on mood, not quality. The same prompt can produce different responses (non-deterministic), adding noise. Carry-over effects: users who had a bad experience with variant A might rate everything lower afterwards. Careful experiment design and sufficient sample sizes are essential.
Before A/B testing with real users, many teams use shadow mode: run the new model alongside the current one, but only show users the current model's responses. Log both responses and compare quality offline (via LLM-as-judge or human review). This catches obvious regressions before any user is affected. Only after shadow mode validation does the new model graduate to a real A/B test.