Question 27
Domain 2: Evaluation, Tuning, and Quality OptimizationWhat A/B testing approach is most appropriate?
Correct answer: A
Explanation
A randomized 50/50 split is the standard A/B testing design because it minimizes selection bias and makes the two groups comparable. Tracking satisfaction scores, accuracy, and resolution time measures both user and performance outcomes, and running the test long enough to reach statistical significance supports a valid comparison using a t-test.
Why each option is right or wrong
A. Implement randomized A/B test with 50/50 traffic split, track satisfaction scores, accuracy, and resolution time, run for 2 weeks to achieve statistical significance, use t-test to compare results.
A randomized allocation with an even 50/50 split is the standard experimental design because it controls for selection bias and gives comparable treatment and control groups under the same conditions. The 2-week duration is appropriate only insofar as it allows enough observations to reach statistical significance, and the comparison of continuous outcomes such as satisfaction scores and resolution time is properly tested with a t-test under the usual assumption of approximately normal sampling distributions.
B. Run Prompt A for 1 week, then Prompt B for 1 week, compare results.
C. Let customer service reps choose which prompt to use.
D. Deploy Prompt B to 10% of traffic for 1 day, then decide.