A Comparative Study of UCB and Thompson Sampling with Structured Rewards: Parameter Sensitivity and Robustness

Duke Kunshan University, Kunshan, China

^* Corresponding author: yutong.chen2@dukekunshan.edu.cn

Abstract

The behavior of multi-armed bandit (MAB) algorithms is closely tied to how their hyperparameters are set, but their stability in structured reward environments has not been examined in depth. To address this gap, we built a bandit framework from the Retail Rocket dataset that represents rewards at both the item and category levels. This setting reflects the correlations that often arise in recommendation tasks. We tested three algorithms: the standard Upper Confidence Bound (UCB), Thompson Sampling (TS), and a Structured UCB that combines individual and category-level statistics. The evaluation covered a horizon of 50,000 rounds with varied hyperparameter settings and repeated runs. Results show that the structured approach lowers cumulative regret and yields higher click- through and conversion rates. It also reduces the sensitivity of UCB and TS to exploration coefficients and prior choices, which are typically difficult to tune. Taken together, the study suggests that incorporating structural information can make bandit learning both more robust and more effective in practice, especially for large-scale recommendation systems.