Performance of Multi-Armed Bandit Algorithms in Dynamic vs. Static Environments: A Comparative Analysis

College of Art and Science, Santa Clara University, 95053 500 El Camino Real, California, USA

^* Corresponding author: Bzhao2@scu.edu

Abstract

This paper conducts a comparative analysis of Multi-Armed Bandit (MAB) algorithms, particularly the Upper Confidence Bound (UCB) and Thompson Sampling (TS) algorithms, and focuses on the performance of these algorithms in both static and dynamic environments. Multi-armed bandit algorithms are instrumental in optimizing decision-making problems. While these algorithms have been studied in a static environment where the reward distribution is constant throughout the problem, real-world issues often have an unstable reward distribution, where the reward distribution may change throughout the process. This paper simulates both static and dynamic environments to evaluate the performance of UCB and TS algorithms by using the MovieLens 1M database. The paper demonstrates that the TS algorithm consistently outperforms the UCB algorithm in both static and dynamic environments. However, both algorithm shows a significantly higher cumulative regret in a dynamic environment compared with a static environment, which is due to the challenges of adapting to changing reward distribution over time. These results provide valuable insight into the application of Multi-Armed Bandit algorithms in real-world environments and highlight the need for further advancement in dynamic adaption for algorithms.