Multi-Armed Bandit Algorithms for Large Language Model Optimization: A Survey of Theory and Applications

Aberdeen Institute of Data Science and Artificial Intelligence, South China Normal University (Nanhai Campus), Foshan 528225, Guangdong, China

^* Corresponding author: x.yang1.23@abdn.ac.uk

Abstract

This article systematically expounds on the application of multi- armed bandit (MAB) algorithms in the optimization of large language models (LLMs). Facing the core challenge of the vast search space and high evaluation cost in the prompt engineering, hyperparameter tuning, and generation strategy selection of LLM, MAB has offered a powerful mathematical framework, offering a theoretically guaranteed efficient way to find the optimal solution with the ‘Explore then Commit’ strategy. The paper details the classic explore-exploit algorithms, including ε -first algorithm (ETC), upper confidence bound algorithm (UCB), asymptotic UCB algorithm and its variants, Thompson sampling (TS), etc., and analyze their specific applications in scenarios such as prompt optimization, hyperparameter tuning, and response generation for LLMs. Through theoretical derivation and case validation, this paper clarifies how to combine the rigor of Bandit theory with the complexity of LLM applications, providing a highly promising technical path for achieving efficient, adaptive, and automated optimization of LLMs.