Fine Tuning for Dqn: Exploring Parameter Impacts on Cart Pole Performance

Nanjing Jinling High School, Nanjing, Jiangsu, China

eloise.xu1106@gmail.com

Abstract

This study investigates the impact of hyperparameter optimization (HPO) on the performance of deep Q-networks (DQN) in the Cart Pole environment, which is a classical reinforcement learning environment. This paper employs Optuna, a Bayesian optimization framework, to automate the HPO process. It systematically explores the critical hyperparameters such as the learning rate and discount factor, and maximum episode duration. The experiment results demonstrated a significant improvement in the average reward compared to the manual tuning, which also fosters convergence and reduced performance instability. The findings highlight the importance of the learning rate and the maximum episode duration in influencing the performance of the agent’s performance, which emphasizes the necessity of automated HPO, as it not only avoids the suboptimal local minima that manual tuning often faces, but also reduces the requirements of expertise, directly resulting access to efficient RL models. While this work focuses on the low-dimensional Cart Pole task, it highlights the framework of automated HPO, with limitations discussed related to computational cost and sparse-reward environmental challenges. This work emphasizes the importance of integrating automated HPO in RL, and bridges the gap in hyperparameter sensitivity analysis for DQN, providing insights into tuning tools while enhancing performances in dynamic environments.