A Study on the Impact of Obstacle Size on Training Models Based on DQN and DDQN

¹ College of Science, Mathematics and Technology, Wenzhou-Kean University, 325006, China
² Waterford Institute, Nanjing University of Information Science and Technology, Nanjing, 210044, China
³ School of Life Health Information Science and Engineering, Chongqing University of Posts and Telecommunications, 400065, China
⁴ School of science, Southwest Petroleum University, 610500, China

^* Corresponding author: 2020212040@stu.cqupt.edu.cn

Abstract

This paper presents a comparative analysis of Deep Q-Network (DQN) and Double Deep Q-Network (DDQN) algorithms in a simulated car racing environment, focusing on how variations in obstacle density affect each algorithm’s learning and trajectory planning, with the goal of enhancing adaptability and safety in autonomous driving systems. In the Gymnasium car racing environment, each episode generates a unique track where obstacles of varying sizes, colours, and quantities are introduced to test the agent's adaptability. Collisions result in immediate penalties and termination of the episode, while avoiding obstacles grants rewards. The model applies penalties to encourage fast completion and minimize the number of frames used. Various parameters such as obstacle size and complexity influence the agent's performance, promoting efficient learning and policy optimization using both DQN and DDQN algorithms under different configurations. After our experiment, comparing with DQN, we found that DDQN could be a better algorithm in car racing scenarios as in some extreme environments, DQN would not very accurately estimate future reward, therefore, agent could not make better decisions because of the misguided evaluation. While DDQN introduces another network for evaluation, which could better decrease overestimation and explore more possible actions to improve training performance. In this experiment, we evaluated both DQN and DDQN algorithms for obstacle avoidance in reinforcement learning, revealing DDQN’s superior performance, especially in complex environment. Initially, DQN and DDQN performed similarly with smaller obstacles, but DDQN gained higher rewards as training progressed. In the case of large obstacles, DDQN always outperforms DQN, which proves the enhancement of strategy optimization. Despite these advantages, DDQN also has training fluctuations that indicate areas for future research, such as enhancing stability, exploring multi-agent learning, and adapting to more complex environments.