Evaluating Conservative Q-Learning Algorithms across Dataset Qualities: A Case Study on Hopper

Zhejiang University-University of Illinois Urbana-Champaign Institute, Zhejiang University, Jiaxing, Zhejiang, 314400, China

^* Corresponding author: This email address is being protected from spambots. You need JavaScript enabled to view it.

Abstract

Offline reinforcement learning (RL) has gained attention as an effective approach for training decision-making policies using pre-existing datasets, eliminating the need for additional interactions with the environment. However, the performance of offline RL algorithms strongly depends on the quality and coverage of the available data, which remains an open challenge in practical applications. Existing studies have primarily focused on algorithmic design but provided limited systematic comparisons across different dataset qualities. This study aims to evaluate and compare three representative algorithms—Q-Learning, Batch-Constrained Q- Learning (BCQ), and Conservative Q-Learning (CQL)—under varying dataset qualities on the Hopper benchmark from the D4RL suite. Using datasets of random, medium, and expert quality levels, this study systematically analyzes the performance of these three reinforcement learning algorithms. Experimental results show that both BCQ and CQL significantly outperform standard Q-learning, particularly on higher-quality datasets, while CQL demonstrates greater robustness. Nevertheless, both CQL and BCQ show a sensitivity for the hyperparameters, indicating room for optimization in stability and computational efficiency.