Analysis and method comparsion of online and offline reinforcement learning

Faculty of Information Science and Engineering, Ocean University of China, Qingdao, Shandong, China

^* Corresponding author’s email: This email address is being protected from spambots. You need JavaScript enabled to view it.

Abstract

In this paper, an exploration of the online and offline precepts of reinforcement learning and the associated algorithms of paradigms is carried out in a systematic manner. Online reinforcement learning is a balance between exploration and exploitation in which there is real-time interaction with the environment, but which incurs significant costs of interaction as well as hazards. In expensive environments, offline reinforcement learning applies trial sets by earlier-collected data to create safety and efficiency procedures. The examples of representative algorithms are Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC) in the online reinforcement learning context, or Conservative Q-Learning (CQL) and Implicit Q- Learning (IQL) in the offline reinforcement learning context. Through comparative analysis, it can be established that online reinforcement learning lacks the ability to use samples efficiently and deploy flexibly, whereas offline reinforcement learning doesn’t have issues with utilizing data and ensuring the safety of data when dealing with distribution change. This paper gives recommendations on where the field of reinforcement learning should go in the present age, based on an overview of four algorithms and analyses of case studies.