Focus on the Optimization of the RLHF Algorithm to Enhance the Training Effect After LLM

¹ School of Systems Science and Statistics, Beijing Wuzi University, Beijing, China
² College of Artificial Intelligence, Tianjin University of Science and Technology, Binhai New Area, Tianjin, China
³ Hefei No.8 Senior High School, Hefei, Anhui, China

^* Corresponding author’s email: This email address is being protected from spambots. You need JavaScript enabled to view it.

Abstract

Post-training alignment of large language models is crucial for ensuring their safety, usefulness, and alignment with human preferences. Although reinforcement learning from human feedback (RLHF) is the mainstream approach, its training process is susceptible to issues such as reward model noise, unstable policy optimization, and catastrophic forgetting, leading to deviations between model outputs and true human preferences. This paper systematically reviews the optimization algorithms for enhancing the stability of RLHF in the past three years and categorizing them into three types: “Modifying signals”, aiming to improve the reliability of the reward signal; “Optimizing algorithms”, focusing on enhancing the performance of the reinforcement learning model to increase its noise resistance; “Developing Models”, reconstructing the training framework and multi-module collaboration from a system perspective to achieve fundamental optimization. This paper elaborates on the principles, representative algorithms, and advantages and disadvantages of each category, and conducts a horizontal comparison and analysis. This review aims to provide researchers with a clear algorithm classification framework and selection reference, while also pointing out the common challenges in current research, such as reward model bias, system implementation complexity, and generalization ability, in the hope of promoting subsequent research progress.