Exploring the Optimization of RLHF and its Variants in Aligning Large Models with Human Preferences

Faculty of Computational Mathematics and Cybernetics, Shenzhen MSU-BIT University, Shenzhen City, Guangdong Province, 518172, China

1120220064@smbu.edu.cn

Abstract

Large Language Modeling has been widely used in learning and life scenarios. How to make model responses more consistent with human preferences has become the core of research. However, issues such as sample efficiency and data collection constrain the model performance, so optimizing related techniques is especially important for further enhancement of large language models. This paper systematically reviews the Token-level direct preference optimization, active learning, data augmentation and multimodal feedback methods. The purpose of this paper is to analyze the application and effectiveness of these methods in solving the problems of sample efficiency and data collection in aligning human preferences to large models.Token-level direct preference optimization effectively balances data utilization and model performance improvement. Active learning makes the reward model learn more comprehensive models. Data enhancement is used to synthesize more high-quality data, which is mainly used to improve sample efficiency and model performance. And the introduction of multimodal feedback provides richer and more detailed information, which can help the model better understand the task and human preferences and improve the learning effect. This paper integrates and summarizes the current research results to help researchers gain a comprehensive understanding of the current state of development in the field.