Optimization and Analysis of Large-scale Model Training Methods

Mechanics and Electronics, Beijing Jiaotong University, Beijing, 100091, China

^* Corresponding Author. Email: This email address is being protected from spambots. You need JavaScript enabled to view it.

Abstract

As AI models have been applied more and more widely in recent years, both the size and quantity of their training data have been expanding exponentially. Training such large-scale models is also facing numerous challenges, and at its root, the current computing efficiency of GPUs is no longer sufficient to meet the demands for training these models. For very large datasets, a single computer cannot train the model efficiently, and for small-scale tasks, the complex computations are also relatively expensive. The following three general-purpose training optimisation algorithms will be introduced to address this problem in this paper: distributed parallel training, efficient self-attention mechanisms, and parameter-efficient fine-tuning. The reasons for improving the computational efficiency of the above methods will be discussed, along with their strengths and weaknesses and applications. In addition, this paper will explore the shortcomings of the three optimisation methods mentioned above and discuss the present problems in AI model training as well as possible directions for future development.