ITM Web Conf.
Volume 61, 2024The 9th International Symposium on Current Progress in Mathematics and Sciences 2023 (The 9th ISCPMS 2023) in conjunction with AUA Academic Conference on the Application of Artificial Intelligences and Data Sciences in a Modern Science for a Better Life
|Number of page(s)
|10 January 2024
Performance comparison between maximum likelihood estimation and variational method for estimating simple linear regression parameter
Department of Mathematics, Faculty of Mathematics and Natural Sciences (FMIPA), Universitas Indonesia, Depok 16424, Indonesia
* Corresponding author: email@example.com
Variational estimation method is a deterministic approximation technique which involves Bayesian framework while giving a point estimate instead of the usual Bayesian interval estimation. The linear regression model, which has always been a popular model, can benefit from the implementation of variational estimation method. In this paper, the theoretical basis on why variational method can reduce overfitting in linear regression is reviewed. Based on the review, in theory, variational method is more robust to overfitting than MLE. This paper also performed a simulation study. The simulation is done in a manner such that the simulation represents the situation of predicting for new or hidden data. The simulation starts from generating random explanatory data and generates the appropriate response data based on linear regression equation. Then, the randomly generated data is used to estimate the linear regression parameters. The simulation is performed to compare the parameters estimation results from variational method with the method of MLE. The comparison is done using the estimation values and the squared differences between true parameters value and the estimates. Empirical findings show that both methods have relatively close estimate values. It can be seen as the simulation study concludes that both variational and ML yield rather close parameters estimates for simple linear regression case. The estimates closeness gets more obvious as the sample size grows. The study also found that Variational method has performs better in terms of parameters estimation in linear regression when the sample size is small or the data has large variance.
Key words: Bayesian regression / maximum likelihood / overfitting / simulation
© The Authors, published by EDP Sciences, 2024
This is an Open Access article distributed under the terms of the Creative Commons Attribution License 4.0, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Current usage metrics show cumulative count of Article Views (full-text article views including HTML views, PDF and ePub downloads, according to the available data) and Abstracts Views on Vision4Press platform.
Data correspond to usage on the plateform after 2015. The current usage metrics is available 48-96 hours after online publication and is updated daily on week days.
Initial download of the metrics may take a while.