Reviewing and matching the estimated power of Machine Learning Models and Statistical Models in Predicting Changes in Profit Components and Selecting the Optimal Model

Objective

The purpose of forecasting profit changes is to inform investors, financial analysts, managers, stock market authorities, creditors and other users in order to judge the entity, to decide whether to buy or sell shares or to grant or not to grant loans and credits. The purpose of this study is to evaluate performance and compare the accuracy of prediction of machine learning models and statistical models in predicting changes in three components of profit including net profit (loss), gross profit (loss) and operating profit (loss).



Methods

In this research, using financial information of 139 manufacturing companies listed in Tehran Stock Exchange during the 15-year period, during 2007-2022 and employing 25 machine learning models and 10 statistical models, the comparison of the efficiency of machine learning models and statistical models in predicting the changes in profit components such as net profit (loss), gross profit (loss) and operating profit (loss) has been discussed. In the present study, Excel software for sorting data, Eviwes software for extracting descriptive statistics, and SPSS Modeler and Rapidminer software were used for predicting earnings changes. Evaluating the performance of machine learning models has been done by two criteria of accuracy (prediction accuracy of model) and AUC (Area under curve) and performance evaluation of statistical models is done only with accuracy. . Finally, in order to select the model that has the best performance for predicting the changes in net profit (loss), gross profit (loss) and operating profit (loss), the optimal model has been selected using ROC curve among machine learning models.



Results

After calculating the average predictive accuracy of machine learning models and statistical models, it was found that the average predictive accuracy of machine learning models for dependent variables (percentage of changes in net profit (loss), percentage of gross profit (loss) changes and percentage of changes in operating profit (loss)) ranged from 83% to 93% and the average accuracy of statistical models for all three components of profit ranged from 76% to 83%. After check outing the non-normalization of the Average prediction accuracy of machine learning models and statistical models for the profit components by Kolmogorov-Smirnov test, non-parametric U-Mann-Whitney test was used to compare the accuracy of prediction of machine learning models and statistical models in predicting for the changes in the components of profit.



Conclusion

The results of testing the research hypotheses indicate high efficiency of machine learning models in predicting the changes in net profit (loss), gross profit (loss) and operating profit (loss) than statistical models. The results of ROC curve also indicate that the decision tree model with 100% accuracy in predicting the direction of net profit (loss) changes and 99.38% accuracy in predicting the direction of gross profit (loss) changes and Rule induction model with 86.76% prediction accuracy in predicting direction of profit (loss) operation, are the best performers and have been selected as the optimal model.