Validation of a Machine Learning Algorithm to enhance decision making in the management of Medial Tibial Stress Syndrome

Background: Medial Tibial Stress Syndrome (MTSS) has been identified as the most costly musculoskeletal injury (MSKI) to the British army (Sharma, Greeves et al. 2015), affecting 35-80% of military trainees. There is no reliable treatment for MTSS and reoccurrence rates are high. Prevention of MTSS is critical to reducing operational burden. Typically injury prediction is complex, multivariate, and has not been capable of discerning individual level risks.

Aim: This study investigated the accuracy of a machine learning approach that combines best known risk factors into an individual risk profiling tool for a common MSKI.

Method: Using a prospective design, this study collected a suite of key variables, determined in a previous study of navy recruits (Garnock, Witchalls et al. 2018), in a new population from the Australian Defence Force Academy. Data was obtained from 107 recruits (35 females and 75 males). Follow-up was conducted at 3 months to determine those in the group that had developed MTSS, when a total of 99 recruits (69 males, 30 females) remained for inclusion in statistical analysis. Six ensemble learning algorithms- Decision Tree, Support Vector Machine, Logistic Regression, K-Nearest Neighbour, Random Forest and Naïve Bayes were deployed and trained 5 times on random stratified samples of 75% of the dataset. The resultant algorithms were tested on the remaining 25% of the dataset and the models were compared for classification accuracy, precision and recall, and Predictive Lift. The most accurate new algorithm was additionally deployed on the previous dataset of navy recruits in order to further validate the accuracy of the model.

Results: Random Forest modelling was the most accurate with a classification accuracy of 92%, Area Under the Curve 98%, precision 91% and recall 92%. When this model was applied to an unrelated dataset it performed with similarly high classification accuracy of 93%, Area Under the Curve 93%, precision 93% and recall 93%. Lift curve analysis showed that treating the top 30% of at risk individuals would result in an 80% reduction in MTSS injuries.

Conclusions: This model is highly accurate in predicting those who will develop the debilitating condition of MTSS. The model provides important preventive capacity which should be trialled as an intervention.

References:

Garnock, C., J. Witchalls and P. Newman (2018). "Predicting individual risk for medial tibial stress syndrome in navy recruits." Journal of Science and Medicine in Sport 21(6): 586-590.

Sharma, J., J. P. Greeves, M. Byers, A. N. Bennett and I. R. Spears (2015). "Musculoskeletal injuries in British Army recruits: a prospective study of diagnosis-specific incidence and rehabilitation times." BMC Musculoskeletal Disorders 16(1): 1-7.