Study design and overall procedure

We will conduct a prospective cohort study, called Injury Prediction with Artificial Intelligence (IPredict-AI), over one athletics season (38 weeks), from September 2022 to July 2023 (figure 1). This study will involve competing athletics athletes licensed with the French Federation of Athletics (FFA). This study protocol is reported according to the STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) items28 and following the Standard Protocol Items: Recommendations for Interventional Trials-Artificial Intelligence (SPIRIT-AI).26

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Study design overview. IPredict-AI, Injury Prediction with Artificial Intelligence; IPrevApp, Injury Prevention Application; I-REF, Injury Risk Estimation Feedback.

Patient and public involvement

Participants and the public were involved in the design and development of the study protocol.

Materials

The inclusion procedure and data collection will be carried out online on a mobile and computer website application (IPrevApp, https://iprevapp.emse.fr) hosted on a local host server AlmaLinux V.8.6 (Sky Tiger) with a database (MariaDB V.10.1.48). A virtual Python environment (through an open-source Python V.3.10.2)29 is implemented to perform the ML analysis using the open-source scikit-learn libraries V.1.1.2,30 on high-performance computing hardware (2 CPU Intel Xeon Gold 6132, 14-core, 2.6 GHz). All statistical analyses regarding the primary and secondary aims will be carried out using the Statistical Software R.31

Population recruitment and inclusion/exclusion criteria

At the beginning of the 2022–2023 athletics season, the FFA will disseminate information about the IPredict-AI study through their website, social media and by email to all athletes licensed at the FFA for competition. These emails will include an invitation to participate in this study at an individual level, as well as a description of the study purpose and procedure, the participation right, a letter of information and a link to register on the website application. Once the athlete will be registered, the athlete’s account will open automatically and will offer access to the IPredict-AI study. The inclusion period will be 3 weeks from 26 September 2022 to 17 October 2022.

The inclusion of athletes will be based on the following criteria: athletes must be licensed at the FFA for competition (sprints, hurdles, jumps, throws, combined events and endurance disciplines), without any counterindications for competitive athletics activity attested by the licence at the FFA, be aged between 15 and 60, have daily access to a digital device (smartphone, computer, tablet) with a network connection (public or private). We will not exclude athletes based on their baseline injury status or history.6

Injury definition

Following Edouard et al,6 we chose the term ‘injury complaint’ since it refers to self-reported information without medical diagnosis.32 Here, an injury is defined as: ‘a pain, physical complaint or musculoskeletal lesion sustained by an athlete regardless of whether it received medical attention or its consequences with respect to impairments in connection with competition or training’.33 Athletes reporting an injury complaint will be asked to provide the following information: the circumstance of injury occurrence (training, competition, not related to athletics activity), mode of onset (sudden or gradual),33 injury location33 and consequence on athletic participation classified into four categories: (1) full participation with no discomfort, (2) full participation with discomfort, (3) reduced participation due to injury complaint and (4) full absence due to injury complaint.6 34 Injury complaints will be differentiated between those related to athletics activity and those outside of athletics activity. For the study outcome, we will restrict the analysis to injury complaints related to athletics activity (training and competition). The term ‘injury complaint leading to participation restriction’ (ICPR) corresponds to the last two categories (c and d).

Data collection

All data will be collected through questionnaires on the website application. To minimise filling errors, we will use only close-ended questions (slider, single-choice, multiple-choice, drop-down list). To avoid missing values during the filling process of a new observation, all questions in the questionnaires will be mandatory.

After being included in the study, athletes will have to fulfil a baseline questionnaire that contains their demographic characteristics (age, weight, height, sex, primary athletics discipline) and their history of injuries and illnesses during the previous athletics season (2021/2022) (table 1).

During the athletics season and throughout the study (from the end of the inclusion to the end of the study follow-up, ie, 38 weeks), the daily follow-up will consist of two questionnaires: one in the morning and one in the evening. These questionnaires will collect parameters related to the athletics activity (eg, volume, intensity, types), athletes’ psychological state (eg, anxiety, stress, self-regulation, emotions), sleep, injuries, illnesses and, only for the 34 last weeks, the level of I-REF use and the frequency of I-REF view. All these parameters are shown in table 1. The daily questionnaires will take between 0.5 and 5 min to be filled out and will be displayed automatically in the website application. They will be accessible during the whole day from 3:00 a.m. to 24 hours. Every night at 3:00, the answers will be reset for the next day.

In this study, missing data will necessarily result from athletes not responding to questions. When omissions occur, no imputation strategy will be used, the questionnaire will be considered incomplete. In addition, some questions and thus variables (eg, competition, training session, stress event) will depend on responses to other questions. Sometimes there will be no response to a certain question and thus no value for a related variable, but this cannot be considered a missing observation. These variables will take a predefined default value within their initial response range (eg, if the athlete replies that he/she did not go to training, the database will record 0 hours of training). The variables which cannot take a default value will be replaced by a constant one (eg, ‘−1’), which reflects the absence of a daily life event.

The response proportion to the questionnaire will be calculated at the individual athlete’s level and expressed as the number of responses obtained in the questionnaires divided by the total number of expected responses.6 Several strategies will be used to promote regular participation and maintain high response rates throughout the study. First, in order to help athletes to integrate the questionnaires into their daily lives, a calendar file in universal ‘.ics’ format will be made available to athletes directly from the website application and will contain all IPredict-AI study reminders for easy addition into a personal calendar. In addition, automatic reminders will be generated if an athlete has not yet completed a questionnaire before the questionnaire closing time. Furthermore, social media, emails and newsletters within the website application will be used to disseminate information regarding the study (eg, tips and tricks regarding the use of the website application, and key details), thus promoting the use of the study’s daily questionnaires throughout the study period. Participating athletes will have access to a visual dashboard on the platform displaying their personal data for the collected variables over time, allowing an individualised real-time summary of the athlete’s answers.

Daily individualised feedback on ICPR risk estimation (I-REF)

The daily feedback on individual athlete ICPR risk estimation (ie, I-REF) will be calculated by a dynamic model (ie, predictive model) using the collected data. More precisely, the feedback presented by the website application will reflect the athlete’s probability of ICPR and will be called I-REF (figure 2). The I-REF value will range between 0% (no risk of injury) and 100% (maximal risk of injury) for the following day (figure 2A). The I-REF value will be displayed on the website application together with the receiver operator characteristic area under the curve (ROC-AUC), which measures the ability of the predictive classifier model to distinguish between the two classes of injury/non-injury (figure 2B). This ROC-AUC ranges from 0 (bad performance of the predictive model) to 1 (perfect performance of the predictive model). The I-REF value will be estimated individually based on the athlete’s features (table 1). In order to mitigate overestimation in ROC-AUC score estimation, we will adhere to the practice of nested cross-validation, which serves to reduce bias in small data sets.35 Specifically, 200 iterations of nested cross-validation will be conducted to calculate each ROC-AUC score. This will involve the random selection of 75% of the data as the outer training set for hyper-parameters tuning, while the remaining 25% will be reserved for validation, serving as the outer test set. Then, for the inner cross-validation, half of the 75% selected for hyper-parameters tuning, that is, 50% (inner train set), will be used for predicting the remaining 50% (inner test set). To evaluate the performance of each model on the outer test set, we will conduct validation by retaining the best model from the inner cross-validations. This process will be repeated four times for each of the 200 iterations to obtain the best model for each outer test set.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

I-REF Module on smartphone. (A) Individual predicted class probability. (B) Ability of the predictive classifier model to distinguish between the two classes injury/non-injury. (C) Individual amount of contribution of each X to predict the Y value, where each variable will be displayed (eg, Variable #12) and will be ordered based on their absolute value influence. Blue variables decrease the risk of injury; red variables increase it. (D) Link for users to a simple explanation of A, B and C. ‘Risque de blessure’ means ‘injury risk’, ‘Fiabilité’ corresponds to the receiver operator characteristic area under the curve and means ‘trustability’, and ‘Variable’ means ‘variable’.

The development of I-REF will be made using tree-based ensemble classifier machine learning techniques. Tree-based models are among the most popular ML predictive models in sports medicine.12 16 Compared with deep learning models such as neural networks, they are more accurate for data sets where features (like ours) are individually meaningful and not strongly related via a temporal or spatial structure (as in the image or speech recognition). Besides accuracy, explainability is another essential aspect of ML models in healthcare. The often-denounced black box effect of ML models is partly resolved by models that have good explainability.35 36 Recent research conducted by Lundberg et al37 has provided a tailor-made method for decision trees, hereby strengthening their use. Baseline and daily questionnaires data will be used as input features (X) (table 1). The output parameter (Y) will be the new injury event (ie, ICPR, as a binary outcome). Our modelling process aims to establish a binary classification based on X from the day of starting data collection to the day of the last model training in order to obtain a predicted classification value of Y (ie, 0: not injured, 1: injured) for the following day. Then, we will estimate the predicted class probabilities ranging between 0 and 100 of this binary classification, which will represent the risk of injury. Finally, to individualise the procedure we will use a recent local interpretable explanation method for ML tree algorithms37 based on classic game-theory additive Shapley values38 to extract the amount of contribution of each X to predict its Y value (figure 2C,D).39

When algorithms are intended to help real-time clinical decision-making as conditions evolve, then these algorithms should make dynamic predictions using new data as it becomes available.40 However, there is no standardised method to set the period between each model training (ie, the time span between each retraining of the model) in connection with model performance when predictions are made in a real-time context.40 A possibility to solve this could be to evaluate model performance metrics at several predetermined, discrete time points, achieving continued monitoring of the predictive performance of the model.27 40 41 We will train the first version of our model within the first 4 weeks of data collection (figure 1), and we will retrain the model each week.

Athletes will be able to check the I-REF value for the first time after 4 weeks of data collection (figure 1). All athletes will be free to consult their own I-REF value, or not, and they will also be free to adapt or not their athletics activity based on it, or not. They will not receive any practical individualised recommendations regarding the risk of ICPR, so the management of this risk will be their own responsibility.

The level of I-REF use by athletes will be assessed by calculating the average score of the self-declared level of consideration that athletes give to I-REF for their athletics activity days. The frequency of I-REF viewing will be calculated as the ratio between the total number of days that athletes check their I-REF divided by the number of days that an I-REF is generated. For the frequency of I-REF viewing, we will consider that an athlete checked the I-REF when he/she enters the I-REF module in the application. This variable will be monitored automatically.

Strategies to limit the bias

We will develop strategies to limit filing errors and missing values (see the ‘data collection’ section). Strategies will also be implemented to limit bias in the predictive ML model and in the ROC-AUC score estimation (see the ‘Daily individualized feedback on ICPR risk estimation (I-REF)’ section). In order to limit the bias implied by the non-randomisation, the key variables, that have been shown to be associated with injury risk: sex, age and history of ICPR during the previous season, will be included in the analyses as independent variables. Consequently, there will be no stratification as these key variables will be already included in the analyses.

Sample size

As this study is an exploratory hypothesis-generating research project formal sample calculation is not necessary.

Study outcomes

The primary outcome will be the ICPR burden, defined as the number of athletics activity days lost due to ICPR per 1000 hours of exposure,6 7 over the period of follow-up with the presentation of I-REF to the athletes (ie, 34 weeks). The secondary outcomes will be (i) the percentage of athletes who will present at least one ICPR during the follow-up, (ii) the time, in hours of athletics activity, to the first ICPR6 and (iii) the number of ICPR per 1000 hours of exposure during the follow-up. All these variables will be gathered from the daily questionnaires in the website application. The follow-up window will be 34 weeks, starting when the first I-REF is calculated and provided to athletes, and finishing at the end of the follow-up (figure 1).

Statistical methods

We will perform the statistical analyses using a well-known software, namely R (V.3.6.3 (2020-02-29, Copyright 2020 The Foundation for Statistical Computing (Comprehensive R Archive Network, http://www.R-project.org)) and the R library ‘survival’).31

For the primary outcome, we will use a linear regression model where the dependent variable will be the ICPR burden and the independent variables will be the level of use of I-REF, sex, age and history of ICPR during the previous season. Using this model, we will explore the relationship between the ICPR burden and the level of use of I-REF by calculating the model coefficient for the variable ‘level of use of I-REF’ and its 95% CI.

For the secondary outcome (i), we will adjust a logistic regression model where the dependent variable will be the occurrence (or non-occurrence) of at least one ICPR over the season. We will calculate the OR with 95% CI as a measure of association. For the secondary outcome (ii), we will use survival analysis. First, non-athletics-related ICPR will be analysed as competing risks (using R package ‘cmprsk’) to explore whether there were significant differences between the athletes who showed higher and lower use of I-REF in terms of the cumulative incidence of ICPR occurred during athletics and outside of athletics. Second, we will adjust a Cox proportional hazards model where the dependent variable will be the time to the first ICPR. We will calculate the HR with 95% CI as a measure of association. Participants will be considered as right-censored (ie, will have incomplete data on the study outcomes at the right side of the follow-up period) if they stop completing the questionnaires, if they have an ICPR that occurred outside of athletics activity or if they have not had an injury at the end of the follow-up. For the secondary outcome (iii), we will use a linear regression model where the dependent variable will be the number of ICPR per 1000 hours of exposure during the follow-up. For all these analyses of secondary outcomes, we will include as independent variables the ones mentioned above in relation to the primary outcome analysis.

Except for the analysis of secondary outcome (ii), which has been explained above, these analyses will be performed on athletes with 100% of the data (complete case analysis). We will also consider performing sensitivity analyses based on consideration of ‘best-case’ and ‘worst-case’ scenarios.

Ethics and dissemination

This prospective cohort study was reviewed and approved by the Saint-Etienne University Hospital Ethical Committee (Institutional Review Board: IORG0007394, IRBN1062022/CHUSTE). Participants’ study information will not be released outside of the study without the written permission of the participants. During the study, participants will give their personal information and complete study questionnaires through a secured individual account on a website application. The database will be hosted on the physical server of Mines Saint-Etienne secured by a private network and individual professional access (login, password). Only P-ED, PE, LN, and the engineer who developed the application will have access to the database before, during and after the trial. Any extraction from this database will necessarily involve the anonymisation of the subjects. The raw database will not be made available online.

The results of the study will be communicated through articles in peer-reviewed journals following the STROBE items26 to produce the manuscripts, and in international scientific congresses. Individuals who have contributed to the design and implementation of the protocol will be eligible to be included in publications as coauthors. The results of the study will also be part of the doctoral thesis of P-ED. In addition, the participants of this study will be informed about the results of the study. Dissemination of the results to the end-users, with the aim of knowledge translation will also be made.