Article Text
PDF
Abstract
Introduction Existing mobility endpoints based on functional performance, physical assessments and patient self-reporting are often affected by lack of sensitivity, limiting their utility in clinical practice. Wearable devices including inertial measurement units (IMUs) can overcome these limitations by quantifying digital mobility outcomes (DMOs) both during supervised structured assessments and in real-world conditions. The validity of IMU-based methods in the real-world, however, is still limited in patient populations. Rigorous validation procedures should cover the device metrological verification, the validation of the algorithms for the DMOs computation specifically for the population of interest and in daily life situations, and the users’ perspective on the device.
Methods and analysis This protocol was designed to establish the technical validity and patient acceptability of the approach used to quantify digital mobility in the real world by Mobilise-D, a consortium funded by the European Union (EU) as part of the Innovative Medicine Initiative, aiming at fostering regulatory approval and clinical adoption of DMOs.
After defining the procedures for the metrological verification of an IMU-based device, the experimental procedures for the validation of algorithms used to calculate the DMOs are presented. These include laboratory and real-world assessment in 120 participants from five groups: healthy older adults; chronic obstructive pulmonary disease, Parkinson’s disease, multiple sclerosis, proximal femoral fracture and congestive heart failure. DMOs extracted from the monitoring device will be compared with those from different reference systems, chosen according to the contexts of observation. Questionnaires and interviews will evaluate the users’ perspective on the deployed technology and relevance of the mobility assessment.
Ethics and dissemination The study has been granted ethics approval by the centre’s committees (London—Bloomsbury Research Ethics committee; Helsinki Committee, Tel Aviv Sourasky Medical Centre; Medical Faculties of The University of Tübingen and of the University of Kiel). Data and algorithms will be made publicly available.
Trial registration number ISRCTN (12246987).
- multiple sclerosis
- parkinson-s disease
- hip
- chronic airways disease
- heart failure
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
A multidisciplinary approach was implemented to define a protocol for the validation of tools for mobility monitoring, covering aspects related to devices, algorithms and users.
A set of rigorous quality assurance procedures have been established to allow for the creation of a high-quality annotated dataset to foster development in the field of digital mobility monitoring.
Mobility data will be collected in the laboratory and in the real-world for five different cohorts of slow-walkers and subsequently will be made publicly available at the end of the study.
The multistage and multidevice experimental procedures required by this validation study can be extremely challenging for both the participants and the assessors.
For the laboratory acquisitions, the level of agreement between the gold standard and the inertial sensor devices might be affected by the limitations associated with a restricted capture volume.
Introduction
The ability to move is a key contributor to physical, mental and social well-being, which is in line with the WHO’s definition of health.1 However, the study of mobility has received relatively little attention, except for diseases characterised by specific mobility dysfunction. The increasing longevity of the world’s population, together with prolonged survival of many patients with long term conditions, means that more people are suffering from loss of mobility, which in turn is a major contributing factor to a loss of independence.2 3 This has a considerable and growing personal, societal and economic impact. Efforts to mitigate this loss of mobility are an increasing priority and promising interventions are now under investigation.
Existing mobility endpoints based on performance, patient self-reporting and one-off assessment are resource-intensive and lack sensitivity,4 which limits therapeutic development and clinical management. A novel approach is needed that is low cost, simple, accurate and that can be used in the real world, including the home and the community.
Poor gait, especially slow walking, is a key determinant of mobility loss. It is associated with greater mortality, morbidity, cognitive decline, dementia and fall risk.2 3 Quantifying gait related mobility outcomes, including features such as step/stride duration and their variability, walking speed and asymmetry features is well established in supervised instrumented assessments.
Wearable devices including inertial measurement units (IMUs) that allow digital mobility outcomes (DMOs) to be described are leading the transition from a laboratory-based assessment of mobility (mobility capacity), to continuous, unsupervised monitoring of mobility in daily life conditions (mobility performance). Nonetheless, the validity of IMU-based methods to characterise real-world mobility, and gait in particular, is still limited, especially in populations suffering from pathological conditions. This is because measuring real-world gait is far from simple or straightforward. In addition, complex factors arise from multiple sources that influence outcome measures, including disease characteristics, patient specific habits, environment/context and the purpose of walking. All these factors limit the validity of existing algorithms developed to quantify targeted DMOs.5 Additionally, validation should include simultaneous evaluation6 of the participants perception and acceptability of the device7 as well as aspects related to wearability and usability.8 9 Finally, a separate assessment of the metrological performance of the sensors contained in the adopted device is required. All these validation steps need to be taken before the DMOs and the associated technologies can be effectively used for clinical4 and regulatory10 purposes (see figure 1 for summary).
Concurrent domains to be assessed as part of a technical validation of digital mobility outcomes (DMOs) obtained from a wearable device.
In this paper, we present the comprehensive, multistage protocol deployed in the technical validation study (TVS) that we have developed as part of the IMI2-JU-funded Mobilise-D project (Number 820820),11 that aims to validate a new digital method for remote monitoring of mobility. In particular, this multistage protocol aims to: (1) verify the metrological performance of the sensors included in a IMU-based monitoring device, using a procedure that could be replicated on any device; (2) establish the validity and reliability12 of the DMOs estimated by the algorithms using data from an IMU-based device, taking into accounts the effects of populations (eg, healthy adults, patients with various conditions), locomotor activities (simple straight walking vs complex walking tasks), contexts (lab based vs real world), durations (device wearing time, DMOs hourly and daily fluctuations, etc) and contextual confounding factors (such as location of walks, weather, use of walking aids, etc); and (3) establish participants’ and assessors’ opinions on the usability and acceptability of the monitoring devices that will be deployed.
This protocol will provide the stakeholders, the first comprehensive, multimodal solution to validate real-world mobility. Notably, the protocol described here has been accepted by the European Medical Agency13 as part of the Mobilise-D process for the regulatory qualification of real-world mobility performance biomarkers in Parkinson’s disease.10
Methods
The monitoring device that will be used in this study to collect mobility data is the DynaPort MM+ IMU (McRoberts, table 1). In a trade-off between usability, accuracy and ability to provide DMOs for both at step and stride level, this is attached to the lower back via an elastic waistband and Velcro strap. The validity of algorithms to accurately estimate DMOs in the real-world will be investigated in 120 participants including: healthy older adults (HA) and in five clinical cohorts (chronic obstructive pulmonary disease, COPD; Parkinson’s disease, PD; multiple sclerosis, MS; proximal femoral fracture, PFF; and congestive heart failure, CHF), chosen as presenting a variety of gait and mobility features.4 The usability and acceptability of the device from the perspective of the participants and the assessors involved in the study will be established via interviews and questionnaires.
Characteristics of the sensors included in the DynaPort MM+ device
Verification of the metrological performances of the device
Verifying the performance of a device needs a robust and comprehensive metrological characterisation of all the sensors that it embeds. This requires a series of standardised procedures (spot-checks) to be implemented to ensure accuracy of raw data that will be used as input to the algorithms. Table 1 shows the sensing characteristics of the device used in this study, the Dynaport MM+ (dimensions: 106.6×58×11.5 mm, size: 55 g).
According to Institute of Electrical and Electronics Engineers Standard for Sensor Performance Parameter Definitions (IEEE 2700–2017)14 a number of parameters are needed to characterise the metrological performance of the accelerometer, gyroscope and magnetometer included in an IMU. These parameters can be computed under both static and dynamic conditions. The main noise parameters used are the first (mean value) and second (variance) order statistics, and the root Allan variance parameters of noise.15 16 The parameters related to the first and second order statistics of noise can also be estimated by means of a short static acquisition (the minimum length of each acquisition is defined by IEEE standard for each sensor). These short static acquisitions can be performed simply using a plastic cube, where the device can be properly secured and then each of the three sensors’ coordinate axes x, y and z are in turn aligned with the direction of gravity (g) as well as its opposite direction (six combinations, figure 2). The root Allan variance parameters are instead computed over a long static acquisition.17 Typically, the acquisitions are performed over a period of 4–8 hours.15 16 All the static acquisitions should be carried out at a constant temperature of 25°C.
Testing configurations used during (A) short static (plexiglass cube and device) and (B) dynamic (turntable and device) acquisitions. Marks (in red) are applied on both on the turntable and on the base to identify start/end, which have to align for the dynamic tests.
In comparison to a static acquisition, characterising the dynamic metrological performance of the sensors embedded in an IMU is less straightforward, since the metrological standards provided by IEEE describe a sequence of operations requiring an expensive and complex testing instrumentation. However, various alternatives have been proposed in the literature. The accuracy of a gyroscope can be quantified during a single-axis rotation by a known angle by computing the ideal angular velocity and comparing it to the average measured angular velocity.18 19 This procedure should be performed using a rotation plate with a rotating speed comparable to what is encountered during human gait (~200°/s) (figure 2).
The tests described above will be performed on 35 different DynaPort MM+ devices deployed in the study. This will allow conformity with manufacturer indications to be verified, highlight the need for sensor recalibrations and provide benchmark standards for any device with equivalent sensing capacity. In turn, this will allow any device with an equivalent or superior solution to be used, in order to facilitate broader adoption of validated algorithms and cope with a continuously changing hardware landscape.
Protocol for validation of the algorithms
Several algorithms to detect the DMOs from a single device have been implemented according to agreed definitions20 and based on existing literature. At this time, these algorithms are being concurrently validated using the approach described by Bonci et al6 using pre-existing datasets, which mostly include lab-based observations only. Following this selection process, the best performing algorithms will be assessed using the data captured with the protocol here described.
Ethics and dissemination
This multicentre study is sponsored and coordinated by The Newcastle upon Tyne Hospitals National Health Service (NHS) Foundation Trust, UK. Participants will be recruited in five sites across Europe: Tel Aviv Sourasky Medical Center, Israel (ethics approval granted by the Helsinki Committee, Tel Aviv Sourasky Medical Center, Tel Aviv, Israel, 0551-19TLV), Robert Bosch Foundation for Medical Research, Germany (ethics approval granted by the ethical committee of the medical faculty of The University of Tübingen, 647/2019BO2), University of Kiel, Germany (ethics approval granted by the ethical committee of the medical faculty of Kiel University, D438/18), The Newcastle upon Tyne Hospitals NHS Foundation Trust, UK and Sheffield Teaching Hospitals NHS Foundation Trust, UK (ethics approval granted by London—Bloomsbury Research Ethics committee, 19/LO/1507).
As per the study register (ISRCTN, 12246987), the data collection was originally planned to start in April 2020 and last 6 months. The pandemic situation meant that the study started in July 2020 and is now planned to finish in September 2021. All experimental procedures are constantly monitored and revised as needed to ensure full compliance with COVID19-related health and safety measures and for safeguarding of both the study participants and the assessors. The data collected as part of this protocol will be made publicly available together with the algorithms used to process the data.
Participants
A convenience sample of 120 participants will be recruited via their clinical care team or research registers to represent the five disease cohorts (COPD, PD, MS, PFF and CHF), as well as HA. Twenty participants will be recruited for each cohort. Each cohort will be recruited across multiple sites to ensure generalisability (eg, differing cultures and contexts). Inclusion and exclusion criteria, grouped by total cohort and disease cohort, are summarised in table 2. Given the novelty of the data, rather than on a power calculation the sample size of 120 has been initially defined according to Consensus-based Standards for the selection of health Measurement Instruments guidelines for measurement properties.21 This sample size, however, will be refined after 50% of the data collection. Given that the DMOs are measured at walking bout level and not at patient level, in this analysis we will use the effective number of walking bouts observed during the 2.5 hours to perform the power calculation. We will base this analysis on a desired Intraclass Correlation Coefficient (ICC) ≥0.7, with Alpha=0.05 and Beta=0.9, and an aimed CI of 0.1. Based on this review, more participants may be recruited.
Inclusion and exclusion criteria adopted for the different disease cohorts
All participants will give written informed consent prior to undergoing a clinic/laboratory-based session to record generic and disease-specific characteristics. This will include participant reported outcomes, assessments and medical notes review. The generic and cohort-specific clinical outcomes that will be collected are summarised in table 3.
Generic and cohort-specific clinical outcomes
Patient and public involvement
Patient and public involvement and engagement has informed the design and conduct of this study. The protocol and patient facing documents were reviewed and changes implemented based on reviewer feedback. We will work with our Patient Advisory Group (with members representing the patient cohorts involved in the study) to review study findings, data interpretation and study reporting and dissemination, including codesign and presentation of dissemination materials for patients and the public. The learnings derived from these activities will further inform the work of the wider Mobilise-D project.
Experimental protocol
Performance of algorithms to determine walking-related DMOs is mostly affected by three factors: (1) the type of motor task (eg, slow as opposed to fast, straight as opposed to curvilinear or inclined walking, etc), (2) the population of interest (eg, healthy vs pathological gait), (3) the context of observation (eg, home vs outdoors). To accommodate these factors, we have developed a comprehensive, multistage protocol that includes a variety of tests conducted both in a laboratory context and in the real world (table 4).
Summary of the experimental protocol used for the validation of the algorithms in the laboratory and in the real world
Laboratory-based assessment
Laboratory-based observations will be used to quantify validity and consistency within and between groups and different types of walking tasks under controlled ideal conditions. Structured and task-based mobility activities and a simulated daily activity session, mimicking habitual movements performed at home or at work will be included. The outcome of this comparison will provide the level of highest expected accuracy and minimum detectable changes for a given DMO.
Measurement tools
Reference system
A stereophotogrammetric (SP) system (100 Hz) will be used as the gold standard in structured and simulated tests of daily activities to validate the DMOs calculated from the DynaPort MM+ raw data. SP systems provide a measurement of the instantaneous position of points in a 3D measurement volume, by means of a set of cameras, each of which can capture the 2D trajectories of markers that are attached to the object of interest. The trajectory reconstructions are affected by systematic and random instrumental errors, normally minimised to a few millimetres via ad hoc calibration procedures and filtering and smoothing techniques.22
To ensure quality and consistency in the SP data collection, accommodating different SP systems across sites, a spot check designed following the methodology proposed by Di Marco et al23 will be used, which will establish the specific level of accuracy for each system. A graphical user interface for automated preprocessing of the SP data will ensure consistency in associated procedures (labelling, gap filling, etc).
A bespoke marker set will be adopted, including four markers on each foot for detecting the gait events and four markers on the lower back device to track the displacement of the DynaPort MM+ device (figure 3).
Illustration of the adopted marker set configuration. Markers were located on the right (RHEEL) and left (LHEEL) heels, toes (RTOE, LTOE) and on the INertial module with DIstance Sensors and Pressure insoles (INDIP) units located on the right and left foot (RINDIP, LINDIP). Two additional reference markers were asymmetrically attached to the side of the foot to favour automatic recognition (RREF. LREF). Four additional markers were located on the DynaPort MM+ sensor (DYNAY, DYNAO, DYNAX, DYNAREF).
Tested devices
During this observation each participant will also be equipped with an additional multisensor system (INertial module with DIstance Sensors and Pressure insoles, INDIP)24–26 and with the Dynaport MM+. The INDIP system (figure 4) includes four inertial modules (one on the lower back, one on the non-dominant wrist and two on the feet), two distance sensors and two force-sensitive resistor pressure insoles including 16 force-resistive sensing elements (manufacturer 221e S.r.l., Italy). The INDIP has been designed to be used as a reference for real-world experiments, and in this phase of the protocol its performance will be validated against the SP system for the populations of interest. Spatio-temporal parameters will be estimated exploiting the sensors redundancy and implementing previously validated sensor fusion algorithms. Gait events will be detected using data from pressure insoles and inertial sensors independently, and then combined to increase robustness and detection accuracy (missed and minimisation of extra events). Spatial variables will be computed from the inertial data of the feet using a Madgwick filter27 28 combined with a zero-velocity update,29 30 subsequently velocity and displacement will be calculated using a direct and reverse approach.24 31–33 The individual components of the INDIP system and the associated algorithms for the estimates of the DMOs have already been extensively validated in previous studies on various healthy and pathological cohorts.31 34 35 The final assembled system in its fully synchronised configuration, developed to address the requirements of this study, is expected to perform equivalently and as such we can anticipate mean absolute percentage errors of 1% on the stride duration, between 2% and 3% in the estimate of the stride length. Preliminary results from in-lab validation showed percentage errors of about 2% for gait speed as estimated during continuous walking, including both straight and curvilinear portions.36
Different components of the INertial module with DIstance Sensors and Pressure insoles system. The figure on the left shows the pressure insoles and the connectors that link them to the distance sensors and the inertial modules. The picture on the right shows how the same system is then attached to the participant’s foot and leg.
The lower back INDIP unit and the DynaPort MM+ will be rigidly attached to each other. The data from the SP (100 Hz), INDIP (IMU and insoles, 100 Hz, Distance sensor, 50 Hz) and DynaPort MM+ (100 Hz) systems will be synchronised using a hardware-based approach for the SP and the INDIP system, and timestamps to align recordings from the INDIP and the DynaPort MM+.
Mobility tasks
Structured mobility tasks
Straight walking: straight walking is the most common test of walking.37 38 The participant walks for a distance of 5 m from a standing start and will be repeated at three different walking speeds: preferred, fast and slow (figure 5A).
Diagrams of the selected tasks: (A) straight walking test, (B) timed up and go, (C) L-test, (D) surface test, (E) Hallway test, (F) schematic of the daily living activities, (G) description of the eight tasks performed during the daily living activities.
Timed Up and Go (TUG): the TUG is a widely used clinical assessment of a person’s mobility.39 The participant is asked to sit in a chair, stand up, walk 3 m in a straight line, make a 180° turn, walk back to the chair, turn and sit down (figure 5B).
L-Test: the participant is asked to sit in a chair, stand up, walk straight, turn 90° to the left around a cone, walk straight to the second cone, make a 180° turn to the left, walk straight before making a final 90° turn to the right and return to the chair to sit down (figure 5C). Besides being a clinically validated test,40 the main purpose of including this test is the variation in curvilinear walking and the inclusion of different types of turns.
Two novel additional tests were also included to simulate confounding factors that could be encountered in the real word:
Surface test: the participant walks around a defined circuit by turning around the cones (figure 5D). The circuit is completed twice, creating the longest walking bout out of all the tasks (approximately 20 m).
Hallway test: the participant walks along a 6 m walkway stepping up and down a step positioned in the walkway. At the end of the walkway, the participant will complete a sharp 180° turn and walk back along the walkway (again stepping up and down off the step) until reaching the end point of the test (figure 5E).
Daily living activities
These lab-based tasks will be used to simulate daily activities expected in the real life, similar to previous studies.41 The participant starts by sitting in chair one and then executes a series of daily living tasks while moving around the room (see figure 5F,G).
Patients will be given regular opportunities for rest periods and will be asked to communicate if they require any additional breaks or would like to stop the assessment at any point. Use of arm rests for the TUG, L-Test and simulated daily living activities, as well as handrails for the hallway test are permitted when needed.
Real-world validation (2.5 hours observation)
This phase of the protocol will quantify validity and consistency across individuals and different types of walking tasks in the real world. It will be performed in a habitual environment (home/work/community/outdoor) chosen by the participants, without specific restrictions. The duration of the observation has been established as a trade-off between experimental, clinical and technical requirements.
Measurement tools
Reference systems
Participants will be asked to wear the INDIP, which in this phase of the protocol will be used as a reference system for the quantification of the DMOs provided by the single sensor algorithms, as applied to the DynaPort MM +data.
In order to quantify the effects of contextual confounding factors, the participants will also be provided with a system detecting outdoors walking, gradient of descent/ascent (walking uphill/downhill). The system is developed as a mobile Android application (Aeqora app) and the device selected was a Samsung S9 with Android 10. The app is composed of three parts: (1) the core tracker, (2) the interface and (3) the server infrastructure collecting data across users. The core tracker, adapted from a library developed by the University of Sheffield,42 uses the mobile phone’s internal sensors to compute the type of activity (eg, walking) and intensity (eg, cadence) to identify geo-located bouts of movement. It operates in the background and senses mobility features through a range of sensors (eg, step counters, activity recognition, accelerometer, gyroscope, etc) as well as from location services (Global Positioning System (GPS), network, Bluetooth, etc). It collects the data and stores the raw sensor data into a local database in real time. A set of mechanisms have been developed to control access to these data, keep it secure and regulate its use. First, no user identity information is sent within a single request as a token identifier is used. Additionally, a security layer is built based on Secure Socket Layer (SSL) and Transport Layer Security (TLS) 3.0 protocol to the data with scalable and efficient encryption algorithms. An SSL/TLS certificate is issued and used to establish identity and trust between server and client apps (desktop and mobile), ensuring privacy and security whenever communicating sensitive data.
Data collected during the experiments will be sent to a cluster of servers that uses algorithms to integrate the phone’s data with contextual information about the locations where the participant will walk: where possible walks will be matched to OpenStreetMap43 roads and paths, to remove GPS noise, the slope variation of each walk is computed on tiles, with a resolution of 5 m within the UK (using Ordnance Survey Terrain 5)44 and 30 m in the other locations (using NASA’s Shuttle Radar Topography Mission data,45 indoors and outdoors walking is recognised. Moreover, weather is associated with participant location based on the most proximate weather station.
The use of walking aids will also be monitored in this phase. For this purpose, a Bluetooth beacon (BlueBeacon Tag, BlueUp) will be attached to the walking aid and its activity will be detected by the phone’s mobile tracker and saved by the app. The distance between the phone and the Beacon and data from the accelerometer contained in the Beacon will be integrated to determine when the aid is in use.
The above contextual factors and the use of walking aids will be included in the analyses to determine the extent to which they affect variation in the DMOs, although the degree of correlation will be adversely affected by the issues in accurately measuring context that are associated with missing data and GPS accuracy.
Mobility tasks
To capture the largest possible range of activities during this assessment, participants will be guided by the following list of activities to be included: if relevant for their chosen environment, rise from a chair and walk to another room; walk to the kitchen and make a drink; walk up and down a set of stairs (if possible); walk outdoors (if possible, for a minimum of 2 min); if walking outside, walk up and down an inclined path. No supervision or structure to how these tasks should be completed will be given to the participants.
Real-world validation: 7 days monitoring
This observation will quantify the effects of device wearing time, hourly and daily fluctuations of DMOs, and contextual confounding factors (such as location of the walk, weather, type of housing, etc).
Measurement tools
The participants will be asked to wear the DynaPort MM+, and to carry a mobile phone equipped with the Aequora App. Bluetooth beacons will also be used to track the use of walking aids. The participants will wear the Dynaport MM+ at all times (including at night, if willingly). As this device is not waterproof, they will be instructed to remove it for showering, bathing, using a sauna and swimming and reattach it afterwards. They will be asked to keep the mobile phone charged, switched on at all times and to carry it with them whenever possible, especially when leaving the house.
Mobility tasks
Participants will be monitored continuously for 7 days, without any specific instruction being provided, except for that of wearing the provided measurement tools.
Assessment of participants’ and assessors’ experience
This part of the study will evaluate the participants’ and assessors’ experience of using the monitoring device. For the participant’s assessment, wear-time of the device during the 7 days monitoring will be collected as a primary measure of compliance. Following the period of the 7 day, free-living data collection, participants will complete two questionnaires to assess the acceptability of the device. The first is a 12-item questionnaire46 investigating usability on a 5-point ordinal scale. The questions are simple and focus on the impact of using a wearable device on participants’ feelings, comfort and the ease of use of the device. The second questionnaire is the Comfort Rating Scale,47 a 6-item measure investigating the comfort of a wearable device on a 21-point ordinal scale from ‘0—low agreement’ to ‘20—high agreement’.
A subset of participants from all recruiting sites and cohorts will complete a semistructured interview (see online supplemental file). For this qualitative part of the study, sampling will continue until saturation, that is, until no additional learning is identified from the data. The interview will explore participants’ opinions on the use of wearable devices and digital technology in healthcare, experiences of managing their condition, experiences of technology, and opinions on data privacy associated with the use of technology in healthcare. Additionally, participants will be asked about their experiences of using the device, including comfort, perceived usefulness and ease of use, barriers and facilitators, and any other usability experiences that they may have encountered. All interviews will be audio recorded, transcribed verbatim and, where required, translated to English.
Supplemental material
To assess the professionals’ experience, assessors from each of the clinical sites will be asked to assess the usability of the device after completion of the data collection. They will be provided with three questionnaires: (1) the System Usability Scale48 a commonly used, validated 10-item questionnaire that asks users to rate a device on a 5-point Likert scale from ‘1 strongly disagree’ to ‘5—strongly agree’. Questions focus on the ease of use of the device, and the integration of various functions within it; (2) the IBM Computer System Usability Questionnaire49 (to assess the DynaPort MM+ software), is composed of 19 items and asks respondents to consider their interaction with a computer system on a 7-point Likert scale from the perspective of data collection; (3) a bespoke questionnaire designed specifically for the TVS to assess the acceptability and effectiveness of the training methods, procedures and any other materials provided within the study. The questionnaire will ask respondents to rate their experiences on a 7-point Likert scale to determine whether any changes to the procedures and materials are required, and whether training was effective in preparing assessors to implement the assessment protocol as planned.
In addition, assessors will complete a semistructured interview with the aim of exploring their experiences of the data collection process. Assessors will be asked about the use of the device (eg, ease of use, intuitiveness, data collection and download procedures, etc), training and materials provided prior to the commencement of the study, and barriers and facilitators to using the device. The topic guide and open-ended questions allow for new areas of conversation to emerge. All interviews will be audio-recorded and transcribed verbatim.
Data management
All data will be uploaded to a central platform ‘e-Science Central’ (e-SC)50 which provides data processing and storage functionality in accordance with principles of reproducible research. The underlying infrastructure complies with the International Organization for Standardization (ISO 27000) standards for Information Security Management Systems and is hosted on Amazon Web Service secure services cloud platform. Data will be integrated on the platform by means of implementation of a standardised file nomenclature system. At point of capture, each file will be labelled in standardised format. For source data, we will adhere to principles defined by the US Food and Drug Administration51 for making them attributable, legible, contemporaneous, original and accurate. In particular, we will use both web-based forms on e-SC and an application from ERT (partner in the project) to capture the electronic clinical outcome assessments. The e-SC forms provide storage of event data, and support for data validation and basic data entry and verification. Both e-SC and ERT systems employ error handling at source which alert the assessors of incorrect data entry (eg, min/max boundaries, required/optional fields). Data captured at source on paper will be copied, signed and scanned, then uploaded to e-SC as a certified copy. The motion capture data will also be transferred to e-SC and stored in an unmodified form. These data will be either uploaded directly to e-SC via the e-SC portal or transferred via an Application Programming Interface. The algorithms being developed and benchmarked will be used to process these files and extract and store the DMOs.
Data analysis plan
Verification of the device
Mean and SD readings of the accelerometer, gyroscope and magnetometer signals captured during static acquisition will be used to assess the reliability of the manufacturer sensor calibration and to detect the presence of abnormal spikes in the sensor signals. Data from long static acquisitions will be used to confirm the stability of sampling frequency, the duration of the battery, and to estimate the Allan deviation (bias instability) of the gyroscope readings over time. Errors of gyroscope readings will be assessed using mean and SD of nominal, measured and relevant errors for angular velocity values during the dynamic acquisitions.
Validation of the algorithms
The data analysis will determine criterion validity (including selected performance metrics and criterion (concurrent) validity metrics of the primary (real-world walking speed) and secondary DMOs listed in table 5. Table 6 summarises the statistical tools that will be used to quantify each of the DMOs. All statistical analyses will be performed using the statistical analysis toolbox of Matlab R2018a. In all tasks and observations, continuous variables (eg, cadence, real-walking speed) will be summarised with descriptive statistics for the values obtained within walking bouts (mean and SD). ICC and its CI (95% CI) will be provided. In addition, the mean, minimum, maximum, SD, median, IQR and root mean square error of DMOs over all available walking bouts will be presented. Categorical variables (eg, laterality of initial contacts) will be summarised with frequency counts and percentages.
List of digital mobility outcomes (primary and secondary digital mobility outcomes (DMOs)) that will be analysed as part of the technical validation study
List of statistical analyses and performance metrics that will be used for the various digital mobility outcomes (DMOs)
Using the gold standard as a reference, true positives (TP), false positives (FP), true negatives (TN) and false negatives (FN) will be identified for the DMOs identified from the single device using a cut-off tolerance window defined as a fixed interval of 0.5 s52 and centred on each event detected by the reference system. The following performance metrics will then be calculated:
Criterion validity will be characterised by evaluating the absolute and relative errors, defined as the relative and absolute differences between the DMOs quantified with the single device and those derived from the reference systems:
The mean, SD and maximum of all errors will be reported for each walking bout. Limits of agreement between single sensor and reference system DMOs will be quantified. In addition, statistically significant differences between the DMOs quantified by the IMU and those by the reference system, parametric (paired t-test) or non-parametric (Wilcoxon signed-rank test) tests will be performed depending on the normality of the distribution of the DMOs. Data distribution will be visually inspected with histograms, and normality tested with the Shapiro-Wilk test.
Concurrent validity between the DMOs quantified by the single device and those derived from the reference systems will be evaluated by quantifying the Intraclass Correlation Coefficient (eg, ICC (2,1)).
All results will be presented separately by cohort (eg, PD) and subgroup (ie, subgroups of the cohorts) stratified by average stride gait speed (eg, fast speed: walking speed >1 m/s, medium speed: walking speed between 0.5 m/s and 1 m/s, slow speed: walking speed <0.5 m/s).2
If participants do not participate in one assessment (eg, one of the tasks in the laboratory) or observation (eg, 2.5 hs), their remaining available data corresponding to remaining assessments/observations will still be included in the analyses. Within each of the contexts/assessments/observations, and assuming that data are missing completely at random, a complete case approach will be used to handle missing data.53
Participants’ and assessors’ experience
Participant and professional questionnaire data will be analysed using descriptive statistics. Interviews will be analysed using thematic analysis.54 Transcripts will be deductively examined for the presence of themes related to the acceptability of the monitoring device to participants (ie, comfort, interference with daily living) based on previous literature. Specifically, perceived usefulness, comfort and ease of use are critical factors of usability; thus, these will be the categories examined within the transcripts. Regarding participants use of technology to manage their healthcare condition, an inductive approach will be taken. A list of codes relevant to the question will be generated and then refined by grouping them into potential themes.
Ethics statements
Patient consent for publication
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @emilyhume, @kluger_felix
Contributors This study is part of a large collaborative initiative, evolving around the design and execution of the study here described. The complexity of the protocol and the highly multidisciplinary content of the study justifies the high number of authors that have been and will be involved in the different stages of its planning, conducting and in the design of the analysis plan. Manuscript initial drafting: CM, TB, SDD, AC, LA, AKe, KS, SB, FC, LRo. Design and deployment of experimental tools and protocols: KS, LS, CM, TB, EB, FC, NI, BS, LVG, IV, EH, NC, FS, AC, MC, SB, LRo, AY, CK, M-EMA, LS, CB, WM, CH, EW, JE, EG, MB, JLH, MN, KT, BV, JMH, AKe, DS, AM, VL. Data collection: LA, TB, PB, MB, EB, FC, EG, CH, EH, WM, CB, LPl, LS, KS, BS, LVG, IV, AY, EW. Data analysis and algorithm development: AP-I, SDD, M-EMA, HS, AS, BE, FK, AKü, MU, AC, JGA, SK, LC, LPa, LRe, HH, FC, EG. Draft revisions: LRo, KA, BE, JMH, BV, HS, AM, JLH, PB, FK, BC, AKe, LC, LPl, EB, KS, SK, LPa, HH, WM, NC, BS, AP-I, IN, VL, SDD. Study design and coordination: CM, LRo, AM, SDD, AC. All authors have read and approved the final manuscript.
Funding We acknowledge support from the Spanish Ministry of Science and Innovation through the “Centro de Excelencia Severo Ochoa 2019-2023” Program (CEX2018-000806-S), and support from the Generalitat de Catalunya through the CERCA Program. SDD, AY and LRo are also supported by the Newcastle Biomedical Research Centre (BRC) based at Newcastle upon Tyne and Newcastle University. CM, BS, LVG and EB are also supported by the Sheffield Biomedical Research Centre (BRC) based at the Sheffield Teaching Hospital and the University of Sheffield. The work was also supported by the NIHR/Wellcome Trust Clinical Research Facility (CRF) infrastructure at Newcastle upon Tyne Hospitals NHS Foundation Trust and the CRF at the Sheffield Teaching Hospital. The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care or the funders.This study was co-funded by the European Union’s Horizon 2020 research and innovation programme and EFPIA via the Innovative Medicine Initiative 2 (Mobilise-D project, grant number IMI22017-13-7-820820). The views expressed are those of the authors and not necessarily those of the IMI, the European Union, the EFPIA, or any Associated Partners. We acknowledge the support of Grünenthal GmbH via the funding of a PhD scholarship directly dedicated to the technical validation protocol.
Competing interests FC is CEO and shareholder of Aeqora Ltd, of which VL is director and shareholder. HS is an employee of AstraZeneca. BE is co-founder and owns shares of Portabiles HealthCare Technologies GmbH. McRoberts is the manufacturer of the DynaPort. MN, JE, and LPl are employees of McRoberts. AM is an employee of Novartis and holds stock in Novartis. LS and CB are consultants of Philipps Healthcare, Bosch Healthcare, Eli Lilly, Gait-up. JMH reports having submitted a patent for assessment of mobility using wearable sensors in 400 Parkinson’s disease; the intellectual property rights 401 are held by the Tel Aviv Medical Center. LPa and LC are co-founders and own shares of mHealth Technologies.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.