Objectives

The main objective of the study is to demonstrate that, using the SMARTSPEECH system, speech performance elicited during regular phone calls through a smartphone can provide principal biomarkers for diagnosis and monitoring the progression of prodromal PD.

The specific objectives are to:

1. Develop a smartphone application that will be able to capture the subjects’ data,

2. Collect up to 2-year longitudinal speech data from subjects with iRBD, patients with early-stage PD and healthy control subjects of comparable age.

3. Collate existing approaches and develop novel methods allowing assessment speech markers of neurodegeneration in PD, including tests of their robustness against noise and recording conditions and selecting the most appropriate parameters for smartphone-based monitoring.

4. Build the concept of a complex system (SMARTSPEECH) for detection of speech abnormalities in PD and other synucleinopathies, including its statistical power evaluation in differentiation between PD, iRBD and control groups and identify its relationship to essential clinical markers reflecting disease progression such as the Movement Disorder Society-Unified Parkinson’s Disease Rating Scale.

5. Analyse the sensitivity of a set of active speech and motor tasks and compare the outcomes to data acquired through SMARTSPEECH passive calls monitoring.

Collection of speech data

To avoid potential conflicts due to different microphone characteristics across various manufacturers, the speech data will be collected using the same smartphone device, HONOR 9X Lite (Shenzhen Zhixin New Information Technology), operating on the Android V.9 system. The application will record the subject’s telephone speech with a high degree of quality using a sampling frequency of 44.1 kHz and 16-bit quantification.

The distant-speaker filtering algorithm

Due to ethical reasons and the General Data Protection Regulation law, the subjects’ speech partners will need to be removed entirely from the recordings, which will be accomplished by employing real-time adaptive filtering of the input audio. The audio will be collected via two different microphones, representing two channels in the stereo mix (see figure 1). The primary microphone (MIC 1) is the closest to the speaker’s mouth and thus captures the subject’s speech with absolute power dominance. However, the distant speaker’s talk might still be present, such as in segments in which the subject is silent or speaks quietly, or when the distant speaker is speaking very loudly. The secondary microphone (MIC 2) mainly captures the speech of the subject, but also captures the distant speaker (coming from a call speaker nearby, hence at a greater power than via MIC1) and surrounding noise. However, these settings are valid only if the loudspeaker is not activated. In this case, the power of both speakers is more or less equal in the channels. Since the algorithm is not guaranteed of functioning properly in such scenario, the recording process will be cancelled immediately when the user activates the loudspeaker.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Schematics of the HONOR 9X smartphone audio inputs and outputs.

In order to remove the distant speaker from the final mix completely, a real-time adaptive algorithm was designed based on the hardware and software settings of the smartphone. Specifically, the computational complexity needed to be optimised because intensive background processes tend to eventually be suppressed by the phone’s operating system. The principle, which is cross-channel thresholding using smoothed energy estimates from a specific spectral band, is based on the Neumann-Pearson Criterion. The algorithm’s illustrative schema is displayed in figure 2, and the procedure is as follows:

The input is a stereo call recording consisting of channel 1 (from MIC 1), , and channel 2 (from MIC 2), . Both are processed in subsequent windows of a given length of L seconds. In the current frame, the signals are decimated by two, thus reducing the computational burden of the entire procedure while maintaining adequate resolution. The signals are then processed through a Butterworth band-pass filter with a range of 300–8000 Hz, as this is the range in which the fundamental speech information is expected to be found. A power estimate is computed from the filtered channels using the L-1 norm, which is simply the sum of the absolute values of the samples in a given segment with a length of w. The estimated power trajectory is then smoothed using a normalised integrator with a forgetting factor λ, producing and . The segments containing the subject’s speech are then detected when the following condition is met:

(1)

where k represents the extent of the channels’ power difference as a ratio of the and SD, calculated as . The SD and the mean in equation (1) are both computed per window. At the end of the window processing section, the timestamps are shifted to compensate for the delay introduced by the normalised integrator. The segments that are not considered to consist of the subject’s speech are masked by zeros, and only the output from the first channel is subject to further processing.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

An illustrative schema of the distant-speaker filtering algorithm.

A final check is made to ensure that the end of the window does not conflict with the subject’s speech segment, which might produce disruptive artefacts. If the result is positive, the output from the current window is discarded, and the process is rerun with an adjusted, shorter L to avoid the conflict. The final output is a mono audio recording containing only the subject’s speech.

After the trial testing was conducted, the default value for L was set to 30 s as the optimal value for reduced computational power, more precise results and lower possibilities of conflicts at the end of the window. The segment length for the L-1 norm was set to samples with an overlap of 50%, which resulted in fast processing with sufficient precision. The forgetting factor was set to producing the optimal smoothing effect for the given scenario.

SMARTSPEECH application for monitoring passive speech via regular smartphone calls

After either an incoming or outgoing call is concluded, the user will be presented with a screen containing an option to delete or save the call (figure 3A). If the second option is selected, the recording will be filtered immediately using a distant-speaker filtering algorithm, and will be saved to the private application folder on the device. It will be possible to replay or delete each recording from the recent calls list (figure 3B) in the application within the following 24 hours. After 24 hours, the recording will be compressed and sent to a secured server using SSH and REST API. The original call will then be deleted from the device. The aim is to record at least four 5 min recordings per month from each participant. A setting section on which the user can configure the application to suit their needs will be included (figure 3C).

• Download figure
• Open in new tab
• Download powerpoint

Figure 3

(A) Option screen to delete or save the recently completed call for speech analysis. (B) List of calls from the previous 24 hours with details and the options of replaying and deletion. (C) The setting screen of the app. (D) the active task introduction screen, which shows a guide to performing the active tasks and the time elapsed since the last active task run.

Active tasks

The application has a module that invites participants to record three active functional vocal tasks once every 14 days; the participants are instructed to capture the data in everyday environments with a low ambient noise level. In addition, the participants are asked to perform five motor tasks that are commonly used in PD research.20–23 These tasks will be used to compare the sensitivity of the active and the passive data. On launching the active tasks protocol the users are presented with the guidelines and elapsed time since last run (figure 3D). Before each task, the user will be presented with short, written instructions, including an option to replay an audio or video example of the given task. In the voice section, the recordings can be replayed after being recorded. If the user is dissatisfied with the result, an option to repeat the task is available.

The entire set of active tasks consists of prolonged phonation (figure 4A), /pa/-/ta/-/ka/ syllable repetition (figure 4B), reading a passage (figure 4C,D), a tapping game (figure 5A,B), alternated tapping (figure 5C,D), a writing task (figure 5E,F), resting hand tremor (figure 6A,B) and a gait with turnaround (figure 6C,D). Each task will be executed twice, and the motor tasks are to be executed first using the right hand and then the left hand.

• Download figure
• Open in new tab
• Download powerpoint

Figure 4

(A) Prolonged phonation task instruction screen (task duration ~15 s). (B) Sequential motion rates instruction screen (task duration 7 s). (C) Reading passage instruction screen. (D) Reading passage sample text that is chosen randomly from six samples and contains approximately 80 words (task duration ~40 s).

• Download figure
• Open in new tab
• Download powerpoint

Figure 5

(A) A tapping game introduction screen. (B) A tapping game task. The delay between spawning the circle and a successful tap is measured. The action takes place 20 times (task duration ~20 s). (C) An alternated tapping introduction screen. (D) An alternated tapping task. The timestamps of the individual taps are measured (task duration 15 s). (E) Writing task instruction screen. (F) A writing task. The time spent rewriting each word is measured. The patient is shown five words in each run. Eight different datasets that are assembled in the same manner are chosen randomly in each run. Examples include the Czech words ‘SESTRICKA’, ‘NEJLEPSI’, ‘LECENI’, ‘ZABAVA’ and ‘PREDSTAVA’. The task can be skipped after 80 s have elapsed (task duration ~40 s).

• Download figure
• Open in new tab
• Download powerpoint

Figure 6

(A) Rest tremor instruction screen. (B) A rest tremor task (task duration 15 s). The patient’s hand tremor is measured using an accelerometer in all three axes. The z-axis is live plotted for user feedback. (C) A gait task instruction screen. (D) A gait task (task duration ~30 s). The z-axis rotation is measured and plotted on the screen for user feedback via an internal gyroscope.

Data validation and monitoring

Once the call and task data arrive on our server, they are saved on a database and the basic information becomes visible on the web interface to allow the data parameters to be reviewed for the quality, frequency and recording duration. In the event that the participant does not make any calls or perform passive tasks within the required period, the participant will be notified via a push notification sent to their email address.

A gateway validation of the incoming data needs to be performed before the data are stored securely on the database to minimise the possibility of corrupted, irrelevant data entering the system, which would produce misleading results.

First, recordings that are shorter than two seconds will be discarded because they do not contain much information that is relevant and often contain errors. Subsequently, the speaker-detection algorithm will be employed to prevent data contamination when the user hands the smartphone to someone else for the call. The Kaldi toolkit24 was used to construct the procedure. We employed a high-order Gaussian mixture model trained on the Czech section of the GLOBALPHONE database25 using a set of mel-frequency cepstral coefficients. New subjects in the study will be enrolled in the model once a sufficient amount of data has been collected. I-vectors are extracted on detection, and are used to compute the probabilistic linear discriminant analysis score. Using this score, a classificator decides whether the recording belongs to the correct subject. The data that will be assessed as not originating from the given subject will be moved to a specific folder and are not included in the data analysis.

Note that the purpose of the algorithm is also a form of quality control, as it excludes recordings that are of poor quality, such as utterances that are significantly affected by disruptive noise at a low signal-to-noise ratio (SNR). In these cases, the subject’s speech is so corrupted that the computed features have little similarity to the subject’s model. In addition, an SNR estimator based on minimum statistics26 is employed in the last validation phase. Segments with an instantaneous SNR lower than the preselected threshold will be excluded from the analysis because the methods for extracting speech features cease to be reliable at certain very low SNR levels (such as pitch detection algorithms at SNR less than 6 dB27).

A voice activity detector is applied to the samples that have passed the validation procedure. The output indicates long calls during which the subject only speaks for a tiny fraction of the overall duration. This allows to have a better grasp of the amount of speech data gathered via the system and to notify the subject if the amount of data is insufficient.

Extraction of acoustic speech and motor features

Hypokinetic dysarthria is due to dysfunction in the basal ganglia motor circuit, leading to impairments in the regulation of the initiation, the amplitude and the velocity of movements.12 28 Therefore, this project aims to collect, develop and extract different acoustic features describing motor aspects of speech that have well-defined PD pathophysiology17 and correspond to the perceptual description of hypokinetic dysarthria defined by Darley et al.29

The following features represent the most anticipated candidates for the final analysis:

1. Disruptions in phonation caused by dysfunctions in the vocal folds can be captured using the acoustic measure of Cepstral Peak Prominence, which correlates with the auditory perception of decreased voice quality/breathiness.30

2. Articulation deficits, which are perceived as a decrease in intelligibility, can be described using metrics such as the vowel space area.31

3. Dysprosody is captured by the reduced amplitude of vocal cord movements, which correlates with the impression of monopitch, and can be assessed using the acoustic measure of pitch variability.14

4. Decreased ability to maintain the speech motor sequence or to alternate quickly between responses can be reflected by the acoustic measures of the net speech rate and duration of pause intervals, which reflect the perceived auditory timing of speech and may describe deficits such as a slow articulation rate and a reduced ability to intermit and initiate speech.32

5. Linguistic deficits, including limited vocabulary and a decrease in its range, which might indicate potential underlying mild cognitive impairment, can be assessed using a set of lexical features such as content density.33

6. To assess the overall degree of speech impairment, powerful deep neural networks could be employed, including spectrogram analysis to evaluate specific features when traditional methods might be insufficient. Insight into the detailed method behaviour would be supportive, and would reveal critical physiological details.34

Several methods for the automatic analysis of key dimensions of speech in patients with PD have already been developed.32 35 36 However, all the methods need to undergo experimental and theoretical testing for noise robustness and reliability to validate their usefulness.27 A set of features commonly used in the existing literature will be extracted from the active motor tasks20–22; the set includes tapping velocity, intratap duration variability, reaction time, tremor acceleration skewness and velocity.

Endpoints

The primary endpoint will be represented by composite dysarthria index, reflecting the severity of speech impairment, which will be based on a combination of several distinct acoustic speech features associated with hypokinetic dysarthria in PD. Secondary endpoints will be represented by individual acoustic speech features associated with hypokinetic dysarthria in PD and linguistic features associated with potential cognitive decline.

Study design and participants

During this project, each participant will be given a full explanation of the project’s purpose and aims, will be informed about the procedure, and will be given the opportunity to ask questions before deciding whether to sign the informed consent form. In total, we plan to recruit 25 iRBD subjects, 25 early-stage patients with PD and 25 healthy controls as part of this longitudinal study. As we expect a drop-out rate of about 20% per year,17 up to 50 patients with iRBD and the same number of early-stage patients with PD will be recruited at the baseline. These data might be used for a better-sampled cross-sectional study. All the iRBD subjects will fulfil the criteria listed in the International Classification of Sleep Disorders, third edition diagnostic criteria,37 including confirmation of REM sleep without atonia via polysomnography. The inclusion criteria for iRBD will be

1. Onset of iRBD after 50 years of age.

2. No history of major neurological disease (such as epilepsy or strokes) or other significant diseases that could affect study participation or voice analysis (eg, active cancer, drug abuse or diseased vocal cords).

3. No significant cognitive decline or severe depression.

4. No history of therapy with antiparkinsonian medication.

All the patients with PD will meet the Movement Disorders Society’s clinical diagnostic criteria for PD,38 and will be investigated during the on-medication state. The inclusion criteria for PD will be:

1. Onset of PD after 50 years of age.

2. Hoehn and Yahr stage 1–2 in the on-medication state.

3. Disease duration more than 5 years after diagnosis.

4. No history of a major neurological disease other than PD (such as epilepsy or strokes) or other significant diseases that could affect study participation or voice analysis (eg, active cancer, drug abuse, or diseased vocal cords).

5. No significant cognitive decline or severe depression.

6. No involvement in any speech therapy during the duration of the project.

The inclusion criterion for controls will be that the participants have no history of neurological or communication disorders, and no history of parasomnias or other sleep disorders. PD and healthy control subjects will be age-matched and gender-matched to the iRBD group.

Each participant will be required to perform passive speech recordings and active tasks using the provided smartphone for 2 years. In addition, each of the subjects will undergo three examinations at the clinic: at the baseline, after 1 year and after 2 years (at the end of the project). The clinical examinations will consist of taking a structured personal history, quantitative testing of motor and non-motor symptoms of PD based on the Movement Disorder Society-Unified Parkinson’s Disease Rating Scale,16 cognitive testing using the Montreal Cognitive Assessment,39 autonomic testing using the Scales for Outcomes in Parkinson’s Disease-Autonomic Dysfunction,40 the evaluation of depressive symptoms using the Beck Depression Inventory II,41 and speech examinations according to the dysarthria research guidelines for acoustic analyses.23 Recruitment of the participants will begin in October 2021.

Patient and public involvement statement

Most of the patients have been involved in previous studies and are familiar with the team, the research topic and the methods. The SMARTSPEECH protocol was designed with the aid of a short questionnaire that was completed by a selected representative group of 33 patients with iRBD or PD, which provided insight into their requirements (eg, a dual-SIM phone and monetary compensation for mobile phone tariffs), doubts and motivations (such as understanding how to operate the phone and data security) and the expected frequency of phone usage and calls.

Sample size estimation

For the primary endpoint of the project, an ad hoc power analysis for a given large effect size (Cohen’s d of 0.8), with the type I error probability (α) set at 0.05 and power of 80%, based on a three-group analysis of variance with one covariate (group), determined a minimum sample size of 66 subjects with at least 22 subjects in each subgroup (ie, 22 patients with iRBD, 22 patients with PD and 22 controls).

Statistical analysis

A one-way analysis of variance with a group (PD vs iRBD vs controls) as a between-subject factor will be used to calculate the differences for each parameter of interest. The Pearson correlation coefficient will be applied to search for correlations among the variables. The minimum level of significance will be set at p<0.05 with an appropriate Bonferroni adjustment. In addition, a binary logistic regression followed by a leave-one-subject-out cross-validation will be used to assess the sensitivity/specificity of the proposed features to differentiate the iRBD subjects from controls, patients with PD from controls and patients with PD from iRBD subjects.