Article Text


Reducing distress and improving social functioning in daily life in people with auditory verbal hallucinations: study protocol for the ‘Temstem’ randomised controlled trial
  1. Alyssa Jongeneel1,2,
  2. Dorien Scheffers3,
  3. Nynke Tromp4,
  4. Chani Nuij1,
  5. Philippe Delespaul5,6,
  6. Heleen Riper1,
  7. Mark van der Gaag1,2,
  8. David van den Berg2
  1. 1 Department of Clinical Psychology, VU University and Amsterdam Public Health Research Institute, Amsterdam, The Netherlands
  2. 2 Research and Innovation, Parnassia Psychiatric Institute, The Hague, The Netherlands
  3. 3 Anxiety Disorders, PsyQ, The Hague, The Netherlands
  4. 4 Department of Industrial Design, Delft University of Technology, Delft, The Netherlands
  5. 5 Department of Psychiatry and Neuropsychology, Maastricht University, Maastricht, The Netherlands
  6. 6 Mondriaan, Heerlen, The Netherlands
  1. Correspondence to Alyssa Jongeneel; a.jongeneel{at}


Introduction Auditory verbal hallucinations (AVH) are prevalent experiences that can induce distress and impede social functioning. While most voice hearers benefit from antipsychotic medication or cognitive–behavioural therapy, additional effective interventions are needed to reduce the burden of experiencing AVH. ‘Temstem’ is an easily accessible and useable smartphone application that was developed by designers in close cooperation with voice hearers and experts. By using language games, Temstem aims to reduce distress and improve social functioning.

Methods This is a single-blind multicentre randomised controlled trial with two arms: ‘Temstem+AVH monitoring’ versus ‘AVH monitoring’ (total n=100). Participants are adult patients who suffer daily from AVH and will be recruited in outpatient units. Primary assessment in daily life is made by the Experience Sampling Method (ESM) and daily monitoring with the PsyMate app. During an ESM period of 6 days, participants assess their mental state (including AVH and context) several times a day by filling in short questionnaires. There are three 6-day ESM periods: at baseline (week 0–1), post-treatment (weeks 5–6) and follow-up (weeks 9–10). In addition, during the entire 10-week study period, all participants monitor their AVH two times a day with a short assessment via the PsyMate app. Participants in the Temstem+AVH monitoring condition are provided with the Temstem app from week 1 to 6. Other assessments made at baseline, post-treatment and follow-up are based on questionnaires and a clinical interview.

Ethics and dissemination The results from this study will provide an evaluation of the effectiveness of Temstem, a non-invasive and easily accessible app for voice hearers, and insight into the determinants of optimal use. Results will be disseminated unreservedly, irrespective of the magnitude or direction of the effects. This study protocol was approved by the Medical Ethics Committee of the VU University Medical Centre (METC number: 2015.435/NL53684.029.15).

Trial registration number ISRCTN75717636; Pre-results.

  • auditory verbal hallucinations
  • intervention
  • smartphone application
  • Experience Sampling

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See:

Statistics from

Strengths and limitations of this study

  • This study tests an intervention for voice hearing in everyday life.

  • Primary assessments are made by Experience Sampling Method (ESM), resulting in ecologically valid data from daily life.

  • The ecological data are validated with previously validated tests.

  • Only daily voice hearing participants, who are able to perform ESM, will participate in this study.


Auditory verbal hallucinations (AVH) can be highly distressing and demoralising, and negatively impact mental health and social functioning. Individuals with distressing AVH tend to have relatively high levels of negative affect, such as depression,1 and tend to socially withdraw.2 3 Adults with AVH have less years of education than ‘healthy’ individuals and they show lower employment rates.4 Also, AVHs are associated with medication non-compliance5 and increased odds of suicide attempt.6

AVHs are a common symptom in psychosis. A review shows prevalence rates of 40%–80% in schizophrenia.7 However, AVHs do not occur exclusively in people with schizophrenia or other psychotic disorders. For example, the estimated prevalence of AVH is 67.7% in bipolar disorder, 40.6% in depressive disorder8 and 13.7% in borderline personality disorder.9 Another review showed that the median lifetime prevalence of voice hearing in the general population is 9.6%.10 Thus, hearing voices is a common symptom and does not seem to be exclusively related to schizophrenia. As Kinderman and colleagues already stated in 2013, we should no longer study diagnostic labels but focus more on specific problems,11 as we will in this study.

Traditionally, treatment for AVH consists of antipsychotic medication and cognitive–behavioural therapy (CBT).12 13 Unfortunately, while many patients benefit from antipsychotic medication, side effects are problematic and treatment adherence is low.12 14 CBT is repeatedly found to be effective in reducing AVH, with small to medium effect sizes.13 15 However, implementation of CBT in patients with a psychotic disorder is generally poor,16 which appears to be a result of high costs, heavy workload of therapists (ie, their time to deliver CBT is limited), time pressure and staff training needs.17

Therefore, additional interventions are needed to reduce the burden of hearing voices; preferably, these should be low-cost solutions that do not rely on additional training of (relatively scarce) therapists. Meanwhile, mental healthcare is adopting technology to support well-being by monitoring health conditions and enhancing self-help.18 Since 2008, thousands of mental health apps have been developed,19 which have the potential to reduce (the negative impact of) different kinds of mental health problems. Although mobile health (mHealth) seems to be promising, few of these apps have been evaluated with high-quality and rigorous study designs.18

In 2012, 72% of the individuals with a severe mental illness in the USA owned a mobile device; this was about 12% lower than in the general population.20 These individuals use their smartphone for a range of functions including calling, texting and emailing.21 22 A study that assessed a specific app (FOCUS, an app to support self-management of illness in persons with schizophrenia) showed that individuals with schizophrenia were able to use the app autonomously, and all participants but one completed the study.22 23 On average, participants used the app 4 days/week for 3–6 months, mostly in response to prompts, but also independently. Interestingly, some support was found for the presence of a therapeutic alliance with a stand-alone mHealth psychotherapy intervention.24 Thus, the use of smartphone interventions among individuals with schizophrenia appears to be feasible and acceptable, and shows clinical potential.

Together with designers and in close cooperation with voice hearers, our group developed a smartphone application called ‘Temstem’. Temstem (a Dutch word that loosely translates as ‘Tame the Voice’) aims to reduce distress and social dysfunction associated with voice hearing. The Temstem app offers two language games that can be played at any moment, that require little cognitive effort and automatically adjust to the performance level of the user. The Temstem app has two different functions and is based on three proposed working mechanisms:

First, Temstem aims to enhance control over AVH with the function Silencing (see ‘Interventions’). A perceived lack of control over the voices is one of the important reasons why people can be distressed by voices.25 In Temstem, the motoric language games were designed to activate the language production and the motor areas in the brain; in this way, Temstem aims to temporarily suppress AVH, giving the user control over the AVH.26 27 Moreover, there is evidence that suggests that interference of subvocal activation (eg, playing a language game) suppresses auditory hallucinations.28 Performing these tasks may also distract the user from the AVH, which has been found to be an effective coping mechanism.29

Second, self-esteem and negative social schemas influence how individuals respond to AVH.30 31 Therefore, Temstem reinforces and supports the user such that it enhances self-esteem, that is, participants indicate in the Temstem app how the voices make them feel (eg, weak, powerless, in danger, worthless, bad, guilty or neutral) and receive specific positive feedback while playing the games. This aims to compete with the negative self-beliefs. If a participant feels worthless, the app gives feedback such as ‘You’re a valuable person’ or ‘You’re okay!’ Improving self-esteem with this type of competitive information was found to improve the acceptance of AVH and reduce negative emotional responses.32

Third, Temstem contains a potential therapeutic mechanism that may reduce AVH frequency. It is known that taxing working memory during recall of negative visual imagery reduces the vividness and emotionality of this imagery and reduces the frequency of reliving symptoms.33 34 This was also found for aversive auditory memories,35 and a recent study showed that taxing working memory with a digital intervention was feasible and effective in reducing emotional intrusive memories.36 Since strong associations have been found between AVH and experiencing severe childhood trauma,37 38 a proportion of voice hearing may actually be considered to be intrusive auditory imagery. We therefore hypothesise that playing language games, while recalling an episodic memory of hearing voices, will reduce the vividness and emotionality of AVH in some people with voices. This may occur, in particular, among individuals with trauma-related AVH.39 40 This dual task is the second function in Temstem, that is, Challenging (see ‘Interventions’).

The target group of Temstem is people with frequent AVH. To use the app optimally, we advise voice hearers to use Temstem every time they hear voices. They can use the Silencing function, for example, to gain control over the voices. When this succeeds, probably after a few minutes of using Temstem, the app can be closed and users can continue with what they were doing. We also recommend to use the Challenging function at least a few times a week. By using this function, we theorise that the voices gradually become less vivid and emotional.

Temstem was developed in an iterative design process in which we used qualitative and quantitative methods to continuously test the usability, underlying principles and functions of the app in a group of voice hearers (not published). The present study tests the clinical effects of Temstem in a randomised clinical trial (RCT).


The aim of this study is to evaluate the effects of Temstem in the daily lives of individuals who suffer from frequent AVH. Since Temstem is used in daily life, primary outcomes will be assessed in daily life with Experience Sampling Method (ESM) and daily monitoring using the PsyMate app. Because monitoring symptoms in daily life can be considered as an intervention,41 42 ‘Temstem+AVH monitoring’ will be compared with ‘AVH monitoring’.

ESM is an ecologically valid means of assessment and overcomes the limitations of experimental and retrospective measures, for example, it does not rely on retrospective memories.43 In statistical analyses, the effects of extreme responses are minimalised when using ESM, and the findings yielded by ESM are more robust than self-report measurements.43 Psychotic phenomena measured by ESM were found to have moderate to strong associations with corresponding items from validated questionnaires.44 However, validation of the ESM items by means of corresponding items in questionnaires or interviews is not well investigated and requires more attention. For this study, to confirm the primary ESM outcomes, most of the outcomes were also assessed using validated instruments (ie, an interview, and several self-report questionnaires, see ‘Instruments’).

Primary objective

The primary objective is to test the effects of Temstem on distress and social functioning. Distress is primarily defined as the total negative affect scores over all ESM assessments at baseline and post-treatment and social functioning as the frequency of being in a social situation and/or pursuing an activity on the ESM time points. We will also study whether there are changes in participants’ subjective experience of distress caused by AVH (1b) and disturbance of social functioning by AVH (2b) on the daily assessments. Also we will test whether the potential effects observed at post-treatment are maintained in the follow-up period, when both groups do not use Temstem.

  • Hypothesis 1: Distress over all daily life assessments will decrease after using Temstem for 5 weeks, as measured by ESM.

    • Hypothesis 1b: Subjectively experienced distress caused by AVH will decrease after using Temstem for 5 weeks, as measured by daily monitoring.

  • Hypothesis 2: Social functioning will improve after using Temstem for 5 weeks, as measured by ESM.

    • Hypothesis 2b: Subjectively experienced negative impact of AVH on social functioning will decrease after using Temstem for 5 weeks, as measured by daily monitoring.

Secondary objective

The secondary objective is to test the effects of Temstem on the frequency and intensity of AVH, the perceived control over AVH, power in relation to AVH, self-esteem, depression and paranoid ideation, and to confirm the primary and secondary outcomes with validated instruments. In addition, using questionnaires, we will test the effects of Temstem on the vividness of AVH, emotionality of AVH, quality of life and social rank to AVH.

  • Hypothesis 3: The frequency and intensity of AVH will decrease after using Temstem for 5 weeks, as measured by ESM/daily monitoring.

  • Hypothesis 4: Temstem will enhance control over AVH, as measured by ESM/daily monitoring.

  • Hypothesis 5: Temstem will enhance power in relation to AVH, as measured by daily monitoring.

  • Hypothesis 6: Temstem will improve self-esteem, depression and/or paranoid ideation as measured by ESM/daily monitoring.

  • Hypothesis 7: The outcomes of hypotheses 1–6 will be confirmed by validated interviews and/or questionnaires.

  • Hypothesis 8: Temstem will reduce the vividness and emotionality of AVH, as measured by a questionnaire.

  • Hypothesis 9: Temstem will improve quality of life and social rank to AVH, as measured by a questionnaire.

Tertiary objective

The tertiary objective is to determine what factors predict and mediate the effects of Temstem and to investigate the usability of Temstem. A reduction in the vividness and emotionality of AVH is one of the proposed working mechanisms of the Temstem app and is hypothesised to mediate outcome. However, because this working mechanism is difficult to assess reliably in a real-life setting, a separate laboratory study (not reported here) will be performed to thoroughly evaluate this working mechanism.

  • Hypothesis 10: The effect of Temstem on distress is predicted by gender, age, education level, severity of childhood trauma, possession of a smartphone, characteristics of AVH and/or the severity of distress at baseline.

  • Hypothesis 11: The effect of Temstem on distress is mediated by increased control over AVH, improved self-esteem and/or a reduction of depression symptoms.

  • Hypothesis 12: Temstem is a useable intervention for this patient group and participants will be satisfied with the application.

Methods and analysis


Adult participants (≥18) will be recruited from outpatient units of 12 mental health organisations in the Netherlands that provide care to adult patients with AVH: Altrecht, Arkin, Bavo Europoort, Dijk en Duin, GGZ Centraal, GGZ InGeest, GGZ Noord-Holland Noord, GGZ Oost Brabant, Parnassia, PsyQ, Stemmenpoli Universitair Medisch Centrum Utrecht, and Universitair Medisch Centrum Groningen. Written informed consent will be obtained from each participant by a research assistant.

Inclusion criteria

To be eligible to participate, patients must meet both of the following criteria:

  • presence of AVH (with distress) for longer than 1 month;

  • presence of AVH during a minimum of 4 days/week in at least 3 of the previous 4 weeks.

Exclusion criteria

  • insufficient competence in the Dutch language;

  • currently undergoing CBT for AVH;

  • estimated IQ under 70;

  • changes in medication regimen in the last month;

  • intensive previous or current use of Temstem;

  • not willing or able to learn to use a smartphone;

  • current involuntary admission in a closed ward.


Experience Sampling Method

Most of the primary and secondary outcomes will be assessed with the PsyMate application, an Experience Sampling Method (ESM) app ( ESM (also referred to as Ecological Momentary Assessment) is a structured diary technique that samples random moments in daily life and allows to monitor mental state, behaviour and context in daily life.41 Participants are prompted to fill in short assessments, which are recorded electronically.45 Monitoring by ESM is increasingly used to assess interventions, because it is ecologically valid and sensitive to change, and can be used to determine how variables are related to each other across time and contexts. ESM is particularly suitable for assessing the impact of particular events or behaviours on internal psychological processes.46 Research repeatedly showed that ESM is valid and feasible in people with severe mental illness; ESM questions were highly correlated with self-report measurements, compliance rates are generally high and participants are satisfied with this method.41 44 45 47 Regular monitoring, without any further intervention, was found to reduce symptoms of depression48 and psychosis.42 Therefore, ESM is considered to be more than a passive monitoring tool and can be perceived as an intervention.

In this study, all participants use the PsyMate app for 6 days at baseline (week 0–1), post-treatment (weeks 5–6) and follow-up (weeks 9–10). The PsyMate gives 10 pseudorandom prompts a day and presents a 2 min questionnaire that the participant has to fill in (within 15 min) after the prompt. This questionnaire consists of 45 items and will be used to measure most of the primary, secondary and tertiary outcomes, including distress, social functioning, frequency of AVH, intensity of AVH, control over AVH, power in relation to AVH, self-esteem, depression, paranoia and type of voices (all scored on 7-point Likert scale, or on a checklist).

During the entire study period of 10 weeks, every morning and evening the participants fill in a short questionnaire on the PsyMate app to monitor AVH. In the morning, five questions assess the sleeping pattern and the influence of AVH on sleep (eg, ‘The voices bothered me last night’). The evening monitoring questionnaire consists of 19–31 items (depending on branching) related to the characteristics of AVH, the responses to AVH (eg, experienced control over AVH) and the distress induced and subjective impact on social functioning by AVH on that day (eg, ‘Today, the voices hindered my functioning’). Negative affect and self-esteem are also assessed each evening for prediction and mediation purposes. See online supplementary appendix 1 for the specific questions included in the ESM and daily monitoring assessments.

Supplementary file 1

Other measurements

Distress due to AVH, the frequency and intensity of AVH and control over AVH will be measured by the Auditory Hallucinations Rating Scale (AHRS)49 50 at baseline, post-treatment and at follow-up (see table 1 for an overview). The AHRS is a subscale of the Psychotic Symptom Rating Scale (PSYRATS). The PSYRATS scales have excellent inter-rater reliability.49 The AHRS subscale has 11 items, rated on a 5-point ordinal scale; the items were shown to be independent of each other, contributing to the assessment of unique characteristics.49

Table 1

Schedule of enrolment, interventions and assessments

Social functioning will be measured by the Sheehan Disability Scale (SDS).51 The SDS assesses subjectively experienced negative impact of AVH on social functioning at baseline, post-treatment and follow-up. The SDS is a 3-item test, which measures the extent to which the sectors (1) work, (2) social life or leisure activities, and (3) home life or family responsibilities are impaired by psychiatric symptoms; it is scored from 0 (not at all) to 10 (extremely). The SDS is reliable and sensitive to individual differences.51 

Power in relation to AVH will be measured with the omnipotence/power subscale of the Beliefs About Voices Questionnaire-Revised (BAVQ-R).52 This subscale consists of six items. Responses are rated on a 4-point scale: disagree (0), unsure (1), agree slightly (2) and agree strongly (3). The BAVQ-R is reliable and sensitive to individual differences.52

Self-esteem is assessed with the Self-Esteem Rating Scale-Short Form (SERS-SF)53 at baseline, post-treatment and follow-up. This is a 20-item questionnaire and consists of a positive and a negative subscale. The total score is the negative subscale subtracted from the positive subscale; items are scored on a 7-point Likert scale. Confirmatory factor analysis showed that a short version of the original instrument had a high validity for all studied samples. The SERS-SF is a valid and reliable self-esteem measure for individuals with severe mental health problems.53

Severity of depression symptoms will be assessed with the Beck Depression Inventory, Second Edition (BDI-II)54 at baseline, post-treatment and follow-up. The BDI-II consists of 21 items rated on an ordinal 4-point rating scale. It has good psychometric properties and is a reliable and valid measure.54 55

The Green et al Paranoid Thoughts Scale56 measures the severity of paranoid ideations at baseline, post-treatment and follow-up. It has 32 items and consists of two scales: social reference and persecution. Items are scored on a 5-point Likert scale. Good internal consistency has been found. Validity was established for both scales and their dimensions, and the scales were sensitive to clinical change. A hierarchical relationship was found between social reference and persecution.56

The visual analogue scale (VAS) is one question or statement rated on a scale (ranging from 0 to 10) and can be widely used for various questions. This study will use a VAS to measure the vividness and emotionality of AVH at baseline, post-treatment and follow-up. Participants are asked to subjectively rate how vivid and how unpleasant the AVHs have been in the past week ranging from 0 (not at all) to 10 (extremely).

Quality of life is assessed by the Manchester Short Assessment of Quality of Life (MANSA).57 The MANSA assesses (in particular) satisfaction with life as a whole, and with several life domains at baseline, post-treatment and follow-up. It consists of 16 questions, of which four are answered with ‘yes’ or ‘no’, and 12 items are rated on a 7-point scale. The psychometric properties of the MANSA, both concurrent validity and reliability, appear satisfactory.57

Social rank to voices (inferiority vs superiority) will be measured by the Social Comparison Rating Scale (SCRS)58 at baseline, post-treatment and follow-up. It consists of 11 statements and the responses are rated on a 10-point scale. The SCRS is based on the social rank theory59 and is a reliable test.58

To investigate traumatic childhood experiences at baseline, the Childhood Trauma Questionnaire (CTQ) short form60 is used. The CTQ is a 25-item questionnaire about traumatic childhood experiences before age 17 years. It consists of five scales: physical, sexual and emotional abuse, and physical and emotional neglect. Items are scored on a 5-point Likert scale; the CTQ is a valid measure of childhood trauma.60

The user-friendliness and usability of Temstem is measured at the post-treatment assessment and only by participants in the Temstem condition. The questionnaire list (self-developed) consists of 17 statements (rated on a 7-point Likert scale), 6 multiple-choice questions and 3 open questions concerning their experiences using Temstem. Examples are: ‘I experienced more control over AVH by using Temstem’ and ‘By using Temstem, I was able to do more activities outdoors.’


This study is a single-blind multicentre RCT with two arms: ‘Temstem+AVH monitoring’ versus ‘AVH monitoring’. Participants in the AVH monitoring condition can choose to start using Temstem after the follow-up period of the study. The two groups will be compared at baseline, post-treatment and at follow-up to examine whether the effects had endured. This trial meets all Standard Protocol Items: Recommendations for Interventional Trials criteria61 (see online supplementary appendix 2).

Supplementary file 2

Power and sample size calculation

We have powered the study conservatively to detect at least small to medium between-group effects (f=0.30) on the primary outcomes. Although multilevel analyses are robust against moderate attrition, we aim to include an extra 20% to compensate for dropout. We used the Powersim program62 running on Stata V.14. Powersim is a Monte Carlo simulation of an empirical power curve and we modelled for the ‘mixed’ multilevel random regression function of Stata V.14, with one predictor and a random effect for the subject level panel variable. We rendered data for normalised ESM samples using parameters from previous research with subjects with psychotic symptoms. Therefore, we are aiming to include 50 participants in both arms, with a total of n=100, for a 0.80 plus power at P=0.05.


The participating mental health organisations will screen their caseloads using a 1 min screener (ie, either sent by mail or provided face-to-face to patients) and will send the screeners to the researchers (see figure 1). Before the researchers contact the patients who meet the inclusion criteria on the screening and have indicated they might be willing to participate in a study exploring an innovative tool for gaining more control over AVH, the practitioner is asked whether there are any barriers to patient participation (eg, unable to provide informed consent). Practitioners can also enrol their patients directly by sending a completed screening list to the researcher on which the clinician indicates whether there are any barriers/objections.

Figure 1

Flow chart screening and assessment. AVH, auditory verbal hallucination; ESM, experience sampling method.

If there are no barriers/objections, patients will be informed about the study by a researcher (by telephone or face-to-face). If all inclusion criteria and no exclusion criteria are met, and the patient is considering participation, information about the study (the Subject Information Form) will be sent to the patient by email or by post. If the patient has had sufficient time (1 week) to consider participation and decides to participate, he will be invited for a face-to-face meeting in which informed consent will be signed. After signing informed consent (see online supplementary appendix 3), the baseline assessment will be performed, which consists of a clinical interview and questionnaires. During this appointment, all participants will receive a smartphone, are instructed how to use it and start with the ESM for 6 consecutive days (week 0–1). If they manage to provide a sufficient number of reports (a minimum of 25 assessments is regarded as reliable) the remainder of the baseline assessment will be conducted. Thereafter, the participants will be randomly assigned to either the ‘Temstem+AVH monitoring’ condition or to the ‘AVH monitoring’ arm.

Supplementary file 3

In the Temstem+AVH monitoring condition, the Temstem app is installed and participants receive instructions from a research assistant on how to use Temstem. Then, they use Temstem daily for 5 weeks (weeks 1–6). In addition, every morning and every evening they monitor their voices with ESM. In the AVH monitoring arm, the participants perform these same assessments but are asked to refrain from using the Temstem app, which the researchers will monitor during the post-treatment and follow-up assessments. Between the baseline and post-treatment appointment, a research assistant will call all participants, irrespective of which condition they are in (for a maximum of three times, this will be documented), to check whether the device and apps are functioning adequately, and whether there are any problems in the use of the device or apps. Then, the post-treatment assessment will be performed, which consists, consecutively, of a clinical interview and questionnaires, a 6-day ESM assessment (weeks 5–6), and a clinical interview and questionnaires. Then, participants in the Temstem+AVH monitoring condition stop using Temstem. In the period between post-treatment and follow-up, the monitoring of voices continues in both conditions. The follow-up assessment is carried out, where a clinical interview and questionnaires are conducted, and the third 6-day ESM assessment is performed (weeks 9–10). After these 6 days, all participants are free to use Temstem, since it is freely downloadable in the App Store and Google Play. Participants in the ‘AVH monitoring’ condition receive instructions on how to use Temstem. Participants without a device of their own can borrow the research smartphone for an additional 5 weeks. All interviews and assessments will take place at the participating mental health organisations, ideally at the location the participant regularly attends for treatment.


Participants will be randomly assigned to one of two conditions: ‘Temstem+AVH monitoring’ or ‘AVH monitoring’. Randomisation is conducted via a scientific randomisation program on the internet ( by the independent randomisation bureau of Parnassia Psychiatric Institute. For each participating centre, five lots for ‘Temstem+AVH monitoring’ and five lots for ‘AVH monitoring’ will be made available. If a centre includes more participants, additional lots will be assigned. All research assistants are blinded for treatment allocation. After completing the baseline assessment, randomisation is performed and the research assistant is informed about the allocation. This unblinded research assistant then instructs the participants how to use Temstem (if applicable). The practitioners are informed about the participation of their client and to which condition the client has been allocated. The unblinded research assistant is then excluded from all further assessments for that specific participant. Therefore, the post-treatment and follow-up assessments will be performed by another blinded research assistant.


All participants receive treatment-as-usual for their psychiatric problems, mostly in community mental healthcare teams. Treatment generally consists of case management, antipsychotic medication and, in some cases, CBT. During their participation in the Temstem trial, participants and clinicians are asked not to start CBT for AVH and to ensure that antipsychotic medications remain unchanged. However, if psychotic symptoms increase in severity, due to any reason, both CBT and antipsychotic medication may be provided.

AVH monitoring

Participants in both the ‘Temstem+AVH monitoring’ and ‘AVH monitoring’ conditions use ESM for three periods of 6 days each. During an ESM period, 10 times a day, a prompt appears with a short questionnaire which the participant fills in within 15 min of the prompt. Additionally, during the entire study period of 10 weeks, participants actively monitor their voices two times a day. As described in the ‘Experience Sampling Method’, ESM may serve as a self-contained intervention or influence experienced distress associated with AVH.


Participants in the ‘Temstem+AVH monitoring’ condition are asked to read the information in the Temstem app concerning voice hearing and the rationale of the application. A researcher supports them in this if necessary. Participants are instructed in the use of the two functions in Temstem and practise the two language games in the app. Subsequently, the researcher encourages daily usage and advises the participant to use Temstem when experiencing voices, and to use both of the functions that are built into the app, that is, ‘Silencing’, which is primarily designed to increase control over AVHs whenever they occur (see the Introduction section, first working mechanism), and ‘Challenging’, in which the user proactively engages with episodic memories of the voices by dual tasking (see the Introduction section, third working mechanism).


If they wish to do so, participants can leave the study at any time for any reason, without any consequences. Also, the investigator can decide to withdraw a participant from the study in case of an urgent medical reason(s). Participants who do not start with, or discontinue using Temstem while allocated to the ‘Temstem+AVH monitoring’ condition, are requested to complete post-treatment and follow-up assessments.

Fidelity checks

At baseline, the AHRS (the only clinical interview in the trial) is audiotaped for all participants. Thereafter, all research assistants rate a random selection of tapes for inter-rater reliability. In addition, every 6 weeks, an intervision meeting is organised for all research assistants to discuss any difficulties in enrolment or assessments; these meetings are supervised by the first author (AJ). In every phase of the study, a check is made to ascertain whether participants use the apps (PsyMate and/or Temstem), whether they use them in the correct way and if there are any problems concerning the app(s).


In case of incidental unblinding, another research assistant will perform the remainder of the assessment and the (potential) subsequent assessment(s).

Adverse events

In the case of any adverse events, the directions and guidelines of the Medical Ethics Committee of the VU University Medical Centre are followed. This Medical Ethics Committee determined that there were no relevant risks attached to participation in this specific study.


Data management

All data are anonymised and directly coded with a number. Hard copy data will be stored anonymously at the secured data centre of the Department of Scientific Research and Innovation of Parnassia Psychiatric Institute. All data entry is double-checked. A data monitor from Parnassia Psychiatric Institute is appointed for the study. A Data Monitoring Committee is not needed since this study has been indicated low risk.

PsyMate data will be stored anonymised on a server owned by Maastricht University (the Netherlands). Servers are protected by firewalls and exclusively accessible via a Virtual Private Network (VPN) connection with a username and password combination. All communications are Secure Socket Layer (SSL) encrypted. Only researchers can access the data of this study. Temstem data will also be stored, coded and cannot be traced back to the participant without the code that the researchers have. The coded data are sent to the server with encryption.

Data analysis

All analyses will be performed according to the intention-to-treat principle. For our primary outcomes, multilevel analyses will be used. Mediation analysis with a multiple mediator model (multiple mediators in the same model) will be calculated with 5000 bootstraps using the PROCESS macro published by Preacher and Hayes.63 For prediction analysis, linear multiple (logistic) regression models will be used. The details of these analyses will be explained/discussed in the publications of the outcomes of the trial.

Protocol modifications

For all modifications of the protocol, were formal amendments are submitted to the Ethics Committee of the VU University Medical Centre, the rules of the METC for communicating this change to relevant parties will be followed.

Ethics and dissemination

This study protocol was approved by the Medical Ethics Committee of the VU University Medical Centre (METC number: 2015.435/NL53684.029.15).

Although AVH may induce considerable distress and impede social functioning, available treatments are still not optimal. Antipsychotic medications have side effects and high attrition rates, and the dissemination of CBT for AVH remains problematic.12–14 16 Temstem is an easily accessible smartphone application that was developed by design engineers in close collaboration with voice hearers and scientists and clinicians specialised in AVH. It was designed to help people experience more control over AVH in their daily life. This study tests the effect of Temstem on distress, social functioning, the frequency/intensity of AVH, control over AVH, power in relation to AVH, self-esteem, depression, paranoid ideations, the vividness/emotionality of AVH and quality of life, and the role of social rank in voice hearing will also be tested. This study will also shed light on the working mechanisms of Temstem and on which individuals with AVH are most likely to benefit from this intervention. Data will be analysed by the intention-to-treat approach and disseminated unreservedly, irrespective of the magnitude or the direction of the effects. The target is to publish the outcomes in peer-reviewed international journals within 1 year after inclusion of the last participant. All relevant parties involved (participants, practitioners, researchers) will receive the main study results. Other healthcare professionals will be notified via workshops, lectures, symposia or conferences. If Temstem is found to effectively reduce the burden of hearing negative voices, it can be used as a non-invasive, complementary tool for existing treatments, or as a stand-alone intervention, to help the individuals who are distressed by AVH. The potential range of the app is extensive; many voice hearers can be reached at low costs. The first-generation e-Health was primarily based on incremental innovation and consisted largely of computerised cognitive behavioural protocols that were used within the mental healthcare setting. Although this has merit, Temstem was developed from a more radical design perspective.64 We believe that this type of innovation can result in disruptive and sustainable innovations that go beyond the current boundaries of mental healthcare and into our everyday lives. Moreover, Temstem is an app for a transdiagnostic experience, voice hearing. At this moment, many more ‘symptom’-focused interventions are being developed which may be used in a modular and personalised fashion, irrespective of classification according to the Diagnostic and Statistical Manual of Mental Disorders (DSM) . A good example is the Feeling Safe Programme that was developed in Oxford.65 This is a modular CBT programme for persecutory delusions that focuses on several relevant factors (eg, sleep dysfunction, worry, negative self-beliefs) and is currently tested in an RCT.65 Temstem could easily be included in such a modular approach. Finally, if effective, Temstem could be translated into different languages. To assure sustainability of the app, the study results will also be used to improve Temstem.


The authors thank all the participants involved in this study, without whom this trial would not have been possible. We also thank Marion Bruns for her invaluable help. Finally, we thank all the research assistants, local researchers, independent specialists, advisors and all others who helped bring this study to a successful conclusion.


  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
  10. 10.
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
  19. 19.
  20. 20.
  21. 21.
  22. 22.
  23. 23.
  24. 24.
  25. 25.
  26. 26.
  27. 27.
  28. 28.
  29. 29.
  30. 30.
  31. 31.
  32. 32.
  33. 33.
  34. 34.
  35. 35.
  36. 36.
  37. 37.
  38. 38.
  39. 39.
  40. 40.
  41. 41.
  42. 42.
  43. 43.
  44. 44.
  45. 45.
  46. 46.
  47. 47.
  48. 48.
  49. 49.
  50. 50.
  51. 51.
  52. 52.
  53. 53.
  54. 54.
  55. 55.
  56. 56.
  57. 57.
  58. 58.
  59. 59.
  60. 60.
  61. 61.
  62. 62.
  63. 63.
  64. 64.
  65. 65.
View Abstract


  • Contributors MvdG, DvdB, DS, NT, PD and AJ contributed to the concept and design of the study. MvdG (principal investigator) and DvdB (project leader) supervise the study. MvdG, DvdB and HR supervise AJ (PhD student). DvdB, DS and AJ coordinate and monitor the study. AJ and CN collect the data. DvdB, AJ and DS will carry out the analyses. All authors have complete access to the final trial data set and have read and approved the final version of the protocol.

  • Funding The Temstem trial is funded by a grant awarded to DvdB and MvdG by the ‘Innovatie Platform Parnassia’ (Monsterseweg 93, 2553 RJ Den Haag).

  • Disclaimer The Innovatie Platform had no role in the design or execution of the study or in the analysis of the results.

  • Competing interests None declared.

  • Patient consent Obtained.

  • Ethics approval Medical Ethics Committee of the VU University Medical Centre (METC number: 2015.435/NL53684.029.15).

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.