The nature of ESM data

The nature of ESM data is appealing but its complexity often challenges researchers and clinicians. The data collection parameters (the actual design) that define the nature of the data can be more important than statistical techniques. Aspects to consider are item selection, item order, the time frame (eg, number of days), the intensity (eg, number of beeps per day; number of questions within a beep), the need for additional information (eg, sleep quality assessed in a morning questionnaire), the signalling algorithm (eg, random, fixed, beep-free periods, anticipation), the addition of event recording (eg, of stressors or panic attacks), application type and data storage.

Items from cross-sectional questionnaires are often unsuitable for repeated assessment in daily life. In one-time questionnaires, a reliable assessment of a construct is achieved by redundancy (multiple items in a sum score). Repeated (up to 10 times a day) answering of similar items is frustrating. Often, metaphors are used, but these lack variation within the day.21 ESM information, such as current mood, activity and company, is assessed with single items, which can be combined to improve reliability: different aspects at one moment or the same items over time. ESM data are ecologically valid but correlational in nature.22 The current activity can be a cause as well as an effect of momentary mood states. Furthermore, different mental states have different natural flows. Anxiety, for example, fluctuates and is more contextually reactive than depression. With an ESM sampling frequency of 10 times a day, highly variable states will not be adequately represented. In that case, the process is under-sampled. Other slow changing states are often over-sampled. The actual ESM protocol is usually a compromise.

Event monitoring is often added to ESM protocols. For example, participants are asked to complete a questionnaire when a certain stressor is present, or when the participant has a panic attack. This requires continual prospective monitoring and results in a high workload for participants. The recorded event initiates a questionnaire. Events should be discrete (have a clear beginning and end) and often require coding instructions (what is a panic attack or a social interaction?). There is no correction for rating misinterpretation. Feasibility limits the number of (different) events that can be reported. Having to respond to additional questions after reporting an event can act as a ‘punishment’ and results in an extinction of the report (not the actual event) under some (stressful) circumstances. The same ‘learning curve’ can occur in branching when different answers lead to different workloads. Often, time sampling is preferred because it is less burdensome, more reliable, allows the (non-exhaustive) assessment of a larger set of events and reports events as well as non-events (eg, when a participant smokes, as well as when a participant does not smoke).

When beeps are programmed at fixed times, predictability increases reactivity and this may induce behavioural changes (eg, postpone shopping or showering). More generally, when beep-free periods can be anticipated (eg, no beep expected within an hour after a beep), reactivity increases. Random sampling avoids this reactivity. Unexpectedly, true random schedules (eg, 3 beeps on 1 day and 15 on another) are not ideal, because long periods with no beeps can result in some participants staying at home (not to miss a beep). This behaviour disrupts normal daily life. Therefore, a stratified random schedule with restricted intervals is advised.4

This list of design modalities is not exhaustive and the actual choices depend on the research question and study population.

Statistical analyses of ESM data

ESM samples daily life and can be used to compute time budgets, for example, the proportion of time spent alone or doing sports. Similarly, the average anxiety level (or anxiety while alone) can be computed, resulting in one statistic per participant as the input for standard statistical analyses. However, in most situations, statisticians prefer to use all available data. In that case, analyses have to account for the fact that in ESM, multiple participants answer a set of questions multiple times, resulting in a multilevel data set. Hence, ESM observations are not independent and standard linear and logistic regression analysis techniques cannot be used.20 Multilevel linear and logistic regression is indicated and is now available in most statistical analysis packages (eg, Stata, R, SPSS). Since observations are nested in participants, the most simple multilevel regression models for ESM data include a random intercept that allows appropriate control for the between participant variability.20 These models therefore include a second error term at the participant level, in addition to the regular error term (at the assessment level) used in unilevel regression. Even when trends over time are not the primary interest, it is advisable to remove the potentially confounding effects of time in ESM data by adding a time variable (detrending23). This is effective when values increase or decrease linearly over the time of the study. Furthermore, since adjacent assessments tend to be more similar to each other than to other assessments further apart in time (autocorrelation), ESM analyses also need more sophisticated adjustment methods.11 For example, autocorrelation can be taken into account by using appropriate correlational structures for the residuals (eg, AR, MA, ARMA).11 Care must be exercised when using such models, since observations are not evenly spaced due to randomisation and data collection usually proceeds over multiple days, resulting in a huge time lag for observations on consecutive days. Therefore, continuous-time structures are usually preferable, although their availability across different software packages differs. It is also highly recommended to allow random slopes in the model where applicable.20 This yields more realistic estimates of SEs, which otherwise might be underestimated. A random slope means that the regression coefficient is not fixed, but is allowed to vary between participants. This variation is realised by adding an extra error term for each predictor. In these models, a simple variance–covariance matrix of the random effects (eg, which assumes that all covariances are zero: the default in some statistical software packages) is not realistic. Therefore, an unstructured variance–covariance matrix, where all variances and covariances are allowed to be different, is preferable. However, this can lead to rather complex and time-consuming models. Sometimes, the statistical program may even fail to provide output. Alternatively, one can consider only using random slopes for the main independent variable(s). If the analysis includes multiple main independent variables, or if the analysis still does not yield valid results after reducing the number of random slopes, it may be necessary to remove essential random slopes. In that case, resampling methods (eg, permutation tests, bootstrapping) can be used to obtain valid tests and CIs.24

Clarifying ESM questions

ESM allows the study of a broad range of research questions. The epidemiological finding that the mental health of people in urban environments is poor25 (a between-subject question) can be explored by assessing whether people in crowded environments feel worse (a within-subject question). ESM allows the exploration or ‘unpacking’ of relevant factors.26 The multilevel data structure of ESM can confuse researchers. Consider, for example, a study that explores the relation between depression and loneliness. What should be studied? Are depressive persons often alone, do lonely people become depressed, does depression make people lonely, are people in general more depressed when alone, and does this relation hold for depressed people? ESM data analyses require more specific research questions. The pitfall lies in the convergence of person-related questions and situation-related questions, both of which are represented in ESM data and can lead to divergent conclusions.22 In addition, results from ESM studies are easily misinterpreted. The conclusion that agoraphobia is a bogus diagnosis because ESM data show that people with this diagnosis feel worse when at home (and better when out) disregards the fact that people may select going out on days they feel best. In correlational relations, causal attributions should be avoided.