You are here

Article Text

Article menu

This article has a correction. Please see:

Download PDFPDF

XML
Communication
Research
Effect of an evidence-based website on healthcare usage: an interrupted time-series study
  1. Wouter A Spoelman1,
  2. Tobias N Bonten1,
  3. Margot W M de Waal1,
  4. Ton Drenthen2,
  5. Ivo J M Smeele2,
  6. Markus M J Nielen3,
  7. Niels H Chavannes1
  1. 1Department of Public Health and Primary Care, Leiden University Medical Center, Leiden, The Netherlands
  2. 2Dutch College of General Practitioners, Utrecht, The Netherlands
  3. 3Department of Primary Care, NIVEL (Netherlands Institute for Health Services Research), Utrecht, The Netherlands
  1. Correspondence to Prof Dr Niels H Chavannes; n.h.chavannes{at}lumc.nl

Abstract

Objectives Healthcare costs and usage are rising. Evidence-based online health information may reduce healthcare usage, but the evidence is scarce. The objective of this study was to determine whether the release of a nationwide evidence-based health website was associated with a reduction in healthcare usage.

Design Interrupted time series analysis of observational primary care data of healthcare use in the Netherlands from 2009 to 2014.

Setting General community primary care.

Population 912 000 patients who visited their general practitioners 18.1 million times during the study period.

Intervention In March 2012, an evidence-based health information website was launched by the Dutch College of General Practitioners. It was easily accessible and understandable using plain language. At the end of the study period, the website had 2.9 million unique page views per month.

Main outcomes measures Primary outcome was the change in consultation rate (consultations/1000 patients/month) before and after the release of the website. Additionally, a reference group was created by including consultations about topics not being viewed at the website. Subgroup analyses were performed for type of consultations, sex, age and socioeconomic status.

Results After launch of the website, the trend in consultation rate decreased with 1.620 consultations/1000 patients/month (p<0.001). This corresponds to a 12% decline in consultations 2 years after launch of the website. The trend in consultation rate of the reference group showed no change. The subgroup analyses showed a specific decline for consultations by phone and were significant for all other subgroups, except for the youngest age group.

Conclusions Healthcare usage decreased by 12% after providing high-quality evidence-based online health information. These findings show that e-Health can be effective to improve self-management and reduce healthcare usage in times of increasing healthcare costs.

  • PRIMARY CARE
  • PUBLIC HEALTH
  • eHealth
  • health care utilization
  • interrupted time series
  • health communication

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/

https://doi.org/10.1136/bmjopen-2016-013166

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • Long-term follow-up (6 years) in a large number of healthcare consultations (18.1 million).

    • An interrupted time series analysis is the strongest research design when randomisation is not possible.

    • A change in trend of the outcome before and after the intervention and additional use of a reference group supports a causal effect of the intervention.

    • Actual healthcare usage rates based on registries were used, instead of self-reported healthcare use.

    • Other events might have affected healthcare usage during our study period, but were carefully considered and analysed in sensitivity analyses which showed similar results as the primary analysis.

    Introduction

    Healthcare costs and usage are rising in western countries and projected to increase even faster due to the ageing population.1–3 Healthcare workers and policymakers are seeking ways to control healthcare costs and usage while simultaneously improving healthcare quality. An important reason for patients to use healthcare is the need for information.4 Many patients look online for health information and use this information to decide whether or not to visit a doctor.5 ,6 So, in theory, patients using online health information are well informed and make a better assessment on the need to visit a doctor, leading to lower usage of healthcare. However, online health information is of inconsistent quality7 and evidence for the effects on healthcare usage is scarce. Previous studies showed heterogeneous results: online information did not change or even increased healthcare usage in some studies.8–17 An important limitation of these studies is that most estimated healthcare usage by patient self-reporting instead of actual healthcare usage statistics. Furthermore, the effects of online health information might be different in subgroups of sex, socioeconomic status (SES) and age. Online health information seekers are mostly young women with a high SES.5 ,18–21 A recent study, investigating the effect of an online support programme on healthcare usage in infants, found no difference in this youngest-age subgroup.16

    The Dutch College of General Practitioners (NHG) launched a non-commercial evidence-based public website in the Netherlands in March 2012.22 The NHG is the Dutch scientific society of general practitioners (GPs), and publishes and updates nationally implemented evidence-based guidelines.23 The content on the website is based on those guidelines. The primary goal of the website was to provide reliable and understandable medical information.22 Since its launch, the website has grown to become one of the most visited healthcare websites in the Netherlands. However, its effect on healthcare usage is not yet known. We therefore conducted a nationwide study to test the hypothesis that providing evidence-based online health information is associated with a reduction of healthcare usage. Second, we investigated whether the effect on healthcare usage differed by sex, age and SES.

    Methods

    Design, participants and care setting

    We used a natural before–after quasi-experimental design to study the number of consultations in primary care before and after the launch of the website (http://www.thuisarts.nl) in March 2012. Observational data were used from the NIVEL Primary Care Database (http://www.nivel.nl/en/dossier/nivel-primary-care-database) from 2009 to 2014. This time period provides data points from 3 years before and nearly 3 years after the website launch, giving enough data points to evaluate changes in trends.24 ,25

    The study was situated in the primary care system of the Netherlands. It consists of 5068 primary care practices for nearly 17 million inhabitants. Dutch GP consultations are fully covered by the mandatory basic insurance for all inhabitants. GPs function as gatekeepers for specialised care and thus are important as a first-line contact for patients seeking health information and play a central role in healthcare usage.

    To ensure data quality and prevent confounding by registration quality, only GPs were included who recorded more than 70% of their consultations with International Classification of Primary Care (ICPC) codes and provided data for the entire calendar year. Nationally representative data26 of on average 230 GPs were sampled in each year, representing 911 177 inhabitants (5.4% of the total Dutch population). GP patient contacts and diagnoses are coded according to the ICPC-1.27 Data were anonymised, but contained records for sex, age and zip codes for each patient. Dutch law allows the use of electronic health records for research purposes under certain conditions. According to this legislation, neither obtaining informed consent from patients nor approval by a medical ethics committee is obligatory for this type of observational studies containing no directly identifiable data (Dutch Civil Law, Article 7:458).

    Data sharing

    Technical appendix, statistical code and data set available from the last author (NHC).

    Intervention

    The website (http://www.thuisarts.nl) was launched in March 2012 by the Dutch minister of health, welfare and sports. Since then, it is a publicly available non-commercial website, providing reliable medical information and advice.22 All information and advice are based on the guidelines of the NHG and the online advice matches the most recent GP guidelines.23 Guideline updates are synchronised with patient situations on Thuisarts.nl by a team of professionals.22 The online content is thus evidence-based, is non-commercial, uses plain understandable language and covers over 600 topics. Each Thuisarts topic consists of several ‘patient situations’; for example: I need to start inhalation medication. Information on what to do and when to contact the GP is given. The situations can include illustrations, short videos, patient decision aids and e-health self-management tools like a self-test on alcohol use. The popularity of the website has grown exponentially, having currently 3 million unique page views each month.

    Measurements

    The primary outcome was defined as the trend in total number of consultations per 1000 patients per month before and after launch of the website. The total number of consultations was calculated in each month and divided by the number of quarterly enlisted patients, multiplied by 1000. Since the outcome was expected to have a seasonal fluctuation, a monthly interval was chosen to enable statistical control for seasonality as described below. The secondary outcome was the trend in consultation rate for a selection of the top 10 popular topics on the website. Top 10 of the topic pages was created before starting the analyses and was based on the number of unique page views, which were extracted using google analytics. A unique page view was defined as the number of times a page was loaded by a visitor's browser. If a visitor viewed a page several times during a single session, it only counted as a one page view. The following topics (ICPC codes) were selected using this criterion before conducting the analyses: herpes zoster (S70), lower back pain (L02 and L03), bladder infection in women (U71), vaginal discharge (X14), shoulder symptoms (L08 and L92), gout (T92), constipation (D12), irritable bowel syndrome (D93), diarrhoea (D11, D70 and D73) and sinusitis (R75).

    Primary analysis

    The primary analysis consists of a comparison of healthcare use before and after the launch of the website. So the most important control period was the period before release of the website. However, since changes in the healthcare system could have affected healthcare usage after release of the website, we additionally incorporated a reference group with health topics that were not exposed to the intervention. These topics were selected before conducting the analyses and included when no online information was available or page views of the topic were very low on the website during the study period. These topics were head trauma (N79 and N80), premenstrual syndrome (X09) and contact eczema (S88).

    Subgroup and sensitivity analyses

    Prespecified subgroup analyses were performed for different types of consultations: short consultations (20 min or less), long consultations (more than 20 min) and consultations by phone. Furthermore, subgroups of sex, SES and age were analysed. SES groups were created based on the available zip codes and using data of the Netherlands Institute for Social Research. These SES scores are based on postal codes (area of residence) and provided as summary scores by the Netherlands Institute for Social Research.28 Furthermore, four sensitivity analyses were performed. First, since website page views increased exponentially, we considered using a lag-time period after the release of the website. Therefore, we calculated the relative and absolute level change 12 and 24 months after the intervention. The absolute level change between 12 and 24 months can thus be considered as the effect of the website with a lag-time of 12 months after the release. In the second sensitivity analysis, we included only the GPs recording >95% of their consultations under an ICPC code, because an improvement in recording over time could have confounded the consultation rate of the secondary outcome. GPs participating over the entire study period were included in the third sensitivity analysis, because a change of participating GP characteristics over time could have confounded the consultation rate over time. Fourth, owing to changes in insurance policies, consultations performed for chronic care (asthma, chronic obstructive pulmonary disease, diabetes and cardiovascular risk management) by nurse practitioners fluctuated during the study period. To exclude any influence of these fluctuations, we performed the primary analyses separately for chronic and non-chronic care consultations.

    Statistical analysis

    All analyses, including subgroup analyses, were predefined in an analysis plan before the start of the analyses. The reference groups ‘premenstrual syndrome’ and ‘contact eczema’ were added post hoc to strengthen the reference group comparison. To evaluate the longitudinal impact of the introduction of the website, an interrupted time series (ITS) analysis was used. ITS is considered a strong design when randomisation is not possible and can thus be used to study causal effects using an observational ‘natural experiment’ approach.29 ITS measures the change in outcome, taking pre-existing trends in the data into account, thereby negating the effect of other factors influencing the outcome, provided that those factors remain constant.24 ,25 So the primary outcome's most important control period was the trend in consultation rate before release of the website. To additionally control for factors that did not remain constant after the release of the website, a reference group was created for consultations not exposed to the intervention.29

    The Cochrane Collaboration guideline on ITS analyses was used, applying an autoregressive integrated moving average (ARIMA) model.24 For the primary and secondary outcomes, the trends in consultation rate before and after the launch of the website in March 2012 were compared by calculating the slope change. Level changes at 12 and 24 months were estimated by forecasting the preintervention model and comparing this with the postintervention model. For each subgroup and the reference group, a separate ARIMA model was fitted. Since an ARIMA model relies on linearity, we checked for stationarity of the mean and variance over time by differencing.30 After differencing once, the time series reached stationarity, indicating linearity and appropriateness of an ARIMA model. We evaluated autocorrelation and seasonality by inspecting the autocorrelation (ACF) and partial autocorrelation functions (PACF), finding autocorrelations at lag 1 and 12. Hence, an ARIMA 1(1,0,0) model and seasonal ARIMA 12(1,0,0) model was fitted. After applying the model, the ACF and PACF of the noise residuals were inspected and tested by using the Box-Ljung statistic, finding no relevant residual autocorrelations, which confirmed the adequacy of the model.25 ,26 All analyses were performed in SPSS V.20.0 (IBM Corporation, New York, USA). Values were considered to be statistically significant at a p value <0.05.

    Results

    The total number of consultations during the 6-year study period was 18.1 million, with a mean (SD) of 290 (21) consultations per 1000 patients per month. The mean (SD) age of the study population during the whole study period was 40 (23) years with 51% females (see online supplementary eTable 1). The total number of unique page views of the website during the study period was 56.6 million, with a monthly mean of 450 000 in 2012 and increased to 2.9 million per month at the end of 2014 (figure 1). Trends of total number of consultations before and after the launch of the website are depicted in figure 1. Before the launch of the website, the consultation rate increased by 0.826/1000/month. However, after the launch of the website, this rate decreased to −0.794/1000/month, giving a slope change of −1.620 (p<0.001; table 1). The slope for all consultation types declined (figure 2), but only the decline for telephone consultations was significant, with a slope change of −1.056 (p=0.001; see online supplementary eTable 2).Trends for the secondary outcome are depicted in figure 3. The consultations for the top 10 topics decreased (slope change: −0.169; p=0.003; table 1), while the consultation rate for the reference group did not change (slope change: −0.001; p=0.96; table 1). When analysed separately, the consultation rate decreased significantly for six of the top 10 topics. The consultation rate did not change significantly for any of the three reference topics (table 1).

    View this table:
    Table 1

    Change in consultation rate before and after the launch of the website in March 2012 for all consultations, the top 10 topics of the website and the reference group

    Figure 1
    Figure 1

    Rate of primary care consultations and page views of the website. Consultation rates with trend lines are displayed on the left y-axis. The number of page views of the website is displayed on the right y-axis and shows a steep increase in March 2012, which represents the launch of the website.

    Figure 2
    Figure 2

    Rate of short consultations, long consultations and consultations by phone with trend lines.

    Figure 3
    Figure 3

    Rate of consultations in the top 10 topics of the website and reference group with trend lines.

    Subgroup and sensitivity analyses

    For all subgroups of sex, age and SES, the stratified analyses showed a significant decrease in consultation rate (table 2), except the youngest age group (0–16 years). In the first sensitivity analysis, calculating level changes 1 and 2 years after launch of the website, the consultation rate had decreased by 6.2% after 1 year and by 11.6% after 2 years (see online supplementary eTable 3). In our study population, this corresponds to a reduction of 62 300 consultations in the month of March 2014. The second sensitivity analysis, including only high-grade recording GPs and the third sensitivity analysis, including only GPs recording throughout the entire study period, confirmed the results of the main analysis (see online supplementary eTables 4 and 5). In the fourth sensitivity analysis, a similar decrease in consultation rate was found for chronic and non-chronic care consultations (see online supplementary eTable 6).

    View this table:
    Table 2

    Subgroup analyses of change in consultation rate before and after the launch of the website in March 2012

    Discussion

    Key results and interpretation

    In this nationwide natural experiment, healthcare usage in primary care decreased after the launch of an evidence-based health information website. This strongly suggests that, by providing evidence-based online health information, healthcare usage can be affected: in our study, consultation numbers decreased by almost 12% after 2 years. When extrapolated to the entire Dutch population, this translates to a reduction of 675 000 consultations in the month of March 2014 only, exactly 2 years after the intervention. Interestingly, the decline was most distinct for the consultations by phone. Those consultations usually encompass non-essential care, and are therefore most likely to be affected by providing online information.31 Importantly, healthcare use in the period before release of the website was not declining and even rising. In addition, a reference group was used to detect effects of other factors influencing consultation rates which were not accounted for by the preintervention trend. After release of the website, this reference group showed no change at all, which supports a causal relation between the launch of the website and the reduction of healthcare usage. The reduction was robustly present in all the subgroups, except in the youngest age group. To the best of our knowledge, this is the first study to evaluate the effects of providing evidence-based online health information on healthcare usage rates in the general population. A previous study implemented online health information and found no change in healthcare usage.17 However, this study was only locally based, underpowered and healthcare usage was assessed by self-reporting, which underestimates actual healthcare usage.32 ,33 Other studies in this field mainly addressed characteristics, motivations and choices of online health information seekers. Most of these studies suggested that online health information seekers tend to make more doctor visits and that online health information is supportive during consultations but does not replace consultations.11 ,13–15 However, these studies are hampered by low response rates and only use self-reported healthcare usage as an outcome. In addition, patient-specific characteristics could have confounded the results of these studies. For example, patients with low health anxiety make fewer doctor visits when looking online than patients with moderate to high anxiety.12 Furthermore, the current online health information is of inconsistent quality,7 highlighting the importance of providing high-quality online information. Our findings in the young-age subgroup are in line with a recent study which found no change in healthcare usage for infants.16 This suggests that the need for assurance by face-to-face contact cannot be replaced by eHealth in this patient group. The decrease in consultation rate was also present in the oldest subgroup. This refutes concerns about accessibility of older patients to online content and supports earlier positive findings of eHealth in this age group.34 ,35

    Strengths and limitations

    The major strengths of this study are its before–after design and additional use of a reference group. The rise in consultation rates before release of the website, the decline after release of the website and the absence of change in consultation rate in the reference group supports a causal relation between the intervention and the observed effects. A limitation is that, owing to nationwide implementation of the website, no reference group could be created which was not exposed to the website at all during the study period. As an alternative, we selected topics which were not on the website (head trauma) or had very low page views during the study period (premenstrual syndrome and contact eczema). They consider health topics for which self-management with online information is feasible and, in our opinion, thus form a valid reference group. Although the absolute number of consultations for these reference topics was substantially lower than the top 10 topics, they provide insight into the stability of healthcare usage without exposure to the website. Another strength is that we investigated the impact of this eHealth intervention using actual healthcare usage rates, instead of questionnaire-based estimates. Importantly, our data are representative of the Dutch population.26 Furthermore, we were able to create a long-term study duration of 6 years, providing enough data points to account for seasonality and preintervention trends. Although ITS is considered the strongest research design when randomisation is not possible and can in theory be used to study causal effects using observational data,29 we carefully considered other events that might have affected healthcare usage during our study period. For example, one of the main incentives for healthcare usage are reimbursements. During our study period, a stepwise increase of the deductible to €360 in 2014 occurred. However, primary care consultations were always fully covered by the health insurance, while secondary care consultations were not. In addition, a stable 3% of the Dutch population avoids primary care due to financial reasons,36 which cannot explain the decrease of 12% in consultation rate that we observed in this study. Another factor possibly influencing the primary care consultation rate is fluctuations in chronic care consultations performed by nurse practitioners. We addressed this by stratifying our analysis for chronic and non-chronic care consultations and found a similar decline in consultation rate for both. Furthermore, we considered a change over time of the general practice characteristics as a confounding variable for the primary outcome and a change in registration grade as a confounding variable for the secondary outcome. After including only GPs who participated throughout the entire study period and after including only high-grade recording GPs, a similar decline in trend was found as in the main analysis. Additionally, an increase in the population's age over time could have affected our results. Our population's age increased by 0.9 years during the study period (see online supplementary eTable 1). In our opinion, this would not have a large impact on our primary results and, if any, this would lead to a larger effect of the intervention. Although the website reached 3 million page views by the end of 2014, the reduction in healthcare use did not show a dose–response relationship. However, a dose–response relationship is unlikely, because only a fixed proportion of healthcare usage can probably be affected by providing high-quality information and patient self-management. This study suggests that this proportion is at least 12%, although future studies with a longer follow-up will provide more insight. A limitation of our study is that we counted unique page views of the website, although counting unique visitors could have added valuable information. Using unique page views to quantify exposure of the website could overestimate or underestimate the number of unique visitors, but still provides important information about the uptake of the website by the public. Finally, mass media coverage about healthcare use of the website may have influenced healthcare consumption during the study period. However, the only mass media coverage about the website took place in March 2012 and no active media or marketing actions were carried out during the rest of the study period.

    Generalisability and future research

    The majority of literature on this topic is focused on whether eHealth can be a replacement or is merely supplemental to regular healthcare. This simplifying approach separates the role of the physician from the patients' self-management. In the Netherlands, 90% of the GPs use http://www.thuisarts.nl as a supportive tool during their consultations and patients subsequently look up the provided information online.22 This emphasises the suitability of eHealth as a tool to integrate self-management in primary care. In our opinion, the reduction of consultations for non-life-threatening conditions in primary care is beneficial and stimulates self-management of patients. Additionally, it could improve efficiency of communication during consultations, because patients are better informed. This model of blended care has also been adopted in other fields of medicine37 ,38 and this study shows its full potential for the first time. Since not all health topics are suitable for blended care, future studies could focus on suitability of specific topics for eHealth and improvement of information for these topics. Furthermore, our results do not imply that quality of care or health of the population is increased by an evidence-based health website. This could be a topic for future research. We expect that our results are generalisable to other countries with other healthcare systems, because patients look for online health information and make a decision to visit a doctor in other countries as well. Still, effect sizes could possibly be stronger in other countries without insurance cover for primary care or weaker when internet penetration is lower.39 Furthermore, evaluating effects of other health websites, for example, in the UK (http://www.nhs.uk/Conditions) and online patient portals (eg, patient.info, patientslikeme.com), might yield further insights into the possibilities of the internet on healthcare consumption. Finally, future studies could investigate the economic impact of this intervention.

    Conclusion

    The effect of providing evidence-based online health information on healthcare usage on a population level was not yet known. This study showed that, 2 years after the launch of an evidence-based health website, nationwide primary care usage decreased by 12%. This effect was most prominent for phone consultations and present in all subgroups of sex, SES and age, except for the youngest age group. This suggests that eHealth can be effective to improve self-management and reduce healthcare usage in times of increasing healthcare costs.

    Acknowledgments

    The authors acknowledge R Wolterbeek of the Department of Medical Statistics of the LUMC for his statistical advice and B van den Hoogenhoff of the Section of Digital media of NHG.

    References

    1. ↵
      Centers for Medicare & Medicaid Services B. National Health Expenditure Data. 2014. https://www.cms.gov/Research-Statistics-Data-and-Systems/Statistics-Trends-and-Reports/NationalHealthExpendData/Downloads/highlights.pdf (accessed 3 Nov 2015).
    2. ↵
      1. Burwell SM,
      2. Frieden TR,
      3. Rothwell CJ
      . Health, United States, 2014: with special feature on adults aged 55–64. National Center for Health Statistics, 2015. http://www.cdc.gov/nchs/data/hus/hus14.pdf (accessed 3 Nov 2015).
    3. ↵
      Europian Union statistical office. Eurostat Health care expenditure. 2015. http://ec.europa.eu/eurostat/web/health/health-care (accessed 3 Nov 2015).
    4. ↵
      1. van de Kar A,
      2. Knottnerus A,
      3. Meertens R, et al
      . Why do patients consult the general practitioner? Determinants of their decision. Br J Gen Pract 1992;42:313–16.
      OpenUrlAbstract/FREE Full Text
    5. ↵
      1. Diaz JA,
      2. Griffith RA,
      3. Ng JJ, et al
      . Patients’ use of the Internet for medical information. J Gen Intern Med 2002;17:180–5. doi:10.1046/j.1525-1497.2002.10603.x
      OpenUrlCrossRefPubMedWeb of Science
    6. ↵
      1. Powell J,
      2. Inglis N,
      3. Ronnie J, et al
      . The characteristics and motivations of online health information seekers: cross-sectional survey and qualitative interview study. J Med Internet Res 2011;13:e20. doi:10.2196/jmir.1600
      OpenUrlCrossRefPubMed
    7. ↵
      1. Eysenbach G,
      2. Powell J,
      3. Kuss O, et al
      . Empirical studies assessing the quality of health information for consumers on the worldwide web: a systematic review. JAMA 2002;287:2691–700. doi:10.1001/jama.287.20.2691
      OpenUrlCrossRefPubMedWeb of Science
    8. ↵
      1. Azocar F,
      2. McCabe JF,
      3. Wetzel JC, et al
      . Use of a behavioral health web site and service utilization. Psychiatr Serv 2003;54:18. doi:10.1176/appi.ps.54.1.18
      OpenUrlCrossRefPubMed
    9. ↵
      1. Beck F,
      2. Richard JB,
      3. Nguyen-Thanh V, et al
      . Use of the internet as a health information resource among French young adults: results from a nationally representative survey. J Med Internet Res 2014;16:e128. doi:10.2196/jmir.2934
      OpenUrlCrossRefPubMed
    10. ↵
      1. Bouche G,
      2. Migeot V
      . Parental use of the Internet to seek health information and primary care usage for their child: a cross-sectional study. BMC Public Health 2008;8:300. doi:10.1186/1471-2458-8-300
      OpenUrlCrossRefPubMed
    11. ↵
      1. Dwyer DS,
      2. Liu H
      . The impact of consumer health information on the demand for health services. Q Rev Econ Finance 2013;53:1–11. doi:10.1016/j.qref.2012.11.002
      OpenUrlCrossRef
    12. ↵
      1. Eastin MS,
      2. Guinsler NM
      . Worried and wired: effects of health anxiety on information-seeking and health care utilization behaviors. Cyberpsychol Behav 2006;9:494–8. doi:10.1089/cpb.2006.9.494
      OpenUrlCrossRefPubMedWeb of Science
    13. ↵
      1. Lee CJ
      . Does the internet displace health professionals? J Health Commun 2008;13:450–64. doi:10.1080/10810730802198839
      OpenUrlCrossRefPubMedWeb of Science
    14. ↵
      1. Nicholson W,
      2. Gardner B,
      3. Grason HA, et al
      . The association between women's health information use and health care visits. Womens Health Issues 2005;15:240–8. doi:10.1016/j.whi.2005.05.004
      OpenUrlCrossRefPubMedWeb of Science
    15. ↵
      1. Suziedelyte A
      . How does searching for health information on the Internet affect individuals’ demand for health care services? Soc Sci Med 2012;75:1828–35. doi:10.1016/j.socscimed.2012.07.022
      OpenUrlCrossRefPubMed
    16. ↵
      1. van der Gugten AC,
      2. Uiterwaal CS,
      3. Verheij TJ, et al
      . E-health and consultation rates for respiratory illnesses in infants: a randomised clinical trial in primary care. Br J Gen Pract 2015;65:e61–8. doi:10.3399/bjgp15X683485
      OpenUrlAbstract/FREE Full Text
    17. ↵
      1. Wagner TH,
      2. Hibbard JH,
      3. Greenlick MR, et al
      . Does providing consumer health information affect self-reported medical utilization? Evidence from the Healthwise Communities Project. Med Care 2001;39:836–47.
      OpenUrlCrossRefPubMed
    18. ↵
      1. Fox S,
      2. Duggan M
      . Health Online 2013. Pew Research Center, 2013. http://www.pewinternet.org/files/old-media//Files/Reports/PIP_HealthOnline.pdf (accessed 3 Nov 2015).
    19. ↵
      1. Kontos E,
      2. Blake KD,
      3. Chou WY, et al
      . Predictors of eHealth usage: insights on the digital divide from the Health Information National Trends Survey 2012. J Med Internet Res 2014;16:e172. doi:10.2196/jmir.3117
      OpenUrlCrossRefPubMed
    20. ↵
      1. Schwartz KL,
      2. Roe T,
      3. Northrup J, et al
      . Family medicine patients’ use of the Internet for health information: a MetroNet study. J Am Board Fam Med 2006;19:39–45. doi:10.3122/jabfm.19.1.39
      OpenUrlAbstract/FREE Full Text
    21. ↵
      1. Wong C,
      2. Harrison C,
      3. Britt H, et al
      . Patient use of the internet for health information. Aust Fam Physician 2014;43:875–7.
      OpenUrlPubMed
    22. ↵
      1. Drenthen T,
      2. Beijaert RP,
      3. Jansen PW, et al
      . [What do you think of Thuisarts.nl? Experiences after 3 years of www.Thuisarts.nl]. Ned Tijdschr Geneeskd 2014;158:A8282.
      OpenUrl
    23. ↵
      Dutch College of General Practitioners. NHG guidelines. 2015. https://www.nhg.org/nhg-standaarden. (accessed 3 Nov 2015).
    24. ↵
      Effective Practice and Organisation of Care (EPOC). Interrupted time series (ITS) analyses. EPOC resources for review authors. Oslo: Norwegian Knowledge Centre for the Health Services, 2015. http://epoc.cochrane.org/epoc-specific-resources-review-authors. (accessed 3 Nov 2015).
    25. ↵
      1. Jandoc R,
      2. Burden AM,
      3. Mamdani M, et al
      . Interrupted time series analysis in drug utilization research is increasing: systematic review and recommendations. J Clin Epidemiol 2015;68:950–6. doi:10.1016/j.jclinepi.2014.12.018
      OpenUrlCrossRefPubMed
    26. ↵
      1. Sitrbu-Wagner I,
      2. Dorsman SA,
      3. Visscher S, et al
      . Landelijk Informatienetwerk Huisartsenzorg. Feiten en cijfers over huisartsenzorg in Nederland. Utrecht/Nijmegen: NIVEL/IG, 2009.
    27. ↵
      1. Lamberts H,
      2. Wood M
      . International classification of primary care. Oxford: Oxford University Press, 1987.
    28. ↵
      The Netherlands Institute for Social Research. Statusscores. 2015. https://www.scp.nl/Onderzoek/Lopend_onderzoek/A_Z_alle_lopende_onderzoeken/Statusscores (accessed 1 Aug 2015).
    29. ↵
      1. Kontopantelis E,
      2. Doran T,
      3. Springate DA, et al
      . Regression based quasi-experimental approach when randomisation is not an option: interrupted time series analysis. BMJ 2015;350:h2750. doi:10.1136/bmj.h2750
      OpenUrlFREE Full Text
    30. ↵
      1. Nelson BK
      . Statistical methodology: V. Time series analysis using autoregressive integrated moving average (ARIMA) models. Acad Emerg Med 1998;5:739–44.
      OpenUrlCrossRefPubMedWeb of Science
    31. ↵
      1. Nagle JP,
      2. McMahon K,
      3. Barbour M, et al
      . Evaluation of the use and usefulness of telephone consultations in one general practice. Br J Gen Pract 1992;42:190–3.
      OpenUrlAbstract/FREE Full Text
    32. ↵
      1. Bhandari A,
      2. Wagner T
      . Self-reported utilization of health care services: improving measurement and accuracy. Med Care Res Rev 2006;63:217–35. doi:10.1177/1077558705285298
      OpenUrlAbstract/FREE Full Text
    33. ↵
      1. Chishti T,
      2. Harris T,
      3. Conroy R, et al
      . How reliable are stroke patients’ reports of their numbers of general practice consultations over 12 months? Fam Pract 2013;30:119–22. doi:10.1093/fampra/cms042
      OpenUrlAbstract/FREE Full Text
    34. ↵
      1. Bolle S,
      2. van Weert JC,
      3. Daams JG, et al
      . Online health information tool effectiveness for older patients: a systematic review of the literature. J Health Commun 2015;20:1067–83. doi:10.1080/10810730.2015.1018637
      OpenUrl
    35. ↵
      1. Crabb RM,
      2. Rafie S,
      3. Weingardt KR
      . Health-related internet use in older primary care patients. Gerontology 2012;58:164–70. doi:10.1159/000329340
      OpenUrlPubMed
    36. ↵
      1. Van Esch TEM,
      2. Brabers AEM,
      3. Van Dijk C, et al
      . Inzicht in zorgmijden: aard, omvang, redenen en achtergrondkenmerken. NIVEL, 2015. http://www.nivel.nl/sites/default/files/bestanden/Inzicht-zorgmijden.pdf (accessed 3 Nov 2015).
    37. ↵
      1. Kouwenhoven-Pasmooij TA,
      2. Djikanovic B,
      3. Robroek SJ, et al
      . Design and baseline characteristics of the PerfectFit study: a multicenter cluster-randomized trial of a lifestyle intervention in employees with increased cardiovascular risk. BMC Public Health 2015;15:715. doi:10.1186/s12889-015-2059-9
      OpenUrl
    38. ↵
      1. Mouthaan J,
      2. Sijbrandij M,
      3. de Vries GJ, et al
      . Internet-based early intervention to prevent posttraumatic stress disorder in injury patients: randomized controlled trial. J Med Internet Res 2013;15:e165. doi:10.2196/jmir.2460
      OpenUrl
    39. ↵
      European Internet Stats. Internet Users in Europe 2015. 2015. http://www.internetworldstats.com/stats4.htm (accessed 7 Oct 2015).
    View Abstract

    Footnotes

    • Twitter Follow Tobias N. Bonten at @tbonten, Ton Drenthen at @Ton_Dr and Niels H Chavannes at @NHChavannes

    • Contributors WAS and TNB designed the study, analysed the data, interpreted the data and drafted the paper. MWMW and NHC designed the study, interpreted the data and revised the paper. TD and IJMS revised the paper. MMJM interpreted the data and revised the paper.

    • Funding The NHG and LUMC funded the analyses for this study. Funding was provided to the Netherlands Institute for Health Services to prepare and extract the data from their data set.

    • Competing interests TD and IJMS are employed by the Dutch College of General Practitioners, which financially supports the website thuisarts.nl.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement Technical appendix, statistical code, and data set available from the last author (NHC).

    Linked Articles

    • Miscellaneous
      Correction
      British Medical Journal Publishing Group
      BMJ Open 2017; 7 - Published Online First: 04 May 2017. doi: 10.1136/bmjopen-2016-013166corr1