Article Text
PDF
Abstract
Objective To assess the effects of using health social media on web activity.
Design Individually randomised controlled parallel group superiority trial.
Setting Twitter and Weibo.
Participants 170 Cochrane Schizophrenia Group full reviews with an abstract and plain language summary web page.
Interventions Three randomly ordered slightly different 140 character or less messages, each containing a short URL to the freely accessible summary page sent on specific times on one single day. This was compared with no messaging.
Outcome The primary outcome was web page visits at 1 week. Secondary outcomes were other metrics of web activity at 1 week.
Results 85 reviews were randomised to each of the intervention and control arms. Google Analytics allowed 100% follow-up within 1 week of completion. Intervention and control reviews received a total of 1162 and 449 visits, respectively (IRR 2.7, 95% CI 2.2 to 3.3). Fewer intervention reviews had single page only visits (16% vs 31%, OR 0.41, 0.19 to 0.88) and users spent more time viewing intervention reviews (geometric mean 76 vs 31 s, ratio 2.5, 1.3 to 4.6). Other secondary metrics of web activity all showed strong evidence in favour of the intervention.
Conclusions Tweeting in this limited area of healthcare increases ‘product placement’ of evidence with the potential for that to influence care.
Trial registration number ISRCTN84658943.
- randomised controlled trial
- social media
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
This is the first randomised controlled trial that we are aware of evaluating the impact of Tweeting health-related web links on access to the target web page and/or related web pages.
This study quantifies the effects of Tweeting evidence and generates many questions for future research.
We used free-to-use software with limited functionality—more sophisticated software may highlight more effects.
We Tweeted links to large academic reviews focusing on one small area of healthcare to a relatively small ‘following’. Different techniques of Tweeting, other areas of healthcare, and a broad set of followers could result in more impact.
Introduction
Cochrane (previously known as Cochrane Collaboration) has more than 37 000 contributors from around 130 countries. They work together to produce credible, accessible health information that is free from commercial sponsorship and other conflicts of interest. The Cochrane Schizophrenia Group (which is part of Cochrane) produces and maintains systematic reviews and meta-analyses of randomised trials that evaluate the effects of interventions for schizophrenia and related psychotic illnesses. Full text versions of this work can be accessed in some high income and most low income countries.1 Each review also has an abstract and a plain language summary (PLS) to make the information in the review more accessible to people without specialised knowledge.2 Both the abstract and PLS are freely available from the Cochrane website.3
Twitter is a popular free-to-use social media platform which allows users to send a 140-character message (a ‘Tweet’) to a group who has chosen to receive (or ‘follow’) these short messages from the sender. Currently there are 288 million monthly active users sending out 500 million Tweets a day.4 Most (77%) accounts are outside the USA and 80% of Tweets are sent from mobile devices. The use of Twitter in healthcare has increased, encompassing, for example, issues relating to public health surveillance, tracking the disease activity of the H1N1 pandemic and isolating the source of a cholera outbreak in Haiti.5–7 Health-related Twitter messages now comprise an estimated 0.0001% of all Twitter traffic.8 Most Cochrane Review Groups have recently developed this method of dissemination.9 However, an investment of effort is required to undertake this activity, and the potential benefits for review groups with limited numbers of reviews and followers are unclear.
Twitter and some other platforms are not available in China. However, 91% of China's population use social media compared to 67% of the population of the USA.10 The Cochrane Schizophrenia Group has been working with a Chinese company, Systematic Review Solutions Ltd,11 to disseminate parallel messages on Weibo, a Twitter-like system, to their followers across China. Weibo is in the top 10 social media sites used in China with over 600 million registered users, of which about 140 million use it regularly (as of March 2014).12
Given the increasing use of social media in healthcare, and the investment involved in doing this, we evaluated the impact of Tweeting précis of Cochrane Schizophrenia Group systematic reviews.
Methods
Study design
This was a prospective two-arm, parallel, open randomised controlled trial with a 1:1 allocation ratio. The protocol with full details of the study design has been published elsewhere.13
Participants
Participants included in this study were published full text Cochrane Schizophrenia Group systematic reviews in The Cochrane Library with PLS which are available for free at http://www.summaries.cochrane.org. We excluded published protocols for Cochrane Schizophrenia Group reviews, any Cochrane Schizophrenia Group review not relevant to schizophrenia, unpublished and withdrawn Cochrane Schizophrenia Group reviews.
This study was conducted entirely on the internet. The participant PLSs were all available on the Cochrane Library website, the intervention was delivered via two social media platforms (Twitter and Weibo) and outcome data were collected using Google Analytics.
This study did not involve any living participants and used information that is available in the public domain. Participants are systematic reviews rather than people and routine web activity data were extracted and recorded through Google Analytics. No ethical approval was required.14 ,15
Randomisation
All participating Cochrane Schizophrenia Group systematic reviews were identified prior to randomisation. We defined mean baseline access activity as high (≥19 unique views per week, n=14), medium (4.3–18.99 unique views per week, n=72) or low (<4.3 views unique per week, n=84) based on the 23-week period 21 September 2013 to 28 February 2014 (defined in the ‘Statistical analysis’ section). Reviews were given a unique code, which along with the baseline activity stratum, was supplied to one of the authors (AAM) who performed the randomisation. Stratifying by baseline access activity, and using computer-generated random number sequences, reviews were first allocated to intervention and control arms, then into pairs of reviews that would have the same reference period for outcome data collection, then to day of the week and week number (1–29) that Tweeting would take place for reviews in the intervention arm. Days for Tweeting were Tuesday, Wednesday or Friday as these are considered to have the heaviest traffic.16 Finally, the sequence of the three Tweets for each review (the Tweet package) in the intervention arm was also randomised. Tweets had been prepared for all reviews. Once allocation codes were supplied to AYMB, independent of other investigators, she scheduled Tweets for the intervention group within Hootsuite.17
Procedures
Reviews in the intervention group were Tweeted in English, by the Cochrane Schizophrenia Group, three times on the same day at 10:30, 13:00 and 15:00 GMT as guided by the SocialBro web tool,18 and 8 h later in China, in Mandarin, via Weibo. There is some evidence that multiple postings, 3–4 times a day, of the same or similar Tweet can be useful for an international following.16 Each of the three Tweets had different accompanying text:
The review title as it appears in summaries.cochrane.org, and a shortened URL to the summaries web page
A pertinent extract from the results or discussion sections of the abstract, and a shortened URL to the summaries web page
An intriguing question or pithy statement directly related to the evidence presented in the abstract, and a shortened URL to the summaries web page (see table 1).
Example of the three tweets relating to the same review
We did not compare the specific content of the different types of accompanying text, which were formulated in order to appeal to various followers of the Cochrane Schizophrenia Group Twitter page and searchers. We compared the combined impact of the package of Tweets versus no Tweets. To assist the logistics of sending out Tweets, we used Hootsuite, a social media management system. This free package allows formulation and scheduling of Tweets and is now available in China and integrates with Weibo (http://blog.hootsuite.com/chinese-localization-weibo/).
Reviews in the control group were not Tweeted by the Cochrane Schizophrenia Group.
Outcomes
The primary outcome was the total number of visits to the relevant Cochrane summary page in the 7 days following the intervention (including the day of Tweeting) as reported on Google Analytics. The secondary outcomes are listed in box 1.
List of secondary outcomes
1. Unique page views
2. Entrances
3. 100% of visits were single page
4. Direct click visits
5. Twitter referrals
6. Any outbound click events
7. Time on page in seconds
For reviews in the control arm, the 7-day follow-up period was the same as for the intervention review with which it had been randomly paired. For the primary outcome, repeated views of the same page during a single user session are counted in the total; unique page views are a secondary outcome. The standard free account in Google Analytics produces various reports in real-time and provides data about whom, when and where someone has visited a site, as well as how they ‘arrived’ at that site.19 ,20 Table 2 outlines the glossary of Google Analytics terms.
Glossary of Google Analytics terms
This includes all traffic to the Cochrane summary page and traffic directly from Twitter. The average half-life of a Tweet (with a web link), defined as ‘
the amount of time at which this link will receive half of the clicks it will ever receive after it's reached its peak’, has been estimated as 2.8 h.21 However, to capture any possible cascade effect of Tweeting, we extended the monitoring period to 7 days.22 Secondary outcomes provide other measures of incoming activity and exiting behaviour. In addition, we recorded country distribution of users clicking on the target sites.
One week after the final Tweets were sent, four different data reports per review were downloaded from Google Analytics for each Cochrane summary page, for the relevant 7-day period. These 680 files were uploaded into an MS Access database and then merged for analysis.
Statistical analysis
The sample size for this study is fixed by the number of published Cochrane reviews under the control of the Cochrane Schizophrenia Group (n=170). Therefore, we estimated the magnitude of the detectable between-group difference in the primary outcome. Google Analytics data for all 170 reviews during the period 21 September 2013 to 28 February 2014 suggested that log-transformed mean number of views per week is normally distributed, with a mean of 1.5 and SD of 0.9. With 5% two-sided α and a sample size of 85 per arm, between-group differences in the range 0.43–0.5 SDs, equivalent to a ratio of geometric means of 1.47 to 1.57, are detectable with 80–90% power.
The primary outcome is a count of the number of times that each review is accessed during its 7-day follow-up period. Due to over-dispersion in this, and in other outcomes that are counts, we used multivariable negative binomial regression modelling to estimate incidence rate ratios and 95% CIs, adjusted for baseline activity stratum and day of the week on which the 7-day data collection period commenced. We also investigated the use of mixed effects models to take account of pairing of intervention and control reviews. This made no material difference and so results from the simpler models are presented. Outcomes with excess zero counts were analysed using zero-inflated negative binomial regression models. Binary variables were analysed using logistic regression. Continuous variables were analysed using linear regression, with log-transformation of outcomes and presentation as the ratio of geometric means if appropriate.
We conducted further secondary analyses of the primary outcome. By including appropriate interaction terms in the primary regression model, we investigated in subgroup analyses whether any effect of the intervention differed according to baseline activity, or by day of the week that Tweets were sent. The latter was not pre-specified before analysis of the data and therefore should be regarded as exploratory. Finally, we estimated the effects of the intervention separately for visits to the review summaries that originated from China and from outside China.
All between-group comparisons analysed reviews as randomised. As the intervention was delivered as intended, and there are no missing outcome data, sensitivity analyses were unnecessary.
Results
At the time of the trial, there were 849 followers on Twitter and 5995 followers on Weibo. Following randomisation of 170 reviews, intervention group Tweets were posted between 1 July 2014 and 13 January 2015 between which participants (systematic reviews) were recruited as outlined in the trial profile (figure 1). Reviews allocated to groups were evenly distributed within each activity stratum (high, medium, low) and this was reflected in the broad categorisation of review content and there were no statistically significant differences between the two groups (table 3).
Description of review by randomised group
Trial profile.
Results are presented in table 4.
Results
For the primary outcome, there were 1162 and 449 visits in total to the 85 review summaries in the intervention and control arms, respectively, and strong statistical evidence of a nearly threefold increase in the intervention arm. There was further evidence of an effect of the intervention on all of the secondary outcomes.
There was no evidence that the intervention was differentially effective at increasing visits to reviews classified according to baseline activity as low (IRR 3.09, 95% CI 2.30 to 4.17), medium (IRR 2.65, 1.94 to 3.61) or high (IRR 1.85, 0.88 to 3.89) (overall p value for interaction=0.31). Nor was there any evidence of a differential effect according to day of the week that Tweets were sent (p value for interaction=0.68).
The data did not suggest that the intervention was any more or less effective in China compared with the rest of the world. The incidence rate ratio for the primary outcome for visits that originated from China was 2.27 (95% CI 0.35 to 14.77), and 2.15 (1.73 to 2.68) for from outside China.
Discussion
Much investment is put into the dissemination of healthcare evidence. Many choices are available to individuals, institutions and industry, and it is difficult to estimate how much of this investment is misplaced. We were able to show that a simple approach on a commonly used set of systems (i) was possible to evaluate and (ii) seemed to have some effect.
This is the first randomised controlled trial that we are aware of evaluating the impact of Tweeting health-related web links on access to the target web page and/or related web pages. This study quantifies the effects of Tweeting evidence and generates many questions for future research. We used free-to-use software with limited functionality—more sophisticated software may highlight more effects. We Tweeted links to large academic reviews focusing on one small area of healthcare to a relatively small ‘following’. Different techniques of Tweeting, other areas of healthcare, and a broad set of followers could result in a greater impact.
There are many more questions on the effects of types of social media targeting and how best to undertake this in order to encourage the implementation of best evidence. Much more sophisticated outcome analytics are available beyond those we used which would add to understanding of how best to target for substantive encouragement of behaviour for best impact.
All hit rates, whatever way they were measured, were largely modest. The degree of effect was consistently substantial but is relative to the modest standard activity. Without our social media activity many of the Cochrane reviews have little or no activity within the target week. The social media activity changes this and there are some indications that the intervention also changed how a person interacts with the site. The user would stay longer, ‘bounce off’ less frequently and exit to another site more frequently. We did not specially investigate if people tended to exit to the full Cochrane Review as access to the full text varies across countries. This study underlines the importance of this summaries web interface for the Cochrane Collaboration. The activities, however, increased by the social media intervention, would seem to indicate improved interest.
Conclusions
Use of social media in an area of healthcare that is not particularly high profile, for the dissemination of evidence that is highly specialised through a small following, nevertheless, seemed to have a genuine effect by which indicators of interest were increased. There were more visits to the target site and the time spent on the page was longer. We cannot tell if that interest translated into better care more in line with best evidence, but ‘product placement’ of good evidence is a significant step in the right direction.
Much effort is expended in disseminating health-related messages across social media with unclear effect. The Cochrane Schizophrenia Group, based in the University of Nottingham, has randomised their reviews to be also disseminated by social media compared with no additional ‘Tweeting’/(‘Weibo-ing’). There is now evidence from a trial that low-grade social media dissemination is powerful for the ‘product placement’ of health information.
Acknowledgments
We are really grateful to Constanze Knahl, student on the EU Erasmus Student Exchange from Neu-Ulm University of Applied Sciences, Ulm, Germany, who constructed the MS Access upload tool without which this study would have been much more difficult.
Footnotes
Twitter Follow Mahesh Jayaram at @maheshjayaram
Contributors CEA wrote the first draft and all authors revised drafts. MJ helped extract data from Google Analytics, clarified queries and drafted the final manuscript. AYMB helped allocate by categorising the reviews and setting up Hootsuite. AAM conducted the statistical analyses. SZ is co-ordinating the China arm of the trial and contributed to revising the manuscript. SS contributed to drafting the protocol and revisions. All authors are accountable for the accuracy and integrity of the work. All authors read and approved the final manuscript.
Funding The trial was sponsored by the Nottinghamshire Mental Health Trust.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement The full dataset and statistical code are available from the corresponding author. We will upload the data on to the University of Nottingham server from where it will be made freely available.