Evaluation of the websites

We determined the quality of information to be low, as indicated by the low 75th percentile cut-offs for EQIP and DISCERN, at 21 and 43, respectively, or 58.3% and 53.8% of their respective full scores. Abundant COVID-19 content is being produced, as evident in figure 1, with the majority being excluded due to its nature rather than interterm duplications. Most frequently excluded websites were either caseload updates or general news articles lacking information on prevention and treatment.

Websites generally scored poorly, with an EQIP mean and median of 18 (IQR 15–20). Despite fewer marks allocated to Structure, websites generally outperform the Content domain (7.30 vs 6.97), suggesting information quality is less adequate than usability. The indicators assessing referencing quality, such as JAMA benchmark and DISCERN Reliability, scored similarly at 2.69 (67.3% of maximum) and 26.93 (67.3% of maximum), respectively, but scored lower in the EQIP Identification domain 3.50 (58.3% of maximum), likely due to more items (ie, whether patients were involved in the material’s production). Generally, high-scoring websites performed better in the Content domain, with OR between 3 and 7. The OR was significantly higher for item 4 (defining the purpose of interventions) (OR 27.78; 95% CI 4.695 to 1000; p<0.001), suggesting that high-scoring websites provided greater reasoning behind preventative and treatment measures. While high-scoring websites scored significantly higher across 23 of the 36 EQIP items, quantitative benefits (item 8) is a notable exception where both cohorts performed poorly (4.1% vs 2.4%; OR 1.695; 95% CI 0.267 to 8.197; p=0.436) and was significantly lower than the scores in available literature.18 23 24 This likely reflects the general lack of COVID-19 knowledge compared with previously explored diseases and treatments. High-scoring websites similarly performed better in Identification (OR ranged between 1.312 and 5.376), with the inclusion of bibliography (item 23) differing most (41.89% vs 11.74%; OR 5.376; 95% CI 1.727 to 7.407; p<0.001) as the majority of websites lacked bibliographies, potentially due to subpar production quality in a high turnover topic. Structure revealed both high-scoring and low-scoring websites to provide clear information (item 30; 98.65% vs 89.88%; OR 8.197; 95% CI 1.294 to 333.3; p=0.013). High-scoring websites provided poor benefit−risk balancing (item 31; 39.19% vs 10.93%; OR 5.208; 95% CI 2.703 to 10.101; p<0.001), as did most other websites assessed (17.5% overall), comparable to existing studies ranging from 11% to 44%. As shown in figure 5, EQIP scores were relatively homogeneous, ranging from 6 to 29, with the majority between 14 and 22. Overall performance agreed with available literature that online health information is inadequate, as median EQIP scores ranged between 15 and 19 and IQR ranged 12–20 and 16–22.18 20 22 23 45 Furthermore, scores for describing intervention sequence (item 6) and quantitative risks (item 10) were much lower (15.9% and 0.9%, respectively) when compared against the study on gallstone disease (27% and 21%, respectively) or liver transplantation (66% and 53%, respectively). This likely reflects the prioritisation of discussing prevention and treatment method efficacy over treatment sequences and risks. Item 23 also scored poorly, both in itself and against literature: only 18.7% of websites provided a short bibliography whereas studies ranged from 19% in liver transplantation to 47% in orthognathic surgery. Interestingly, COVID-19 websites scored well in dating (item 19; 87.2%) compared literature, likely reflecting the demanding and time-sensitive nature to understand how the COVID-19 pandemic affects individuals. Unsurprisingly, encyclopaedias (n=5) scored the highest in content (10.80), identification (4.80) domains and overall EQIP (24.20), holding the highest proportion of websites discussing both prevention and treatment (n=4; 80%). Encyclopaedias’ high scores are attained through their endeavours to provide neutral summaries which meet the majority of the EQIP criterion, as well as the items in both JAMA benchark and DISCERN tool.46 Nonetheless, while overall EQIP scores do not differ substantially from the existing literature, it is important to recognise that the quality of information is significantly influenced by the rapid turnover of information, a phenomenon not present in any of the previous studies.

• Download figure
• Open in new tab
• Download powerpoint

Figure 5

High-score distribution of the final dataset compared against low-scoring websites for EQIP Tool, JAMA benchmark and DISCERN Tool. EQIP, Ensuring Quality Information for Patients; JAMA, Journal of American Medical Association.

JAMA benchmark scores, unlike EQIP and DISCERN, were more consistent as only four points are available. Hence, high-scoring websites often fulfilled all four JAMA criteria, whereas the majority of other entries scored two or three. Low-scoring websites scored significantly lower (p<0.001) in Authorship and Attribution, with only 44.13% and 17.81% fulfilling the criteria, reflecting congruence with the findings from EQIP regarding bibliography inclusion. Contrarily, website ownership and funding assessment, under Disclosure (86.23%), and assessments of publishing and updating dates, under Currency (81.78%), varied to a lesser degree, although still scoring significantly lower (p<0.001). Comparative to the related EQIP sections, Disclosure and Currency likely scored better by assessing attributes independent to the content itself. Professional societies (n=4; 3.50), closely followed by encyclopaedias (n=5; 3.40), scored the highest overall, whereas the mean JAMA score was only 2.69, with 4 websites scoring 0 points. This is likely attributable to the lack of additional assessment criteria by the JAMA benchmark as it was developed during the infancy of web-based resources. Regardless, the tool is still effective at identifying high-quality content as high-scoring websites scored significantly better across each item (p<0.001).

DISCERN’s mean score of 38.00 across 16 items averaged 2.38 out of 5 per item. The DISCERN handbook details a rating of one when the information does not provide the appropriate information, three where it addresses it partially and five for a complete and adequate inclusion. Based on these guidelines, the majority of websites meet the listed criteria to a minimal extent. Similar to EQIP, DISCERN scores vary significantly, ranging from 19 to 75, with the majority distributed between 25 and 47. Of the 16 items, 12 presented statistically significant differences between high-scoring and low-scoring websites. Interestingly, the Reliability section provided the greatest difference in scoring, seen between item 4, clear display of information sources (mean 4.16 vs 2.55; p=0.019), and item 5, a clear indication of where sources were used (mean 3.54 vs 2.28; p=0.007). This corroborates with results from EQIP and JAMA assessments in highlighting the inadequacies of informative material production.

Quality of treatment information section scores varied less, although all with statistically significant differences. The greatest differences were in item 9, describing how each treatment works (1.76 vs 1.43; p=0.005) and item 10, detailing treatment benefits (1.78 vs 1.43; p<0.001). Despite accommodating the treatment uncertainty during the assessment, almost all items evaluating treatment scored poorly, including high-scoring websites, and may be reflective of its lower journalistic priority or demand as the majority of websites were tailored towards discussing prevention. As a whole, DISCERN scores specifically addressing treatment options were poor, scoring 45.07 (SD 11.68) for websites that only discuss treatment as well as 42.84 (SD 10.93) for websites that discuss both prevention and treatment. These scores are also lower than those of other DISCERN studies, which have mean scores between 45.8 and 56.1, with SD between 8.76 and 13.6.47–49 While the majority of DISCERN scores were low, websites scored similarly in the Reliability section. The mean scores for items 1, 2 and 3 were the highest of all indicators, averaging 4.37, 4.29 and 4.33, whereas the items in treatment section scored between 1 and 2, with a maximum mean score of 1.66 in Item 14. The treatment section of DISCERN shows much more variation, with prevention alone (8.60; n=202) scoring lowest, treatment alone (18.09; n=46) scoring highest and websites discussing both (13.47; n=73) in between. This suggests that many websites do not include treatment information and, of those that do, websites tend to avoid discussion rather than provide the limited information available.

In short, all three tools used are validated based on international recommendations and provide a comprehensive assessment of online information: EQIP delivers an all-rounded evaluation of health information, DISCERN excels at scrutinising treatment and JAMA benchmark assesses all of a website’s content as a whole. Additionally, as design alone is known to improve the perception of information credibility,50 the higher emphasis on quality of content in EQIP and DISCERN will likely prevent well-presented and user-friendly websites with poor content from attaining high scores over poorly presented websites with good content. Altogether the tools suggest that the majority of COVID-19 websites are generally of poor quality and that quality fluctuates highly. This may subsequently impair the ability of the public to filter out websites of low validity and reliability and, hence, increase their risk of unsafe health behaviours during the pandemic. Through the use of these tools, the same standards can be held across different sources of information to produce higher quality educational material to improve both education and awareness.

Reasons for poor quality information

Most sources scored poorly due to several possible reasons. First, as COVID-19 was only identified in early December 2019,51 and recognised as a pandemic in March 2020,52 the general lack of information to produce patient information prevents accurate and reliable conclusions to be drawn. While research efforts have since gained traction, production of research, reviewing and publication is a lengthy process comparative to the rapid spread of COVID-19. Journals have since implemented fast tracking of COVID-19 research.53–55 Similarly, Governments and Health Departments have also cooperated to support and fast-track COVID-19 studies.56 The combined efforts have facilitated the publication of over 5000 COVID-19 articles in the WHO database alone.57 It is also important to note the role of preprint servers during the pandemic as they are accessible to the public. While improving accessibility helps facilitate peer reviewing; non-peer-reviewed articles can potentially be used, or cherry picked, by non-professionals that can adversely affect public understanding.58–61

Health literacy also plays a crucial role in how COVID-19 information influences health behaviour. The associations between health literacy and health behaviour are well documented: low literacy is common among older adults engaging in poor health behaviour (p<0.005) ranging from lifestyles, such as physical activity, dietary habits and obesity, to social factors, such as loneliness and social isolation.62–64 The effects of primary preventative measures, such as social distancing, self-isolation and other hygiene recommendations, may become impaired by the abundance of poorly written and incorrect information online. In particular, the elderly, most vulnerable population, suffer from even higher risks due to their lower health literacy.65 Although health literacy, and by extension, health behaviours, can be improved through education,66 effective dissemination of credible information is critical during the pandemic. The public, and journalists, need to exercise caution when accessing research and preprints during this period as inadequate health literacy may lead to counterproductive effects. Similarly, mass production of online information greatly increases the difficulty in distinguishing reliable information from the sea of misinformation, and hence, a greater emphasis must be placed on authors and journalists to deliver unbiased, credible and accurate information to the public.67 68

On publication, articles are reviewed and summarised by journalists who bridge the knowledge gap between scientists and the wider public. This allows efficient dissemination of critical research to those who lack the scientific background to critically appraise and evaluate research. Notably, while 52% (n=169) of websites were ‘News Services’, only 7.1% (12) were considered high scoring, reflecting that very few provide a comprehensive account of COVID-19 information, possibly explained by the difficulty in matching the pace of COVID-19 research and technical inadequacies in delivering accurate and concise scientific information. As health information-seeking behaviour of the public will likely be based on news services,69 the highly variable and generally poor content is problematic.

Previous studies have identified that many health journalists lack the training required to accurately disseminate health news, leading to potentially harmful health effects.70 71 The low scores across all tools indicate overall inadequacy of both reliability and accuracy. A survey of medical journalists across 37 countries highlighted the 3 most common barriers against quality content: lack of time, space and knowledge.72 While journalism stresses the ability to summarise content quickly and concisely, experienced journalists often report the lack of knowledge as a barrier. The lack of expert knowledge in a rapidly progressing scientific field can impact the quality of conveyed information significantly. Furthermore, journalists reported difficulty in finding experts to explain the jargon, further impacting quality. This presents a large barrier towards disseminating quality COVID-19 information as the redirected efforts of many countries and institutions’ scientists into research reduces availability to assist with medical journalism.73 74 Expertise in a relevant academic background likely helps improve the content, as indicated by the highest-scoring entry, whose author holds a PhD in molecular genetics which scored 22 in EQIP, 4 in JAMA and 74 in DISCERN.75 In short, our findings highlight the importance of addressing health outcomes through health literacy of both the public and health journalists.

Our analysis identified the majority of websites as sources from Governments or health departments across various states of the USA, many of which have based information and advice from the Centers for Disease Control and Prevention (CDC). As both the nature and purpose of these sources vary, the visiting population and demographics would also vary due to differences in information-seeking behaviours. A recent study has identified a deficit of awareness and trust in information originating from the Government such as those of the CDC, revealing that while up to 83.6% of American adults are aware of the CDC, only 64.6% trust this source.76 American adolescents were, however, less aware of the CDC (55.8%), but were more trusting of their information (72.2%). As previously established, trust in a Government is predictive of health outcomes as it affects behaviours such as service usage and vaccination rates.77–79 Similarly, as access and usage of online health information vary between different demographics, it is paramount that we create and provide targeted and effective educational material for public use.80 81

A comparison between the global Government/Health Department and News Services websites revealed significant differences between EQIP Content, Total EQIP, Total and DISCERN Treatment. The EQIP Content scores reflect differences in where information is obtained, with Government/Health Departments using primary research, whereas journalists tend to use secondary research. However, News Services scored higher in DISCERN Treatment, potentially due to the Government’s reluctance to prematurely disclose treatment information at early stages of discovery, whereas journalists may freely report results of all potential studies. Interestingly, the majority of US Government websites had some form of copy-pasted information from the CDC, likely with the aim of maintaining consistency and centralisation information sources. In contrast, the majority of other sources rewrite information based on a variety of sources. American sources display a similar pattern of variation but the EQIP Identification scores of Government/Health Departments show statistically significant differences and scores lower than News Services (mean 3.34 vs 3.78; p=0.011). The verbatim use of CDC information on these Government websites may have neglected the importance of clarity, transparency and critical appraisal of literature and focused on disseminating information instead.

Is the current information online adequate?

Google Trends identified the search popularity of treatment to have increased rapidly, outgrowing prevention searches since early March. This suggests that initial demands for preventative information have been sufficiently met and interests now shift towards treatment. The 275 websites addressing prevention scored below the mean EQIP (17.12 vs 17.78; p<0.001), JAMA (2.53 vs 2.69; p=0.001) and DISCERN scores (34.64 vs 38.00; p<0.001), showing that quality of preventative information remains subpar across all indicators. Similarly, treatment websites demonstrated their efficacy in sourcing, over discussing information, as they excelled in Identification (3.87 vs 3.50; p<0.001) but scored poorly and below the mean in EQIP (16.89 vs 17.78; p<0.001). This is also reflected by its above-average JAMA (3.13 vs 2.69; p=0.001) and DISCERN (45.07 vs 38.00; p<0.001) scores. Contrarily, the 73 websites that discussed both prevention and treatment of COVID-19 consistently scored higher than the mean across all indicators: EQIP (20.15 vs 17.78; p<0.001), JAMA (2.85 vs 2.69; p=0.001) and DISCERN (42.84 vs 38.00; p<0.001). These websites explored multiple aspects of the virus and likely use a variety of sources, thus producing higher quality articles through a better understanding of the topic. To combat increasingly dangerous COVID-19 myths, such as injection of disinfectants as treatment, continued maintenance and improvement to online available resources is paramount.82 83

Limitations

Although the ‘Google’ search engine is the most commonly used, it is not wholly representative as searches are often affected by the location of the requesting server and previous internet usage. While querying from every country is not practical, the server used in the study was not used to conduct any other COVID-19-related searches, thus impact should be minimised. A further search was conducted on 10 July 2020 on the top 10 scoring EQIP and DISCERN websites through VPN servers in Texas, London, Toronto and Sydney. Only three of the websites were found within the first 10 pages of results using the same search terms on the original server, with London and Sydney each returning two of the results, Toronto returning none (online supplemental table 4) and a remaining URL redirecting to another page. The search also confirms differences in results between the location of the search; however, all three results from the Texas server were covered by London and Sydney. As websites can be updated or removed any time, our results are representative only at the time of the search, demonstrated by the distinct lack of the original websites in our second search in July. Similarly, search terms obtained using ‘Google Adwords Keyword Planner’ and ‘Google Trends’ may not truly be indicative of search patterns of the wider public. Hence, 12 different variations of ‘COVID-19’ names and phrases potentially used by patients were used as search terms. While social media is increasingly used to share health information, our study focused on search engines because previous studies have identified a lack of trust in social media information and which acts as a barrier against public engagement.84 Similarly, while video-based information constitutes a valuable source of information, to the best of our knowledge, no tools have been validated in assessing video-based information, particularly for COVID-19. Forced inclusion of video content would likely yield inaccurately low scores across current tools as videos typically do not include as much written information. Another limitation is the exclusion of non-English language websites, especially reducing the representation of publicly available information given the international nature of the pandemic. A minority of URLs, particularly of News Services, regularly update their content or redirect visitors, potentially affecting the second round of evaluation and subsequent statistical analysis. Additionally, modified EQIP Tool, JAMA benchmark and DISCERN Tool were not designed to specifically assess the highly variable information produced during pandemics. However, the EQIP Tool was designed to assess any type of patient information and demonstrated high inter-rater reliability.17 Similarly, JAMA benchmark was designed to evaluate website reliability alone and DISCERN examines both reliability and content accuracy, thus, the combinational use of tools enhances the accuracy and objective assessment of these websites.