Article Text

Download PDFPDF

Original research
Using variation between countries to estimate demand for Cochrane reviews when access is free: a cost–benefit analysis
  1. Perke Jacobs,
  2. Gerd Gigerenzer
  1. Harding Center for Risk Literacy, Max Planck Institute for Human Development, Berlin, Germany
  1. Correspondence to Perke Jacobs; jacobs{at}mpib-berlin.mpg.de

Abstract

Objectives Cochrane reviews are currently of limited use as many healthcare professionals and patients have no access to them. Most member states of the Organisation for Economic Co-operation and Development (OECD) choose not to pay for nationwide access to the reviews, possibly uncertain whether there is enough demand to warrant the costs of a national subscription. This study estimates the demand for review downloads and summary views under free access across all OECD countries.

Design The study employs a retrospective design in analysing observational data of web traffic to Cochrane websites in 2014. Specifically, we model for each country downloads of Cochrane reviews and views of online summaries as a function of free access status and alternative sources of variation across countries. The model is then used to estimate demand if a country with restricted access were to purchase free access. We use these estimates to perform a cost-benefit analysis.

Results For one group of eight OECD countries, the additional downloads under free access are estimated to cost between US$4 and more than US$20 each. Three countries are expected to save money under free access, as existing institutional subscriptions would no longer be needed. For the largest group of 17 member states, free access is estimated to cost US$0.05–US$2 per additional review download. On average, the increase in review downloads does not appear to be associated with a decrease in the number of summary views. Instead, translations of plain-language summaries into national languages can serve as an additional strategy for dissemination.

Conclusions We estimate that free access would cost less than US$2 per additional download for 20 of the 28 OECD countries without national subscriptions, including Canada, Germany and Israel. These countries may be encouraged by our findings to provide free access to their citizens.

  • public health
  • health economics
  • medical education & training

Data availability statement

The data used for the analysis are proprietary and were kindly provided by the Cochrane Collaboration and Wiley. Therefore, we are unable to share them publicly.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Strengths and limitations of this study

  • Direct use of observational data on worldwide downloads and views of Cochrane reviews and summaries.

  • Model evaluation based on out-of-sample predictive accuracy rather than in-sample fit statistics.

  • Limitation is the imbalance of data resulting in large confidence intervals and the lack of time-series data from countries changing their subscription status.

Medical research accounts for a substantial proportion of research and development (R&D) expenditures. In the USA, total spending on medical and health R&D increased between 2013 and 2016 to US$172 billion, led by industry with 67 per cent and the federal government with 22 per cent.1 Worldwide, biomedical publications are increasing year by year; for instance, about one million articles are added to this literature annually.2 Faced with this large volume of articles, no healthcare worker is able to stay fully informed about recent research. The problem of quantity is amplified by one of quality; many of the clinical trials published are unreliable or of uncertain reliability and most healthcare professionals, including physicians and nurses, do not have the time and/or training to evaluate the quality of a research article.3 Additionally, direct-to-consumer ads, websites and television shows compete for the attention of healthcare professionals and patients, disseminating a mix of evidence and unwarranted claims based on commercial interests or personal opinion.4 In the USA, an estimated 20–50 per cent of healthcare service use is inappropriate, wasteful or harmful for patients.3

To address these issues, over 10 000 medical researchers have built an international network, named Cochrane after the British epidemiologist Archie Cochrane, to assist healthcare professionals and patients in making well-informed decisions about healthcare interventions. This network produces systematic reviews of the available evidence on the benefits and harms of medical interventions and tests, such as measles, mumps and rubella vaccination, check-ups, prostate cancer screening and statins. Since 1992, these Cochrane reviews have been written by some 30 000 medical researchers and are generally recognised as the gold standard of medical evidence.5 6 The reviews are intended to be regularly updated as new findings become available and provide three important services for healthcare professionals.7 First, they offer an overall assessment of the available evidence by evaluating individual studies according to the quality of their evidence and statistically integrating their results, which often vary due to their small sample sizes. Second, in contrast to a self-survey of the literature, systematic reviews allow professionals to absorb the relevant information about the benefits and harms of specific treatments under the typical conditions of time pressure. Finally, Cochrane reviews offer plain-language summaries and summary-of-findings tables that highlight key findings and can be easily understood by persons without statistical training, which makes them suitable for both professionals and lay people alike. For these reasons, many professionals consult the Cochrane reviews regarding interventions. Yet here is where the problem arises.

Whereas plain-language summaries are openly available online, access to the full-text reviews is often restricted, despite their containing large amounts of relevant information for patients and healthcare professionals. Institutions in many low-income and middle-income countries are granted free or inexpensive access through the WHO’s HINARI Access to Research for Health Programme (see also www.who.org/hinari), but healthcare professionals and patients outside of an institutional context are excluded. Most countries in North America and Europe (including the USA and Germany), by contrast, are not eligible and fall into one of two groups: those with and those without a national subscription. The latter group far exceeds the former, with only eight countries subscribing nationally in 2014, six of which are members of the Organisation for Economic Co-operation and Development (OECD, see box 1). Specifically, Australia, Denmark, Ireland, Norway, New Zealand and Great Britain offered free access nationwide, as did Egypt and India, which are not OECD member states. In addition, one US state, Wyoming, and three Canadian provinces, New Brunswick, Nova Scotia and Saskatchewan, had statewide subscriptions in 2014. Given their small shares of the country’s total population, we treated the USA and Canada as having no subscription. Whereas a national subscription grants all domestic internet users free access to Cochrane Reviews, users in countries without a national subscription need to pay for alternative access options. These prices are shown in box 2.

Box 1

Organisation for Economic Co-operation and Development (OECD) member states

The 34 OECD member states are Australia (AUS), Austria (AUT), Belgium (BEL), Canada (CAN), The Czech Republic (CZE), Denmark (DNK), Estonia (EST), Finland (FIN), France (FRA), Germany (DEU), Greece (GRC), Hungary (HUN), Iceland (ISL), Ireland (IRL), Israel (ISR), Italy (ITA), Japan (JPN), Republic of Korea (KOR), Luxembourg (LUX), Mexico (MEX), Netherlands (NDL), New Zealand (NZL), Norway (NOR), Poland (POL), Portugal (POR), Slovak Republic (SVK), Slovenia (SVN), Spain (ESP), Sweden (SWE), Switzerland (CHE), Turkey (TUR), UK (GBR) and USA.

Box 2

Access to the Cochrane Library

Individual users can read reviews at US$6 each, download reviews at US$38 each or obtain a personal subscription at US$365 annually. In addition, academic and corporate institutions with fewer than 1001 employees can obtain licenses at annual prices of US$2582 and US$3812, respectively. All prices retrieved from www.cochranelibrary.com/help/how-to-order and links therein on 5 April 2020.

This article examines the expected demand for full-text reviews and plain-language summaries under free access for countries that have no national subscription. Absent institutional access, many healthcare professionals and patients may be unwilling or unable to purchase alternative access but would use reviews if access was free. Governments in countries without a national subscription, however, may be reluctant to subscribe nationally without knowing the expected benefit of such a policy.

In this article, we define the benefit of a national subscription as the increase in the downloads of Cochrane reviews. This benefit depends on the elasticity of demand, that is, users’ responsiveness to changes in the price of review downloads. National subscriptions reduce the marginal cost a user incurs for downloading a review to zero. Using the standard model of supply and demand, we would expect review downloads to increase as more users can afford to download. When access is restricted, these potential users are either unable or unwilling to pay for review downloads and resort to less detailed or potentially misleading sources of information. Free access would attract downloads from these users and those who learnt about the service through its growing popularity.

An increase in review downloads can be expected to have a converse effect on its (imperfect) substitutes. On the one hand, this would be desirable if increased reviews manifested in reduced use of misleading sources of information. For example, misleading information, such as exaggerating benefits and downplaying harms of drugs or cancer screening, is the norm on (commercial) websites and in patient brochures.8 9 On the other hand, an increase in review downloads may also subtract from plain-language summary views; ignoring this substitution effect would overestimate the effect of a national subscription. We expect this effect to be limited because some users may prefer or need the detail of the reviews whereas others may prefer the conciseness and availability of plain-language summaries, particularly when summaries are translated into their native language. Translations from English into other national languages primarily address a lay audience (or healthcare professionals who do not understand statistics) with little or no command of English. We, therefore, expect that translating additional plain-language summaries can counteract the drop in summary views under free access, as they attract additional users who were previously unable to use the service.

To test these hypotheses, the goal of this article is to estimate the impact of national subscriptions on the number of downloads and views of (translated or untranslated) online summaries for individual OECD countries.

Method

The data used for the analysis were drawn from both Cochrane and publicly available databases. We obtained from Cochrane data of web traffic on their websites in 2014, including the Cochrane Library hosted by Wiley and third-party sites such as EBSCO and OVID. From these data, we derived our two variables of interest for this study: the number of review downloads and the number of summary views, stratified by country. Each of these variables captures one way in which Cochrane reviews can be used. Full-text reviews are likely, but not exclusively, downloaded by healthcare professionals who understand technical details. Naturally, these professionals often function as multipliers who pass on information to patients. In contrast, patients without medical training are more likely to consult plain language or other summaries available on different Cochrane websites. These summaries are intended for a lay audience and are sometimes translated for this purpose. Jointly, the number of downloads and summary views give a comprehensive picture of how Cochrane reviews are accessed.

Our analysis exploited the variation in the use of Cochrane reviews across a range of countries to estimate the effect of different subscription schemes. Specifically, we compared the groups of countries with and without free access on their number of downloads and used the difference to calculate the expected effect of a national subscription on countries without one. Taking into consideration that each country’s use of Cochrane reviews is not exclusively affected by their subscription scheme, we collected supplemental data on other determinants of review downloads and summary views. For example, we expected that more populous countries download, all else being equal, more reviews than less populous countries. Our analysis hence needed to isolate the effect of subscription type from that of population size and other country characteristics.

Table 1 lists all variables considered in the analysis. The number of review downloads, number of summary views and subscription status refer to 2014, whereas supplemental data10 11 are as recent as 2016 but may go back as far as 2008, especially in less-developed countries. One variable, subscriptions, was available only as intervals of the form 0 to 50, 50 to 100. For the analysis, we used the centre of each interval as an estimate of each country’s number of existing subscriptions. For some countries, the available data were incomplete. Excluding these countries, we obtained a total set of 158 countries for the analysis. Binary variables are coded as zero and one for no and yes, respectively.

Table 1

Overview of variables, variable types and data sources

We used two linear models to isolate the effects of a national subscription on review downloads and summary views, respectively. The first model, DOWNLOADS, decomposes the number of downloads into the effects of the different country characteristics listed in table 1. Formally, the number of review downloads of country i is given by

Embedded Image

where Embedded Image denotes the intercept, Embedded Image denotes the partial effect of variable j, and Embedded Image denotes an error term that is assumed to be independently, identically and normally distributed. The purpose of the analysis was to estimate the parameters Embedded Image to Embedded Image chief among them was Embedded Image, the effect of a national subscription.

In addition to estimating the model shown here, we also estimated an augmented model that includes interaction effects of free with english and population. Likewise, we estimated three different nonlinear models that predict downloads by combining a set of regression trees such as random forests.12 Because many variables were not normally distributed but included considerable outliers, all five models were tested with and without logarithmic transformation of all continuous variables, yielding a total set of 10 models that were tested. These models were compared on the quality of their out-of-sample predictions using 17-fold cross-validation, where the test set was restricted to the 34 OECD member states. The model presented above, with logarithmic transformation, produced a root-mean squared error (RMSE) of 182 662 downloads, whereas the closest competitor exhibited an RSME of 187 298 downloads. A sensitivity check using the model with the next lowest out-of-sample error yielded comparable results. Further, a visual check of the model assumptions revealed no irregularities.

The second model, VIEWS, decomposed summary views into the effects of the different country characteristics listed in table 1. Unlike reviews, summaries are sometimes translated into other national languages, but the number of translated summaries varies across countries. To separate the effect of language from that of national subscriptions, we used the same linear model as before to estimate the number of summary views based on country characteristics but replaced the binary variable Embedded Image with Embedded Image, which gives the number of plain-language summaries available in the national language. Formally, summary views are then described as follows:

Embedded Image

where Embedded Image denotes the intercept, Embedded Image denotes the partial effect of variable j, and Embedded Image denotes an error term that is assumed to be independently, identically and normally distributed. Again, the purpose of the analysis was to estimate the parameters Embedded Image to Embedded Image, with particular interest in variables Embedded Image and Embedded Image. As before, we chose this model from a set of 10, including six random-forest and four linear models. Two of the linear models slightly outperformed the selected model in 17-fold cross-validation, with training sets restricted to OECD countries. These models used non-logarithmic versions of the variables included and yielded RMSE of around 301 000 views whereas the chosen model yielded an error of around 317 000 views. Nonetheless, we chose the selected model because the logarithmic versions seemed more adequate, particularly because the model led to slightly better estimates for the majority of countries, although predictions for a few countries were less precise. A sensitivity check showed that this choice was conservative in the sense that the combined effects of Embedded Image and Embedded Image, which are most relevant to our argument, are somewhat smaller in the model chosen than in the model with the lowest out-of-sample error.

Patient and public involvement

This research involves secondary data and no patients were involved in the design or conduct of the study.

Results

In this section, we compare countries with and without a national subscription on their review downloads and summary views. We present the results of our two statistical models and use these models to calculate the expected number of reviews for all OECD countries. Finally, we provide rough estimates of the monetary costs of a national subscription.

Review downloads

The black and grey circles in figure 1 show the total number of review downloads in 2014 for all OECD member states. The position of each circle on the x-axis indicates the number of downloads per 1000 persons and the size of the circle indicates the total number of downloads. Among countries without free access, shown by the black circles, the Netherlands, Sweden and Switzerland had the highest and Mexico, Slovakia and the Czech Republic the lowest number of downloads per capita. Although there was a tendency for more prosperous countries to have more downloads per capita, exceptions can be found. Most notably, there were seven downloads per 1000 persons in Chile, but only 0.25 per 1000 in Japan. On average, countries without a national subscription downloaded 2.33 reviews per 1000 persons.

Figure 1

Observed and expected annual review downloads per 1000 persons for OECD member states. OECD, Organisation for Economic Co-operation and Development. AUS, Australia; AUT, Austria; BEL, Belgium; CAN, Canada; CHE, Switzerland; CZE, The Czech Republic; DEU, Germany; DNK, Denmark; ESP, Spain; EST, Estonia; FIN, Finland; FRA, France; GRC, Greece; HUN, Hungary; ISL, Iceland; IRL, Ireland; ISR, Israel; ITA, Italy; JPN, Japan; KOR, Republic of Korea; LUX, Luxembourg; MEX, Mexico; NDL, Netherlands; NZL, New Zealand; NOR, Norway; POL, Poland; POR, Portugal; SVK, Slovak Republic; SVN, Slovenia; SWE, Sweden; TUR, Turkey.

The Netherlands had 10 downloads per 1000 persons, making it the country with by far the highest download rate among those without free access. For countries with a national subscription, the grey circles show downloads per capita. Each of these countries had more downloads per capita than the Netherlands, on average 19.2 reviews per 1000 persons. Download rates were particularly high for anglophone countries, suggesting a linguistic advantage.

To illustrate the effect of a national subscription, it can be instructive to compare countries that differ in their subscription status but are similar in many other respects. For example, Denmark and Norway, with free access, had roughly twice as many downloads as Finland, which was without free access. Likewise, UK, with free access, had roughly the same total as the USA, without free access, despite its population being only a fifth of the latter’s.

Although these comparisons provide a first indication that more reviews were downloaded when access was free, the DOWNLOADS model offers a more rigorous estimation of the effect of a national subscription when it is isolated from other factors. Comparing the SD of Embedded Image with the prediction error for OECD countries indicates that the model yielded fairly accurate predictions. However, a visual inspection of the predictions per country (not shown) revealed that the models underestimated downloads for Chile and the Netherlands, whose downloads appeared to be driven by idiosyncratic factors omitted here. At the same time, the countries that were best predicted appear to be those with free access.

Columns 5 and 6 of table 2 present the estimates of Embedded Image to Embedded Image, which indicate the approximate percentage increase in review downloads associated with a one-percent increase in the variable of interest (see box 3). For example, a gross domestic product (GDP) increase of one percent is associated with a rise in review downloads of half a percent on average. As expected, most variables are positively associated with review downloads. The only exceptions are OECD membership, which appears to have no discernible effect beyond the effect of GDP, and the number of physicians. Increasing the number of physicians by 10 per cent is estimated to reduce downloads by about 3 per cent on average. This negative effect may appear surprising. One possible explanation is that an increase in the number of physicians per capita implies fiercer competition among them, given that the number of patients is fixed. Such increased competition may incentivise physicians to favour profitable over effective treatments, lowering demand for medical evidence.13 A second possible explanation is that countries with more physicians are more likely to have alternative resources available such as national guidelines for professional practice, including those by the US Preventive Services Task Force. A comparison of guidelines in the UK and the USA points in this direction.14

Table 2

Estimated coefficients and diagnostics of ordinary least squares regression models

Box 3

Elasticities

Coefficients in a regression of one logarithmic variable on another are referred to as elasticities. For example, in the regression model Embedded Image, elasticity Embedded Image gives the approximate percentage change in y associated with a one-percent change in x. To see this, recall that Embedded Image for small values of Δ, so raising by one percent increases Embedded ImageEmbedded Image Embedded Image. Multiplying x by 1.01 therefore increases Embedded Image by 0.01. Correspondingly, α gives the increase in Embedded Image and percentage increase in y.

For the present purpose, interest lies in the estimated effect of a national subscription. All else equal, the model estimated that the number of review downloads increased, on average, to Embedded Image percent when access was free. However, concluding that a national subscription increases downloads tenfold would be premature. Under a national subscription, institutional and individual subscriptions are no longer needed and should no longer be considered in the model. We, therefore, need to subtract the estimated effect of those subscriptions from that of a national subscription to obtain the incremental effect. The model then estimates that for countries with 25 or 150 subscriptions, the number of downloads would increase to Embedded Image per cent and to Embedded Image per cent, respectively. As usual, these estimates indicate the average increase in the number of downloads, and observed increases may vary for countries that are dissimilar to those that had a national subscription in our data.

The estimated coefficient for a national subscription exhibits a large SE of 0.53 (see table 2). Although we can reject the null that Embedded Image, we suspect that the lack of precision is due to the fact that only eight countries are currently subscribed whereas 150 countries are not. Given this imbalance, a large SE is not surprising. For an alternative assessment of the accuracy of the estimated coefficient Embedded Image, we calculated the RMSE in cross-validation specifically for those countries with free access. To this end, we predicted downloads for each country separately, based on parameters estimated from the data of all other countries. Across the resulting 158 models, the estimated effect of free access varied only slightly between 2.33 and 2.56. Using these estimates, the bottom of table 2 shows that the model’s predictions were considerably more precise among countries with free access than among those without. These findings indicate that the estimated effect of free access is closer to its true value than its SE may suggest.

Using the estimated coefficients and the data on existing subscriptions, we can calculate for each OECD country the number of expected downloads under a national subscription. These projections are shown by the white circles in figure 1. The logarithmic nature of the model implies that the number of additional downloads generated by a national subscription is driven by the existing download volume: countries with larger download volumes (eg, anglophone, populous and prosperous) are expected to profit more from their introduction.

Consider two cases that illustrate the expected effects of a national subscription. First, recall the case of the USA with as many downloads as the UK (around 1.4 million), despite having a population that is five times larger. The results of our analysis showed that a national subscription would be expected to generate an additional 1.6 million downloads per year, doubling the national total. Second, among non-anglophone countries, Germany had a download level of only 116 000 reviews, less than twice as many as Denmark, despite its population being around 13 times larger. A national subscription would be estimated to increase national totals in Germany to 408 000, increasing the rate of downloads per person to half of the rate in Denmark.

Summary views

Our second analysis concerned the effect of a national subscription on plain-language summary views. The black and grey circles in figure 2 show the number of plain-language summary views in 2014 per 1000 persons for all OECD member states. Among countries without a national subscription, France, Canada and Spain had the highest number of views per capita, and Turkey, South Korea and Japan had the lowest. The average number of summary views for countries without a national subscription was 2.18 summaries per 1000 persons.

Figure 2

Observed and expected annual summary views per 1000 persons for OECD member states. OECD, Organisation for Economic Co-operation and Development. AUS, Australia; AUT, Austria; BEL, Belgium; CAN, Canada; CHE, Switzerland; CZE, The Czech Republic; DEU, Germany; DNK, Denmark; ESP, Spain; EST, Estonia; FIN, Finland; FRA, France; GRC, Greece; HUN, Hungary; ISL, Iceland; IRL, Ireland; ISR, Israel; ITA, Italy; JPN, Japan; KOR, Republic of Korea; LUX, Luxembourg; MEX, Mexico; NDL, Netherlands; NZL, New Zealand; NOR, Norway; POL, Poland; POR, Portugal; SVK, Slovak Republic; SVN, Slovenia; SWE, Sweden; TUR, Turkey.

In contrast, there were on average 5.39 summaries per 1000 persons in countries with a national subscription, indicating an effect of such a subscription. Although the levels for countries with and without national subscription overlap, the highest level (9.1 views per 1000 persons) was reached by Australia, which held a national subscription. Within this group of subscribing countries, Denmark had the fewest views per capita, in keeping with the level of structurally similar countries such as Finland and Sweden. Among national subscribers, anglophone countries appear to have consumed more: not only were more reviews downloaded, as noted before, but also more summaries were viewed.

The views model offers a more detailed examination of the effects of a national subscription and of language. Although the model diagnostics indicated that the model yielded acceptable predictions, predicting the number of summary views was apparently more difficult than predicting downloads. Most notably, the model overestimated the number of views from Japan, Germany and South Korea, where there appeared to have been constraining factors omitted from the model. Columns 7 and 8 of table 2 report the estimated model parameters. Whereas most variables had their expected positive effect on the number of summary views, a higher density of physicians and OECD membership decreased the number, although this latter effect is imprecisely estimated. We were particularly interested in the estimated effects of free access and translations.

Given the substituting nature of full-text downloads and summaries, we had expected a negative effect of a national subscription on summary views. Surprisingly, the estimated effect was positive, indicating at first sight that the additional popularity of the service compensates for summary views supplanted by review downloads. However, there are two caveats to this conclusion. First, the effect was imprecisely estimated so that the degree of compensation cannot be firmly established to be positive or negative. More importantly, subtracting the effects of existing subscriptions can lead to a negative net effect for countries with more than Embedded Image existing subscriptions. Generally, we conclude that the negative effect of a national subscription on summary views appears to be small, if at all present.

In contrast, the effect of translations was precisely estimated and positive. The point estimate indicates that increasing the number of summary translations by 100 per cent increases views by approximately Embedded Image per cent. Although the magnitude of this effect appears small, it is worth pointing out that some countries had only few translations. For example, only 128 of 5952 summaries have been translated into German. A translation of all summaries is then estimated to increase summary views by 57.8 per cent.

To illustrate the interaction of the effects of free access to full-text reviews and summary translations, we used the model estimates to calculate for all OECD countries the number of expected summary views under a national subscription and full translation. These projections are shown by the white circles in figure 2 and vary considerably for two reasons. First, as before, the logarithmic nature of the model implies that those with many views benefit more strongly than those with few views. Second, countries vary in their progress on summary translations, and those with few translations have more room for improvement than those with many translations.

Implied costs

Like all policy instruments, national subscriptions to Cochrane reviews ought to be subjected to a cost-benefit analysis. We have seen above that the benefits in terms of additional full-text downloads and summary views vary across countries but can be substantial in some cases. Here, we set these benefits in relation to the monetary costs of a national subscription. These costs depend on the price of a national subscription and the amount spent on existing subscriptions that would be obsolete under a national subscription. We will discuss each of these factors in turn.

Although Cochrane does not publish rates for national subscriptions, the annual rate is believed to be around US$0.01 per capita (Gert Antes, former director of Cochrane Germany, personal communication). On the basis of this estimate, the first column of table 3 lists the total costs of a national subscription for each country according to its population size. For example, a national subscription for small countries such as Finland or Austria would cost less than US$100 000 annually while larger countries such as Germany or Japan would require around one million dollars per year.

Table 3

Estimated costs of national subscriptions across OECD countries

At the same time, a national subscription implies that existing subscription holders no longer need their subscriptions. This may further lower the cost of a national subscription. Unfortunately, we do not know each country’s total spending on individual downloads, personal licenses or institutional subscriptions. However, our data include an interval of the total number of subscriptions, which can be used to estimate existing total spending. For this purpose, we assumed that observed downloads increase linearly within each subscription interval and estimated for each country i the number of subscriptions, si, from the number of review downloads, di, using

Embedded Image

where Embedded Image and Embedded Image denote the lowest and highest possible number of subscriptions in the interval of country i, and Embedded Image and Embedded Image denote the minimum and maximum number of downloads for countries with the same interval. The approximated number of subscriptions is then multiplied by US$2582, which is the price of the least expensive institutional subscription (see box 2). For the country with the fewest downloads in the interval, si is set at the lower bound of the interval plus ten percent of its range to avoid inconsistencies at the interval bounds. Conversely, for the country with the most downloads, si is set at the upper interval bound minus ten percent of its range. To summarise this procedure, consider for example, three countries in the 50–100 subscriptions interval with 100, 1100 and 350 downloads, respectively. Based on these data, they would be assumed to have Embedded Image, Embedded Image and Embedded Image subscriptions, respectively. With only one country per interval, Embedded Image is set at the centre of the interval.

The second column of table 3 lists the approximate existing total spending for each country. It shows that some countries without a national subscription, such as Germany and Japan, have spent large amounts of money on Cochrane licences for research institutions or medical organisations. Under a national subscription, these individual licenses would become obsolete. However, to determine the actual financial burden of a national subscription, it may be important to consider the mix of private and public institutions among existing subscribers. Unlike potential savings by public institutions, which may be subtracted from the total costs, savings by private institutions would in fact raise costs to governments through foregone sales taxes. However, we suspect that the large majority of existing subscribers are publicly funded, implying that omitting the need for existing spending on Cochrane licenses would reduce the effective cost of a national subscription.

The third column of table 3 subtracts the estimated existing costs from the estimated total and divides it by the estimated increase in review downloads shown in figure 1. Integrating costs and benefits, this column can be used to separate the countries into three groups. First, three countries, Czech Republic, Mexico and Slovakia, would pay around US$20 per additional download, a sum that falls short of the price of an individual download but exceeds the cost of merely viewing a review. Similarly, Greece, Hungary, Poland and Turkey would pay US$4–US$7, considerably more per additional download than most other countries. Second, three countries, Iceland, Sweden and the Netherlands, are predicted to save money through a national subscription. The majority of countries, including Canada, France, Germany, Italy and the USA, fall in between these extreme groups, with costs per additional download ranging between US$0.05 and US$2.

Discussion

Cochrane reviews are currently of limited use, as many healthcare professionals and most patients do not have free access to them. In spite of efforts to promote informed healthcare professionals and patients, governments have been reluctant to purchase national subscriptions. We calculated estimates of the increase in full-text downloads and summary views of Cochrane Reviews in OECD countries if they were to purchase a national subscription. We then integrated these estimated benefits with the estimated costs of a national subscription and provided a measure of the effective costs.

Our findings are encouraging. Although the estimated increases in full-text downloads vary between countries, figure 1 shows that considerable improvements are possible. Indeed, the majority of countries is projected to multiply their downloads by a factor above two, including countries with few downloads in the absence of a national subscription.

In addition, our analysis of summary views showed that a national subscription is not associated with a reduction in summary views. Instead, the effect of a national subscription could be both positive and negative, depending on the country. However, figure 2 illustrates that translations of summaries into the national language can attenuate possible negative effects and offer a second avenue for disseminating Cochrane evidence. As we used each country’s national language to determine the number of available translations, the model did not control for national differences in English proficiency. Therefore, the model may have overestimated the effect of additional translations for countries in which English is widely and well understood, such as Scandinavian countries or the Netherlands. Nonetheless, the results indicate that translations have the potential to increase summary views in many countries, including some without exceptional English proficiency. For example, Slovenia, Greece, Italy and Germany hold the potential for considerable improvements through comprehensive translation of existing summaries. We therefore conclude that translations of Cochrane summaries offer an additional tool for disseminating Cochrane evidence that can be used independently of a national subscription.

Integrating these estimated benefits with the costs of national subscriptions, we find that for all but seven OECD member states, the net costs would be small. Whereas seven countries can expect to face insufficient demand to justify the purchase of a national subscription, according to our estimates in table 3, many countries would pay less than US$1 for each additional download and three countries would save money under a national subscription. Thus, for most countries, national subscriptions to Cochrane reviews present an inexpensive way of disseminating medical evidence. The question of to what degree this evidence will be used cannot be answered by the present study, although Cochrane reviews have in the past had direct impact on policy-making,15 and, when translated into fact boxes and other understandable forms, can foster physicians’ and patients’ understanding and decision making.16–18

The estimates of our analysis are based on an ordinary least squares regression model of observational data and are not without caveats. Most importantly, observational data are ill suited to establish causal relationships. That is, our analysis cannot formally answer the question whether national subscriptions lead to increases in downloads, whether the reverse is true, or whether both variables have a common cause. Instead, we have found that it is more plausible that a national subscription leads to a given number of downloads than vice versa because subscriptions have a causal effect on the costliness of a download. However, it is important to note that there remains the possibility that both subscription and downloads are caused by a third variable that we have not accounted for in our models. Despite all efforts to control for potential confounds such as economic strength or research activity, comparisons across countries retain the possibility that relevant differences between countries remain unnoticed or unobserved. To corroborate our findings, we, therefore, encourage studies that examine the effects of a national subscription by comparing downloads before and after its introduction within the same country.

A second limitation of this study concerns the uncertainty of our cost estimates. When calculating the expected costs per additional download, both the numerator and the denominator were based on estimates. The costs in the numerator were based on estimates of the costs of existing subscriptions and the denominator was based on our model estimates. Although we could compute confidence intervals for the denominator, we cannot quantify the uncertainty of the numerator. These estimates are based on the number subscriptions and a conservative estimate of their costs. These estimates are conservative but their uncertainty remains unclear until more detailed data on subscriptions become available.

Our cost-benefit analysis provides estimates of the effective costs per download gained through a national subscription. The analysis remains agnostic as to how highly additional downloads are valued and leaves such judgements to policy-makers. However, we emphasise the importance of evidence for directing healthcare resources to where they are most effective. This is especially true in healthcare systems where various actors are incentivised to overstate the effectiveness of different health interventions. In these environments, it is key that healthcare professionals and patients are empowered to base their decisions on evidence instead of advertisements. However, to be effective, good evidence requires not only high-quality studies but also easy access to their conclusions.

Data availability statement

The data used for the analysis are proprietary and were kindly provided by the Cochrane Collaboration and Wiley. Therefore, we are unable to share them publicly.

Ethics statements

Acknowledgments

We thank the ABC Research Group for helpful comments, Rona Unrau for copy editing the manuscript, and the Max Planck Society for funding this work.

References

Footnotes

  • Contributors GG and PJ conceived and designed the study, PJ collected the data and performed the analyses, GG and PJ wrote the manuscript.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.