Intended for healthcare professionals

Analysis

What is missing from descriptions of treatment in trials and reviews?

BMJ 2008; 336 doi: https://doi.org/10.1136/bmj.39590.732037.47 (Published 26 June 2008) Cite this as: BMJ 2008;336:1472
  1. Paul Glasziou, professor of evidence based medicine1,
  2. Emma Meats, research assistant1,
  3. Carl Heneghan, senior clinical research fellow1,
  4. Sasha Shepperd, NIHR research scientist in evidence synthesis 2
  1. 1Centre for Evidence-Based Medicine, Department of Primary Health Care, University of Oxford, Oxford OX3 7LF
  2. 2Department of Public Health, University of Oxford
  1. Correspondence to: P Glasziou paul.glasziou{at}dphpc.ox.ac.uk
  • Accepted 21 March 2008

Replicating non-pharmacological treatments in practice depends on how well they have been described in research studies

Have you ever read a trial or review and wondered exactly how to carry out treatments such as a “behavioural intervention,” “salt reduction,” or “exercise programme”? Although CONSORT and related initiatives have focused on the assessment of validity and presentation of results,1 2 less attention has been given to the adequacy of the description of the treatment used. For pharmacological treatments the description would need to include the dose, titration, route, timing, duration, and any monitoring used. For complex treatments the problems are even greater.

Why are full descriptions of treatment important?

The uptake of positive findings from trials is often slow and sometimes negligible.3 Reasons for this slow uptake include clinicians not becoming aware of the results, perceiving the results as either invalid or not relevant to their patients, or simply not remembering to use the treatment.4 5 An additional barrier, which has received less attention, is clinicians’ ability to carry out the treatment on the basis of the information provided in the published reports. For example, after receiving numerous requests for additional details from doctors and patients, the author of a randomised trial on graded exercise for chronic fatigue syndrome6 subsequently published a supplementary article with a more detailed “prescription.”7 Similarly, it is not possible to set upa stroke unit, offer low fat diets, or give smoking cessation advice without sufficient details on the components that were planned and delivered.8

Extent of the problem

To assess the extent of problems with descriptions of treatment we prospectively assessed 80 consecutive studies selected for abstraction in the journal Evidence-Based Medicine from October 2005 to October 2006. The journal is aimed specifically at doctors working in primary care and general medicine, and it provides summaries of research that is highly relevant to clinical practice. To select studies, the staff of the journal hand search 140 or so high impact clinical journals, selecting only articles of sufficient validity and relevance to warrant changes in clinical practice.9 The 5% of articles that pass the validity criteria are scored for clinical relevance by active primary care (and appropriate specialty) clinicians. A dozen or so clinicians, from a pool of several thousand, score each article. The articles that were scored as most relevant to practice are then abstracted (fig 1). For each study two general practitioners (PG, CH) were independently asked whether they could use this treatment with a patient if they saw them tomorrow.

Figure1

Fig 1 Selecting studies for inclusion in one year’s issues of Evidence-Based Medicine

Of the 80 published reports of treatment, 55 were single randomised trials and 25 were systematic reviews; they were published in New England Journal of Medicine (10), Cochrane Database of Systematic Reviews (9), Lancet (7), JAMA (7), Archives of Internal Medicine (6), BMJ (5), Annals of Internal Medicine (5), and several other journals (31). Most (65) were of treatments directly applicable in general practice; the remainder were relevant to general practice but were targeted at surgery (6), emergency medicine (5), internal medicine (3), and dental medicine (1). More than half (44/80) were of drug treatments. Non-drug treatments were education and training (15), devices or surgery (10), psychological treatments (4), service delivery (3), and a mix of other interventions (4).

Elements of the intervention were missing in 41 of 80 of the published descriptions. Information was better in reports of individual trials than in systematic reviews, and for drug treatments than for non-drug treatments. Information was also better for control interventions, with 58 being sufficiently well described (22 as drugs and 36 as non-drug controls); the remaining 22 were simply described as placebos.

What elements of treatment do authors miss?

The missing element was most often the description of the process, but several studies were missing handouts or booklets—for example, the name or source for a self help booklet trialled for irritable bowel syndrome. Table 1 shows the missing elements encountered and their resolutions.

Table 1

 Some reasons for non-reproducibility for studies

View this table:

Can authors supply the missing information?

When information on any aspect of treatment needed for clinical use was missing we tried to find a solution. We first retrieved references from the paper relevant to the treatment description, searched the internet and other bibliographic databases as required, and emailed study authors (up to three times). We gained some missing information for most study reports; 52 of 59 authors replied. By this simple measure the completeness of treatment descriptions improved from 49% to 76%, and improved similarly across all categories (fig 2).

Figure2

Fig 2 Percentage of studies with sufficient description of treatment initially (based only on the published paper) and after supplementary information was obtained.

Some authors were extremely helpful in providing additional information. Some volunteered manuals and videos that would form part of the treatment; they were often willing to allow free access to such resources via the internet but had not included such a suggestion in the original paper. For example, one study of “cognitive therapy” had been specifically adapted and tailored for schizophrenia; this required specialised training for which an author was willing to provide considerably more detail than had been published in the journal: “[I] would be delighted for you to include the manual and information booklets in downloadable format . . . it is really important that they are available in the public domain.” Others were less able to help: for a software program to guide nurses in telephone counselling, the authors stated that it was available but not for public use—but this had not been indicated in the publication. Others had concerns about providing sufficient details because of the skill level needed to carry out the treatment. For example, the responding author of a smoking cessation worksheet said: “[The authors of the worksheet] have asked that we do not post this online. The rationale is that people really have to be thoroughly trained in terms of how to use the form.” No details on how to obtain such training had been described in the paper.

Despite our attempts to get further information, elements of the treatment were still missing in 19 studies. Some examples of these are detailed in table 2.

Table 2

 Problems with reproducing the study’s treatment after attempts to obtain further information

View this table:

For systematic reviews, several authors were helpful in providing details and appropriate choices, but many were reluctant to suggest which version(s) of the treatment might be appropriate to use in practice. Many believed that the review already had sufficient information: “All details and recommendations can be extracted from the meta-analysis” and “The references for these studies are found in the article”—but they did not say which treatment should be implemented. One author narrowed the range later, saying “cognitive behavioral therapy was found to have uniform benefits” but did not indicate the type and quantity needed. One author thought it was simply not possible to specify a treatment: “No regimen can or should be based on a single trial, even a systematic review.” Authors understood the need for a specific regimen(s) to be selected but were reluctant to choose one. For example, one suggested a regimen but did not want it publicised: “My hospital’s regimen . . . is definitely not something I would recommend in writing, as it is a local protocol.” Others believed selecting a regimen was not appropriate: one author stated: “It would not be valid to propose one type of planned management over another based solely on our review.”

Improving reporting

While some of the missing detail is intrinsic to the complexity of certain treatments, providing some additional and readily available information would allow a greater use of published research in clinical practice. Many authors were willing to provide extra details and materials so that their treatment could be used, but clearly this had not been required by most journals.

So how might reporting be improved? For trials, the CONSORT statement asks for “Precise details of the treatments intended for each group and how and when they were actually administered,”10 but further guidance would be helpful. A more detailed checklist about the “who, what, when, and where” of the treatment is desirable but may need to be tailored to different types of interventions. For non-pharmacological treatments the details are often complex, and a recent extension to CONSORT (available at www.consort-statement.org/index.aspx?o=1068) requests extra details on the components, the procedure, including tailoring to individuals, standardisation, and adherence.

Full descriptions of treatment should include any procedures used; the timing of treatment, including duration and intervals of dosing or sessions; any materials needed (such as patient handouts or devices); and accessibility of any materials or instructions, including overcoming language barriers. This may need to be supplemented with graphical methods for depicting the flow and timing of sessions of treatment11 and copies of materials or handouts used. Electronic publishing provides the ideal format for distributingsuch extra material.12 For systematic reviews, statistical and clinical approaches need to be integrated to help reviewers select the most appropriate treatment(s) from among those included in the review.

In systematic reviews, the high level of abstraction used in selecting “similar” treatments causes a problem. Even when reviews and trials include enough details of the treatment, the clinical reader may find it difficult to select the version to adopt in practice, especially if there is heterogeneity of effect among the treatments. Several formal and informal options can be used to decide the appropriate treatment in practice. For example, selecting the treatment with the largest apparent benefit seems sensible but has several dangers. In particular, a small trial may have a large effect by chance. The treatment used in the largest trial is not necessarily the most effective one. The QUOROM statement asks reviewers to report “details of intervention” of trials,13 and the Cochrane Handbook provides more on how to describe complex interventions (Section 7.3.4) and their fidelity,14 but neither currently gives guidance on how to select and report the appropriate versions of the treatments reviewed.

Access to such complete descriptions should not be restricted. Development of a repository of treatment descriptions, particularly for non-drug treatments, would help clinicians and be supported by many authors. An example of such a process is the Centers for Disease Control and Prevention’s Replicating Effective Programs (www.cdc.gov/hiv/projects/rep), which aims to provide detailed packages and resources in HIV prevention. Without such descriptions and programmes, tens of millions of pounds of research effort could be wasted each year because effective treatments can’t be implemented or will lack fidelity when applied.

Summary points

  • To use treatments tested in trials, clinicians need sufficient details of the “how to”

  • Many current trials and reviews often omit crucial details of treatments

  • Providing some additional treatment details could improve the uptake of trial results in clinical practice

Footnotes

  • We thank all the authors who responded so helpfully; and Iain Chalmers, Sara Schroter, Muir Gray, Doug Altman, Brian Haynes, Mike Clarke, and Andy Oxman for helpful comments on the paper. CH and SS are funded by Department of Health Research Development Award and Evidence Synthesis Award respectively. PG had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

  • Contributors: PG conceived the study; EM was project manager; CH and PG assessed the target treatments and SS assessed the control treatments; EM with assistance from SS coded, entered, and analysed the data; all authors contributed to the concepts and writing of the paper. PG is guarantor.

  • Funding: BMJ Publishing Group partially funded this research.

  • Competing interests: None declared.

  • Provenance and peer review: Not commissioned; externally peer reviewed.

References

View Abstract