Article Text
PDF
Abstract
Objective To assess risk of bias and to investigate methodological issues concerning the design, conduct and analysis of randomised controlled trials (RCTs) testing acupuncture for knee osteoarthritis (KOA).
Methods PubMed, EMBASE, Cochrane Central Register of Controlled Trials and four major Chinese databases were searched for RCTs that investigated the effect of acupuncture for KOA. The Cochrane tool was used to examine the risk of bias of eligible RCTs. Their methodological details were examined using a standardised and pilot-tested questionnaire of 48 items, together with the association between four predefined factors and important methodological quality indicators.
Results A total of 248 RCTs were eligible, of which 39 (15.7%) used computer-generated randomisation sequence. Of the 31 (12.5%) trials that stated the allocation concealment, only one used central randomisation. Twenty-five (10.1%) trials mentioned that their acupuncture procedures were standardised, but only 18 (7.3%) specified how the standardisation was achieved. The great majority of trials (n=233, 94%) stated that blinding was in place, but 204 (87.6%) did not clarify who was blinded. Only 27 (10.9%) trials specified the primary outcome, for which 7 used intention-to-treat analysis. Only 17 (6.9%) trials included details on sample size calculation; none preplanned an interim analysis and associated stopping rule. In total, 46 (18.5%) trials explicitly stated that loss to follow-up occurred, but only 6 (2.4%) provided some information to deal with the issue. No trials prespecified, conducted or reported any subgroup or adjusted analysis for the primary outcome.
Conclusion The overall risk of bias was high among published RCTs testing acupuncture for KOA. Methodological limitations were present in many important aspects of design, conduct and analyses. These findings inform the development of evidence-based methodological guidance for future trials assessing the effect of acupuncture for KOA.
- acupuncture
- knee osteoarthritis
- quality
- methodological characteristics
- trial design
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
The literature search was comprehensive, and detailed inclusion and exclusion criteria were developed to ensure transparency and reproducibility in the judgements.
A standardised and pilot-tested questionnaire was used to systematically assess the methodological issues concerning the design, conduct and analysis of randomised controlled trials testing acupuncture for knee osteoarthritis.
The insufficient reporting of methodological details among those published trials limited our assessment of the methodological quality and identification of specific methodological gaps.
In addition, this study did not examine the specification of patient population in those trials, and the findings may not be applicable to other acupuncture trials.
Background
Knee osteoarthritis (KOA) is a major health concern worldwide and its disease burden continues to increase due to population ageing.1 2 Standard treatments for KOA, such as non-steroid anti-inflammatory drugs, glucosamine and intra-articular injections, are often of limited effects.3–6 Acupuncture has received increasing attention,7 8 and is becoming popular in many Western countries.2 Approximately one million American patients with musculoskeletal disorders are now treated with acupuncture annually.9
A number of randomised controlled trials (RCTs) have been conducted to test the effects of acupuncture on KOA. However, the evidence so far is weak, and the majority of trials had low methodological quality.10–12 Due to the lack of compelling evidence, acupuncture has not been recommended as a treatment option for KOA in the most recent clinical practice guidelines.2 13–15
Although there is strong consensus that efforts should be made to improve the quality of such trials, no established methodological guidance is available to inform researchers about how to rigorously design, conduct and analyse a trial of acupuncture for KOA. Unlike classical drug trials, RCTs of acupuncture have many specific features. For instance, acupuncture, as a typical complex intervention, is often expertise/preference-based, varies in practice and contains multiple components. Patient characteristics and study settings are also complex. Blinding may be challenging. Thus, there are special issues concerning the design, conduct and analysis of RCTs of acupuncture for KOA. Our ultimate goal is to develop a methodological guidance for rigorous design, conduct and analysis of acupuncture RCTs for KOA. Understanding methodological characteristics of previously published RCTs testing acupuncture for KOA would be the first and critical step.
Therefore, we conducted a cross-sectional study of published RCTs specifically testing acupuncture for KOA, so as to examine their methodological details. The findings provide critical information and valuable insights to identify important methodological gaps and support the development of methodological guidance.
Methods
Eligibility criteria
An eligible study had to meet all of the following inclusion criteria: (1) a two-arm RCT, regardless of study type (superiority, non-inferiority or equivalence); (2) enrolled patients diagnosed with KOA; (3) tested the effect of acupuncture in an uncounfunded manner. That is, acupuncture may be used alone or in combination with other treatments, as long as the effects of acupuncture can be parcelled out. The types of acupuncture may include electro-acupuncture, filiform needle, fire needle, silver needle, dry needle, laser acupuncture, ear acupuncture and scalp acupuncture; (4) compared with another type of acupuncture, pharmacological intervention, placebo acupuncture, sham acupuncture, waiting list or physical treatments (eg, exercises and weight loss); and (5) reported findings in the full text of the paper.
We excluded study protocol from our study. Trials reported in research letters or conference abstracts were also excluded.
Literature search
We searched PubMed, EMBASE, Cochrane Central Register of Controlled Trials as well as four Chinese databases, namely Chinese Biomedical Database, China National Knowledge Infrastructure, Wanfang and China Science and Technology Journal Database (VIP). The search was conducted from inception to February 2017 and for articles published in either English or Chinese. A combination of keywords and Medical Subject Headings related to KOA, acupuncture and RCT was used after consulting an experienced librarian (online supplementary appendix 1). We also manually screened reference lists of eligible papers for articles which were not identified by the computerised search.
Supplementary file 1
Study process
Using standardised and pilot-tested forms, two authors (PLJ and JLL) screened the titles and abstracts independently and in duplicate for potential eligibility. They subsequently read the full texts to determine final eligibility. Then, from the eligible RCTs, two other authors (PLJ and XZ) assessed their risk of bias and collected methodological details on design, conduct and analysis using a study-specific questionnaire comprising 48 items (see the ‘Study questionnaire’ section below). A third reviewer (XS) was involved as the adjudicator in case of discrepancy throughout the process.
Risk of bias assessment
We assessed the risk of bias of included RCTs using the Cochrane Risk of Bias tool. This tool contains seven items: (1) randomisation sequence generation; (2) allocation concealment; (3) blinding of participants and personnel; (4) blinding of outcome assessors; (5) incomplete outcome data; (6) selective reporting; and (7) other sources of bias. Each item was assigned one of three responses: ‘low risk of bias, high risk of bias or unclear risk of bias’ according to the Cochrane handbook.
Study questionnaire
In order to fully examine the methodological details of eligible RCTs, the research team, consisting of clinical trial experts, statisticians and acupuncturists, developed a study-specific questionnaire.
In developing the questionnaire, we first reviewed existing tools for assessing methodological quality and reporting (eg, Cochrane Risk of Bias Tool and CONSORT), and drafted a list of items. Based on those items, the research team, through brainstorming, added other potentially useful items. Afterwards, the research team undertook six group discussions to ascertain relevancy of each questionnaire item and clarify its definition. Items that only addressed issues of reporting were deleted due to out of the study scope. The resulting questionnaire was pilot-tested using 10 studies randomly selected from the eligible articles. Thereafter, the items were revised based on the pilot test results.
The final version of the study questionnaire (online supplementary appendix 2) consisted of 48 items covering the following aspects: general information (eg, publication year, scope of journal and funding sources), study design and conduct (eg, randomisation, allocation concealment, standardisation of acupuncture procedures, blinding and sample size calculation) and data analysis (methods for dealing with loss to follow-up, subgroup analysis and adjustment for confounders).
Supplementary file 2
Statistical analysis
We qualitatively summarised the general characteristics about the design, conduct and data analysis of included studies. Number and percentage were used for multinomial or dichotomous outcomes, whereas median and IQR were calculated for continuous variables.
In order to examine the factors associated with major methodological quality indicators, we conducted four separate multivariable regression analyses for the following: adequate generation of randomisation sequence (yes vs no or unclear), allocation concealment (yes vs no or unclear), blinding of participants and personnel (yes vs no or unclear) and blinding of outcome assessors (yes vs no or unclear). For each of the regression analyses, we included four prespecified factors, namely, language (English vs Chinese), multiple centre study (yes vs no or unclear), funding support (yes vs no or unclear) and sample size (categorised according to the median).
Results
Our search resulted in 4527 reports, of which 557 were potentially eligible after title and abstract screening. On reviewing their full texts, 248 RCTs were finally included (figure 1). No additional RCTs were identified through screening of their reference lists.
PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow chart for search and selection processes.
General characteristics of included trials
The 248 RCTs (219 in Chinese and 29 in English) were published between 1992 and 2016. Table 1 summarises general characteristics of the included trials. The majority were single-centre trials (92.7%) published in a non-acupuncture related journal (76.6%). The median sample size was only 76 (range 20–632), and the median duration of follow-up was 4 weeks (range 1–48 weeks). Only one-quarter of the included trials reported sources of funding and all of them were funded by non-profit funding agencies. The median number of authors was 3 (range 1–13).
General characteristics of included trials
Risk of bias assessment of included trials
We found that almost half of the 248 trials were at low risk of bias for randomisation sequence generation (119, 48.0%) and incomplete outcome data (105, 42.3%). But only a small proportion of trials were at low risk of bias for allocation concealment (31, 12.5%), blinding of participants and personnel (15, 12.5%), blinding of outcome assessors (15, 12.5%), selective reporting (9, 3.6%) and other sources of bias (7, 2.8%) (figure 2, online supplementary appendix 3). In general, the trials had a high risk of bias.
Supplementary file 3
Risk of bias assessment of randomised controlled trials of acupuncture for knee osteoarthritis (n=248).
Methodological details of included trials
Randomisation and allocation concealment
Of those 248 RCTs, 99 (39.9%) did not provide details on randomisation, and a large proportion (217, 87.5%) failed to specify the method for allocation concealment (table 2). Among those which provided details on randomisation (149, 60.1%), only 39 (26.2%) used a computer-generated randomisation sequence (the preferred approach), 70 (47.0%) used a random number table, whereas others (40, 26.8%) adopted inadequate methods (eg, coin tossing, visit order) to generate the randomisation sequence. Very few trials used blocked randomisation (15, 6.0%) and stratification at randomisation (9, 3.6%).
Randomisation and allocation concealment of included trials
Of the 31 trials (12.5%) that provided details about allocation concealment, only 1 (3.2%) reported central randomisation, whereas 30 trials (96.8%) used sequentially numbered, opaque or sealed envelope methods. No details were available from other trials (217, 87.5%). Given their applied methods of generation, we judged that most of trials were unlikely to conceal treatment allocation (see table 2).
Intervention and blinding
All 248 trials described interventions for each group (table 3). Types of acupuncture used in the interventions group were filiform/ordinary needle (107, 43.1%), electro-acupuncture (98, 39.5%), fire needle (14, 5.7%), laser acupuncture (4, 1.6%) and non-conventional acupuncture (20, 8.1%), such as silver, Fu’s subcutaneous, blade and longitudinal needle. For the control group, the majority (184, 74.2%) administered a single intervention: acupuncture and moxibustion (51, 20.6%), pharmacological intervention (45, 18.1%), needle-knife (29, 11.7%), physical treatments (22, 8.9%), acupuncture (21, 8.5%), sham acupuncture (8, 3.2%), placebo acupuncture (5, 2.0%), moxibustion (2, 0.8%) and waiting list (1, 0.4%). Multiple interventions, such as pharmacological intervention in conjunction with acupuncture/physical treatments, were used in the remaining trials (64, 25.8%).
Interventions and blinding of included trials
Standardisation of acupuncture procedures was poor across trials. Only 25 (10%) trials stated that acupuncture procedure was standardised, and only 18 (7.3%) explicitly specified that the standardisation was conducted according to manual, specific guidelines or a standardised protocol. A small proportion of the trials (18, 7.3%) clearly stated that acupuncture was performed by acupuncturists, physicians and/or physiotherapists.
The great majority of trials (233, 94%) stated that blinding was in place, but most of them (204, 87.6%) did not clarify who was blinded. The blinding of patients, acupuncturists, outcome assessors or data analysts was mentioned in <10% of the RCTs. Only 12 trials (4.8%) stated that blinding was achieved by sham procedures, such as placebo acupuncture or sham acupuncture (table 3).
Sample size calculation
Only a small number of trials (17, 6.9%) included details regarding the sample size calculation (table 4). Among those 17 trials, 10 performed the calculation based on a defined primary outcome, 9 defined the α level, 12 gave the β value, 9 took into account the potential loss to follow-up and 11 used data from a pilot/previous study. None of the trials preplanned an interim analysis and associated stopping rule.
Sample size calculation of included trial
Primary outcome and data analysis
Only 27 (10.9%) RCTs specified a primary outcome of interest (eg, pain, function and quality of life, table 5). Among them, six mentioned the statistical analytic method, and seven reported intention-to-treat analysis for the primary outcome.
Primary outcome and data analyses of included trials
In total, 145 trials (58.5%) provided information regarding loss to follow-up, and 46 (31.7%) trials explicitly stated that loss to follow-up occurred. Among these 46 RCTs, 37 (80.4%) trials reported loss to follow-up separately for each arm, but only one compared baseline characteristics of loss to follow-up patients between treatment and control groups. Only six trials (13.1%) provided some information to deal with the problem. However, they did not specify the exact analytical methods for dealing with loss to follow-up in the primary outcome. Furthermore, none of trials prespecified, conducted or reported any subgroup analysis or adjustment for confounding variables (table 5).
Factors associated with methodological quality indicators
The multiple regression analysis showed that no prespecified factors were statistically associated with better use of method for generating randomisation sequence; results of which are summarised in table 6. However, trials published in English (OR 4.99, 95% CI 1.72 to 14.47) and conducted in multiple centres (OR 4.49, 95% CI 1.39 to 14.53) were associated with a higher likelihood of achieving allocation concealment. Similarly, trials published in English were more likely to achieve blinding of participants and personnel or outcome assessors (OR 230.74, 95% CI 19.93 to 2671.65; OR 199.82, 95% CI 22.97 to 1738.10, respectively) after adjusting for the influence of funding, sample size and study centres. Multicentre trials were associated with a higher likelihood of achieving blinding of outcome assessors (OR 65.47, 95% CI 6.03 to 710.78).
Factors associated with methodological quality indicators
Discussion
Main findings
In this study, we found that the published RCTs testing acupuncture for KOA were at high risk of bias. The methodological quality issues were, however, complicated with inadequate reporting of trials. We found that 66.1% of trials were unclear in the specification of acupuncture procedure, 92.7% in procedure standardisation, 39.9% in randomisation and 87.5% in concealment. The apparent lack of rigorous methods was in part due to the poor reporting of trials.
However, one may reasonably infer that adequate use of rigorous methodologies for design, conduct and analyses is less likely among trials with serious limitations in reporting. We found that, among those trials which provided methodological details, many used inadequate methods to generate a randomisation sequence, such as coin tossing and alternate visit number; 12.5% of trials performed allocation concealment, but only one used central randomisation, the optimal approach to achieve adequate allocation concealment. In addition, very few trials considered block randomisation and stratification at randomisation, the two important approaches to achieving prognostic balance. These two techniques are particularly useful for acupuncture trials, since the study population is often present with a variety of characteristics
Acupuncture represents a practitioner-dependent, complex intervention.16 Acupuncture interventions are subjected to a high degree of variability in terms of the acupuncturist’s expertise and styles.16 17 Careful consideration of standardisation of acupuncture procedures is an important aspect in conducting high-quality trials, and explicit description of the acupuncture procedure is essential to determine whether the results are generalisable.18 Our study found that many types of acupuncture were used. However, only 10% of trials explicitly stated whether and how acupuncture procedures were standardised, and 7.3% of trials specified who would be qualified for operating the procedures.
Another relevant issue is blinding. Our study found that 94% of trials reported blinding, but 88% of them did not specify who were blinded; only 5% of trials attempted to use sham procedures. Blinding appeared to be challenging in the setting of acupuncture trials. Sometimes, caregivers, who administer acupuncture procedures, and patients may not be blinded if interventions between treatment and control groups have varying features. However, trialists should always consider blinding of both caregivers and patients under all circumstances. Unfortunately, nearly no trials clearly justified why blinding was not implemented. In addition, others parties such as outcome assessors can always be masked. This is particularly relevant for trials of KOA, since the main outcomes are often patient centred (such as pain, disability, quality of life and global perceived effect) and subjective.19
The lack of specifying primary outcomes and calculating sample sizes are two other concerning issues. Our study showed that only 10.9% of trials specified a primary outcome, much lower than previous studies (45% and 73%),20 21 and only 6.9% clearly articulated sample size calculation. Such lack of explicit specification of primary outcomes suggested that trialists did not carefully consider this issue during design, despite the specification of primary outcome is highly relevant to the main study objective and subsequent sample size calculation. Moreover, there is strong evidence that significant outcomes are more likely to be reported.22 When the primary outcomes were not prespecified, the authors tended to selectively report outcomes based on statistical significance instead of the predefined clinical importance,20 thereby portraying an overestimation of the treatment effects. Similarly, very few (10 trials) determined the sample size based on the defined primary outcome, and included previous data to support sample size calculation. None of the trials considered interim analyses and associated stopping rule, even though a few trials had adopted large sample sizes and should have the opportunity to conduct close monitoring of trials through interim analysis.
Analysis of trial data has posed another issue. Our study found that most trials failed to adequately deal with loss to follow-up. Very few had prespecified how the primary outcome was analysed. Even more surprisingly, no trial explicitly considered subgroup analyses and adjustment for potential confounding effects. Although these conditions are not always present in RCTs, particularly those with smaller sample sizes, their absence of consideration further implicates the weak planning and analysis of these trials.
Overall, our study revealed that the published trials testing acupuncture for KOA are at high risk of bias and identified that many methodological details are not carefully considered during the various stages of planning, design, conduct and analysis.
Implications for future trials
Our findings have important implications for future trials of acupuncture for KOA. First, one should centrally randomise patients based on a computer-generated randomisation sequence. Blocked randomisation with variable sizes may also improve balance of sample sizes between groups. In case of important prognostic factors, stratification should be introduced to further improve the balance of these factors.
All acupuncture trials of KOA should predefine primary outcomes in the protocol and follow the CONSORT guidelines (2010) to register the trial at inception. Given the predefined primary outcome, careful calculation of sample sizes should be performed with an appropriate alpha level and statistical power, and including important clinical parameters such as expected treatment effect, baseline risk and expected loss to follow rate. In all possible cases, trial investigators should justify the choices of the primary outcome and parameters for the sample size calculation.
Designing and practicing acupuncture interventions are important. Trial investigators should be explicit in their design of acupuncture procedures. In any case, one should consider standardising issues such as the type of acupuncture, acupoints and other procedural issues (eg, number of needle insertions/treatment sessions and frequency and duration of treatment sessions). The experience of practitioners would be another factor for consideration, and when possible, for standardisation. Ideally, the qualification, training and experience of the acupuncturists providing the treatment should be more or less similar, especially for an exploratory trial.
In parallel, trial investigators should judge whether a blinding technique can be used according to the actual setting because in many situations of acupuncture trials, it is impractical or infeasible to blind the participants or study personnel. Nevertheless, blinding should be encouraged in an explanatory trial. They may adopt appropriate blinding method, such as sham acupuncture, and monitor the performance of blinding regularly to ensure that it is well implemented.
In specifying interventions, trial investigators should also account for the impact of co-interventions, and justify their use during the course of a trial. They must make an effort to minimise cross-overs, so as to maintain interventions as planned, and develop strategies to maximise patient follow-up. In all cases, these issues should be documented in the study protocol.
A statistical analysis plan should be in place prior to undertaking data analysis. Trial investigators should prespecify the analytic strategies to deal with loss to follow-up and the missing data problem. These should be explicit for both primary and secondary outcomes. It would be highly desirable to prespecify any subgroup hypotheses, if relevant, and plan the corresponding adjusted analyses.
There are other issues that trial investigators should take into consideration prior to their studies, such as safety monitoring and specification of interim analysis and stopping rules.
Strengths and limitations
The main strength of this study was its comprehensive and systematic investigation of the methodological details about the design, conduct and analysis of acupuncture RCTs for KOA. Nonetheless, several limitations should be considered. The data extraction form was developed by our research team. Despite the use of trained assessors and pilot testing of the questionnaire, it is likely that some extracted items from the original article source might still incur a certain degree of misinterpretation which could have affected the results. The insufficient reporting of methodological details among the included trials limited our assessment of the methodological quality and identification of specific methodological gaps. In addition, this study did not examine the specification of patient population in the included trials, and the findings may not be applicable to other acupuncture trials. Lastly, this study did not investigate methodological characteristics of trials published in languages other than Chinese and English, which were expected to be scarce in the literature.
Conclusion
The overall risk of bias was high among published RCTs testing acupuncture for KOA. When examining methodological details, important issues were present in every aspect of design, conduct and analyses. These included, but were not limited to, randomisation, standardisation and operation of procedures, blinding, outcome specification and assessment, sample size calculation, setting up early stopping rules and data analysis (such as dealing with loss to follow-up, subgroup and adjusted analyses). Identifying and understanding these important methodological gaps have profound implications for future trials. Our findings support the development of an evidence-based methodological guidance for the design, conduct and analysis of future trials testing acupuncture for KOA.
References
Footnotes
PJ, LT and JY contributed equally.
Contributors XS conceived the study and acquired the funding. PLJ, LT, JJY, DYK and XS developed and tested the questionnaire. PLJ, XZ, JJY and JLL acquired the data and performed risk of bias assessment. PLJ, LT and YNL performed data analysis. PLJ drafted the manuscript. XS, LT, AHL and JJY critically revised the manuscript. All authors read and approved the final manuscript. XS is the guarantor.
Funding This project was supported by the National Natural Science Foundation of China grant No. 81590955.
Disclaimer The funding has no role on any processes of the study (ie, study design, statistical analysis and results reporting).
Competing interests None declared.
Patient consent Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement All relevant data are within the paper and its Supporting Information files.