Objectives There is an increasing prevalence of hypothyroidism and there is a growing body of meta-analyses (MAs) on the association between hypothyroidism and other diseases. However, the methodological quality of the MAs significantly varies. Thus, this study aimed to evaluate and summarise data on the methodological quality of MAs on the associations between hypothyroidism and other diseases using the Assessment of Multiple Systematic Reviews (AMSTAR) scale, providing suggestions for clinical decision-making processes.
Design To assess the methodological quality of MAs using the AMSTAR scale.
Data sources A systematic literature search was performed in PubMed, EMBASE, the Cochrane Library, web of science and Chinese Biomedicine Literature Database.
Eligibility criteria We included MAs that had assessed the association between hypothyroidism and other diseases in humans and that had full texts regardless of the publication status. No restriction applied on language or date.
Data extraction and synthesis Two reviewers independently screened the titles and abstracts of all searched literature to acquire potentially eligible publications. The full texts of possible eligible publications were downloaded and assessed. Inconsistent comments were resolved through discussions with a third reviewer.
Results 52 studies were included. The average AMSTAR score of the included articles was 8.6 (range: 5–10), and those of English and Chinese MAs were 8.8 and 7.0, respectively. A total of 52 MAs were evaluated, and 19 (36.5%) and 33 (63.5%) of these MAs were of moderate and high quality, respectively. None of the MAs were of low quality. Only two MAs had an a priori design. Items 3, 5 and 9 had the highest compliance (50/52, 96.2%), and aside from item 1, items 7 and 8 had the lowest compliance (33/52,63.5%). According to the results of these MAs, hypothyroidism was significantly associated with cardiovascular diseases, metabolic diseases, neuropsychiatric disorders, breast cancer and pregnancy outcome.
Conclusions The methodological quality of the included MAs on the association between hypothyroidism and other diseases was moderate to high. MAs with high qualities confirmed that hypothyroidism was significantly associated with cardiovascular diseases, metabolic syndrome, preterm birth and neonatal outcomes. Consideration of scientific quality when formulating conclusions should be made explicit and more attention should be paid to improving the methodological quality of MAs, and increasing their applicability for clinical decision-making.
- methodological quality
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
This study first assessed the methodological quality of meta-analyses (MAs) on the association between hypothyroidism and other diseases.
We used the Assessment of Multiple Systematic Reviews (AMSTAR) scale to assess the methodological quality of MAs.
The included MAs were randomly selected without restriction and followed rigorous inclusion and exclusion criteria.
One limitation of this study is that the AMSTAR appraisal process was difficult to implement when the reporting quality was poor.
We tried our best to make a conversion between OR and relative ratio by extracting raw data from MAs, but some data were not obtained.
Hypothyroidism is defined as an increase in serum thyroid-stimulating hormone levels, with decreased (overt hypothyroidism [OH]) or normal (subclinical hypothyroidism [SCH]) serum thyroid hormone levels.1 Epidemiological data have indicated that the prevalence of hypothyroidism in the general population ranged from 4.6% to 23.5%,2 3 and is predominant in elderly individuals and women.4 OH is associated with weight gain, dyslipidaemia and hypertension, which are confounders of atherosclerosis and subsequently causes coronary heart disease (CHD).5 Prospective cohort studies have shown that SCH can increase the risk of heart failure (HF),6CHD events and mortality.7 Recently, Canpolat et al 8 have revealed that impaired gastric motility and resultant upper gastrointestinal symptoms were observed in individuals with SCH, and thyroid hormone replacement was beneficial in improving symptoms and changes in clinical indicators. In addition, studies on the effects of SCH on cognitive impairment,9 10 metabolic syndrome (MetS)11 12 and fractures13 14 have shown conflicting results.
Meta-analyses (MAs) have been becoming a popular and powerful type of evidence, and they have several advantages, including overcoming the limitation of small sample sizes and pooling individual study results to generate a single best estimate.15 High-quality MAs can provide guideline developers better evidence to formulate guidelines, show the advantages and disadvantages of an intervention and guide health professionals, clinicians and stakeholders with the best interventions for targeted patients.16 The Cochrane Collaboration is an international network of healthcare professionals who prepare and regularly update systematic reviews (SRs), and Cochrane reviews were generally of high quality.15 In 2016, 73 reviews, which were completed by 11 Cochrane Review Groups, were used in 14 of 18 (78%) WHO guidelines.17 By contrast, in MAs, a poor methodology can negatively affect decision-making processes.15 A great number of MAs that evaluate the role of hypothyroidism in the development and progression of other diseases have been published. However, the methodological quality of these MAs had not been evaluated, and the conclusions of these MAs were debatable.
Assessment of Multiple Systematic Reviews (AMSTAR), which was developed based on previous tools, empirical evidence and expert consensus and can be applied to SRs/MAs of both randomised controlled and non-randomised studies, is the most recent, reliable and valid tool for evaluating MAs.18 19 Gagnier et al have assessed the methodological quality of SRs and MAs in the top five orthopaedic journals using AMSTAR. The included articles did not conform to the accepted standards of quality. Moreover, the validity of the published SRs is questionable, and their contribution to clinical decision-making is suboptimal.19 Remschmidt et al have used the AMSTAR to investigate the methodological quality of the SRs and MAs on influenza vaccination and identify influencing factors.20 Tian et al compared the methodological quality of SRs from China and the USA using the AMSTAR scale and concluded that the overall methodological quality of the SRs from China was similar to that from the USA.21
Thus, this study aimed to evaluate and summarise data on the methodological quality of MAs on the associations between hypothyroidism and other diseases using the AMSTAR scale, providing suggestions for clinical decision-making processes.
Materials and methods
Data sources and study selection
We performed a systematic literature search from inception to 1 April 2018 in PubMed, Embase, the Cochrane Library, Web of Science and Chinese Biomedical Literature Database to identify MAs on the associations between hypothyroidism and other diseases. The combinations of the following keywords were used in the search strategy: ‘thyroid’, ‘hypothyroidism’, ‘subclinical hypothyroidism’ and ‘meta-analysis’. Free-text words and MeSH terms were entered depending on the characteristics of the database. Search strategy is shown in online supplementary appendix 1. To minimise the risk of missing relevant literature, reference lists from retrieved articles were hand screened for additional applicable studies. We did not apply any restriction to language or date.
Inclusion and exclusion criteria
We included MAs that fulfilled the following criteria: (1) MAs that assessed the association between hypothyroidism and other diseases in humans (2) and that had full texts regardless of publication status, (3) studies in abstract form or meeting reports after the authors were contacted and full texts were obtained within a month, (4) recent studies with the same topic from the same author and (5) duplicate copies of MAs in both Chinese and English, including English articles.
Two reviewers (FS and YQ) independently screened the titles and abstracts of all searched literature to acquire potentially eligible publications. The full texts of possible eligible publications were downloaded and assessed. Inconsistent comments were resolved through discussions with a third reviewer (LT).
Assessment of methodological quality of the included MAs
Two authors (FS and YQ) independently assessed the methodological quality of the included MAs. The AMSTAR scale, which uses an 11-item questionnaire, was used in this study to assess the methodological quality of MAs because it is the most recent, reliable and valid tool.22 The items in the AMSTAR scale are the following: an a priori design, duplicate study selection and data extraction, comprehensive literature search, use of the status of publication as an inclusion criteria, a list of included and excluded studies, characteristics of included studies, documented assessment of the scientific quality of the included studies, appropriate use of the scientific quality in forming conclusions, appropriate use of methods to combine the findings of studies, assessment of the likelihood of publication bias and documentation of potential conflicts of interest. The items may be answered with a ‘yes’, ‘no’, ‘can’t answer’ or ‘not applicable’ . One point was provided when the answer was ‘Yes’; otherwise, no score was provided. The AMSTAR quality score was the summation of the number of ‘yes’. According to the number of criteria met, the included articles were ranked into three levels: ‘high’ (range: 9–11), ‘moderate’ (range: 5–8) and ‘low’ (range: 0–4).
Agreement of assessment
To obtain more reliable data on the methodological quality of MAs, the agreement of the methodological quality assessment results between the two independent reviewers (FS and YQ) was investigated through a plot test. In this process, 10 MAs were selected from the included studies. The two reviewers independently assessed the methodological quality of these MAs using the AMSTAR scale. We calculated the agreement proportion and Cohen’s kappa (k) value for each of the 11 AMSTAR items. A k value between 0.81 and 1.00 indicated a good agreement.23 During the first assessment, the agreement for items 7 and 8 between the two reviewers were poor (0.67 and 0.70, respectively), and good agreement was obtained after a discussion between the reviewers.
Given its methodological focus, we did not evaluate patient-related outcomes. Therefore, we also chose not to involve patients’ input in its design. However, the aim of this study is to indirectly benefit the welfare of patients by promoting the development of high-quality MAs.
Initially, we yielded 3107 potentially relevant articles by searching the five electronic databases and other sources. After identifying duplications and screening the titles and abstracts, 3039 articles were excluded. We attempted to find the full texts of the remaining 68 articles for intensive reading. Of these, 12 articles that met the criteria were published as abstracts. Moreover, we aimed to obtain the full texts by contacting the authors. However, the authors did not respond within a month. Thus, these articles were excluded. Two Chinese articles24 25 were excluded from publishing in both Chinese and English languages. In addition, two MAs26 27 were excluded from the old version. Thus, we only included the recently updated ones.28 Finally, 52 eligible MAs were included in our analysis. The flowchart of the review selection process is presented in figure 1.
Characteristics of the included MAs
Among the 52 included eligible MAs, 4628–73 (46/52, 88.5%) were published in English, and the rest74–79 (6/52, 11.5%) were published in Chinese. The 52 included MAs contained 685 studies, with an average of 13.2 (range: 3–39) studies per article. The publication years of the included articles ranged from 2008 to 2018, and 82.7% of the articles were published in the last 5 years. The characteristics of the 52 MAs are shown in online supplementary appendix 2.
Ten39 40 43 49 57 58 65 68–70 MAs were about the associations between OH and other diseases, and 3528–36 38 42 44–47 50–56 59–64 66 67 75–79 MAs have shown the associations between SCH and other diseases. Seven37 41 48 71–74MAs have investigated the effects of both OH and SCH on other diseases. Except for the general population, pregnant women with hypothyroidism were also included in the target groups of the included MAs.35 37 40 41 48 51 61 64 67 71 74 The characteristics of these MAs are shown in table 1.
Methodological quality of the included MAs
The average AMSTAR score of the included articles was 8.6 (range: 5–10), and those of English and Chinese MAs were 8.8 and 7.0, respectively. A total of 52 MAs were evaluated, and 19 (36.5%) and 33 (63.5%) of these MAs were of moderate and high quality. None of the MAs was of low quality. The numbers and percentages of each response (‘yes’, ‘no’, ‘can’t answer’ and ‘not applicable’) among the MAs for each AMSTAR item are shown in table 2. Two MAs had an a priori design. Items 3, 5 and 9 had the highest compliance (50/52, 96.2%), and aside from item 1, items on whether the scientific quality of the included studies was assessed and documented and whether the scientific quality of the included studies has been used appropriately in formulating conclusions had the lowest compliance 35/52 (67.3%). For item 7, the Newcastle–Ottawa Scale, the most commonly used tool for observational studies, was used.
Association between hypothyroidism and other diseases
Conclusions and the number of MAs, case–control (CC)/cohort (CO)/cross-sectional (CS) studies and patients, and average AMSTAR score about the association between hypothyroidism and other diseases are shown in tables 3 and 4.
According to the AMSTAR score and the number of patients included in these MAs, the association between hypothyroidism and other diseases was summarised by showing the relative ratio (RR)/OR/HR/weighted mean difference/standardised mean difference (tables 5 and 6). Regarding the association between SCH and other diseases, diabetic peripheral neuropathy had the highest RR value (2.27) and OR value (1.87). The association between OH and breast cancer had the highest OR (1.79). However, this value is not considered statistically significant.
In view of the growing expansion of MAs that are related to the association between hypothyroidism and other diseases, the methodological quality of these articles was investigated. To the best of our knowledge, this study first used the AMSTAR tool to evaluate the quality of MAs in this field. The AMSTAR tool is the recommended scale for assessing design progress and minimising bias in research.
The methodological quality of the included MAs was moderate to high. Various flaws regarding the information in the included articles were found, and the main problems were the following.
First, to increase scientific credibility and improve research standards, a standard protocol for MAs must be designed. A protocol can make the research process prospective, strict and transparent. In this study, only two MAs61 72 provided an a priori study design. Whether the authors had been influenced by the published articles during a certain process in preparing the MAs is challenging to assess. Therefore, more attention should be provided in drafting the protocol.
Second, to avoid inappropriate inclusion or exclusion of articles and minimise selection bias, the AMSTAR scale recommends that at least two independent reviewers should obtain results and data. In this study, eight (15.4%) studies did not follow this recommendation, which may increase selection bias and decrease the quality of the included MAs.
Moreover, the AMSTAR scale suggested that a comprehensive search strategy with a systematic search plan and wide retrieval range should be developed. The former refers to keywords and retrieval types, and the latter indicates that at least two databases should be searched. In addition, item 3 emphasised that supplemental searching is also significantly important. Similarly, to ensure a comprehensive search and decrease selection bias, the fourth AMSTAR scale recommends that the status of publication should not be restricted, which include grey articles and published language. The authors are all willing to obtain positive outcomes, and journals published articles with positive outcomes. However, articles with negative outcomes are not usually published or only published in informal journals. The inclusion of grey articles is significant in decreasing selection bias. For item 3, one44 article was not in accordance to this item. However, for item 4, 15.4% (8/52)28 37 45 48 51 57 76 77 of the studies did not consider using published literature, which might result in selection bias.
The AMSTAR scale also suggests that a list of both the included and excluded articles should be provided. The included and excluded articles list can explicitly show the qualities of the selected articles. However, due to restricted page layouts, apart from the Cochrane MAs, most articles only listed the included articles. In this study, 51 MAs included an articles list, and one49 article had neither included nor excluded a list of articles.
The sixth item focuses on whether the characteristics of the included studies were provided. In this study, most (96.1%) of the included articles described subject, gender, age and other characteristics of included articles.
Items 7 and 8 recommend the assessment of the scientific quality of the original articles and consideration of the scientific quality when formulating conclusions, which should be based on scientific and cautious research results to provide an objective and reasonable suggestion for clinical professionals and stakeholders. Nineteen (36.5%)31 33 36 37 39 44 50 53 56 57 59 61 62 70–72 77–79 MAs did not assess the scientific quality of the included studies, which might decrease the credibility of the conclusions.
Thus, publication bias should be assessed in every MA according to the Cochrane handbook for the SRs of interventions,80 which can be performed by using a certain statistical test method or chart.81 Among the 52 included articles, seven29 40 50 54 57 62 65 failed to assess publication bias, and all of which were Chinese articles. Another area of concern is the lack of reports on the conflict of interest among the authors of the MAs. Readers should know whether the results of the MAs are influenced by any funders. In our results, 23.1% (12/52)33 50 62 67–69 74–79 of the authors did not declare any conflict of interest.
The conclusions of the included articles were summarised, and the number of randomised controlled trials/CC/CO/CS studies and patients included in each MA and the average AMSTAR score were analysed.
For the association between OH and other diseases, one MA39 has confirmed that breast cancer was significantly associated with OH. This MA included one, two and three CC, CO and CS studies, respectively. The AMSTAR score was moderate. However, another MA65 has shown that OH was not considered a risk factor of breast cancer, and such study included 24 571 participants and 12 CC study. Moreover, its AMSTAR score was high. Thus, the credibility of the conclusion is relatively high. One article43 has included 10 CO studies and shown that OH was significantly associated with increased cardiac death, hospitalisation and all-cause mortality in patients with HF. The large sample size and high AMSTAR score increased its credibility. Based on the effect of OH on gestational diabetes, one MA37 has included 225 427 patients in three CO studies and shown that OH was a risk factor of gestational diabetes. The AMSTAR score was moderate.
Six MAs40 41 48 64 71 74 have shown that pregnant women with OH were at high risk for preterm birth. The average number of patients was more than one million, and the average AMSTAR score was 8.8 (range: 8–10). By contrast, one MA40 has confirmed that maternal OH might be associated with high and low birth weight and it might reduce the risk of large for gestational age. The AMSTAR score of this article was 8. These data can provide greater power regarding the credibility of the conclusion.
Additionally, OH was significantly associated with stroke,32 left ventricular diastolic function,33 carpal tunnel syndrome,49 obstructive sleep apnea57and plasma homocysteine;58 the AMSTAR scores of MAs were from moderate to high quality. More high-quality evidence is needed to support these results.
For the association between SCH and other diseases, four MAs36 59 66 75 reported that SCH was associated with a significant increase in carotid intima-media thickness. These MAs have an average of 2420 patients, and an average AMSTAR score of 8.75. Larger study containing more patients will make the result more credible. All three MAs30 52 54 have shown that the lack of association between SCH and fractures. A large number of patients and a high AMSTAR score increased the credibility of the conclusion. In addition, one MA60 has confirmed that SCH was associated with increased risk for fractures; therefore, more high-quality evidence is needed.
Two36 42 high-quality MAs have shown that serum total cholesterol, low-density lipoprotein cholesterol and triglyceride levels were significantly higher in patients with SCH than those with euthyroidism. However, no significant difference was observed in serum high-density lipoprotein cholesterol levels. SCH was associated with a significant decrease in fasting plasma glucose levels but not significantly associated with Body Mass Index.36 The AMSTAR score of this article was high. However, a small sample size decreased its credibility.
The included MAs had different conclusions regarding the role of SCH in blood pressure. One MA31has shown that SCH was associated with a significant increase in diastolic blood pressure. However, the other two36 56 had a different conclusion. The former included 6 CS studies with 17 323 patients, and the two other MAs had an average of 1.7 CC studies, 1.7 CO studies and 5 CS studies with an average of 25 810 patients, and the average AMSTAR score for both MAs was 8.0. All three MAs have shown that SCH was associated with an increased systolic blood pressure.
Six MAs28 44 50 63 70 77 have assessed the effects of SCH on CHD and cardiovascular mortality. Among these, three28 44 77 have evaluated the association between SCH and CHD and confirmed that SCH was not significantly associated with an increased risk for CHD, whereas the other two studies50 63 had a conflicting result. Two50 70 MAs have shown that SCH was associated with an increased cardiovascular mortality rate from cardiovascular disease, and another two28 44 have revealed that SCH was not significantly associated with an increased mortality rate from cardiovascular disease. In addition, three studies28 44 50 have concluded that SCH is not significantly associated with an increased risk for all-cause mortality. All the studies had moderate AMSTAR scores. Thus, the association between them should be further validated.
Two MAs34 35 have confirmed that SCH was not associated with MetS, and the AMSTAR scores were high. Three MAs29 45 47 have investigated the effect of SCH on cognitive function and shown that SCH was not significantly associated with cognitive impairment. A large number of studies and patients and a high AMSTAR score increased the credibility of this conclusion. In addition, two MAs38 53 have reported that SCH significantly increased the risk of complications from diabetes.
One MA48 has shown that maternal SCH is not significantly related with the occurrence of preterm birth, whereas the other four41 64 71 74 with more patients had a conflicting result, which showed that SCH increased the risk of preterm birth. Two MAs51 64 have shown that maternal SCH is significantly associated with the risk for intrauterine growth restriction, and these MAs had a moderate AMSTAR score. Three MAs35 64 76 have shown that maternal SCH also had a significant adverse effect on the intelligence of an offspring. However, the sample size of these MAs was small.
Two MAs68 69 have indicated that individuals with OH have an increased risk of developing glaucoma. These MAs included 9 CC studies, 4 CO studies and 11 CC studies, and had an average AMSTAR score of 9. One MA72 has shown no significant association between non-alcoholic fatty liver disease as well as SCH and OH. However, another study73 has revealed that both individuals with SCH and OH are at higher risk for non-alcoholic fatty liver disease than those with euthyroid; therefore, more high-quality evidence is required.
In summary, when evaluating the association between hypothyroidism and other diseases, the hypothyroidism activity index, follow-up duration, baseline demographic data and clinical characteristics should also be considered in evaluating the role of hypothyroidism in other diseases.
The methodological quality of the included MAs on the association between hypothyroidism and other diseases was moderate to high. MAs with high qualities confirmed that hypothyroidism was significantly associated with cardiovascular diseases, MetS, preterm birth and neonatal outcomes. Consideration of scientific quality when formulating conclusions should be made explicit and more attention should be paid to improving the methodological quality of MAs, increasing their applicability for clinical decision-making.
Strengths and limitations
To the best of our knowledge, this study first assessed the methodological quality of MAs on the association between hypothyroidism and other diseases. The included MAs were randomly selected without restriction and followed rigorous inclusion and exclusion criteria. Our study has several limitations. First, the AMSTAR appraisal process was difficult to implement when the reporting quality was poor. This could be attributed to space restrictions in some journals. Second, we tried our best to make a conversion between OR and RR by extracting raw data from MAs, but some data were not obtained.
Contributors LT, QG and CG conceived and designed the study. FS and YQ performed the experiments. FS wrote the paper. LT and CG reviewed and edited the manuscript. All authors read and approved the manuscript.
Funding This research received the grant from National Natural Science Foundation of China and Province Natural Science Foundation of Gansu (grant nos and 17JR5RA041).
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available.
Patient consent for publication Obtained.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.