Introduction For patients with a first unprovoked venous thromboembolism (VTE), the optimal duration of anticoagulation is a crucial clinical dilemma which has yet to be resolved. The decision to stop anticoagulant therapy (AT) after the initial 3–6 months or to continue AT indefinitely, is primarily governed by the long-term risk of recurrence when treatment is discontinued. This risk however, is not well established, hindering decision making.
Methods and analysis We will conduct a systematic review and a meta-analysis of studies involving patients diagnosed with a first, symptomatic unprovoked VTE or VTE provoked by minor transient risk factors, who have completed at least 3 months of initial AT; and who were followed-up for standardised time intervals of 1, 2, 5, 10 and 20 years (±3 months) after stopping AT. We will search (from inception to January 2017) MEDLINE, Embase and the Cochrane library for randomised controlled trials and prospective observational studies. Two reviewers will conduct all screening and data collection independently. The primary outcome of the rate of recurrent VTE at the standardised time intervals will be calculated for each study from the total number of recurrent events and the corresponding number of patient-years of follow-up. We will use a random-effects model to pool study results and report a weighted estimate of the absolute rate of recurrent VTE (events per 100 patient-years) over standardised time intervals of 1, 2, 5, 10 and 20 years after discontinuing anticoagulants.
Ethics and dissemination Ethical approval is not applicable for this study. Findings from this study will be disseminated through peer-reviewed journal publication as well as relevant national and international conference presentations.
PROSPERO registration number CRD42017056309.
- venous thrombosis
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
To the best of our knowledge, this is the first meta-analysis aimed at establishing the absolute, long-term risk of recurrence after discontinuing anticoagulant therapy (AT) in patients with a first unprovoked venous thromboembolism (VTE).
Subgroup analyses will address the influence of important patient characteristics (eg, differences between sexes, and comparison between different presentation of initial VTE and plasma D-dimer levels measured after stopping anticoagulation) on the long-term risk of recurrence after stopping AT.
The variability in clinical definitions of unprovoked VTE and major bleeding is a potential limitation that can cause challenges to be encountered in this review. Included studies will be prudently examined for their definition of unprovoked VTE and major bleeding and results will be interpreted by taking the potential for heterogeneity into consideration.
Deep vein thrombosis (DVT) and pulmonary embolism (PE), jointly denoted as venous thromboembolism (VTE), comprise a treatable yet burdensome condition.1 2 It is recommended that anticoagulant therapy (AT) be continued for at least 3 months in all patients with VTE.3 Thereafter, approximations of the projected long-term risk of recurrent VTE off anticoagulation, risk of major bleeding resulting from continued AT as well as patient preferences are crucial to decide the optimal duration of treatment.
The primary driver of the risk of recurrence after discontinuing anticoagulants is the aetiology of the first VTE episode.4 In patients with VTE provoked by a major transient risk factor (eg, postmajor surgery), initial treatment can be limited to 3 months, as the annual risk of recurrence is only 1% after stopping anticoagulation.3
Nevertheless, approximately half of VTE cases occur without an identifiable major or minor transient provoking risk factor (ie, unprovoked),1 2 in which the risk of recurrent VTE after cessation of AT is roughly estimated to be 5%–10% after 1 year, and 30% after 5 years,3 with a case-fatality rate (CFR) of 3.6%.5 In such patients, extending treatment with vitamin K antagonists (VKAs) or direct oral anticoagulants (rivaroxaban, apixaban, edoxaban, dabigatran), beyond the initial 3 months, significantly reduces the risk of recurrence by >80%–90%.6 The literature suggests that the risk of VTE recurrence after stopping AT is not significantly influenced by the duration of initial treatment (ie, 3 months vs a longer course of AT achieves a similar risk of recurrence after treatment is discontinued),7 and a longer, fixed-time duration of treatment will only delay recurrent events.1 8 9 As such, after 3–6 months of initial treatment, patients should either stop or continue AT indefinitely.1 3 4
Given the seemingly high risk of recurrence after cessation of AT, current guidelines suggest that all patients with unprovoked VTE, with non-high bleeding risk, should be considered for indefinite anticoagulation.3 10 This recommendation however, is based on limited data. Furthermore, less than half of all patients with unprovoked VTE are expected to have a recurrence within 10 years of stopping anticoagulation.11 12 As such, considering indefinite AT in all patients with unprovoked VTE inevitably exposes a large proportion of such patients to an unwarranted risk of major bleeding,1 associated with an estimate CFR of 11.3%,5 which is twofold to threefold higher than the CFR of recurrent VTE.
Hence, a high-priority question remains to be answered: should patients with unprovoked VTE continue AT indefinitely, or can they safely discontinue anticoagulants after completing 3–6 months of initial therapy? As mentioned previously, although not the only important factor to consider (ie, risk of major bleeding from continuing anticoagulation, and patient preferences), this decision predominately rests on the poorly established, long-term risk of VTE recurrence after stopping anticoagulation.
Based on the projected long-term mortality rates associated with recurrent VTE and major bleeding, the International Society on Thrombosis and Haemostasis (ISTH) suggests that discontinuation of AT is justified when the annual risk of recurrence is lower than 5% (upper bound limit of 8%) in the first year, and 15% (upper bound limit of 24%) in the first 5 years after stopping treatment.13 Consequently, in an effort to identify patients with unprovoked VTE in whom indefinite AT may not be indicated, various clinical and laboratory variables have been explored to stratify such patients into low risk and high risk of recurrence off anticoagulation. Of such variables, the strongest predictors of the risk of recurrence appear to be gender,14 site of index VTE7 and plasma D-dimer levels measured 1–2 months after AT is discontinued.15 However, risk stratification using individual predictors alone has failed to identify a group of low-risk patients in whom anticoagulation can be safely discontinued. Incorporation of such variables has led to the derivation of promising clinical prediction scores including the Vienna Prediction Model,16 the DASH score17 as well as the ‘Men continue and HERDOO2’ rule, which is the only clinical decision rule to date that has been prospectively validated.18 19
A previous individual patient-data meta-analysis of seven randomised controlled trials (RCTs) conducted by Boutitie et al 7 quantified the risk of recurrent VTE within 24 months after stopping AT in this patient population, but over varying durations of initial treatment. Furthermore, the latest trial of patients with unprovoked VTE included in the analysis by Boutitie et al was published in 2003. Since then, numerous clinical trials and prospective observational studies have been published, with follow-up beyond 24 months and up to 20 years after discontinuing anticoagulants. First, this offers an opportunity to more accurately define the risk of VTE recurrence within 12 and 24 months after stopping AT. Second, and more importantly, for deciding whether patients with unprovoked VTE should continue treatment indefinitely, understanding the long-term risk of recurrent VTE after discontinuing anticoagulation is crucial; a meta-analysis of the risk of recurrence in this patient population, over a longer time interval (up to 20 years), at prescribed time points after stopping AT and irrespective of the duration of initial therapy, has never been conducted.
The objective of this systematic review is to establish the absolute, long-term risk of recurrent VTE at standardised time intervals of 1, 2, 5, 10 and 20 years after stopping anticoagulation in patients with a first episode of unprovoked VTE.
The proposed project is called Meta-Analysis of the long-term Risk of recurrent Venous thromboEmboLism after stopping anticOagulation for acute Unprovoked venous thromboemboliSm (MARVELOUS). This protocol has been registered in the PROSPERO international register of prospective systematic reviews database (CRD42017056309). Its contents have been drafted with contributions from all members of the authorship team.
Studies to be incorporated into this systematic review will be selected based on the criteria specified.
The study population of interest will include adults (aged 18 years or older) who have experienced a first episode of objectively confirmed, symptomatic VTE that is either unprovoked or provoked by weak/minor transient risk factors (as defined per ISTH guidance on categorisation of VTE)20; and who have completed at least 3 months of initial AT. Eligible patients should initially be treated with either 1) rivaroxaban or apixaban or 2) intravenous heparin/low molecular weight heparin injections administered for at least 5 days followed by VKA, dabigatran or edoxaban. Studies that limited their analyses to populations with certain diseases or conditions such as VTE in critically ill patients, or patients with VTE provoked by a major transient and/or persistent risk factor (ie, active cancer), as defined per ISTH guidance on categorisation of VTE,20 will be excluded. Moreover, there will be no restrictions based on the type of study setting (eg, urban vs rural centre, community hospital vs academic hospital, North America vs Europe).
Studies in which AT is stopped in eligible patients who have completed at least 3 months of initial therapy, as per the treatment strategy described above; and followed-up for the duration of standardised time intervals of 1, 2, 5, 10 and 20 years will be included. Studies that do not systematically stop AT in eligible patients, or studies that conclude follow-up at the time of stopping AT will be excluded. To accommodate studies in which follow-up pertaining to exactly the time intervals specified is unavailable, a rule of (year±3 months) will be applied. For example, in reporting the event rate within the first year after stopping anticoagulation, studies in which eligible patients were followed up for a minimum of 9 months after stopping AT would be included, whereas studies with only 1 month or 6 months of follow-up after stopping AT will be excluded. Lastly, studies in which the decision to stop AT was influenced by a potential predictor (ie, D-dimer level to decide whether to stop or continue anticoagulation), will be excluded.
Given the well-established evidence that extended anticoagulation is highly effective at reducing the risk of recurrence,6 and that there is no significant influence of different durations of initial treatment on the risk of recurrence after discontinuing anticoagulants,7 there is no predefined comparator for this meta-analysis. As such, each study arm of an eligible RCT will be treated as two separate cohorts with follow-up beginning at the time that anticoagulants were discontinued.
The primary outcome measure will be the rate of symptomatic, objectively confirmed, recurrent VTE after stopping AT. The secondary outcome will be the event rate of major bleeding episodes after stopping AT.
We will include RCTs and prospective cohort studies. Studies with other types of designs, including retrospective cohort studies, case-control studies, cross-sectional studies, case series and case reports will be excluded.
A comprehensive literature search will be conducted in the MEDLINE and Embase databases on the Ovid platform. The Cochrane Central Register of Controlled Trials will also be searched for relevant literature. There will be no restrictions on language or time period. Search terms will be categorised into four major concept groups: disease terms, intervention terms, those related to secondary prevention and type of study design. Medical subject heading terms will be used and supplemented by keywords, with vocabulary and syntax adjusted across databases. A search strategy (available in online supplemetary appendix 1) using the MEDLINE database (from inception to January 2017) was developed by an experienced information specialist. This strategy will be modified as needed to search other databases. Additional references will be sought by hand-searching the bibliographies of relevant articles.
Study selection process
Literature search results will be deduplicated in RefWorks21 and uploaded to Covidence,22 an online software used to conduct abstract and full-text screening as well as data extraction. The inclusion and exclusion criteria used for screening will be pilot-tested on a subset of search results. Two independent reviewers will conduct title and abstract and full-text screening procedures. Conflicts in screening will be resolved by consensus or by a third person. Search results and study selection will be illustrated in a Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram,23 with reasons specified for excluding articles during full-text screening.
A data extraction form will be developed to facilitate the process of data collection from the studies that have been included in the review. The form will be pilot-tested on Covidence to identify any data that may be irrelevant or missing from the form. Two reviewers will independently extract all data and a third person will verify a subset of the studies to ensure that information about the general characteristics and outcomes of the study are accurate. Discrepancies between reviewers will be resolved by consensus or by a third person if necessary. The following information will be attempted to be collected from all included studies (as/if available):
Study information: reference ID, authors, year of publication, journal information, publication status.
Study characteristics: study design, country, study period, sample size, funding, definition of VTE that is unprovoked or provoked by minor transient risk factors, types of thrombophilia excluded, duration of initial AT.
Participant characteristics (% males, mean age, site of initial VTE event (number of isolated proximal DVT, isolated distal DVT, isolated PE, DVT±PE, PE±DVT, PE+DVT)), and number of patients lost to follow-up.
Intervention characteristics: type of initial anticoagulant agent, dose, duration of therapy before stopping AT, discontinue treatment versus placebo versus aspirin, as applicable.
Outcomes: number of patients at risk, and the corresponding patient-years of follow-up at standardised time intervals of 1, 2, 5, 10 and 20 years from the date of stopping AT, criteria used for diagnosis of recurrent VTE, total number of recurrent VTE events (isolated proximal DVT, isolated distal DVT, isolated non-fatal PE, fatal PE, DVT±PE, PE±DVT, PE+DVT), criteria used for major bleeding and the total number of major bleeding events, at standardised time intervals of 1, 2, 5, 10 and 20 years from the date of stopping AT.
Potential sources of confounding (eg, age, weight) for subgroup analyses of gender, site of initial VTE, post-treatment D-dimer level, will be identified, and taken into account if possible.
Assessing the quality of studies
Critical appraisal of the methodological quality of studies is an essential element of systematic reviews. Hence, the methodological quality and risk of bias of eligible studies will be carefully and rigorously assessed. The risk of bias for each study will be ascertained by two reviewers, and a subset of studies will be verified by a third person. Disagreements between reviewers will be resolved by consensus or by a third person if required. RCTs will be appraised using the Cochrane Collaboration’s tool (if/when applicable) for assessing risk of bias24 and reported on a per study basis. For studies that have used a cohort design, the Newcastle-Ottawa Scale (NOS)25 will be used. Additionally, since each eligible arm of included RCTs will be treated as a single, individual cohort, and no comparisons will be made between outcomes of different treatment arms of an RCT, the risk of bias of RCTs will also be appraised by the NOS, if/when applicable. Findings from risk of bias assessments will inform sensitivity analyses in order to investigate whether differences in the methodological quality of studies have influenced the findings of the present systematic review.
Management of missing data
If data are not directly reported, they will be requested from the primary investigator of the study. In the case where the total patient-years accumulated at each specified time interval are unavailable, they will be estimated from the number of patients at risk at each specified time interval (when available/provided) with the assumption that patients who did not complete a follow-up period (eg, died or were lost to follow-up) were observed for half of the interval. This assumption will be verified with the corresponding study investigator. Otherwise, analysis will be conducted on the final available data, and the potential impact of the missing data will be discussed as a limitation.
Characteristics of eligible studies will be summarised and presented in a table in the final report. One of the main objectives of this systematic review is to combine data from pertinent RCTs and prospective cohort studies to generate a pooled estimate of the absolute rate of recurrent VTE over standardised time intervals of 1, 2, 5, 10 and 20 years after stopping AT. Prior to pooling results, the research team will assess studies for clinical and methodological heterogeneity through comparison of important study characteristics including those related to the study design, patients and interventions. The degree of statistical heterogeneity will be measured and interpreted using a combination of Cochrane’s Q (statistically significant at p<0.10) and the I2 statistic (>50% considered substantial). An I2 value >75% is indicative of a very high degree of heterogeneity, and if encountered the data will not be pooled. If homogeneity among studies is judged as satisfactory, then the results from trials will be pooled using standard meta-analysis procedures.
Should pooling of results be limited by heterogeneity, subgroup analyses as well as meta-regression will be explored to account for differences in study characteristics. In this scenario, important variables that will be considered for subgroup analyses include type of study, type of initial anticoagulant administered, definition of unprovoked VTE and risk of bias; these variables are expected to account for clinical or methodological heterogeneity. A random-effects meta-regression has been planned using the variables specified; this promotes the consideration of residual heterogeneity that may not be accounted for by study-level factors.26 Based on published recommendations for meta-regression,26 there should be at least 10 studies for each continuous variable and at least 4 studies per subgroup for categorical variables in the model.
Data from RCTs and prospective cohort studies will also be analysed and pooled separately if applicable. In the event that quantitative pooling of cohort study data is inappropriate (due to clinical and methodological heterogeneity), findings from these studies will be compiled to synthesise a narrative. StatsDirect27 will be used for meta-analyses and SAS software (V.9.4; SAS Institute, Cary, North Carolina, USA) will be used for meta-regression.
Measures of treatment effect
Dichotomous outcome data (rate of recurrent VTE and major bleeding) will be quantified in terms of absolute risk. Due to variation in the duration of follow-ups in different studies, and the well-recognised fact that the risk of recurrence off anticoagulation varies considerably with time, standardised time intervals will be used to collect outcome data. The rate of recurrence in the study cohorts will be calculated from each study based on the number of recurrent VTE events and the total number of patient-years of follow-up accumulated during the standardised follow-up periods after stopping AT. For RCTs, each study arm will be treated as a separate cohort. The total number of recurrent VTE events and number of patient-years across all studies will be pooled together to obtain a weighted estimate of the absolute rate (events per 100 patient-years) over standardised time intervals of 1, 2, 5, 10 and 20 years. Additionally, the absolute rate of recurrent VTE at the standardised time intervals will be used to estimate the cumulative probability of recurrent VTE, if possible, at the standardised time intervals of 1, 2, 5, 10 and 20 years. In the context of the study question, reporting the absolute and the cumulative risk is essential to describe the effect of no intervention within a specified period of time.
Secondary outcomes and planned Subgroup analyses
As stated above, rate of major bleeding after stopping anticoagulation will be assessed as a secondary outcome. Additionally, subgroup analyses of the rate of VTE recurrence based on patients stratified by gender (men vs women), plasma D-dimer level measured after stopping anticoagulation (normal/negative vs positive/abnormal as defined by individual studies) and the site of initial VTE (isolated proximal DVT vs isolated distal DVT vs isolated PE vs DVT±PE, PE±DVT, PE+DVT) are planned. Moreover, a subgroup analysis will be performed based on risk of bias of included studies.
Planned sensitivity analyses
We acknowledge the potential for outliers among the larger collection of studies with respect to the effect estimates. As part of the sensitivity analyses, such outliers will be removed from meta-analysis and the change in summary estimates will be recorded. Substantial changes in summary estimates will prompt the investigation of the outlier, with a particular focus on possible factors that explain the variation.
Additionally, although not an anticoagulant, aspirin has been shown to reduce the risk of recurrent VTE by approximately 30% as compared with placebo in patients with unprovoked VTE.28 As such, a sensitivity analysis excluding data pertaining to studies in which patients received aspirin after discontinuing AT will be performed.
Further sensitivity analysis will entail addressing the risk of bias present among included studies. Studies that are adjudicated to have high risk of bias will be excluded and a summary effect estimate will be generated from pooling studies adjudicated to have low risk of bias. The overall estimate and the results from the studies with low risk of bias will be compared to gauge the impact of potential biases on the primary results.
Evaluating the quality of evidence
The risk of bias for outcomes will be considered during the appraisal of the quality of the evidence. The overall risk of bias in the body of evidence will be assessed using the approach specified by the Grading of Recommendations Assessment, Development, and Evaluation (GRADE).29 The GRADE judgement takes several factors into account including the risk of bias, imprecision, inconsistency, size of the effect, effect of residual confounding, presence of a dose-response relationship and publication bias.30 The collective effect of these factors are combined to arrive at a conclusion regarding the quality of the overall body of evidence, rather than the individual studies. Based on the GRADE assessment, the quality of the body of evidence will be designated into one of the following categories:
High: we are confident that the true effect lies close to that of the estimate of the effect.
Moderate: the true effect is likely to be close to the estimate of the effect; however, there is a chance that it is considerably different.
Low: confidence in the estimate of the true effect is limited; the true effect may be substantially different from the estimate.
Very low: we have very little confidence in the estimated effect; it is likely to be significantly different from the true effect.
In this systematic review and meta-analysis, we aim to establish the absolute, long-term risk of recurrent VTE over standardised time intervals of 1, 2, 5, 10 and 20 years after stopping AT in patients with a first unprovoked VTE. A clear-cut estimate of this risk will substantially help synthesise and supplement the existing body of evidence needed for informing the clinical management of this patient population. Additionally, findings from our analysis will serve as basis for an economic evaluation of short-term versus indefinite anticoagulation to answer the question of the optimal duration of AT for such patients. Thus, our systematic review is a crucial step forward in providing evidence-based knowledge needed for clinical decision making in counselling and managing patients with a first unprovoked VTE. Nevertheless, it is important to highlight that our study does not address the other key aspect of the decision process regarding treatment duration for this patient population, namely the long-term risk of major bleeding under ongoing anticoagulation, which remains uncertain.
Limitations and challenges
Heterogeneity of major bleeding and unprovoked VTE definitions across studies
Clinical definitions of major bleeding are variable across RCTs and cohort studies. This is a potential limitation that can cause challenges to be encountered in this review. Studies will be carefully examined for their definition of major bleeding and results will be interpreted by taking the potential for heterogeneity into consideration. The definition of major bleeding will be classified into three categories based on reviewers’ judgement: according to the ISTH,31 similar to the ISTH definition or different from ISTH.
Furthermore, since the definition of unprovoked VTE, as well as the strategy used for the diagnosis of recurrent VTE may vary between studies published over time, studies will also be carefully examined for their definition of unprovoked VTE and recurrent VTE, and if possible/applicable, results will be interpreted by taking the potential for heterogeneity into consideration. The definition of recurrent VTE will be classified into three categories based on reviewers’ judgement: according to the ‘gold standard’, similar to the ‘gold standard’, different from the ‘gold standard’. The definition of recurrent VTE that would be used as ‘gold standard’ is described in online supplemetary appendix 2.
This protocol was developed using the PRISMA for Protocols checklist.32 33 The systematic review will be reported based on the PRISMA statement and its quality will be monitored using the Measurement Tool to Assess the Methodological Quality of Systematic Reviews tool.34 Any modifications made to this protocol will be reported and justified in the final report.
Ethics and dissemination
Ethical approval and patient consent are not required since this is a meta-analysis based on published studies. The results of this study will be submitted for presentation at relevant national and international conferences, and for publication in a peer-reviewed journal.
FK, MC, CK, SS and MAR are members of the Canadian Venous Thromboembolism Clinical Trials and Outcomes Research (CanVECTOR) network. FK is supported by a Canada Graduate Scholarship from the Canadian Institutes of Health Research (CIHR), and Drug Safety and Effectiveness Cross-Disciplinary Training (DSECT) award from the CIHR.
Twitter @Faizank91 @RodgerMarc
Contributors FK and MAR conceived the idea and design for this systematic review. FK, AR, BH, GW and MAR developed the methodology for the systematic review protocol. The contents of this manuscript were drafted by FK, AR, MC, BH and MAR with input from all members of the authorship team. The manuscript was reviewed by CK, SS, FC, PP, SE, CB, GA, HRB, TAB, GP, LP, ES and GAW for important intellectual content. All authors read and approved the final manuscript.
Competing interests MC reports receiving research support form Leo Pharma and BMS, and honoraria from Pfizer, Bayer, BMS, and Sanofi, outside the submitted work. SS reports receiving honoraria from Boehringer Ingelheim, Bayer HealthCare, Daiichi Sankyo and Sanofi, and research support from Boehringer Ingelheim, Baxter and Octapharma, outside the submitted work. FC reports having received research grant support from Pfizer, honoraria for board memberships or symposia from Bayer and AstraZeneca, and travel support from Bayer, Daiichi Sankyo, Leo Pharma, Intermune, and Actelion, outside the submitted work. PP reports receiving consultancy and lectures fees from Bayer Pharma, Sanofi, Daiichi-Sankyo and Pfizer, outside the submitted work. CB reports receiving lectures fees from Bayer HealthCare, Bristol Meyer Squibb and Boehringer Ingelheim, outside the submitted work. GA reports personal fees from Bristol-Myers-Squibb, Pfizer, Bayer Healthcare, Boehringer Ingelheim, and Daiichi Sankyo, outside the submitted work. HRB reports receiving research support and consultancy fees from Sanofi-aventis, Bayer HealthCare, Bristol-Myers Squibb, Daiichi-Sankyo, GlaxoSmithKline, Pfizer, Roche, Isis, Thrombogenics, and Boehringer Ingelheim, outside the submitted work. TAB reports receiving personal fees from Bayer, Bayer Australia, Novo Nordisk, and Glaxo Smith Klein, outside the submitted work. GP reports Advisory Board for Alfa-Wassermann, Daiichi-Sankyo, Pfizer and Roche, and speaker fees from Werfen, outside the submitted work. BH reports receiving honoraria from Cornerstone Research Group for provision of methodologic advice related to systematic reviews and meta-analysis. MAR reports receiving research support from Biomerieux, outside of the submitted work. All other authors (FK, AR, CK, SE, LP, ES and GAW) declare that they have no relevant competing interests.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.