Author + information
- †Institute for Clinical Evaluative Sciences, University of Toronto, Toronto, Canada
- ‡Peter Munk Cardiac Centre, University Health Network, Toronto, Canada
- §Institute for Health Policy, Management, and Evaluation, University of Toronto, Toronto, Canada
- ‖Joint Department of Medical Imaging, University of Toronto, Toronto, Canada
- ↵∗Reprint requests and correspondence:
Dr. Douglas S. Lee, Senior Scientist, Institute for Clinical Evaluative Sciences, Associate Professor of Medicine, University of Toronto, Institute for Clinical Evaluative Sciences, 2075 Bayview Avenue, Room G-106, Toronto, Ontario M4N 3M5, Canada.
Despite the remarkable advances in the treatment of acute and chronic heart failure (HF), prognosis assessment remains an ongoing challenge. Multiple cardiovascular diseases lead to the development of HF. Treatment options for these diseases are evolving, which leads to greater numbers of patients reaching the final state of HF at a later age with a greater comorbidity burden. The multiple concomitant diseases that coexist in HF patients also have an impact on prognosis.
Although HF patients demonstrate progressive decline, there is heterogeneity in the trajectory of the decline among HF patients. Along this trajectory, there are multiple factors that can influence prognosis, which poses great challenges in risk prediction. Reasons for this phenomenon are multiple, complex, and overlapping, but it is recognized that HF demonstrates acute decompensations that punctuate a chronic and progressive decline. The predictors of outcomes in these 2 stages, acute versus chronic, are different and distinct (1). Consequently, there is a need for different predictive models to guide prognosis estimation in the acute emergency department/hospital-based setting and in the clinic.
Given the high mortality and morbidity and the potential costs and health care burden associated with HF, there is great need for methods of prognostication at multiple points in the life course of the patient. Improved ability to identify high- or low-risk patients allows for matching of more or less intensive care to the baseline risk of the patient, termed the risk-treatment mismatch (2).
In this issue of JACC: Heart Failure, Collins et al. (3) present a predictive model intended for use in acute HF. The STRATIFY decision tool assesses 30-day probability of severe adverse events in acutely decompensated HF patients seen in an emergency department (3). The model was derived from prospectively collected data on 1,033 patients admitted with acute HF enrolled over a period of almost 5 years from 4 emergency departments in the United States (3). The model differs from a large prior risk prediction decision support algorithm that was derived and validated in >12,000 patients, the Emergency Heart Failure Mortality Risk Grade (EHMRG) (4). EHMRG was designed to predict the specific outcome of 7-day mortality among those who were nonpalliative and not chronically dialysis dependent.
The primary outcome of the STRATIFY model was a 6-level ordinal variable that consisted of the most severe physician-rated adverse events, including, in descending order of importance, death, cardiopulmonary resuscitation, mechanical cardiac support, mechanical ventilation, emergent dialysis, and acute coronary syndrome or need for coronary revascularization. On the basis of clinical relevance and availability, the authors identified 57 candidate predictor variables. The final model included 13 variables, which were selected using pre-conditioning and best model performance based on the Akaike information criterion after backward variable elimination. The variables in STRATIFY represented readily available clinical characteristics, including demographics, medical history, laboratory values, and electrocardiographic parameters, and were combined to estimate the risk of 30-day adverse events.
Of the 13 variables included in the model, 2 (blood urea nitrogen and troponin) were significantly associated with adverse events, whereas the remainder (age, body mass index, diastolic blood pressure, respiratory rate, sodium, brain natriuretic peptide, oxygen saturation, QRS duration, use of supplemental oxygen, use of dialysis, and use of angiotensin-converting enzyme inhibitor) were not independent predictors of adverse events. The discriminative performance of STRATIFY was modest (optimism-corrected C statistic 0.68) but improved for women (C statistic 0.74) and patients with preserved left ventricular ejection faction (C statistic 0.72). A contrasting model is the EHMRG30-ST, an extension of the EHMRG risk algorithm designed specifically to predict the occurrence of death within 30 days of emergency department presentation among nonpalliative patients (5). For this specific outcome, there was very good discrimination for 30-day mortality, with a C statistic of 0.801 (5).
There are a number of strengths to the STRATIFY model for HF. It is derived from a prospective, real-world population, which is in contrast to clinical trial cohorts, which are potentially highly selected. This leads to enhanced generalizability and more accurate risk predictions in external cohorts. The model was developed with a high degree of statistical rigor, with a relatively small amount of missing data in the candidate variables.
The model is interesting because 11 of the 13 predictors were not significantly associated with the outcome based on nominal level of statistical significance. Only 2 of the variables in the STRATIFY model were significantly associated with the composite ordinal outcome: blood urea nitrogen and troponin. It is true that both clinical and statistical significance are considered when prediction models are developed, but the effect of including variables that are predominantly not statistically significant is unclear. A significant proportion of the 30-day events were coronary disease related, and it is unclear whether troponin elevations are on the causal pathway to the outcome. Similarly, elevated blood urea nitrogen and prior dialysis are likely highly correlated with the need for emergent dialysis. Because the above 2 events are the least severe events on the ordinal outcome scale, one might question what significant model covariates might remain if the primary outcome was limited to death or more serious outcome events.
The ordinal scale of outcomes was determined by physicians. It included severe events such as death, less severe morbid events, and procedures such as dialysis and coronary revascularization procedures. A hierarchical approach to outcomes evaluation was used in which death was the most important outcome, but one questions whether coronary revascularization is an adverse event, because many patients have ischemic HF. In these patients, coronary revascularization is considered a therapy used to treat the underlying cause of HF and not an adverse event (6). Furthermore, because the objective of STRATIFY is to enhance shared decision making, it would be of great interest to know whether patients would rate outcomes in the same order of priority as physicians. In particular, would HF patients consider coronary revascularization and dialysis as an adverse event in the same way that physicians did?
The implications of the model for health policy are unclear. Collins et al. (3) state that the negative predictive values were 100%, 96%, and 93% at risk thresholds of 1%, 3%, and 5%, respectively. This leads to the question of what is the right threshold that optimizes the tradeoff between an acceptably low risk of events and the proportion of patients who could be discharged home. At a 5% threshold, 13% of patients could be discharged home, but is a 5% risk of events low enough to be accepted by the patient and the physician? At the lowest risk threshold, the negative predictive value is 100%, but then few patients would be considered at low enough risk to be discharged home without hospital admission. At the opposite end of the spectrum, it is unknown whether STRATIFY will be useful to predict higher-risk patients. There is some uncertainty at the upper end of the risk range given the small numbers in the highest-risk categories.
It is unknown whether clinicians will use a predictive risk index. This will depend, in part, on its clinical utility in helping them make better decisions for acute HF. It is also unknown whether the model will truly assist shared decision-making practices. The authors have derived a new model for HF prognosis and performed an internal validation using the bootstrap; however, there is a need to prospectively validate the STRATIFY model in an external clinical dataset to demonstrate that the present model also predicts outcomes in an independent cohort of patients. The model’s impact will also need to be demonstrated. Specifically, whether use of the model changes physician behavior, patient disposition, and outcomes needs to be demonstrated.
Given the complexity of prognosis assessment in HF, methods to improve risk prediction are indispensable. They are potentially useful adjuncts to decision making by clinicians, patients, and families throughout the life course. However, to move the field forward, we need more evidence that use of risk prediction methods has an impact on patient care, physicians’ decision making, outcomes, or costs. Derivation of predictive tools such as STRATIFY is a promising initial step in this process toward demonstration of potential future impact.
↵∗ Editorials published in JACC: Heart Failure reflect the views of the authors and do not necessarily represent the views of JACC: Heart Failure or the American College of Cardiology.
Both authors have reported that they have no relationships relevant to the contents of this paper to disclose.
- American College of Cardiology Foundation
- Alba A.C.,
- Agoritsas T.,
- Jankowski M.,
- et al.
- Collins S.P.,
- Jenkins C.A.,
- Harrell F.E. Jr..,
- et al.
- Greig D.,
- Austin P.C.,
- Zhou L.,
- et al.
- Braga J.R.,
- Tu J.V.,
- Austin P.C.,
- et al.