Skip to main content

Simultaneous external validation of various cardiac arrest prognostic scores: a single-center retrospective study

Abstract

Background

This study aimed to compare and validate the out-of-hospital cardiac arrest (OHCA); cardiac arrest hospital prognosis (CAHP); non-shockable rhythm, unwitnessed arrest, long no-flow or long low-flow period, blood pH < 7.2, lactate > 7.0 mmol/L, end-stage chronic kidney disease, age ≥ 85 years, still resuscitation, and extracardiac cause (NULL-PLEASE) clinical; post-cardiac arrest syndrome for therapeutic hypothermia (CAST); and revised CAST (rCAST) scores in OHCA patients treated with recent cardiopulmonary resuscitation strategies.

Methods

We retrospectively collected data on adult OHCA patients admitted to our emergency department between February 2015 and July 2018. OHCA, CAHP, NULL-PLEASE clinical, CAST, and rCAST scores were calculated based on the data collected. The predictive abilities of each score were tested using the area under the curve (AUC) of the receiver operating characteristic (ROC) curve.

Results

We identified 236 OHCA patients from computer-based medical records and analyzed 189 without missing data. In OHCA patients without bystander witnesses, CAHP and OHCA scores were not calculated. Although the predictive abilities of the scores were not significantly different, the NULL-PLEASE score had a large AUC of ROC curve in various OHCA patients. Furthermore, in patients with bystander-witnessed OHCA, the NULL-PLEASE score had large partial AUCs of ROC from sensitivity 0.8–1.0 and specificity 0.8–1.0.

Conclusions

The NULL-PLEASE score had a high, comprehensive predictive ability in various OHCA patients. Furthermore, the NULL-PLEASE score had a high predictive ability for good and poor neurological outcomes in patients with bystander-witnessed OHCA.

Background

Out-of-hospital cardiac arrest (OHCA) occurs annually in 250,000–300,000 patients worldwide [1]. The management of cardiac arrest, including modern cardiopulmonary resuscitation (CPR), extracorporeal CPR, emergency cardiovascular treatment, and targeted temperature management, is progressing [2]; however, in patients with a successful return of spontaneous circulation (ROSC), in-hospital survival and neurologically intact survival rates remain disappointingly low [3].

Accurate prognostication of survival and good neurological outcome after ROSC is very important to reduce unnecessary treatments and counselling the patients’ families. Several clinical scores have been reported to predict the neurological outcome of patients with OHCA at an early stage [4,5,6,7,8,9]. In 2006, the OHCA score was the first practical score developed to predict prognosis in OHCA patients at intensive care unit admission and has been used for many years [4]. This score was developed based on OHCA-patient data from 1999 to 2003. Therefore, the OHCA score may not reflect recent changes in resuscitation strategies and improvements in the outcome. The cardiac arrest hospital prognosis (CAHP) score was designed to more accurately recognize neurological prognosis with a nomogram, including an independent prognostic factor, and was presented in 2016 [6]. The OHCA and CAHP scores require “no-flow interval,” which is the time from cardiac arrest to the initiation of CPR, to calculate the prediction scores. Therefore, they cannot be applied to patients without witnesses of cardiac arrest [4, 6]. The “non-shockable rhythm, unwitnessed arrest, long no-flow or long low-flow period, blood pH < 7.2, lactate > 7.0 mmol/L, end-stage chronic kidney disease, age ≥ 85 years, still resuscitation, and extracardiac cause” (NULL-PLEASE) clinical score was devised to identify patients unlikely to survive out-of-hospital cardiac arrest using several unfavorable cardiac arrest- or patient-related characteristics [7]. Recently, the post-cardiac arrest syndrome for therapeutic hypothermia (CAST) score and revised CAST (rCAST) score were developed to predict the neurologic prognosis in patients after resuscitation following cardiac arrest, prior to inducing therapeutic hypothermia [10, 11]. The variables required to calculate each predictive score are summarized in Table 1 [4, 6, 7, 10, 11].

Table 1 The variables needed to calculate each predictive score

Although various scores have been proposed, it is unclear which prognostication tool is superior in guiding decision-making regarding individual patients with OHCA. Therefore, this study aimed to compare and validate the OHCA, CAHP, NULL-PLEASE clinical, CAST, and rCAST scores in OHCA patients treated with recent CPR strategies.

Methods

Patient selection and data collection

This single-center retrospective study evaluated electronic medical records from Hokkaido University Hospital, a tertiary care center in Sapporo City, Japan, which covers 1121 km2 with a population of approximately 2.0 million. The study protocol was approved by our institutional review board, and the requirement for informed consent was waived owing to the retrospective design.

Patients with OHCA who were admitted to our emergency department (ED) between February 2015 and July 2018 were included in the present study. Patients were excluded based on the following criteria: (a) age < 18 years, (b) death at the ED, and (c) transferred from other hospitals. From the electronic medical records, we collected pre-hospital data recorded according to the Utstein style [12,13,14] and in-hospital data regarding CPR, laboratory tests, and treatments. Furthermore, neurological outcomes were evaluated using the cerebral performance category (CPC) scale [15] at 1 month after cardiac arrest. The primary outcome was defined as a good neurological outcome (CPC 1 and 2).

Statistical analysis

Data for continuous variables are presented as medians with interquartile ranges. Categorical data are presented as frequencies and percentages. Patient characteristics and outcomes were compared between the two groups using the Mann–Whitney U test (for numerical variables) and Fisher’s exact test (for categorical variables). The overall predictive abilities of the various scores for good neurological outcome were tested using the area under the curve (AUC) of the receiver operating characteristic (ROC) curve. To evaluate the predictive ability for good neurological outcome with high specificity, the partial AUC (pAUC) of the ROC curve from a specificity of 0.8–1.0 in each predictive score was calculated. Furthermore, to evaluate the predictive ability for poor neurological outcome with high sensitivity, the pAUC of the ROC curve from sensitivity 0.8–1.0 in each predictive score was calculated. All analyses were performed using R statistical software version 3.6.3 (The Institute of Statistical Mathematics, Tokyo, Japan). All reported p values were two-tailed, and differences with p < 0.05 were considered statistically significant.

Results

During the study period, 236 OHCA patients were admitted to our ED. This study included patients with extrinsic cardiac arrest and thus included all OHCA patients regardless of the cause of cardiac arrest. We excluded 47 patients because the neurological outcome 1 month after cardiac arrest was unclear. Therefore, 189 patients were included in the present study. The characteristics of patients with good (CPC 1 and 2) and poor (CPC 3, 4, and 5) neurological outcomes are presented in Table 2.

Table 2 Baseline characteristics of the out-of-hospital cardiac arrest patients

In Table 3, the AUCs of the ROC curves of each predictive score in all patients and various subgroups are presented. Because the two predictive scores require no-flow time, which we were only able to evaluate in patients with bystander-witnessed OHCA, AUCs of ROC curves of CAHP and OHCA scores were not evaluated in all patients. In the group of patients in which therapeutic hypothermia was not induced, the predictive ability of the CAST and rCAST scores was low, and the rCAST score showed significantly lower predictive ability than the OHCA, CAHP, and NULL-PLEASE scores (p = 0.037, 0.016, and 0.027, respectively). In the other subgroups, all predictive scores had sufficiently large AUCs of ROC curves. Although statistical significance was only observed in the group of patients in which therapeutic hypothermia was induced, the NULL-PLEASE score tended to show a high predictive ability in the overall OHCA patient cohort and the subgroups. Furthermore, when comparing subgroups within the same score, the accuracy of the rCAST score in the therapeutic hypothermia (+) group was significantly higher than that in the overall cohort and therapeutic hypothermia (−) group. (p = 0.020 and 0.002, respectively). In subgroup analyses of the other scores, therapeutic hypothermia and veno-arterial extracorporeal membrane oxygenation (VA-ECMO) did not affect the scores’ accuracy.

Table 3 Area under the receiver operating characteristic curves of each predictive score in all patients and various subgroups

In patients with bystander-witnessed OHCA, the pAUC of the ROC curve from sensitivity 0.8–1.0 in each predictive score was presented to evaluate the predictive ability for a poor neurological outcome with high sensitivity (Table 4). Although the pAUC of each predictive score was not statistically different, the pAUCs of NULL-PLEASE and CAHP scores were larger than those of the other scores.

Table 4 Predictive ability for poor neurological outcome in patients with by-stander witnessed out-of-hospital cardiac arrest

To evaluate the predictive ability for good neurological outcome with high specificity, the pAUCs of ROC curves from specificity 0.8–1.0 in each predictive score are presented in Table 5. The pAUCs of NULL-PLEASE and OHCA scores were large, whereas that of the rCAST was small. Furthermore, the pAUC of the OHCA score was statistically larger than that of the rCAST score (p = 0.0204).

Table 5 Predictive ability for good neurological outcome in patients with by-stander witnessed out-of-hospital cardiac arrest

Discussion

To date, several scores that predict the prognosis of patients with cardiac arrest have been published; however, no study has used the same patient group to verify the accuracy of each score at one time. The present study was the first to simultaneously validate various cardiac arrest prognostic scores. All prognostic scores that were evaluated in the present study had a sufficiently high predictive ability. Among them, the NULL-PLEASE score could be easily calculated in various OHCA patients, including those without bystander witnesses. Furthermore, the NULL-PLEASE score had a high predictive ability for good and poor neurological outcome in patients with bystander-witnessed OHCA.

Previous studies have revealed various factors, such as older age, cardiac arrest occurring at home, initial rhythm other than ventricular tachycardia/ventricular fibrillation, longer duration of no flow, longer duration of low flow, treatment with adrenaline (epinephrine), pupillary response, and a serum lactate level, as prognostic factors for OHCA patients [16,17,18,19,20]. Despite the prognosis scores proposed using these predictors [4,5,6,7,8,9], prognostication of OHCA patients remains challenging, and no single risk-assessment tool has been recommended for the prognostic classification of OHCA patients. Although various prognostic scores have been reported, the target patients were different in each instance [4,5,6,7,8,9]. For example, the targets for the NULL-PLEASE score were all OHCA patients, whereas those for the OHCA and CAHP scores were restricted to bystander-witnessed OHCA patients [4, 6, 7]. The targets of the CAST and rCAST scores were restricted to OHCA patients in whom therapeutic hypothermia was induced [10, 11]. However, where possible, we were able to evaluate the prognostic scores in various OHCA patient subgroups, regardless of the original targets.

The prognostic scores require various variables for their calculation. Although no-flow time is required in OHCA and CAHP scores, the variable cannot be obtained in OHCA patients without bystander witnesses [4, 6]. Therefore, OHCA and CAHP scores could not be calculated in OHCA patients without bystander witnesses [4, 6]. In addition, some scores require information that may be difficult to obtain, such as medical history, neurological findings, and findings of brain computed tomography [5, 7, 9, 10]. For these reasons, score calculation is often complicated and/or impossible. In the present study, the OHCA score was the simplest, whereas the CAST score was the most troublesome.

We examined the accuracy of the aforementioned predictive scores across all OHCA patients and various subgroups. As a result, the NULL-PLEASE score had a high, comprehensive predictive ability in all OHCA patients and various subgroups. Furthermore, it had a high predictive ability for good and poor neurological outcome in patients with bystander-witnessed OHCA. Moreover, the NULL-PLEASE score can apply to various OHCA patients because all the variables required to calculate it can be easily collected in clinical settings. Therefore, the NULL-PLEASE score is a useful predictive score in various clinical settings.

Originally, the CAST and rCAST scores were targeted at OHCA patients in whom therapeutic hypothermia was induced [10, 11]. In clinical settings, OHCA patients who have regained consciousness or are strongly predicted to have a poor prognosis tend to be excluded from therapeutic hypothermia. Therefore, the characteristics of OHCA patients who undergo therapeutic hypothermia tend to be restrictive. Furthermore, in OHCA patients who underwent therapeutic hypothermia, the prior probability for good or poor neurological outcome was completely different from that in all OHCA patients. In the present study, although CAST and rCAST scores had high predictive ability in OHCA patients who underwent therapeutic hypothermia, this ability was not observed in other OHCA patients, especially those who did not undergo therapeutic hypothermia. The pAUC from sensitivity 0.8–1.0, which indicates the predictive ability for poor neurological outcome, of the rCAST score in the therapeutic hypothermia (+) group was also significantly higher than those in the overall patient cohort and the therapeutic hypothermia (−) group (data not shown). Therapeutic hypothermia had no effect on the accuracy of the other scores. These results were likely affected by the differences in the aforementioned prior probability. Therefore, rCAST score should not apply patients who were not induced therapeutic hypothermia.

Although CAST and rCAST scores can be calculated online (http://www.castscore.sakura.ne.jp/), other predictive scores cannot. Furthermore, almost all variables required to calculate predictive scores were the same. Therefore, we created a website to conveniently calculate and compare multiple prognostic scores for OHCA patients (https://hokudai-qq.com/score). Comparing different scores simultaneously and selecting the appropriate score for patients with cardiac arrest will be helpful in clinical settings. However, although these scores are useful in explaining the prognosis to relatives, they are not perfectly accurate and should not be used as the basis for clinical decision-making.

Limitation

This study was conducted retrospectively in a single institution, and the number of target patients was small. In addition, there was potential for selection bias and confounding due to unknown or unmeasured variables. In addition, about 20% of patients were excluded due to unclear follow-up status, leading to inclusion bias.

Conclusions

Among the predictive scores evaluated in the present study, the predictive abilities of the scores were not significantly different; nevertheless, that of the NULL-PLEASE score was high in various OHCA patients. Furthermore, the variables used to calculate the NULL-PLEASE score can be easily collected in clinical settings. Therefore, the NULL-PLEASE score is a useful predictive score in clinical settings.

Availability of data and materials

Data are available from the corresponding author upon reasonable request.

Abbreviations

AUC:

Area under the curve

CAHP:

Cardiac arrest hospital prognosis

CAST:

Cardiac arrest syndrome for therapeutic

CPC:

Cerebral performance category

CPR:

Cardiopulmonary resuscitation

ED:

Emergency department

NULL-PLEASE:

Non-shockable rhythm, unwitnessed arrest, long no-flow or long low-flow period, blood pH < 7.2, lactate > 7.0 mmol/L, end-stage chronic kidney disease, age ≥ 85 years, still resuscitation, and extracardiac cause

OHCA:

Out-of-hospital cardiac arrest

pAUC:

Partial area under the curve

rCAST:

Revised CAST

ROC:

Receiver operating characteristic

ROSC:

Return of spontaneous circulation

VA-ECMO:

Veno-arterial extracorporeal membrane oxygenation

References

  1. 1.

    Berdowski J, Berg RA, Tijssen JG, Koster RW. Global incidences of out-of-hospital cardiac arrest and survival rates: systematic review of 67 prospective studies. Resuscitation. 2010;81:1479–87.

    Google Scholar 

  2. 2.

    Mosier JM, Kelsey M, Raz Y, Gunnerson KJ, Meyer R, Hypes CD, et al. Extracorporeal membrane oxygenation (ECMO) for critically ill adults in the emergency department: history, current applications, and future directions. Crit Care. 2015;19:431.

    Article  Google Scholar 

  3. 3.

    Lemiale V, Dumas F, Mongardon N, Giovanetti O, Charpentier J, Chiche JD, et al. Intensive care unit mortality after cardiac arrest: the relative contribution of shock and brain injury in a large cohort. Intensive Care Med. 2013;39:1972–80.

    Article  Google Scholar 

  4. 4.

    Adrie C, Cariou A, Mourvillier B, Laurent I, Dabbane H, Hantala F, et al. Predicting survival with good neurological recovery at hospital admission after successful resuscitation of out-of-hospital cardiac arrest: the OHCA score. Eur Heart J. 2006;27:2840–5.

    Article  Google Scholar 

  5. 5.

    Rittenberger JC, Tisherman SA, Holm MB, Guyette FX, Callaway CW. An early, novel illness severity score to predict outcome after cardiac arrest. Resuscitation. 2011;82:1399–404.

    Article  Google Scholar 

  6. 6.

    Maupain C, Bougouin W, Lamhaut L, Deye N, Diehl JL, Geri G, et al. The CAHP (cardiac arrest hospital prognosis) score: a tool for risk stratification after out-of-hospital cardiac arrest. Eur Heart J. 2016;37:3222–8.

    Article  Google Scholar 

  7. 7.

    Potpara TS, Mihajlovic M, Stankovic S, Jozic T, Jozic I, Asanin MR, et al. External validation of the simple NULL-PLEASE clinical score in predicting outcome of out-of-hospital cardiac arrest. Am J Med. 2017;130:1464.e13-e21.

    Article  Google Scholar 

  8. 8.

    Coppler PJ, Elmer J, Calderon L, Sabedra A, Doshi AA, Callaway CW, et al. Validation of the Pittsburgh cardiac arrest category illness severity score. Resuscitation. 2015;89:86–92.

    Article  Google Scholar 

  9. 9.

    Kiehl EL, Parker AM, Matar RM, Gottbrecht MF, Johansen MC, Adams MP, et al. C-GRApH: a validated scoring system for early stratification of neurologic outcome after out-of-hospital cardiac arrest treated with targeted temperature management. J Am Heart Assoc. 2017;6:e003821.

    Article  Google Scholar 

  10. 10.

    Nishikimi M, Matsuda N, Matsui K, Takahashi K, Ejima T, Liu K, et al. CAST: a new score for early prediction of neurological outcomes after cardiac arrest before therapeutic hypothermia with high accuracy. Intensive Care Med. 2016;42:2106–7.

    Article  Google Scholar 

  11. 11.

    Nishikimi M, Ogura T, Nishida K, Takahashi K, Nakamura M, Matsui S, et al. External validation of a risk classification at the emergency department of post-cardiac arrest syndrome patients undergoing targeted temperature management. Resuscitation. 2019;140:135–41.

    Article  Google Scholar 

  12. 12.

    Perkins GD, Jacobs IG, Nadkarni VM, Berg RA, Bhanji F, Biarent D, et al. Cardiac arrest and cardiopulmonary resuscitation outcome reports: update of the utstein resuscitation registry templates for out-of-hospital cardiac arrest: a statement for healthcare professionals from a task force of the International Liaison Committee on Resuscitation (American Heart Association, European Resuscitation Council, Australian and New Zealand Council on Resuscitation, Heart and Stroke Foundation of Canada, InterAmerican Heart Foundation, Resuscitation Council of Southern Africa, Resuscitation Council of Asia); and the American Heart Association Emergency Cardiovascular Care Committee and the Council on Cardiopulmonary, critical care, perioperative and resuscitation. Circulation. 2015;132:1286–300.

    Article  Google Scholar 

  13. 13.

    Cummins RO, Chamberlain DA, Abramson NS, Allen M, Baskett PJ, Becker L, et al. Recommended guidelines for uniform reporting of data from out-of-hospital cardiac arrest: the Utstein Style. A statement for health professionals from a task force of the American Heart Association, the European Resuscitation Council, the Heart and Stroke Foundation of Canada, and the Australian Resuscitation Council. Circulation. 1991;84:960–75.

    CAS  Article  Google Scholar 

  14. 14.

    Jacobs I, Nadkarni V, Bahr J, Berg RA, Billi JE, Bossaert L, et al. Cardiac arrest and cardiopulmonary resuscitation outcome reports: update and simplification of the Utstein templates for resuscitation registries: a statement for healthcare professionals from a task force of the International Liaison Committee on Resuscitation (American Heart Association, European Resuscitation Council, Australian Resuscitation Council, New Zealand Resuscitation Council, Heart and Stroke Foundation of Canada, InterAmerican Heart Foundation, Resuscitation Councils of Southern Africa). Circulation. 2004;110:3385–97.

    Article  Google Scholar 

  15. 15.

    Teasdale G, Jennett B. Assessment of outcome and impaired consciousness. A practical scale. Lancet. 1974;2:81–4.

    CAS  Article  Google Scholar 

  16. 16.

    Herlitz J, Engdahl J, Svensson L, Angquist KA, Young M, Holmberg S. Factors associated with an increased chance of survival among patients suffering from an out-of-hospital cardiac arrest in a national perspective in Sweden. Am Heart J. 2005;149:61–6.

    Article  Google Scholar 

  17. 17.

    Lin S, Callaway CW, Shah PS, Wagner JD, Beyene J, Ziegler CP, et al. Adrenaline for out-of-hospital cardiac arrest resuscitation: a systematic review and meta-analysis of randomized controlled trials. Resuscitation. 2014;85:732–40.

    CAS  Article  Google Scholar 

  18. 18.

    Shinozaki K, Oda S, Sadahiro T, Nakamura M, Hirayama Y, Watanabe E, et al. Blood ammonia and lactate levels on hospital arrival as a predictive biomarker in patients with out-of-hospital cardiac arrest. Resuscitation. 2011;82:404–9.

    CAS  Article  Google Scholar 

  19. 19.

    Sasson C, Rogers MA, Dahl J, Kellermann AL. Predictors of survival from out-of-hospital cardiac arrest: a systematic review and meta-analysis. Circ Cardiovasc Qual Outcomes. 2010;3:63–81.

    Article  Google Scholar 

  20. 20.

    Tamura T, Namiki J, Sugawara Y, Sekine K, Yo K, Kanaya T, et al. Early outcome prediction with quantitative pupillary response parameters after out-of-hospital cardiac arrest: a multicenter prospective observational study. PLoS ONE. 2020;15:e022824.

    Google Scholar 

Download references

Acknowledgements

We would like to thank Cruller (https://cruller.jp) for creating our website. We would also like to thank Editage (https://online.editage.jp/) for English language editing.

Funding

None.

Author information

Affiliations

Authors

Contributions

TT, KK, and TY contributed to study conception and manuscript preparation. KO and KM contributed to analysis of the data. MH and TW contributed to manuscript preparation and revision for intellectual content. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Mineji Hayakawa.

Ethics declarations

Ethics approval and consent to participate

This study was approved by the Ethics Committee of Hokkaido University (Approval number: 190513).

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Tsuchida, T., Ono, K., Maekawa, K. et al. Simultaneous external validation of various cardiac arrest prognostic scores: a single-center retrospective study. Scand J Trauma Resusc Emerg Med 29, 117 (2021). https://doi.org/10.1186/s13049-021-00935-w

Download citation

Keywords

  • Out-of-hospital cardiac arrest
  • Prediction
  • Prognosis
  • Neurological outcome