Simultaneous external validation of various cardiac arrest prognostic scores: a single-center retrospective study

Background This study aimed to compare and validate the out-of-hospital cardiac arrest (OHCA); cardiac arrest hospital prognosis (CAHP); non-shockable rhythm, unwitnessed arrest, long no-flow or long low-flow period, blood pH < 7.2, lactate > 7.0 mmol/L, end-stage chronic kidney disease, age ≥ 85 years, still resuscitation, and extracardiac cause (NULL-PLEASE) clinical; post-cardiac arrest syndrome for therapeutic hypothermia (CAST); and revised CAST (rCAST) scores in OHCA patients treated with recent cardiopulmonary resuscitation strategies. Methods We retrospectively collected data on adult OHCA patients admitted to our emergency department between February 2015 and July 2018. OHCA, CAHP, NULL-PLEASE clinical, CAST, and rCAST scores were calculated based on the data collected. The predictive abilities of each score were tested using the area under the curve (AUC) of the receiver operating characteristic (ROC) curve. Results We identified 236 OHCA patients from computer-based medical records and analyzed 189 without missing data. In OHCA patients without bystander witnesses, CAHP and OHCA scores were not calculated. Although the predictive abilities of the scores were not significantly different, the NULL-PLEASE score had a large AUC of ROC curve in various OHCA patients. Furthermore, in patients with bystander-witnessed OHCA, the NULL-PLEASE score had large partial AUCs of ROC from sensitivity 0.8–1.0 and specificity 0.8–1.0. Conclusions The NULL-PLEASE score had a high, comprehensive predictive ability in various OHCA patients. Furthermore, the NULL-PLEASE score had a high predictive ability for good and poor neurological outcomes in patients with bystander-witnessed OHCA.


Background
Out-of-hospital cardiac arrest (OHCA) occurs annually in 250,000-300,000 patients worldwide [1]. The management of cardiac arrest, including modern cardiopulmonary resuscitation (CPR), extracorporeal CPR, emergency cardiovascular treatment, and targeted temperature management, is progressing [2]; however, in patients with a successful return of spontaneous circulation (ROSC), in-hospital survival and neurologically intact survival rates remain disappointingly low [3].
Accurate prognostication of survival and good neurological outcome after ROSC is very important to reduce unnecessary treatments and counselling the patients' families. Several clinical scores have been reported to predict the neurological outcome of patients with OHCA at an early stage [4][5][6][7][8][9]. In 2006, the OHCA score was the Open Access *Correspondence: mineji@dream.com 1 Department of Emergency Medicine, Hokkaido University Hospital, N14W5 Kita-ku, Sapporo 060-8648, Japan Full list of author information is available at the end of the article first practical score developed to predict prognosis in OHCA patients at intensive care unit admission and has been used for many years [4]. This score was developed based on OHCA-patient data from 1999 to 2003. Therefore, the OHCA score may not reflect recent changes in resuscitation strategies and improvements in the outcome. The cardiac arrest hospital prognosis (CAHP) score was designed to more accurately recognize neurological prognosis with a nomogram, including an independent prognostic factor, and was presented in 2016 [6]. The OHCA and CAHP scores require "no-flow interval, " which is the time from cardiac arrest to the initiation of CPR, to calculate the prediction scores. Therefore, they cannot be applied to patients without witnesses of cardiac arrest [4,6]. The "non-shockable rhythm, unwitnessed arrest, long no-flow or long low-flow period, blood pH < 7.2, lactate > 7.0 mmol/L, end-stage chronic kidney disease, age ≥ 85 years, still resuscitation, and extracardiac cause" (NULL-PLEASE) clinical score was devised to identify patients unlikely to survive out-ofhospital cardiac arrest using several unfavorable cardiac arrest-or patient-related characteristics [7]. Recently, the post-cardiac arrest syndrome for therapeutic hypothermia (CAST) score and revised CAST (rCAST) score were developed to predict the neurologic prognosis in patients after resuscitation following cardiac arrest, prior to inducing therapeutic hypothermia [10,11]. The variables required to calculate each predictive score are summarized in Table 1 [4,6,7,10,11]. Although various scores have been proposed, it is unclear which prognostication tool is superior in guiding decision-making regarding individual patients with OHCA. Therefore, this study aimed to compare and validate the OHCA, CAHP, NULL-PLEASE clinical, CAST, and rCAST scores in OHCA patients treated with recent CPR strategies.

Patient selection and data collection
This single-center retrospective study evaluated electronic medical records from Hokkaido University Hospital, a tertiary care center in Sapporo City, Japan, which covers 1121 km 2 with a population of approximately 2.0 million. The study protocol was approved by our institutional review board, and the requirement for informed consent was waived owing to the retrospective design.
Patients with OHCA who were admitted to our emergency department (ED) between February 2015 and July 2018 were included in the present study. Patients were excluded based on the following criteria: (a) Table 1 The variables needed to calculate each predictive score All laboratory data (pH, lactate, albumin, hemoglobin, serum creatinine) were taken on arrival at the hospital OHCA score out-of-hospital cardiac arrest score, NULL-PLEASE score non-shockable rhythm, unwitnessed arrest, long no-flow or long low-flow period, blood pH < 7.2, lactate > 7.0 mmol/L, end-stage chronic kidney disease, age ≥ 85 years, still resuscitation, and extracardiac cause score CAST score post-cardiac arrest syndrome for therapeutic hypothermia score, rCAST score revised post-cardiac arrest syndrome for therapeutic hypothermia score, CAHP score cardiac arrest hospital prognosis score, EMS emergency medical services, CPR cardiopulmonary resuscitation, ROSC return of spontaneous circulation, GCS Glasgow coma scale, CT computed tomography  [12][13][14] and in-hospital data regarding CPR, laboratory tests, and treatments. Furthermore, neurological outcomes were evaluated using the cerebral performance category (CPC) scale [15] at 1 month after cardiac arrest. The primary outcome was defined as a good neurological outcome (CPC 1 and 2).

Statistical analysis
Data for continuous variables are presented as medians with interquartile ranges. Categorical data are presented as frequencies and percentages. Patient characteristics and outcomes were compared between the two groups using the Mann-Whitney U test (for numerical variables) and Fisher's exact test (for categorical variables).
The overall predictive abilities of the various scores for good neurological outcome were tested using the area under the curve (AUC) of the receiver operating characteristic (ROC) curve. To evaluate the predictive ability for good neurological outcome with high specificity, the partial AUC (pAUC) of the ROC curve from a specificity of 0.8-1.0 in each predictive score was calculated. Furthermore, to evaluate the predictive ability for poor neurological outcome with high sensitivity, the pAUC of the ROC curve from sensitivity 0.8-1.0 in each predictive score was calculated. All analyses were performed using R statistical software version 3.6.3 (The Institute of Statistical Mathematics, Tokyo, Japan). All reported p values were two-tailed, and differences with p < 0.05 were considered statistically significant.

Results
During the study period, 236 OHCA patients were admitted to our ED. This study included patients with extrinsic cardiac arrest and thus included all OHCA patients regardless of the cause of cardiac arrest. We excluded 47 patients because the neurological outcome 1 month after cardiac arrest was unclear. Therefore, 189 patients were included in the present study. The characteristics of patients with good (CPC 1 and 2) and poor (CPC 3, 4, and 5) neurological outcomes are presented in Table 2.
In Table 3, the AUCs of the ROC curves of each predictive score in all patients and various subgroups are presented. Because the two predictive scores require noflow time, which we were only able to evaluate in patients with bystander-witnessed OHCA, AUCs of ROC curves of CAHP and OHCA scores were not evaluated in all patients. In the group of patients in which therapeutic hypothermia was not induced, the predictive ability of the CAST and rCAST scores was low, and the rCAST score showed significantly lower predictive ability than the OHCA, CAHP, and NULL-PLEASE scores (p = 0.037, 0.016, and 0.027, respectively). In the other subgroups, all predictive scores had sufficiently large AUCs of ROC curves. Although statistical significance was only observed in the group of patients in which therapeutic hypothermia was induced, the NULL-PLEASE score tended to show a high predictive ability in the overall OHCA patient cohort and the subgroups. Furthermore, when comparing subgroups within the same score, the accuracy of the rCAST score in the therapeutic hypothermia (+) group was significantly higher than that in the overall cohort and therapeutic hypothermia (−) group. (p = 0.020 and 0.002, respectively). In subgroup analyses of the other scores, therapeutic hypothermia and veno-arterial extracorporeal membrane oxygenation (VA-ECMO) did not affect the scores' accuracy.
In patients with bystander-witnessed OHCA, the pAUC of the ROC curve from sensitivity 0.8-1.0 in each predictive score was presented to evaluate the predictive ability for a poor neurological outcome with high sensitivity (Table 4). Although the pAUC of each predictive score was not statistically different, the pAUCs of NULL-PLEASE and CAHP scores were larger than those of the other scores.
To evaluate the predictive ability for good neurological outcome with high specificity, the pAUCs of ROC curves from specificity 0.8-1.0 in each predictive score are presented in Table 5. The pAUCs of NULL-PLEASE and OHCA scores were large, whereas that of the rCAST was small. Furthermore, the pAUC of the OHCA score was statistically larger than that of the rCAST score (p = 0.0204).

Discussion
To date, several scores that predict the prognosis of patients with cardiac arrest have been published; however, no study has used the same patient group to verify the accuracy of each score at one time. The present study was the first to simultaneously validate various cardiac arrest prognostic scores. All prognostic scores that were evaluated in the present study had a sufficiently high predictive ability. Among them, the NULL-PLEASE score could be easily calculated in various OHCA patients, including those without bystander witnesses. Furthermore, the NULL-PLEASE score had a high predictive ability for good and poor neurological outcome in patients with bystander-witnessed OHCA.
Previous studies have revealed various factors, such as older age, cardiac arrest occurring at home, initial rhythm other than ventricular tachycardia/ventricular fibrillation, longer duration of no flow, longer duration of low flow, treatment with adrenaline (epinephrine), pupillary response, and a serum lactate level, as prognostic factors for OHCA patients [16][17][18][19][20]. Despite the prognosis scores proposed using these predictors [4][5][6][7][8][9], prognostication of OHCA patients remains challenging, and no single risk-assessment tool has been recommended for the prognostic classification of OHCA patients. Although various prognostic scores have been reported, the target patients were different in each instance [4][5][6][7][8][9]. For example, the targets for the NULL-PLEASE score were all OHCA patients, whereas those for the OHCA and CAHP scores were restricted to bystander-witnessed OHCA patients [4,6,7]. The targets of the CAST and rCAST scores were restricted to OHCA patients in whom therapeutic hypothermia was induced [10,11]. However, where possible, we were able to evaluate the prognostic scores in various OHCA patient subgroups, regardless of the original targets. The prognostic scores require various variables for their calculation. Although no-flow time is required in OHCA and CAHP scores, the variable cannot be obtained in OHCA patients without bystander witnesses [4,6]. Therefore, OHCA and CAHP scores could not be calculated in OHCA patients without bystander witnesses [4,6]. In addition, some scores require information that may be difficult to obtain, such as medical history, neurological findings, and findings of brain computed tomography [5,7,9,10]. For these reasons, score calculation is often complicated and/or impossible. In the present study, the OHCA score was Table 3 Area under the receiver operating characteristic curves of each predictive score in all patients and various subgroups There was no significant difference between the scores in all subgroups *p < 0.05 compared with the rCAST score in patients with therapeutic hypothermia **p < 0.01 compared with the rCAST score in patients with therapeutic hypothermia the simplest, whereas the CAST score was the most troublesome.
We examined the accuracy of the aforementioned predictive scores across all OHCA patients and various subgroups. As a result, the NULL-PLEASE score had a high, comprehensive predictive ability in all OHCA patients and various subgroups. Furthermore, it had a high predictive ability for good and poor neurological outcome in patients with bystander-witnessed OHCA. Moreover, the NULL-PLEASE score can apply to various OHCA patients because all the variables required to calculate it can be easily collected in clinical settings. Therefore, the NULL-PLEASE score is a useful predictive score in various clinical settings.
Originally, the CAST and rCAST scores were targeted at OHCA patients in whom therapeutic hypothermia was induced [10,11]. In clinical settings, OHCA patients who have regained consciousness or are strongly predicted to have a poor prognosis tend to be excluded from therapeutic hypothermia. Therefore, the characteristics of OHCA patients who undergo therapeutic hypothermia tend to be restrictive. Furthermore, in OHCA patients who underwent therapeutic hypothermia, the prior probability for good or poor neurological outcome was completely different from that in all OHCA patients. In the present study, although CAST and rCAST scores had high predictive ability in OHCA patients who underwent therapeutic hypothermia, this ability was not observed in other OHCA patients, especially those who did not undergo therapeutic hypothermia. The pAUC from sensitivity 0.8-1.0, which indicates the predictive ability for poor neurological outcome, of the rCAST score in the therapeutic hypothermia (+) group was also significantly higher than those in the overall patient cohort and the therapeutic hypothermia (−) group (data not shown). Therapeutic hypothermia had no effect on the accuracy of the other scores. These results were likely affected by the differences in the aforementioned prior probability. Therefore, rCAST score should not apply patients who were not induced therapeutic hypothermia.
Although CAST and rCAST scores can be calculated online (http:// www. casts core. sakura. ne. jp/), other predictive scores cannot. Furthermore, almost all variables required to calculate predictive scores were the same. Therefore, we created a website to conveniently calculate and compare multiple prognostic scores for OHCA patients (https:// hokud ai-qq. com/ score). Comparing different scores simultaneously and selecting the appropriate score for patients with cardiac arrest will be helpful in clinical settings. However, although these scores are useful in explaining the prognosis to relatives, they are not perfectly accurate and should not be used as the basis for clinical decision-making.

Limitation
This study was conducted retrospectively in a single institution, and the number of target patients was small. In addition, there was potential for selection bias and confounding due to unknown or unmeasured variables. In addition,  about 20% of patients were excluded due to unclear followup status, leading to inclusion bias.

Conclusions
Among the predictive scores evaluated in the present study, the predictive abilities of the scores were not significantly different; nevertheless, that of the NULL-PLEASE score was high in various OHCA patients. Furthermore, the variables used to calculate the NULL-PLEASE score can be easily collected in clinical settings. Therefore, the NULL-PLEASE score is a useful predictive score in clinical settings.