External validation of multimodal termination of resuscitation rules for out-of-hospital cardiac arrest patients in the COVID-19 era

Abstract

Background

Futile resuscitation for out-of-hospital cardiac arrest (OHCA) patients in the coronavirus disease (COVID)-19 era can lead to risk of disease transmission and unnecessary transport. Various existing basic or advanced life support (BLS or ALS, respectively) rules for the termination of resuscitation (TOR) have been derived and validated in North America and Asian countries. This study aimed to evaluate the external validation of these rules in predicting the survival outcomes of OHCA patients in the COVID-19 era.

Methods

This was a multicenter observational study using the WinCOVID-19 Daegu registry data collected during February 18–March 31, 2020. The subjects were patients who showed cardiac arrest of presumed cardiac etiology. The outcomes of each rule were compared to the actual patient survival outcomes. The sensitivity, specificity, false positive value (FPV), and positive predictive value (PPV) of each TOR rule were evaluated.

Results

In total, 170 of the 184 OHCA patients were eligible and evaluated. TOR was recommended for 122 patients based on the international basic life support termination of resuscitation (BLS-TOR) rule, which showed 85% specificity, 74% sensitivity, 0.8% FPV, and 99% PPV for predicting unfavorable survival outcomes. When the traditional BLS-TOR rules and KoCARC TOR rule II were applied to our registry, one patient met the TOR criteria but survived at hospital discharge. With regard to the FPV (upper limit of 95% confidence interval < 5%), specificity (100%), and PPV (> 99%) criteria, only the KoCARC TOR rule I, which included a combination of three factors including not being witnessed by emergency medical technicians, presenting with an asystole at the scene, and not experiencing prehospital shock delivery or return of spontaneous circulation, was found to be superior to all other TOR rules.

Conclusion

Among the previous nine BLS and ALS TOR rules, KoCARC TOR rule I was most suitable for predicting poor survival outcomes and showed improved diagnostic performance. Further research on variations in resources and treatment protocols among facilities, regions, and cultures will be useful in determining the feasibility of TOR rules for COVID-19 patients worldwide.

Background

The novel coronavirus disease (COVID-19) continues to be in a pandemic status and remains an important health concern worldwide [1]. Due to the presenting symptoms, complications, and transmission efficiency of COVID-19, we changed our standard treatment processes and personal protection strategies. In particular, resuscitation of cardiac arrest patients posed an especially high risk of disease transmission to healthcare professionals [2, 3]. For in-hospital cardiac arrest (IHCA) patients, resuscitation was typically performed after establishing whether the patient has COVID-19. However, for out-of-hospital cardiac arrest (OHCA) patients, resuscitation was performed prior to any testing or confirmation. Consequently, the OHCA group was of high risk to rescuers.

In the prehospital setting, emergency medical personnel screen patients for COVID-19 based on their symptoms (fever, dyspnea, cough, and sore throat) and inquired about any recent history of contact with infected individuals. However, this method has limitations because many carriers are asymptomatic [4, 5]. In addition, these asymptomatic COVID-19 patients admitted to the intensive care unit (ICU) pose a challenge to determining the need for wearing personal protective equipment (PPE) and utilizing medical resources. Patients who failed to experience a return of spontaneous circulation (ROSC) in the event of a cardiac arrest were also often no longer transported to the hospital due to their low survival probability [6]. Accordingly, recent cardiopulmonary resuscitation (CPR) guidelines were changed to recommend that all healthcare providers assess the safety of the scene and wear higher levels of PPE for protection against infectious airborne droplets and detailing the criteria for the basic life support termination of resuscitation (BLS-TOR) rule. These guidelines stipulate that before terminating resuscitative efforts, the healthcare workers must check that (1) the arrest was not witnessed by emergency medical service (EMS) personnel or a first responder, (2) ROSC was not observed prior to the patient being transported to the emergency medical center (EMC), and (3) no electrical shocks were delivered [7,8,9,10,11,12,13,14].

We applied the rules that were typically selected depending on the country or region where the derivation and validation phases were conducted. These included the (1) International BLS and advanced life support (ALS) rules derived and validated in the United States and Europe [8]; (2) Goto and KANTO-SOS rules developed in Japan and Asian countries [13, 14]; and (3) Korean OHCA registry-based TOR models, Korean Cardiac Arrest Research Consortium (KoCARC) TOR rules, and two new TOR models that were used in our previous studies [7, 11].

In Daegu metropolitan city, the number of patients with COVID-19 increased rapidly at the ​​community level from February 18, 2020 concurrent with the pandemic outbreak in South Korea. During its 7-week peak, we resuscitated 184 OHCA patients at six EMCs. This was the first study to determine the efficacy of these TOR guidelines for an emerging infectious disease [7, 11, 12].

We were collecting prospective data for a registry of OHCA patients since 2017. However, as emergency rooms were shutdown, we recognized a need for additional research to improve the outcomes of OHCA patients while concurrently protecting emergency department (ED) staff. Thus, this registry-based observational study aimed to evaluate the external validation of these rules in predicting the survival outcomes of OHCA patients in the COVID-19 era.

Methods

Study design and variables

This study was approved by the Institutional Review Board of Kyungpook National University Hospital (Approval no.: KNUH 2020–04-032).

We conducted a multicenter registry-based observational study using the WinCOVID-19 citywide OHCA registry data. In March 10, ED personnel from six hospitals in Daegu voluntarily discussed about data of COVID-related factors. We evaluated patient demographics (age and sex), prehospital variables (e.g., place of arrest, initial cardiac rhythm, whether the arrest was witnessed by a bystander or EMS personnel, whether a bystander performed CPR, whether an automated external defibrillator was used and prehospital defibrillation occurred, and whether ROSC occurred), and time intervals. Response time was defined as the period from contacting the EMS to them arriving at the scene, while scene time was defined as the period from the EMS arriving at the scene to them departing from the scene. Resuscitation management and survival outcomes, including ROSC, survival to admission, survival to discharge, and neurological outcomes at discharge, after cardiac arrest were recorded as hospital-level variables. Based on this collective data, we determined whether the patients met the existing TOR guidelines and qualified to be enrolled in this study. In South Korea, EMS personnel cannot terminate resuscitation unless OHCA patients show obvious signs of death [7, 11]; therefore, all EMS-treated OHCA patients should be transported to the hospital. Notably, these personnel use manual defibrillators and analyze electrocardiograms prior to the patients’ arrival at the hospital.

To evaluate COVID-related variables, we obtained data, including the clinical symptoms or signs (sore throat and cough), vital signs (fever was defined as a temperature of ≥37.5 °C]), recent exposure history, and chest X-ray findings, of patients with confirmed COVID-19 on admission from the electronic medical records.

Study setting and patients

Daegu is a metropolitan city with a population of 2.44 million people and is composed of two regional level I EMCs, four local level II EMCs, and 19 emergency facilities or clinics. In particular, there are six EMCs where ALS, post-cardiac arrest care, and cardiovascular interventions are available. In this study, we evaluated adult patients (age > 18 years) who presented with presumed OHCA and used the EMS system in Daegu. Those who did not receive any resuscitation and those who experienced cardiac arrest in a primary care clinic or long-term care hospital were excluded.

Main outcome measures

The primary outcome measures were (1) the proportions of survival to hospital discharge defined as discharged to go home or transferred to another facility after admission to the hospital and (2) survival with a favorable neurological outcome quantified using Cerebral Performance Category scores, with scores of 1 and 2 being the most favorable.

Meanwhile, the secondary outcome measures included external validation of previously researched TOR rules (Table 1). We identified OHCA patients who met all the TOR criteria and calculated the sensitivity, specificity, false positive ratios (FPR), false positive values (FPVs), positive predictive values (PPVs), negative predictive values (NPVs), and their respective 95% confidence intervals (CI) to identify patients with a risk of poor survival [7, 8, 11, 13, 14].

Table 1 Criteria of the BLS-TOR and KoCARC BLS-TOR rules

Statistical analyses

The normality of the variables was determined using the Shapiro–Wilk test. Descriptive statistics were presented as medians with interquartile ranges (IQR, 25th and 75th percentiles), whereas categorical variables were presented as amounts and percentages. Significant differences among the survival outcomes were determined using the Mann-Whitney test for continuous variables and chi-square tests or Fisher’s exact test for categorical variables.

Logistic regressions were performed for patients who met the criteria of the previous TOR rules. The outcome of interest was survival to hospital discharge (Table 1). To determine the model calibration, the Hosmer-Lemeshow goodness of fit test was calculated. Odds ratio (OR) greater than 1 indicated an unfavorable effect on survival; ORs and 95% CIs were calculated for all covariates. The characteristics of the adjusted ORs were described using Forest plots. A receiver operating characteristic (ROC) curve, with the area under the curve (AUC), was used to determine the accuracy of such variables in predicting unfavorable survival outcomes at discharge. All tests were two-tailed, and p < 0.05 was considered statistically significant.

The diagnostic performances of all existing TOR rules were analyzed. Previous TOR rules were created based on FPR (1-specificity, i.e., the probability that the rule will suggest terminating resuscitation in case of patient survival) and PPV (i.e., the probability that the rule will suggest terminating resuscitation in case of patient death) of the rules. Current guidelines recommend that when cessation of life-sustaining care is considered, the tool used to predict poor outcomes must be accurate and reliable, with an specificity (of close to 100%) and a PPV (of > 99%, a narrow 95% CI) [7, 15].

All statistical analyses were performed using IBM SPSS Statistics v. 25 (IBM Corp., Armonk, NY, USA) and MedCalc v. 17.4.4 (MedCalc Software, Mariakerke, Belgium).

Results

Patient characteristics

Demographic and prehospital characteristics of the 2017–2018 Korean nationwide OHCA and Daegu Citywide OHCAs were presented in Supplementary Table 1. The prehospital and hospital characteristics of variables are almost similar. Of the 184 adults who developed OHCA between February 18, 2020 and March 31, 2020 (i.e., the peak of the COVID-19 outbreak), 170 patients (male, 63.2%) were included in this study (Fig. 1). In total, 76.0% of the arrest was witnessed by EMS personnel, 34.3% of the patients received CPR from a bystander, 4.7% experienced ROSC prior to hospital arrival, 58.5% showed abnormal chest radiography findings, and 5.8% were confirmed to have COVID-19 via reverse transcription polymerase chain reaction. The patients’ baseline characteristics are summarized in Table 2.

Fig. 1

Study flowchart

Table 2 General characteristics of the COVID-19 OHCA study population

Factors associated with survival

The rates of survival to discharge and survival with favorable neurological outcomes were 4.7 and 2.9%, respectively. Compared to the patients who died, the survivors were younger, experienced more prehospital defibrillations, and achieved ROSC before arriving at the ED (Table 2). Advanced age, arrest not being witnessed by emergency medical technician (EMT) personnel, receiving no shocks, and not developing ROSC before arriving at the ED were related to unfavorable survival outcomes. Only ROSC before arriving at the ED was related to survival to discharge (Fig. 2). Interestingly, age, witness status, and prehospital shock delivery were significantly associated with survival outcomes in the univariate analysis, but not in the multivariable analysis (Table 3).

Fig. 2

Forest plot for multivariate analysis of predictors of unfavorable survival of COVID-19 patients with OHCA. Data are adjusted for sex, age, location of event, witness status, bystander CPR, and any shockable rhythm

Table 3 Logistic regressions of factors associated with unfavorable survival outcomes

External validation of the TOR rules

On applying the existing international TOR rules to our registry, we found that 122 patients (71.8%) met all three criteria, but one survivor was discharged (Fig. 1). Another patient met the criteria for the traditional BLS-TOR and KoCARC TOR II rules, but he was also discharged (Table 4). The ROC curves of multimodal TOR categories for poor outcomes are shown in Fig. 2. Overall, KoCARC TOR rule I and the current TOR rule II, with AUCs of 0.951 and 0.967, respectively, were the most effective for predicting in-hospital mortality (Fig. 3). With respect to the specificity, FPR, FPV (narrow range of 95% CI) and PPV (> 99%), only KoCARC TOR rule I predicted poor survival outcomes, and it showed a higher diagnostic performance than did the other TOR rules (Table 4).

Table 4 External validations of multimodal TOR rules for predicting death prior to discharge (n = 170)

Fig. 3

Receiver operating characteristic curves for BLS-TOR rules to predict poor survival outcomes at hospital discharge

Discussion

This study validated the existing multimodal TOR rules using the WinCOVID-19 Daegu registry data on OHCA patients. Among the nine existing TOR rules, KoCARC TOR rule I was found to be the best indicator of poor outcomes during the COVID-19 outbreak as evidenced by its highest specificity and PPV. To our best knowledge, this was the first study to determine the efficacy of these TOR guidelines for an emerging infectious disease.

Resuscitation of infected individuals greatly increases the risk of virus transmission to healthcare providers [1, 6, 16]. Current guidelines recommend emergency personnel to confirm the presence of COVID-19 in OHCA patients and wear high-level PPE, even in doubtful cases, before performing CPR. However, these measures may have limited benefits because COVID-19 has a very high rate of disease transmission and approximately 25% of the patients have asymptomatic infection [4]. Therefore, the COVID-19 era is a challenging and confusing time for healthcare workers in the medical and emergency fields [5]. Some OHCA patients were unexpectedly confirmed to be positive post-CPR or postmortem, contributing to emergency room shutdowns and temporary closures [17]. As of September 2020, more than 20,000 people have been confirmed to have COVID-19 in South Korea, and 385 of these patients have died. In addition, more than 50% of these cases were in Daegu, the area of interest in this study, and its neighboring area Gyeongbuk [10]. Further, more than 130 healthcare workers and medical staff have been infected. One physician died after treating two confirmed patients.

Our findings, although preliminary, showed that the survival outcomes of OHCA patients in Daegu during the peak of the COVID-19 outbreak (4.1%) were significantly lower than those reported nationwide (9.8%) or in the city (8.8%) in 2018 [18]. Many variables were changed in the COVID-19 era, including prehospital ROSC and pre-hospital transport time (see Additional file 1). Although we could not describe the impact of the COVID-19 pandemic on the chain of survival and its negative effects on high-quality CPR, the risk-benefit ratio for CPR should be reconsidered [2]. Despite that there were several factors associated with good prognosis in this study, including bystander CPR, EMS CPR, prehospital ROSC, and VF, no new factor was found for COVID-19. Other studies have also raised concerns on how CPR must be performed for IHCA patients with confirmed COVID-19. Considering the lower survival rate, physicians should establish goals of care or CPR preferences to reduce futile resuscitation by stratifying the survival probability of the IHCA patients, regardless of their COVID-19 status, at the time of hospital admission [5, 19]. It is also important to consider prehospital TOR for out-of-hospital resuscitation in an infectious disease epidemic area.

The previous TOR rules can be divided into two sets of variables: one can be applied at the pre-hospital level and the other can be evaluated immediately after arriving at the ED. In this study, we selected and analyzed the external validation of all nine multimodal TOR rules for OHCA patients during the COVID-19 epidemic period. These rules were commonly selected depending on the country or region where the derivation and validation phases were conducted. These included the (1) International BLS (a combination of three criteria: arrest not being witnessed by EMTs, not receiving prehospital shock delivery, and not experiencing prehospital ROSC, Table 1) and ALS rules derived and validated in the United States and Europe [8]; (2) Goto and KANTO-SOS rules developed in Japan and Asian countries [13, 14]; and (3) the Korean OHCA registry-based TOR models, KoCARC TOR rules, and two new TOR rules that were used in our previous studies [7, 11]. The international BLS-TOR rules that can be enforced at the prehospital stage has high sensitivity and specificity, but also relatively high FPR (1-specificity) [9, 15]. Therefore, a continuous development of the TOR model has been proposed [7, 9]. A previous study that included acquired ECG asystole rhythm as a criterion also proposed a new TOR model applicable at the prehospital stage and another TOR model applicable immediately after arriving at the ED [11].

The previous four rules have been partially validated in other countries and in the setting of mechanical CPR or comprehensive post-resuscitation care [12, 20,21,22,23]. Previous validations of the TOR rule reported survival rates of less than 1% among TOR rule–positive patients in North America. In contrast, high FPR of survival has been reported in Asian countries (28.7% in Singapore, 25.9% in Taiwan, and 30.4% in South Korea). This discrepancy may be due to different prehospital practices and a relatively higher prevalence of non-shockable rhythm in patients in Asian countries [7]. However, the high false positive cases of survival in these Asian countries, where the withdrawal of life-sustaining treatment is not commonly applied, are likely to be biased. Kajino et al. [24] validated the TOR rules for predicting poor neurologic outcomes in a Japanese population and concluded that more specific TOR rules for each region should be developed, despite the good performance of the TOR rules in their study. However, even if the COVID-19 outbreak was not considered, these previous results implied that the extrapolation to and implementation of different TOR rules in regions with different organization of EMS treatment protocols, legislation, and socioeconomic characteristics might be problematic because the TOR rules would need to be adjusted to meet the regional situation.

In this study, we validated the existing multimodal TOR rules using the WinCOVID-19 Daegu registry data on OHCA patients. Our results indicated that of the 170 OHCA patients, we failed to screen one survivor of the seven survival discharges for the international BLS and KoCARC II rules. However, the remaining seven TOR rules were classified correctly. Current guidelines recommend that diagnostic tests that guide the cessation of life-saving efforts be accurate and reliable, with an FPV and FPR value close to 0% [7, 9]. Among the nine rules, KoCARC TOR rule I was found to be the most effective indicator for poor outcomes, as indicated by the lowest FPV (0% with narrow 95% CI) and highest PPV (> 99%). This rule included the combination of three factors, namely, not being witnessed by EMT, presenting with an asystole at the scene, and experiencing no prehospital shock or ROSC. It did not include the patient’s age or ED parameters, thereby making it easy to use in prehospital settings and applicable for OHCA patients in this current pandemic [7].

This study had several strengths and limitations. First, as with other multicenter observational studies, the integrity of the data could be biased. In addition, the observation period was only 2 months; hence, the effects on long-term survival outcomes, which are most important for OHCA research, remain unknown. In this regard, we will continue to conduct investigations until the COVID-19 pandemic has been effectively contained. Second, it has been speculated that poor survival outcomes are associated with fewer resuscitations and prolonged EMS scene and transport time to the hospital. Moreover, unfavorable neurological outcomes, as a primary outcome, is more suitable than survival to discharge. Because only four patients in our study displayed favorable neurological outcomes, we could not perform any secondary analyses or external validations. Third, the ratio of the survival outcomes was markedly lower in our study than that in previous Korean OHCA reports during the COVID-19 outbreak. This could be due to the direct adverse effects of COVID-19 on the cardiovascular system, prolonged EMS scene time, and differences in treatment in different hospitals [25]. Fourth, the number of COVID-19 cases sharply decreased in April in Daegu. Thus, the sample size was too small for a large-scale multi-factor analysis. Finally, facility or regional differences in EMS resources, CPR quality, and post-cardiac arrest care might affect the survival outcomes during the COVID-19 outbreak. TOR rules in the COVID-19 era and socio-ethical issues must be discussed further, and a consensus process must be developed.

Conclusions

Among the nine previously existing TOR rules, KoCARC TOR rule I is the most suitable for predicting poor survival outcomes, and it showed improved diagnostic performance in the COVID-19 era. With regard to the specificity, FPV and PPV criteria, KoCARC TOR rule I was superior to all other TOR rules. Further research on variations in resources and treatment protocols (CPR quality and post-cardiac arrest care) among facilities, regions, and cultures will be useful in determining the feasibility of TOR rules for COVID-19 patients worldwide.