External validation of the ROSC after cardiac arrest (RACA) score in a physician staffed emergency medical service system

Background The return of spontaneous circulation (ROSC) after cardiac arrest (RACA) score may have implications as a quality indicator for the emergency medical services (EMS) system. We aimed to validate this score externally in a physician staffed urban EMS system. Methods We conducted a retrospective cohort study. Data on resuscitation attempts from the Helsinki EMS cardiac arrest registry from 1.1.2008 to 31.12.2010 were collected and analyzed. For each attempted resuscitation the RACA score variables were collected and the score calculated. The endpoint was ROSC defined as palpable pulse over 30 s. Calibration was assessed by comparing predicted and observed ROSC rates in the whole sample, separately for shockable and non-shockable rhythm, and separately for resuscitations lead by a specialist, registrar or medical supervisor (i.e., senior paramedic). Data are presented as medians and interquartile ranges. Statistical testing included chi-square test, the Mann-Whitney U test, Hosmer-Lemeshow goodness of fit test and calculation of 95% confidence intervals (CI) for proportions. Results A total of 680 patients were included of whom 340 attained ROSC. The RACA score was higher in patients with ROSC (0.62 [0.46–0.69] than in those without (0.46 [0.36–0.57]) (p < 0.001). Observed against predicted ROSC indicated reasonable calibration overall (p = 0.30), with better calibration in patients with a shockable initial rhythm (p = 0.75) than in patients with a non-shockable rhythm (p = 0.04). There was no statistical difference between observed and predicted ROSC rates in resuscitations attended by a specialist (50% vs 53%, 95% CI 45–55) or registrar (55% vs 53%, 95% CI 48–62), but rates were lower than predicted in resuscitations lead by a medical supervisor (36% vs 49%, 95% CI 25–47). Discussion Developing a practical severity-of-illness scoring system for out-of-hospital cardiac arrest patients would allow patient heterogeneity adjustment and measurement of quality of care in analogy to commoly used severity-of-illness- scores developed for the similar purposes for the general intensive care unit population. However, transferring RACA score to another country with different population and EMS system might affect the performance and generalizability of the score. Conclusions This study found a good overall calibration and moderate discrimination of the RACA score in a physician staffed urban EMS system which suggests external validity of the score. Calibration was suboptimal in patients with a non-shockable rhythm which may due to a local do-not-attempt-resuscitation policy. The lower than expected overall ROSC rate in resuscitations attended by medical supervisors requires further study.


Background
The incidence of out-of hospital cardiac arrest (OHCA) in Europe is estimated to be 37-55/100,000 inhabitants per year [1,2]. Because of the major burden that OHCA patients cause to EMS systems and hospitals worldwide quality assurance is paramount [3]. This has resulted in attempts to develop means for predicting survival and comparing pre-hospital care of OHCA patients [4][5][6][7][8][9][10][11]. Outcome of OHCA vary: in high-quality EMS systems spontaneous circulation may be achieved in up to 53% of patients at least until hospital admission and discharge rates are reported to be between 14 and 20% while much lower rates are reported from other EMS systems [1,3,12,13]. The comparability of different cohorts has been questioned and direct outcome comparisons may also be affected by alternating definitions of inclusion and exclusion criteria [4]. There seems to be a considerable need for practical scoring system allowing comparison between these different EMS systems and patient cohorts and thus, serving as a quality indicator.
In 2011 Gräsner and colleagues developed and internally validated a score to predict occurrence of return of spontaneous circulation (ROSC) after out-of hospital cardiac arrest, the so called return of spontaneous circulation after cardiac arrest (RACA) score [4]. The return of spontaneous circulation after cardiac arrest core was developed with data from the German Resuscitation Registry and incorporates multiple pre-resuscitation variables found to have a significant positive or negative impact on the probability of ROSC. Importantly, the RACA score is based on variables available on EMS arrival at the scene; e.g., patient age, gender, aetiology, place and EMS delay. However, RACA score is not designed to be calculated at the scene, influence medical management or to be used as a prognostic tool, in order to determine the success or failure of resuscitation of individual patient.
Since the original study was performed with data from the German Resuscitation Registry, its applicability in other countries with different EMS systems and populations is currently unknown. The aim of this study was to validate externally the RACA score in a physician staffed urban EMS system. We hypothesized that the RACA score would predict ROSC with good calibration and discrimination in a cohort of patients from the Helsinki EMS Utstein-style based cardiac arrest registry.

Setting and patient sample
Helsinki, the capital city of Finland has a population of approximately 580,000 inhabitants served by a threetiered EMS based at eight regional rescue stations. Out-of-hospital resuscitations are attended by a nearest ambulance (BLS, basic life support or ALS, advanced life support) + a physician unit or medical supervisor ALS unit + a first responding rescue unit (i.e., fire engine) when appropriate. Resuscitation is always lead either by the physician or the medical supervisor on-duty. EMS physicians are either specialists or experienced registrars of anaesthesiology with intensive care training. Medical supervisors are specially trained senior paramedics capable of executing ALS procedures and further postresuscitation care including intubation and ventilatory support, vasoactive medication and sedation, for example. Local resuscitation protocols followed European Resuscitation Council 2005 and 2010 guidelines during the study period [14]. The Helsinki EMS has prospectively collected OHCA data according to the Utstein guidelines since 1994. All data are gathered and validated by Helsinki EMS physicians or on-duty medical supervisors.
In the current retrospective cohort study we included all resuscitation attempts occurring from 1.1.2008 to 31. 12.2010. For further analysis we excluded resuscitations where EMS physician decided to stop before resuscitation was commenced or any ALS measures initiated. Refraining from further resuscitation was based on a local do-not-attempt-resuscitation (DNAR) policy, patients medical history (e.g., terminal illness), very long delay on starting CPR, patients of whom CA was due to apparently fatal condition or other rare clinical situation where further resuscitation was considered futile. In the Helsinki EMS cardiac arrest registry, a resuscitation attempt is defined as an attempt of intubation (or other advanced airway), defibrillation or the use of adrenaline and/or amiodarone in addition to chest compressions. Similarly, if ROSC is achieved with chest compressions only resuscitation is considered attempted and the case included in the study.
Notably, Helsinki EMS DNAR guidelines includes situations when resuscitation can be stopped after being initiated. Discontinuing of resuscitation is considered in asystole if the arrest is unwitnessed, if the delay of the ambulance exceeds 10 min from call or if ROSC is not achieved despite 20 min of ALS. Similarly, in witnessed pulseless electrical activity (PEA) resuscitation is stopped if the delay of the ambulance exceeds 15 min, ROSC cannot be achieved within 20 min of ALS or in unwitnessed PEA ROSC is not achieved within 10 min of ALS [15]. These time frames are not applied on hypothermic patients or if the reason for cardiac arrest is drowning or penetrating trauma.
In the development of the RACA score the selection of potential predictive factors was performed in a multivariate analysis approach considering available literature. For each categorical variable one condition was defined as standard category, which did not receive a specific coefficient in the model but was defined as a reference for the other conditions of the respected variable. In the original study, following conditions were defined as "standard category": female gender, age < 80 years, cardiac aetiology, non-witnessed CA, location at home and workplace, ventricular fibrillation as a primary rhythm, and no bystander CPR. Only conditions associated with a negative or positive impact on ROSC were given regression coefficients respectively, and included in the RACA score equation [4].
Notably, there are twenty different possible aetiologies for cardiac arrest in the Helsinki EMS cardiac arrest registry and in RACA equation there are only three (trauma, hypoxia, intoxication). From the twenty possible aetiologies the physician chooses the most probable reason for the cardiac arrest based on clinical findings and patient pre-arrest symptoms on the scene. For this study the following aetiologies found in the Helsinki EMS cardiac arrest registry were included in the RACA category hypoxia: drowning, suffocation, carbon monoxide poisoning, worsening of asthma/chronic obstructive pulmonary disease (COPD), pneumonia and pulmonary embolism. If the presumed aetiology data was missing, the RACA "other" category was used, as in the Helsinki cardiac arrest registry the aetiology of OHCA is presumed to be of cardiac origin unless otherwise stated. Accordingly, the RACA "other" category was also used for all aetiologies other than "trauma", "hypoxia" or "intoxication", which are included in the RACA score equation. For example, cardiac arrests with presumable aetiology of non-traumatic bleeding or cerebrovascular disorder were categorised as "other" as those are not included in the RACA score equation. "Trauma" and "intoxication" as aetiologies for OHCA are found as such in the Helsinki EMS cardiac arrest registry. The Helsinki registry summarizes all medical locations and does not include data required for the RACA score, i.e., "nursing home", "doctor's office" or "medical institution". Thus for this study the exact location of these attempts were identified from EMS case notes.

Primary and secondary outcomes
Primary outcome was of ROSC, defined as palpable pulse at any point during resuscitation for over 30 s.

Statistical analysis and research permit
Data were analyzed using SPSS version 19.0 (SPSS, Chigaco, Ill., USA). Results are presented as frequencies and percentages or medians (interquartile range). We assessed calibration by comparing observed against predicted ROSC rates by using Hosmer-Lemeshow C test (goodness-of-fit). Expected probabilities of ROSC were calculated using equation of the German study [3] as illustrated above. Discrimination was examined by calculation of the area under receiver operating characteristic (ROC) curve. Area under the curve is presented with 95% confidence intervals. The study plan was approved by the Institutional Review Board of the Helsinki University Hospital. Informed consent was waived due to the retrospective setting of the study.

Results
After exclusions a total of 680 patients were included in the study and ROSC was achieved in 340 resuscitations (50%). The total incidence of attempted resuscitations was 39/100,000 inhabitants per year. In 377 (36%) cases resuscitation was not attempted and these were excluded from the study.

Outcome
Baseline characteristics and ROSC rates for different subgroups of patients are presented in Table 1. Primary rhythm was shockable in 275 (40%) of all patients. There were over twice as many male (70%) CA patients as compared to female (30%), but no significant difference on ROSC rates between genders was found (male 48% vs. female 54%, p = 0.132). In this study, the age group >80 years was not associated with significantly lower chance of ROSC (age >80 years 48% vs. age <80 years 49%, p = 0.91).

RACA score
The RACA score was higher in patients with ROSC (0.62 [0.46-0.69] than in those without (0.46 [0.36-0.57]) (p < 0.001). For the entire study population RACA score showed moderate discrimination (AUC 0.71, CI 0.67-0.75) for predicting ROSC (Fig. 1). Predicted ROSC rate in the whole sample was 52% while observed ROSC rate was 50% (CI 46-54). The mean predicted ROSC rate for the 275 resuscitations with shockable initial rhythm was 66% and ROSC was achieved in 67% (CI 61-73) of the patients. For patients with a nonshockable rhythm results were 43% vs. 38% (CI 33-42), respectively (Fig. 2). Observed and predicted ROSC with the RACA indicated reasonable calibration overall (p = 0.30), with better calibration in patients with a shockable initial rhythm (p = 0.75) than with a nonshockable rhythm (p = 0.04). Concordance between observed and predicted ROSC rates divided into deciles are shown in Fig. 3. In the whole study population RACA score tended to overestimate ROSC rates in general while observed ROSC rates were markedly higher in the two highest deciles. Similarly, the score underestimated ROSC rates in the highest deciles in both shockable and nonshockable rhythms but for non-shockable rhythms outcome tended to be worse than predicted.

Discussion
In this study, we validated RACA score externally in a physician staffed urban EMS system and found good overall calibration and moderate discrimination. We did, however, find that the score had suboptimal calibration in patients with non-shockable primary rhythm. To the best of our knowledge, this is the first study externally that validates the RACA score in an EMS system. Developing a practical severity-of-illness scoring system for out-of-hospital cardiac arrest patients would allow patient heterogeneity adjustment and measurement of quality of care in analogy to commonly used severity of illness scoring systems such as APACHE (Acute Physiology and Chronic Health Evaluation) and SAPS (Simplified Acute Physiology Score) developed for the similar purposes for the general intensive care unit population [16,17]. This would allow comparisons between different EMS systems and patient cohorts as baseline factors affecting outcome would be taken into account. However, scoring system updates are also often required, as performance of the score tends to deteriorate over time due to changes in casemix and patient management. This is common practise in the use of severity of illness scores in intensive care, where the commonly used APACHE IV and SAPS III scores have reached their third and fourth generation, respectively [18]. The RACA score was developed in 2011 based on prospectively registered data from patients treated between 1998 and 2008, and may indeed benefit from a recalibration in the future or with more recent data. For example, failure of prehospital ROSC with ongoing CPR on admission is considered as a negative outcome [4]. Developments of OHCA management include extracorporeal membrane oxygenation (E-CPR) and out-and in-hospital transfers with ongoing (mechanical) CPR to the cardiac catheterization laboratory for intra-arrest interventions [19]. It is highly likely that such changes will influence the accuracy of the RACA score.
The purpose of the RACA score was to derive a score enabling prediction of initial resuscitation success  adjusted to different conditions, using data available on EMS arrival at the scene [4]. External validation of such score in other population is necessary before prediction model can be used more widely, as the model reflects the risk of a patient in the system and population in which it was developed. The original study by Gräsner et al. was based on data from German Resuscitation Registry and its applicability has only been tested in a German populations and EMS systems [4]. However, transferring RACA score to another country with different population and EMS system might affect the performance and generalizability of the score. Compared with the original study by Gräsner et al. [4] our cohort was different in terms of primary rhythm (ventricular fibrillation (VF) 28% vs 39%, pulseless electrical activity (PEA) 11% vs 36%, asystole (ASY) 46% vs 22%), non-witnessed CA (41% vs 15%), bystander cardiopulmonary resuscitation (CPR) (15% vs 50%) and ROSC rate (44% vs 50%). Nonetheless, we found a similar performance of the RACA score in terms of discrimination (AUC 0.73 vs. 0.71) supporting the generalizability of the score. Overall, observed ROSC rates corresponded well with predicted rates from the RACA score and especially well within the shockable rhythm -subgroup. In patients with a non-shockable primary rhythm the score showed poor calibration. One reason for this could be the donot-attempt resuscitation (DNAR) practice of Helsinki EMS, limiting of both the number of resuscitation attempts included in this study and the duration of resuscitation attempts -especially in non-shockable rhythms [15]. This most likely reduces the ROSC rate in our population as in the original study by Gräsner et al. there was a ROSC rate of approximately 35% in patients with EMS delay of 20 min or more from the onset of CA -situation where resuscitation would be stopped for non-shockable rhythms according to Helsinki EMS DNAR practice [4]. A previously published study showed unpredicted survivors especially among patients with unwitnessed asystole when the Helsinki EMS DNAR practice was followed -although, the prognosis in this subgroup seems very poor in general [15]. This might to some extent explain poor calibration found in this subgroup.
The RACA score has been used in German studies since its development. In a previously published study RACA score was used to compare predicted and observed non-traumatic out-of-hospital cardiac arrest ROSC rates over three 5-year time periods in the EMS of Bonn, Germany, showing lower than observed ROSC rates in all time periods [20]. It was also used in another German study to compare ROSC rates in out-of-hospital cardiac arrest patients with difficult or unsuccessful intubations where predicted and observed ROSC rates were similar in the group of difficult intubations, while the unsuccessful intubation group had lower than predicted ROSC rates [21]. ROSC rate was higher than predicted by the RACA score in a study comparing seven different centres in Germany when used as one part of EMS quality assessment [22]. In general, RACA score has showed slight tendency to underestimate rather than overestimate ROSC rates in German cohorts, whereas in our study it rather tended to overestimate ROSC rates -except in patients with high probability of ROSC.
We found no influence between observed and predicted ROSC rates in resuscitations lead by specialist or registrar as was the case in the original study between different specialties of the physician. Similarly, there was no influence whether first responder was rescue unit or not, or if delay of the ALS unit was over vs. under median time. However, we found a negative impact on resuscitations lead by a medical supervisor. This might partly be due to the fact there being two persons less on scene as opposed to resuscitations attended by a physician staffed unit. Furthermore, in comparison with EMS physicians, medical supervisors had more limited exposure and experience regarding ALS procedures and cardiac arrest patients in overall. During the study period, medical supervisors attended to 78 OHCA patients compared to 602 EMS physician lead resuscitations, as shown in Table 2. There was also a negative impact on adrenaline use during resuscitation, which could be explained by the lower than expected ROSC rates in patients with nonshockable rhythms. Similarly, the positive impact on not using adrenaline is most likely explained by the high prevalence of shockable rhythm in our cohort as ROSC is often achieved by defibrillation only -supported by high ROSC rate of 77% in this subgroup.
There are some limitations of our study. Firstly, as a principle, all OHCAs in the Helsinki cardiac arrest registry are presumed as of cardiac origin unless otherwise stated, and in this study "other" category was used accordingly, as only "intoxication", "trauma" and "hypoxia" have significant impact on the probability of ROSC in RACA score equation. Some selection bias due to incorrect presumed aetiology cannot, however, be ruled out.
Secondly, the duration of each resuscitation was not observed although it may have impact on the probability of ROSC. This may be highlighted when comparing EMS systems using different protocols in regards to cessation of a resuscitation attempt. However, this variable is not included in the RACA score equation and was therefore out of scope in this study.

Conclusions
This study conducted in one urban EMS system found a good overall calibration and moderate discrimination of the RACA score suggesting external validity of the score. Calibration was better in patients with a shockable rhythm and suboptimal in patients with non-shockable rhythm which can be due to a local do-not-attempt-resuscitation policy. The lower than expected overall ROSC rate in resuscitations attended by medical supervisors requires further study. Suboptimal calibration in certain subgroups of patients and only moderate discriminative power may limit the use of RACA score in different EMS systems.