Development and validation of a score to identify in the Emergency Department patients who may benefit from a time-critical intervention: a cohort study
© Challen et al. 2015
Received: 2 June 2015
Accepted: 8 September 2015
Published: 17 September 2015
Risk stratification methods developed on the basis of predicting illness severity are often used to prioritise patients on the basis of urgency. Illness severity and urgency may not be interchangeable. Severe illness places patients at risk of adverse outcome, but treatment is only urgent if adverse outcome can be prevented by time-sensitive treatment. We aimed to develop a score to identify patients in need of urgent treatment, on the basis of potential to benefit from time-sensitive intervention, and to compare this with a severity score identifying patients at high risk of death.
A sequential cohort of adults presenting to one Emergency Department by ambulance and admitted to hospital was prospectively collected (2437 derivation, 2322 validation). Data on outcomes representing potential to benefit was collected retrospectively on a random subset (398 derivation, 227 validation). Logistic regression identified variables predictive of death and potential to benefit from urgent treatment.
Death was predicted using age, respiratory rate, diastolic blood pressure, oxygen saturations, temperature, GCS and respiratory disease (AUROC 0.84 (95 % CI 0.8–0.89) derivation and 0.74 (0.69–0.81) validation), while potential to benefit was predicted by pulse, systolic blood pressure and GCS (AUROC 0.74 (0.67–0.80) derivation and 0.71 (0.59–0.82) validation).
A score developed to predict the need for urgent treatment has a different composition to a score developed to predict illness severity, suggesting that triage methods based on predicting severity could lead to inappropriate prioritisation on the intended basis of urgency.
There were 5.3 million emergency hospital admissions in England in 2012–3, mainly from the 21 % of Emergency Department (ED) attendances resulting in hospital admission . Similarly, in 2010–11, only 71,801 (45 %) of 160,460 critical care admissions were planned . EDs and admissions units therefore need rapid and accurate methods to identify those in urgent need of treatment.
Within emergency care illness severity and urgency are often assumed to be interchangeable concepts. However, severity reflects the risk of a poor outcome or impact of a condition upon the patient , whereas urgency reflects the potential to benefit from timely care; so a patient with metastatic cancer would be severely ill but a patient with facial angioedema would be of high urgency. A measure of patient urgency should identify patients whose outcome will be improved with prompt care and/or those whose outcome will worsen without this care. We have demonstrated that many risk scores have been developed to predict adverse outcomes not amenable to intervention, and are therefore severity rather than urgency scores .
Recent commentary has drawn attention to the disconnect between the identification of patients at risk of a particular outcome and the potential of those patients to benefit from available interventions . Although systems to identify deteriorating patients and responding rapidly are intuitively appealing , meta-analysis of these amongst in-patients has failed to identify a benefit in terms of patient outcome [7, 8]. This may reflect use of scores that measure severity rather than urgency.
We therefore aimed to develop a score to identify in the ED patients of high urgency with the potential to benefit from time-sensitive interventions, and to compare this with a score to identify patients at high risk of death.
Study design and setting
This study was developed alongside the DAVROS project which developed and validated a risk-adjustment method for research and audit in emergency care and has been described in full elsewhere . Data were collected from February-May (derivation) and October-December (validation) 2008 at Northern General Hospital, Sheffield, the only adult ED serving the half million population of Sheffield. It is an acute teaching hospital with 1100 beds. The ED at the time of the study had 98,000 attendances per year. Interventional cardiology, critical care and acute theatres are all available on site.
Patients were included if they were transported by an emergency ambulance, then either died in the ambulance or ED or were admitted to hospital. Patients who had no vital signs at the time of ambulance arrival (even if resuscitation was attempted) were excluded. Patients aged under 65 with trauma were excluded prospectively as risk prediction in the trauma population has been extensively studied and is strongly influenced by variables such as injury site and type which are not replicated in non-trauma patients (see https://www.tarn.ac.uk). Patients aged over 65 were excluded retrospectively if their reason for admission was purely for trauma. The cohort was restricted to patients presenting by ambulance to ensure a clear point of entry to the emergency care system. Children and patients with purely obstetric presentations were excluded as their physiology is different and therefore predictors of adverse outcome are likely to be different, as are mortality and critical illness rates.
The whole dataset was used to develop the tool to predict death. Patient casenotes were screened in date order until adequate numbers of patients with outcomes of interest were included for the tool to predict potential to benefit. This group constituted the “potential to benefit” subset of the study population.
Ethical approval for both studies was obtained from Leeds (East) REC (09/H1306/2). Approval to use patient identifiable data without specific consent was gained from the Patient Information Advisory group for the original study and from the National Information Governance Board (ECC 5-07(f)/2009) for this study.
Predictor variables were extracted from ED records within 2 days of presentation and entered directly into an online database. Data entry staff were clerical personnel who were specifically trained for the study but had no involvement in the analysis phase. Study staff randomly sampled and rechecked entered data to ensure data quality. Age, blood pressure, Glasgow coma scale, oxygen saturation (breathing air and/or breathing supplemental oxygen), pulse rate, respiratory rate and temperature as recorded on arrival at the ED were chosen as they are widely recorded in a relatively standardised manner. Pulse pressure was calculated from systolic and diastolic blood pressures immediately prior to data analysis. Also extracted from the ED notes was the presence of specific co-morbidities (active malignancy, chronic respiratory disease, heart disease, asthma, diabetes, epilepsy, and warfarin or steroid use), as recorded by the attending clinician. Presenting complaint was recorded verbatim as described to reception staff but not further analysed as there was no way of confirming consistency of recording.
inevitable death: the patient died and a decision was made within 24 h of admission not to attempt CPR;
potentially preventable death, where the patient died but no decision was made to withdraw or limit care;
potentially prevented death, where the patient survived and received a potentially life-saving intervention (defined below);
Interventions defined a priori as potentially life-saving
• Use of airway adjunct or procedure to maintain patent airway.
• Use of intravenous/intramuscular adrenaline to treat or prevent airway compromise.
• Bag-valve-mask ventilation (unless during procedural sedation), intermittent positive pressure ventilation, or non-invasive ventilation.
• Decompression of tension pneumothorax.
• Drainage of significant pleural effusion (>1 litre).
• Insertion of chest drain for pneumothorax in patients with pre-existing lung disease.
• Intravenous therapy except steroids for asthma.
• Cardioversion (chemical or DC) of ventricular tachycardia or supraventricular tachycardia or atrial fibrillation with accessory pathway.
• Emergency endoscopy or surgery for upper GI bleed or use of Sengstaken tube or use of vasopressin/terlipressin.
• Infusion of >2 litres of fluid or transfusion for haemodynamic instability.
• Laparotomy for GI bleed/gynaecological bleed (including ectopic)/AAA.
• Sepsis care bundle.
• Thrombolysis for AMI or PE, or percutaneous revascularisation.
• Therapeutic (not diagnostic) pericardiocentesis.
• Transcutaneous or external pacing or administration of atropine (except in theatre).
• Vasopressor use (except bolus dosing in theatre).
• Administration of naloxone or flumazenil (unless related to procedural sedation).
• Administration of 10 %/50 % dextrose.
• Administration of >1 dose benzodiazepines/other anticonvulsants for fitting.
• Neurosurgical intervention.
• Active rewarming (not including Bearhugger).
• Laparotomy for sepsis/infarction/obstruction.
• New initiation of renal replacement therapy.
• Specific poisons antidotes including N-acetylcysteine.
Sample size was based on ten observed outcome events per predictor variable included in the analysis . Given the different frequencies of death and potential to benefit the sample sizes required for the two outcomes therefore varied.
Risk of each outcome in various groupings of predictor variables was explored visually using histograms (data available). Groups of similar risk were collapsed for further analysis. Given the absence of linear and monotonic relationships with risk, we wished to avoid arbitrary dichotomisation of data and therefore avoided decision tree analysis. Univariate association between potential predictor variables and outcome was assessed using logistic regression in SPSS, and first order interaction of variables significant at p < 0.15 was examined. Variables found to be significantly predictive of outcome at p < 0.1 were block entered into the multivariate analysis. Linear coefficients were then recalculated for independently predictive variables and an equation to predict poor outcome generated from those coefficients using the general formula p(outcome) = exb/1 + exb, where xb is the linear predictor.
The equations generated were applied to the validation sets without re-estimation of coefficients and performance assessed using ROC curves.
Full cohort for death (n = 2437)
Subset for potential to benefit (n = 398)
Full cohort for death (n = 2322)
Subset for potential to benefit (n = 227)
Mean (sd; range)
69 (19; 18–103)
66.5 (20.2; 18–102)
70 (19; 18–103)
71.3 (18.3; 19–96)
20 (7; 6–45)
20 (6.6; 8–80)
20 (5.8; 12–40)
136 (29; 24–266)
133 (29; 45–243)
139 (28.4; 44–261)
138 (28.6; 60–249)
75 (15; 30–153)
75 (15; 36–130)
76 (15; 11–151)
74 (15; 36–142)
88 (24; 21–215)
92 (24; 35–188)
88 (23; 20–180)
89 (21.9; 35–152)
36.6 (1.2; 26.0–41.0)
36.5 (1.2; 26.0–40.0)
36.5 (1.1; 25.2–40.5)
36.6 (1.40; 26.4–39.6)
Median (IQR; range)
SaO2 breathing air
97 (95–98; 50–100)
97 (94–98; 66–100)
97 (95–98; 45–100)
96 (94–98; 45–100)
SaO2 breathing oxygen
98 (95–100; 24–100)
97 (96–99; 71–100)
98 (95–100; 60–100)
98 (96–100; 84–100)
15 (15–15; 3–15)
15 (15–15; 3–15)
15 (15–15; 3–15)
15 (15–15; 7–15)
Potentially preventable death
Univariate logistic regression analysis of predictor variables for potential to benefit
95 % CI for exp(B)
Pulse (ref <71)
Respiratory rate (ref <16)
Systolic BP (ref 121–180)
Pulse pressure (ref 51–76)
GCS (ref 13–15)
SaO2 (ref low risk (99–100 breathing air))
High (<95 breathing air or <96 with supplemental O2)
Moderate (95–98 breathing air or >95 with supplemental O2)
Analysis of interactions showed significant interactions in: pulse by respiratory rate, systolic blood pressure and SaO2, respiratory rate by pulse pressure and SaO2, systolic blood pressure by pulse pressure and SaO2 and pulse pressure by SaO2. Briefly it appears that hypoxia confers increased risk if tachycardia is absent or if tachypnoea is present, and that normal pulse pressure confers increased risk in the presence of systolic hypotension.
Multivariate analysis of predictor variables for potential to benefit
95 % CI for exp(B)
Pulse (ref <71)
Respiratory rate (ref <16)
Systolic BP (ref 121–180)
Pulse pressure (ref 51–76)
GCS (ref 13–15)
SaO2 (ref low risk)
Respiratory rate/SaO2 interaction
The performance of this equation was assessed using a ROC curve, which had an area under the curve (c-statistic) of 0.737 (95 % CI 0.671–0.804). When applied to the validation set it had an area under the curve (c-statistic) of 0.707 (95 % CI 0.594–0.820).
This had a c-statistic of 0.847 (95 % CI 0.8–0.894). When applied to the validation set, the equation had an area under the curve (c-statistic) of 0.741 (95 % CI 0.685–0.806).
We have demonstrated that a combination of pulse, systolic blood pressure and Glasgow Coma Score can predict provision of time-sensitive interventions with moderate discrimination. Different variables (age, respiratory rate, diastolic blood pressure, oxygen saturations, temperature, Glasgow Coma Score and a pre-existing diagnosis of respiratory disease) predict death within seven days. Thus a score developed to predict the need for urgent treatment will differ from a score developed to predict illness severity, suggesting that triage methods based on severity could lead to inappropriate prioritisation on the intended basis of urgency.
This is the first study specifically to address the identification of at-risk patients in an ED population not preselected for diagnosis or severity, and to use the provision of a life-saving intervention as an outcome measure. It therefore addresses the issues facing emergency clinicians more closely than the existing literature.
Although our definitions of time-sensitive interventions might be criticized for being defined arbitrarily, the concept of an “Emergency Care Sensitive Condition” is being more widely embraced and supports our methodology . It is apparent that patterns of physiological derangement differ between patients who die and those who receive life-saving intervention . The development of a tool to identify this second group was therefore necessary.
Patients who were not admitted to hospital and those who self-presented were excluded from the initial study dataset as the aim was to develop a risk-adjustment tool for emergency admissions to hospital. This limits the generalisability of this study in terms of developing a clinical score. Ideally a full cohort of presenting patients would be studied and those discharged from the ED followed up to analyse post-discharge adverse events. However that is logistically unfeasible, as rates of short-term death after discharge from the ED are 30–50/100,000 [17, 18], and only 11 % of self-presenting patients are admitted to hospital , so the required cohort size would have been impractical. Obviously our findings cannot be applied to other patient groups such as children or trauma patients.
Our definition of an inevitable death might be considered overly restrictive but we chose this deliberately to include the widest possible group as having potential to benefit.
As a single-site study it may be that these results are not generalisable; interpretation of vital sign derangement is not only affected by patient factors but also by the health care system, staffing levels and types and time available for patient care . Thus a process of external validation might find that life-saving interventions are provided differently in other settings.
We initially wished to examine the role of clinician gestalt in detection of the at-risk patient. Early data collection included a “yes/no” question to the transporting paramedic as to whether the patient was critically ill; this had to be abandoned due to poor rates of completion.
Implications for clinical practice and policy
The variables we have identified as predictive of a need for life-saving intervention are not the same as those in use in many standardized early warning scores. This may reflect the inappropriateness of developing early warning scores using data sets in which death is the main outcome. This score may be more appropriate than existing scores but should not yet be widely applied in standard practice. It needs wider validation, ideally including comparison with unstructured clinician (doctor or triage nurse) gestalt and with NEWS as the currently mandated standard of care. There must also be consideration as the score is applied of whether the outcomes used to develop definitions of urgency are still valid; the interventions listed in the Appendix are acknowledged to be based on incomplete evidence; it is to be hoped that as the evidence base for emergency care is developed these can be refined (for example, intravenous magnesium in acute severe asthma would no longer be considered a potentially life-saving intervention ).
These results highlight the potential flaws in applying clinical scores to predict outcomes other than those for which they were originally derived. As the health economist Tony Culyer said “capacity to benefit is not identical to need” , and clinicians should be clear about the reasons for which a score is being used. Equally, if a scoring system is used as casemix adjustment in an attempt to assess or improve quality, it should be clear that it identifies conditions which are amenable to alteration with good care .
This leaves the working emergency clinician in the situation of not having a score developed for and demonstrated to work in the emergency setting. There are two options: firstly to use an existing score but to recognise its limitations in the ED; secondly not to use a score but to rely on the unstructured judgement of clinical staff. In the current culture of high regard for standardised paperwork easily amenable to retrospective audit this is unlikely to be managerially palatable. Given the current state of equipoise over the utility of standardised scores in terms of patient benefit clinicians should also be encouraged to participate in formal research to address the issue.
Implications for research
Researchers in other settings have demonstrated the value of changes in physiological scores in prognostication ; ideally ongoing research would examine the prognostic value of response (or non-response) to treatment provided prehospitally or in the ED.
We have defined a priori a group of interventions which appear on best available evidence to be potentially life-saving and time-sensitive. However, the evidence supporting these is incomplete and our beliefs underlying many frequently-used interventions would bear further scrutiny; patients who could benefit could then be more reliably identified. We made no attempt to assess functional outcome of those patients defined as having potential to benefit; future researchers may wish to consider whether morbidity should also be a component of benefit.
Identification of and response to the patient at risk requires more than a reliable scoring system; complex psychosocial and cognitive factors affect decision-making, particularly in the pressured ED environment , and examination of the interaction of these factors should be a priority for future research .
In summary, we have developed a score that predicts urgency (potential to benefit from time-sensitive treatment) with moderate discrimination. The score has different constituent variables and weights than a score developed to measure severity (risk of death). Early warning scores developed to predict death may not be useful predictors of urgency.
Thanks to Richard Wilson and Martina Santarelli for their contribution to the original DAVROS project, and to Jon Nicholl for his constructive criticism of both projects.
Dr Challen received an MRC PhD stipend for this work but the study sponsor had no role in the study design, the collection, analysis and interpretation of data, the writing of the manuscript or the decision to submit the manuscript for publication.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Health and Social Care Information Centre. Provider level analysis for HES admitted patient care 2011–12 and 2012–13. Leeds: HSCIC; 2013.Google Scholar
- Hospital Episode Statistics. Adult Critical Care in England – April 2010 to March 2011: Experimental Statistics. London: Health and Social Care Information Centre; 2012.Google Scholar
- Finlayson TL, Moyer CA, Sonnad SS. Assessing symptoms, disease severity, and quality of life in the clinical context: a theoretical framework. Am J Manage Care. 2004;10:336–44.Google Scholar
- Challen K, Goodacre SW. Predictive scoring in non-trauma emergency patients: a scoping review. Emerg Med J. 2011;28:827–37.View ArticlePubMedGoogle Scholar
- Grady D, Berkowitz SA. Why is a good clinical prediction rule so hard to find? Arch Intern Med. 2011;171:1701–2.View ArticlePubMedGoogle Scholar
- Hillman KM. Rapid response systems: you won’t know there is a problem until you measure it. Crit Care. 2011;15:1001.PubMed CentralView ArticlePubMedGoogle Scholar
- Gao H, McDonnell A, Harrison DA, Moore T, Adam S, Daly K, et al. Systematic review and evaluation of physiological track and trigger warning systems for identifying at-risk patients on the ward. Intensive Care Med. 2007;33:667–9.View ArticlePubMedGoogle Scholar
- Chan PS, Jain R, Nallmothu BK, Berg RA, Sasson C. Rapid response teams: a systematic review and meta-analysis. Arch Intern Med. 2010;170:18–26.View ArticlePubMedGoogle Scholar
- Goodacre S, Wilson R, Shephard N, Nicholl J. Derivation and validation of a risk adjustment model for predicting seven day mortality in emergency medical admissions: mixed prospective and retrospective cohort study. BMJ. 2012;344:e2904.PubMed CentralView ArticlePubMedGoogle Scholar
- Edbrooke DL, Minelli C, Mills GH, Iapichino G, Pezzi A, Corbella D, et al. Implications of ICU triage decisions on patient mortality: a cost-effectiveness analysis. Crit Care. 2011;15:R56.PubMed CentralView ArticlePubMedGoogle Scholar
- Cardoso LT, Grion CM, Matsuo T, Anami EH, Kauss IA, Seko L, et al. Impact of delayed admission to intensive care units on mortality of critically ill patients: a cohort study. Crit Care. 2011;15:R28.PubMed CentralView ArticlePubMedGoogle Scholar
- Kaji AH, Schriger D, Green S. Looking through the retrospectoscope: reducing bias in Emergency Medicine chart review studies. Ann Emerg Med. 2014;64:292–8.View ArticlePubMedGoogle Scholar
- Platts-Mills TF, Travers D, Biese K, McCall B, Kizer S, LaMantia M, et al. Accuracy of the Emergency Severity Index triage instrument for identifying elder Emergency Department patients receiving an immediate life-saving intervention. Acad Emerg Med. 2010;17:238–43.View ArticlePubMedGoogle Scholar
- Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol. 1996;49:1373–9.View ArticlePubMedGoogle Scholar
- Berthelot S, Lang ES, Quan H, Stelfox HT. Identifying emergency-sensitive conditions for the calculation of an emergency care inhospital standardized mortality ratio. Ann Emerg Med. 2014;63:418–24.View ArticlePubMedGoogle Scholar
- Churpek MM, Yuen TC, Edelson DP. Predicting clinical deterioration in the hospital: The impact of outcome selection. Resuscitation. 2013;84:564–8.PubMed CentralView ArticlePubMedGoogle Scholar
- Sklar DP, Crandall CS, Loeliger E, Edmunds K, Paul I, Helitzer DL. Unanticipated death after discharge home from the Emergency Department. Ann Emerg Med. 2007;49(6):735–45.View ArticlePubMedGoogle Scholar
- Gabayan GZ, Derose SF, Asch SM, Yiu S, Lancaster EM, Poon KT, et al. Patterns and predictors of short-term death after Emergency Department discharge. Ann Emerg Med. 2011;58:551–8.PubMed CentralView ArticlePubMedGoogle Scholar
- Jones D, Mitchell I, Hillman K, Story D. Defining clinical deterioration. Resuscitation. 2013;84:1029–34.View ArticlePubMedGoogle Scholar
- Goodacre S, Cohen J, Bradburn M, Stevens J, Gray A, Benger J, et al. The 3Mg trial: a randomised controlled trial of intravenous or nebulised magnesium sulphate versus placebo in adults with acute severe asthma. Health Technol Assess. 2014;18(22):1–168.View ArticlePubMedGoogle Scholar
- Cookson R, Claxton K, editors. The Humble Economist: Tony Culyer on Health, Health Care and Social Decision Making. York: University of York; 2012.Google Scholar
- Kellett J, Woodworth S, Wang F, Huang W. Changes and their prognostic implications in the abbreviated VitalpacTM early warning score (ViEWS) after admission to hospital of 18,853 acutely ill medical patients. Resuscitation. 2013;84:13–20.View ArticlePubMedGoogle Scholar
- Croskerry P. From mindless to mindful practice — cognitive bias and clinical decision making. N Engl J Med. 2013;368:2445–8.View ArticlePubMedGoogle Scholar
- Gaddis GM, Greenwald P, Huckson S. Toward improved implementation of evidence-based clinical algorithms: clinical practice guidelines, clinical decision rules, and clinical pathways. Acad Emerg Med. 2007;14:1015–22.View ArticlePubMedGoogle Scholar