- Original research
- Open Access
Development and validation of a score to identify in the Emergency Department patients who may benefit from a time-critical intervention: a cohort study
Scandinavian Journal of Trauma, Resuscitation and Emergency Medicine volume 23, Article number: 67 (2015)
Risk stratification methods developed on the basis of predicting illness severity are often used to prioritise patients on the basis of urgency. Illness severity and urgency may not be interchangeable. Severe illness places patients at risk of adverse outcome, but treatment is only urgent if adverse outcome can be prevented by time-sensitive treatment. We aimed to develop a score to identify patients in need of urgent treatment, on the basis of potential to benefit from time-sensitive intervention, and to compare this with a severity score identifying patients at high risk of death.
A sequential cohort of adults presenting to one Emergency Department by ambulance and admitted to hospital was prospectively collected (2437 derivation, 2322 validation). Data on outcomes representing potential to benefit was collected retrospectively on a random subset (398 derivation, 227 validation). Logistic regression identified variables predictive of death and potential to benefit from urgent treatment.
Death was predicted using age, respiratory rate, diastolic blood pressure, oxygen saturations, temperature, GCS and respiratory disease (AUROC 0.84 (95 % CI 0.8–0.89) derivation and 0.74 (0.69–0.81) validation), while potential to benefit was predicted by pulse, systolic blood pressure and GCS (AUROC 0.74 (0.67–0.80) derivation and 0.71 (0.59–0.82) validation).
A score developed to predict the need for urgent treatment has a different composition to a score developed to predict illness severity, suggesting that triage methods based on predicting severity could lead to inappropriate prioritisation on the intended basis of urgency.
There were 5.3 million emergency hospital admissions in England in 2012–3, mainly from the 21 % of Emergency Department (ED) attendances resulting in hospital admission . Similarly, in 2010–11, only 71,801 (45 %) of 160,460 critical care admissions were planned . EDs and admissions units therefore need rapid and accurate methods to identify those in urgent need of treatment.
Within emergency care illness severity and urgency are often assumed to be interchangeable concepts. However, severity reflects the risk of a poor outcome or impact of a condition upon the patient , whereas urgency reflects the potential to benefit from timely care; so a patient with metastatic cancer would be severely ill but a patient with facial angioedema would be of high urgency. A measure of patient urgency should identify patients whose outcome will be improved with prompt care and/or those whose outcome will worsen without this care. We have demonstrated that many risk scores have been developed to predict adverse outcomes not amenable to intervention, and are therefore severity rather than urgency scores .
Recent commentary has drawn attention to the disconnect between the identification of patients at risk of a particular outcome and the potential of those patients to benefit from available interventions . Although systems to identify deteriorating patients and responding rapidly are intuitively appealing , meta-analysis of these amongst in-patients has failed to identify a benefit in terms of patient outcome [7, 8]. This may reflect use of scores that measure severity rather than urgency.
We therefore aimed to develop a score to identify in the ED patients of high urgency with the potential to benefit from time-sensitive interventions, and to compare this with a score to identify patients at high risk of death.
Study design and setting
This study was developed alongside the DAVROS project which developed and validated a risk-adjustment method for research and audit in emergency care and has been described in full elsewhere . Data were collected from February-May (derivation) and October-December (validation) 2008 at Northern General Hospital, Sheffield, the only adult ED serving the half million population of Sheffield. It is an acute teaching hospital with 1100 beds. The ED at the time of the study had 98,000 attendances per year. Interventional cardiology, critical care and acute theatres are all available on site.
Patients were included if they were transported by an emergency ambulance, then either died in the ambulance or ED or were admitted to hospital. Patients who had no vital signs at the time of ambulance arrival (even if resuscitation was attempted) were excluded. Patients aged under 65 with trauma were excluded prospectively as risk prediction in the trauma population has been extensively studied and is strongly influenced by variables such as injury site and type which are not replicated in non-trauma patients (see https://www.tarn.ac.uk). Patients aged over 65 were excluded retrospectively if their reason for admission was purely for trauma. The cohort was restricted to patients presenting by ambulance to ensure a clear point of entry to the emergency care system. Children and patients with purely obstetric presentations were excluded as their physiology is different and therefore predictors of adverse outcome are likely to be different, as are mortality and critical illness rates.
The whole dataset was used to develop the tool to predict death. Patient casenotes were screened in date order until adequate numbers of patients with outcomes of interest were included for the tool to predict potential to benefit. This group constituted the “potential to benefit” subset of the study population.
Ethical approval for both studies was obtained from Leeds (East) REC (09/H1306/2). Approval to use patient identifiable data without specific consent was gained from the Patient Information Advisory group for the original study and from the National Information Governance Board (ECC 5-07(f)/2009) for this study.
Predictor variables were extracted from ED records within 2 days of presentation and entered directly into an online database. Data entry staff were clerical personnel who were specifically trained for the study but had no involvement in the analysis phase. Study staff randomly sampled and rechecked entered data to ensure data quality. Age, blood pressure, Glasgow coma scale, oxygen saturation (breathing air and/or breathing supplemental oxygen), pulse rate, respiratory rate and temperature as recorded on arrival at the ED were chosen as they are widely recorded in a relatively standardised manner. Pulse pressure was calculated from systolic and diastolic blood pressures immediately prior to data analysis. Also extracted from the ED notes was the presence of specific co-morbidities (active malignancy, chronic respiratory disease, heart disease, asthma, diabetes, epilepsy, and warfarin or steroid use), as recorded by the attending clinician. Presenting complaint was recorded verbatim as described to reception staff but not further analysed as there was no way of confirming consistency of recording.
The patient group of interest in this study is those where urgent intervention had the potential to affect survival, ie where a patient death was prevented or could potentially have been prevented by urgent intervention. We therefore classified 7-day outcomes as:
inevitable death: the patient died and a decision was made within 24 h of admission not to attempt CPR;
potentially preventable death, where the patient died but no decision was made to withdraw or limit care;
potentially prevented death, where the patient survived and received a potentially life-saving intervention (defined below);
The 7 day timescale was selected to achieve a balance between allowing sufficient time for an illness course to declare itself and minimising confounding by chronic disease and/or iatrogenic factors. Mortality was recorded by study staff using hospital record review, supplemented by a search of Coroner’s Office records. It was assumed that all patients admitted to intensive care received potentially life-saving intervention(s) [10, 11]. Interventions that had the potential to prevent death were identified from hospital casenotes by KC using a priori definitions (Table 1) based on evidence-based guidelines for acute care (Appendix). No judgement was made as to whether the intervention was helpful in individual cases, and outcome data was abstracted before examination of predictor variable data, to minimise abstractor bias . It must be noted that many of these interventions are not supported by high quality evidence. They are, however, similar to the list of life-saving interventions developed independently and concurrently by another research group . It was not possible to examine primary care records to see if patients were admitted within the 7 days to a different hospital; however given local geography and facilities this was felt to be unlikely.
Sample size was based on ten observed outcome events per predictor variable included in the analysis . Given the different frequencies of death and potential to benefit the sample sizes required for the two outcomes therefore varied.
Risk of each outcome in various groupings of predictor variables was explored visually using histograms (data available). Groups of similar risk were collapsed for further analysis. Given the absence of linear and monotonic relationships with risk, we wished to avoid arbitrary dichotomisation of data and therefore avoided decision tree analysis. Univariate association between potential predictor variables and outcome was assessed using logistic regression in SPSS, and first order interaction of variables significant at p < 0.15 was examined. Variables found to be significantly predictive of outcome at p < 0.1 were block entered into the multivariate analysis. Linear coefficients were then recalculated for independently predictive variables and an equation to predict poor outcome generated from those coefficients using the general formula p(outcome) = exb/1 + exb, where xb is the linear predictor.
The equations generated were applied to the validation sets without re-estimation of coefficients and performance assessed using ROC curves.
The derivation cohort included 2437 patients (1131 male, mean age 69) with 398 in the potential to benefit subset (183 male, mean age 66.5), while the validation cohort included 2322 patients (1093 male, mean age 70) with 227 in the potential to benefit subset (95 male, mean age 71.3) (fig 1 for CONSORT diagram). Table 2 compares demographic characteristics of both populations and subsets.
After grouping variables and collapsing groups with similar risk, variables significantly predictive of potential to benefit on univariate analysis are in Table 3. Full details of all variables assessed are in Additional file 1: Table S1.
Analysis of interactions showed significant interactions in: pulse by respiratory rate, systolic blood pressure and SaO2, respiratory rate by pulse pressure and SaO2, systolic blood pressure by pulse pressure and SaO2 and pulse pressure by SaO2. Briefly it appears that hypoxia confers increased risk if tachycardia is absent or if tachypnoea is present, and that normal pulse pressure confers increased risk in the presence of systolic hypotension.
Multivariate logistic regression analysis including interactions of oxygen saturations with pulse and respiratory rate produced estimated odds ratios (Exp(B)) and p values as in Table 4, with pulse, systolic blood pressure and GCS remaining significant. These were block re-entered into a multivariate analysis to develop linear coefficients for the final model:
where xb = (pulse[71–110]*2.373) + (pulse[>110]*9.533) + (sbp[<100]*3.614) + (sbp[100-120]*1.488) + (sbp[>180]*2.888) + (gcs[3-8]*13.28) + (gcs[9-12]*2.770).
The performance of this equation was assessed using a ROC curve, which had an area under the curve (c-statistic) of 0.737 (95 % CI 0.671–0.804). When applied to the validation set it had an area under the curve (c-statistic) of 0.707 (95 % CI 0.594–0.820).
The same process when performed to predict all deaths at 7 days (multivariate analysis in Additional file 1: Table S2, full details available) generated the equation below. Of note is the inclusion of age, temperature and co-morbidity which predict death but not potential to benefit.
where xb = (age[50–69]*0.608) + (age[70–85]*0.348) + (age[>85]*0.924) + (respiratory rate[19-23]*1.753) + (respiratory rate[>23]*2.791) + (DBP[<65]*3.172) + (DBP[>90]*5.204) + (sao2[high risk]*3.202) + (sao2[moderate risk]*0.932) + (temperature[<36]*4.714) + (GCS[3-12]*29.372) + (GCS[13-14]*5.311) + (respiratory disease [present]*12.355) + (respiratory disease [present] and temperature[<36]*7.466).
This had a c-statistic of 0.847 (95 % CI 0.8–0.894). When applied to the validation set, the equation had an area under the curve (c-statistic) of 0.741 (95 % CI 0.685–0.806).
We have demonstrated that a combination of pulse, systolic blood pressure and Glasgow Coma Score can predict provision of time-sensitive interventions with moderate discrimination. Different variables (age, respiratory rate, diastolic blood pressure, oxygen saturations, temperature, Glasgow Coma Score and a pre-existing diagnosis of respiratory disease) predict death within seven days. Thus a score developed to predict the need for urgent treatment will differ from a score developed to predict illness severity, suggesting that triage methods based on severity could lead to inappropriate prioritisation on the intended basis of urgency.
This is the first study specifically to address the identification of at-risk patients in an ED population not preselected for diagnosis or severity, and to use the provision of a life-saving intervention as an outcome measure. It therefore addresses the issues facing emergency clinicians more closely than the existing literature.
Although our definitions of time-sensitive interventions might be criticized for being defined arbitrarily, the concept of an “Emergency Care Sensitive Condition” is being more widely embraced and supports our methodology . It is apparent that patterns of physiological derangement differ between patients who die and those who receive life-saving intervention . The development of a tool to identify this second group was therefore necessary.
Patients who were not admitted to hospital and those who self-presented were excluded from the initial study dataset as the aim was to develop a risk-adjustment tool for emergency admissions to hospital. This limits the generalisability of this study in terms of developing a clinical score. Ideally a full cohort of presenting patients would be studied and those discharged from the ED followed up to analyse post-discharge adverse events. However that is logistically unfeasible, as rates of short-term death after discharge from the ED are 30–50/100,000 [17, 18], and only 11 % of self-presenting patients are admitted to hospital , so the required cohort size would have been impractical. Obviously our findings cannot be applied to other patient groups such as children or trauma patients.
Our definition of an inevitable death might be considered overly restrictive but we chose this deliberately to include the widest possible group as having potential to benefit.
As a single-site study it may be that these results are not generalisable; interpretation of vital sign derangement is not only affected by patient factors but also by the health care system, staffing levels and types and time available for patient care . Thus a process of external validation might find that life-saving interventions are provided differently in other settings.
We initially wished to examine the role of clinician gestalt in detection of the at-risk patient. Early data collection included a “yes/no” question to the transporting paramedic as to whether the patient was critically ill; this had to be abandoned due to poor rates of completion.
Implications for clinical practice and policy
The variables we have identified as predictive of a need for life-saving intervention are not the same as those in use in many standardized early warning scores. This may reflect the inappropriateness of developing early warning scores using data sets in which death is the main outcome. This score may be more appropriate than existing scores but should not yet be widely applied in standard practice. It needs wider validation, ideally including comparison with unstructured clinician (doctor or triage nurse) gestalt and with NEWS as the currently mandated standard of care. There must also be consideration as the score is applied of whether the outcomes used to develop definitions of urgency are still valid; the interventions listed in the Appendix are acknowledged to be based on incomplete evidence; it is to be hoped that as the evidence base for emergency care is developed these can be refined (for example, intravenous magnesium in acute severe asthma would no longer be considered a potentially life-saving intervention ).
These results highlight the potential flaws in applying clinical scores to predict outcomes other than those for which they were originally derived. As the health economist Tony Culyer said “capacity to benefit is not identical to need” , and clinicians should be clear about the reasons for which a score is being used. Equally, if a scoring system is used as casemix adjustment in an attempt to assess or improve quality, it should be clear that it identifies conditions which are amenable to alteration with good care .
This leaves the working emergency clinician in the situation of not having a score developed for and demonstrated to work in the emergency setting. There are two options: firstly to use an existing score but to recognise its limitations in the ED; secondly not to use a score but to rely on the unstructured judgement of clinical staff. In the current culture of high regard for standardised paperwork easily amenable to retrospective audit this is unlikely to be managerially palatable. Given the current state of equipoise over the utility of standardised scores in terms of patient benefit clinicians should also be encouraged to participate in formal research to address the issue.
Implications for research
Researchers in other settings have demonstrated the value of changes in physiological scores in prognostication ; ideally ongoing research would examine the prognostic value of response (or non-response) to treatment provided prehospitally or in the ED.
We have defined a priori a group of interventions which appear on best available evidence to be potentially life-saving and time-sensitive. However, the evidence supporting these is incomplete and our beliefs underlying many frequently-used interventions would bear further scrutiny; patients who could benefit could then be more reliably identified. We made no attempt to assess functional outcome of those patients defined as having potential to benefit; future researchers may wish to consider whether morbidity should also be a component of benefit.
Identification of and response to the patient at risk requires more than a reliable scoring system; complex psychosocial and cognitive factors affect decision-making, particularly in the pressured ED environment , and examination of the interaction of these factors should be a priority for future research .
In summary, we have developed a score that predicts urgency (potential to benefit from time-sensitive treatment) with moderate discrimination. The score has different constituent variables and weights than a score developed to measure severity (risk of death). Early warning scores developed to predict death may not be useful predictors of urgency.
Health and Social Care Information Centre. Provider level analysis for HES admitted patient care 2011–12 and 2012–13. Leeds: HSCIC; 2013.
Hospital Episode Statistics. Adult Critical Care in England – April 2010 to March 2011: Experimental Statistics. London: Health and Social Care Information Centre; 2012.
Finlayson TL, Moyer CA, Sonnad SS. Assessing symptoms, disease severity, and quality of life in the clinical context: a theoretical framework. Am J Manage Care. 2004;10:336–44.
Challen K, Goodacre SW. Predictive scoring in non-trauma emergency patients: a scoping review. Emerg Med J. 2011;28:827–37.
Grady D, Berkowitz SA. Why is a good clinical prediction rule so hard to find? Arch Intern Med. 2011;171:1701–2.
Hillman KM. Rapid response systems: you won’t know there is a problem until you measure it. Crit Care. 2011;15:1001.
Gao H, McDonnell A, Harrison DA, Moore T, Adam S, Daly K, et al. Systematic review and evaluation of physiological track and trigger warning systems for identifying at-risk patients on the ward. Intensive Care Med. 2007;33:667–9.
Chan PS, Jain R, Nallmothu BK, Berg RA, Sasson C. Rapid response teams: a systematic review and meta-analysis. Arch Intern Med. 2010;170:18–26.
Goodacre S, Wilson R, Shephard N, Nicholl J. Derivation and validation of a risk adjustment model for predicting seven day mortality in emergency medical admissions: mixed prospective and retrospective cohort study. BMJ. 2012;344:e2904.
Edbrooke DL, Minelli C, Mills GH, Iapichino G, Pezzi A, Corbella D, et al. Implications of ICU triage decisions on patient mortality: a cost-effectiveness analysis. Crit Care. 2011;15:R56.
Cardoso LT, Grion CM, Matsuo T, Anami EH, Kauss IA, Seko L, et al. Impact of delayed admission to intensive care units on mortality of critically ill patients: a cohort study. Crit Care. 2011;15:R28.
Kaji AH, Schriger D, Green S. Looking through the retrospectoscope: reducing bias in Emergency Medicine chart review studies. Ann Emerg Med. 2014;64:292–8.
Platts-Mills TF, Travers D, Biese K, McCall B, Kizer S, LaMantia M, et al. Accuracy of the Emergency Severity Index triage instrument for identifying elder Emergency Department patients receiving an immediate life-saving intervention. Acad Emerg Med. 2010;17:238–43.
Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol. 1996;49:1373–9.
Berthelot S, Lang ES, Quan H, Stelfox HT. Identifying emergency-sensitive conditions for the calculation of an emergency care inhospital standardized mortality ratio. Ann Emerg Med. 2014;63:418–24.
Churpek MM, Yuen TC, Edelson DP. Predicting clinical deterioration in the hospital: The impact of outcome selection. Resuscitation. 2013;84:564–8.
Sklar DP, Crandall CS, Loeliger E, Edmunds K, Paul I, Helitzer DL. Unanticipated death after discharge home from the Emergency Department. Ann Emerg Med. 2007;49(6):735–45.
Gabayan GZ, Derose SF, Asch SM, Yiu S, Lancaster EM, Poon KT, et al. Patterns and predictors of short-term death after Emergency Department discharge. Ann Emerg Med. 2011;58:551–8.
Jones D, Mitchell I, Hillman K, Story D. Defining clinical deterioration. Resuscitation. 2013;84:1029–34.
Goodacre S, Cohen J, Bradburn M, Stevens J, Gray A, Benger J, et al. The 3Mg trial: a randomised controlled trial of intravenous or nebulised magnesium sulphate versus placebo in adults with acute severe asthma. Health Technol Assess. 2014;18(22):1–168.
Cookson R, Claxton K, editors. The Humble Economist: Tony Culyer on Health, Health Care and Social Decision Making. York: University of York; 2012.
Kellett J, Woodworth S, Wang F, Huang W. Changes and their prognostic implications in the abbreviated VitalpacTM early warning score (ViEWS) after admission to hospital of 18,853 acutely ill medical patients. Resuscitation. 2013;84:13–20.
Croskerry P. From mindless to mindful practice — cognitive bias and clinical decision making. N Engl J Med. 2013;368:2445–8.
Gaddis GM, Greenwald P, Huckson S. Toward improved implementation of evidence-based clinical algorithms: clinical practice guidelines, clinical decision rules, and clinical pathways. Acad Emerg Med. 2007;14:1015–22.
Thanks to Richard Wilson and Martina Santarelli for their contribution to the original DAVROS project, and to Jon Nicholl for his constructive criticism of both projects.
Dr Challen received an MRC PhD stipend for this work but the study sponsor had no role in the study design, the collection, analysis and interpretation of data, the writing of the manuscript or the decision to submit the manuscript for publication.
The authors declare that they have no competing interests.
KC and SG conceived the study and designed the trial. KC supervised the conduct of the trial and data collection. SG and MB provided statistical advice on study designed and supported KC in the analysis of the data. KC drafted the manuscript and all authors contributed substantially to its revision. KC takes responsibility for the paper as a whole. All authors read and approved the final manuscript.
Acute care guidelines used to identify potentially lifesaving interventions
Acute illness: NICE clinical guideline 50
Acute onset atrial fibrillation: NICE clinical guideline 36, SIGN 94 and 129
Alcohol-related disease: NICE clinical guideline 100
Anaphylaxis: College of Emergency Medicine
Arrhythmia: SIGN 94
Asthma: SIGN 101
Cardiac arrest: SIGN 94
COPD: NICE clinical guideline 101
DVT: NICE clinical guideline 144 and SIGN 122
Headache: College of Emergency Medicine and American College of Emergency Physicians
Hypertension: American College of Emergency Physicians
LVF: American College of Emergency Physicians and European Society of Cardiology
NSTEMI/Unstable angina: NICE clinical guideline 94, SIGN 93, American College of Emergency Physicians and European Society of Cardiology
PE: NICE clinical guideline 144, SIGN 122, American College of Emergency Physicians and European Society of Cardiology
Pneumonia: American College of Emergency Physicians and British Thoracic Society
Pneumothorax (spontaneous): British Thoracic Society
Poisoning: NICE clinical guideline 16, American College of Emergency Physicians and College of Emergency Medicine
Seizure: American College of Emergency Physicians and College of Emergency Medicine
Sepsis: NICE clinical guideline 151
STEMI: SIGN 93 and European Society of Cardiology
Stroke/TIA: NICE clinical guideline 68, SIGN 108.
Transient loss of consciousness: NICE clinical guideline 109
Type 1 diabetes: NICE clinical guideline 15.
Upper GI bleeding: NICE clinical guideline 141, SIGN 105
Urinary tract infection: SIGN 88
About this article
Cite this article
Challen, K., Bradburn, M. & Goodacre, S.W. Development and validation of a score to identify in the Emergency Department patients who may benefit from a time-critical intervention: a cohort study. Scand J Trauma Resusc Emerg Med 23, 67 (2015). https://doi.org/10.1186/s13049-015-0150-y
- Emergency Department
- Glasgow Coma Score
- Urgent Treatment
- Early Warning Score
- Critical Care Admission