Reliability and validity of triage systems in paediatric emergency care
© van Veen and Moll; licensee BioMed Central Ltd. 2009
Received: 9 June 2009
Accepted: 27 August 2009
Published: 27 August 2009
Triage in paediatric emergency care is an important tool to prioritize seriously ill children. Triage can also be used to identify patients who do not need urgent care and who can safely wait. The aim of this review was to provide an overview of the literature on reliability and validity of current triage systems in paediatric emergency care
We performed a search in Pubmed and Cochrane on studies on reliability and validity of triage systems in children
The Manchester Triage System (MTS), the Emergency Severity Index (ESI), the Paediatric Canadian Triage and Acuity Score (paedCTAS) and the Australasian Triage Scale (ATS) are common used triage systems and contain specific parts for children. The reliability of the MTS is good and reliability of the ESI is moderate to good. Reliability of the paedCTAS is moderate and is poor to moderate for the ATS.
The internal validity is moderate for the MTS and confirmed for the CTAS, but not studied for the most recent version of the ESI, which contains specific fever criteria for children.
The MTS and paedCTAS both seem valid to triage children in paediatric emergency care. Reliability of the MTS is good, moderate to good for the ESI and moderate for the paedCTAS. More studies are necessary to evaluate if one triage system is superior over other systems when applied in emergency care.
Large numbers of patients visit the emergency department. Consulting patients in the order of attending will, in a crowded emergency department (ED), lead to long waiting times for seriously ill patients. It is important to prioritise patients who are seriously ill and would be at increased risk of morbidity or even mortality due to delay in the initiation of treatment.
The aim of triage is to determine and classify the clinical priority of patients visiting the ED.  During a short assessment the nurse will identify signs and symptoms that determine the patient's urgency. The physician will see the patients in order of their urgency level. Patients requiring immediate care are identified. Moreover, patients are identified who can safely wait longer or who can be seen by another caregiver such as the general practitioner or nurse practitioner.
Triage systems are developed by expert opinion. [2–5], the lowest level of evidence, and are mainly based on the adult population visiting the ED. The Paediatric Canadian Triage and Acuity Scale (PaedCTAS) was especially modified for the paediatric population.  Several studies have investigated the reliability and validity of triage systems in children. [6–17]
The aim of this review is to provide an overview of the current scientific knowledge of triage systems for the broad population of children visiting the ED.
We performed a search for literature in May 2009 using Cochrane and the following MeSH terms in Pubmed, "triage" [MeSH Terms] AND "emergency medical services" [MeSH Terms] AND ("infant" [MeSH Terms] OR "child" [MeSH Terms] OR "adolescent" [MeSH Terms]) AND (validity [All Fields] OR accuracy [All Fields]). Secondly we performed a wider search for "triage" [MeSH Terms] AND system [All Fields] AND "emergency medical services" [MeSH Terms] AND ("infant" [MeSH Terms] OR "child" [MeSH Terms] OR "adolescent" [MeSH Terms])
Studies were selected if they described a triage system for the broad population visiting the emergency care or reported about a study on reliability or validity of a triage system for emergency care, applied to the paediatric population. Studies on triage for a subpopulation were not included as well as for triage systems applied in the developing world. We included papers published between 1999 and 2009. Finally, reference lists of the included papers were checked for relevant publications using the same selection criteria.
The narrow search gave 44 hits, of which 12 were selected because of the title; one article was excluded following reading of the abstract. The broad search resulted in 112 hits of which six extra articles were selected.
Triage systems in paediatric emergency care
Worldwide, the Manchester Triage System (MTS) [1, 5, 18], the Emergency Severity Index (ESI) [19, 20] the Canadian triage and acuity scale (CTAS)  and the Australasian triage scale (ATS)  are consensus based and commonly used triage systems in emergency care. Although different criteria per triage system are used, they all sort patients into five urgency categories.
Manchester Triage system (MTS)
A second version of the MTS was published by the Manchester Triage group in 2006.  Some discriminators were modified or added (for example 'pain' in level 4 was modified to 'recent pain' for flowcharts in which pain is one of the discriminators). 
In a large validation study we identified subgroups of patients in which the validity of the MTS for children was low, such as young patients, patients with a non-traumatic presenting problem and older patients with fever. 
Emergency Severity Index (ESI)
The ESI is a 5-level triage system, developed in the United States. Level 1 stands for the highest acuity level and level 5 for the lowest acuity. Patients requiring immediate life-saving interventions are allocated into level 1 and must be seen immediately. Patients in a high risk situation, who are confused, lethargic, disoriented, have severe pain or distress or have deviated vital signs/PO2 are attributed to level 2. A physician should see these patients within ten minutes. Level 3 is for patients who are expected to require two or more resources. Level 4 is attributed if one resource is expected to be required and 5 if no resources are expected to be required. Resources can be diagnostics (for example lab tests, ECG, X-rays, CT scan etc), treatment (for example IV fluids, laceration repair) or specialty consultation. Patients triaged as level 3–5 can safely wait for several hours. 
In the fourth version of the ESI, a specific flowchart for children with fever was added. It uses age, the height of fever, the cause of fever and whether the child is immunized to determine urgency. Children younger than 28 days with a temperature >38.0°C are allocated to level 2. Children with fever aged 28 days – 3 months are assigned to level 2 or 3, depending on the hospital's institutional protocol. Children aged 3–36 months who are under immunized or who have no obvious source of fever and a temperature >39.0°C are allocated to level 3. 
Canadian Triage and Acuity Scale (CTAS)
In 2001 a specific guideline to triage children was added to the CTAS, (paedCTAS). Per presenting problem, specific criteria are provided to allocate patients to different urgency levels. For example for children presenting with respiratory distress, for level 1 signs are: inability to speak, cyanosis, lethargy or confusion, tachycardia or bradycardia, and hypoxemia with O2 saturation <90%. For level 2 the signs audible stridor, intermittent respiratory distress and audible wheezing, tachypnea, or cough are listed in order to select patients with respectively upper respiratory distress, congenital vascular anomalies and foreign bodies or lower airway concerns. Level 3 is for patients with moderate respiratory distress such as patients with pneumonia, bronchiolitis or croup. Level 4 and 5 do not contain criteria for patients with respiratory distress.
Medical care should be delivered immediately for level 1, within 15 minutes for level 2, within 30 minutes for level 3, within 60 minutes for level 4 and within 120 minutes for level 5.  A detailed recent description of the paedCTAS can be found at the website http://www.caep.ca/template.asp?id=B795164082374289BBD9C1C2BF4B8D32
Australasian triage scale (ATS)
Formerly known as the National Triage Scale, the ATS provides criteria per urgency level. Most criteria are general but three criteria are specific for children: shocked child/infant should be allocated to level 1, all 'stable neonates' are allocated to level 3 as well as 'children at risk'. 
Pain in triage
In the MTS as well as the paedCTAS pain plays an important role in urgency classification. Both systems allocate patients with severe pain to a level 2 urgency. Patients with moderate pain and patients with mild/acute pain (paedCTAS) or recent pain (MTS) are triaged into level 4. [3, 5] The ESI allocates patients with severe pain to level 2. A lower pain score does not influence the ESI urgency level.  The Manchester pain scale correlated well with the Oucher pain scale, which is a common used and validated pain scale in emergency care. 
Referral of low urgency patients to other caregivers
Besides prioritising urgent patients, triage systems are used to identify patients with a low urgency. These patients can safely wait, do not need urgent care and could as well be seen by another health professional. One study showed that the CTAS, when applied to adults and children is not valid to safely identify low urgency patients with the aim to refer them to other caregivers.  For other triage systems such as the MTS and the ESI, this question still needs to be answered.
Research on reliability and validity of triage systems
Validity of a triage systems is determined by reliability (inter-rater agreement and intra-rater agreement) and whether or not the triage system can predict the true urgency (internal validity) The external validity determines the value of the system in different settings.  The inter-rater agreement is determined by the agreement in triage urgency level if multiple nurses triage one patient or patient scenario. The intra-rater agreement presents the agreement in triage urgency level if one triage nurse triages one case scenario at different points in time. The inter- and intra-rater agreement is dependent on the uniformity and completeness of a triage system and on how the triage nurse applies the system. Good training and instruction of the triage nurses can optimise the usage and interpretation of triage systems.
Inter- and intra-rater agreement are usually analysed using Cohen's kappa. Kappa provides a measure of agreement between observers, corrected for agreement expected by chance.  In case of an ordinal scale, which is the case when 5-level triage systems are studied, quadratic and linear weighted kappa analysis provide different weights per amount of disagreement.  If the inter-rater agreement between multiple observers is studied, the intraclass correlation coefficient (ICC) can be used. It can easily be calculated using SPSS and is equivalent to a quadratic weighted kappa, under certain conditions. 
To assess validity, a 'gold standard' as a proxy for urgency has to be defined. Since it is difficult to determine the 'true urgency', different approaches are currently used to assess validity. Outcome measures such as hospitalisation, ICU admission, resource uses, total length of stay at the ED or costs of an ED consultation are used. [6, 8, 13]
We studied the validity of the MTS in children in a large prospective observational study by comparing the MTS urgency level with a predefined, independently assessed reference standard for urgency.  We defined the highest urgency level for patients with deviated vital signs according to the PRISM (Paediatric Risk of Mortality),  patients with a potentially life threatening diagnosis were defined as level 2, patients were allocated to level 3 or 4 depending on if they were hospitalised after ED consultation and the amount of diagnostics and therapeutic interventions performed at the ED. Patients allocated to level 5 did not meet the criteria for level 1 or 2, were not hospitalised, and no diagnostics or therapeutic interventions were performed during their ED visit. A detailed description of the reference standard was published before.  It is important to triage a patient and to assess the reference independently, in order not to overestimate validity. 
Assessing urgency per case by experts is another way to assess validity. However, these judgements are quite dependent on the used protocols in the hospital and the personal experience of the expert.
Validity can be expressed in sensitivity and specificity of a triage system. Sensitivity presents the ability for a triage system to identify high urgent patients. Specificity presents the ability for a triage system to identify patients with low urgent problems. The 'Likelihood Ratio for a positive test results' (LR+) represents the ratio between the chance on a high urgency test result in patients with a true high urgency and the chance of a high urgency test results in patients with a true low urgency. [25, 30]
Validity is analysed in some studies by assessing agreement between the triage system urgency and a reference urgency, using kappa statistics. [6, 13] Van der Wulp et al suggested a triage weighted kappa in which under-triage (when the triage urgency is lower than the reference urgency) is weighted as more severe than over-triage (when the triage urgency is higher than the reference standard urgency).  Lee at al proposed a weighted scheme (error weights) for a 3-level triage system, in which under-triage was weighted twice as over-triage. They calculated sensitivity, specificity, positive and negative predictive value incorporating these error weights. 
Reliability and validity of triage systems in paediatric emergency care
Studies on reliability of the ESI, CTAS, MTS and ATS in paediatric emergency care
7 paper, 7 computer based scenarios
K 0.40 (paper)
K 0.58 (computer)
97 nurses (44%)
Written case scenarios
ESI version 3, children
Written case scenarios
ESI version 3, children
Kw 0.59 (95% CI 0.55–0.63)
18 nurses (62%)
Written case scenarios
Kw 0.51 (95% CI 0.50–0.52)
Lineair Kw 0.55 (95% CI 0.48–0.61)
Quadratic Kw 0.61 (95% CI 0.42–0.80)
The Netherlands 
48 nurses (87%)
MTS adults and children
Written case scenarios
The Netherlands 
43 nurses (100%)
MTS in children
Written case scenarios
Quadratic Kw 0.83 (95% CI 0.74–0.91)
Quadratic Kw 0.65 (95% CI 0.56–0.72)
Studies on validity of the ESI, CTAS, MTS in paediatric emergency care
Before and after design, prospective study
Admission rate, medical interventions, and
PRISA score, comparison with previous used triage tool (4 level)
Previous triage tool had better ability to predict admission than paediatric CTAS
Admission, ICU admission
Length of stay (LOS)
Good correlation between urgency and admission, ICU admission and LOS
Costs of resource utilization
PaedCTAS urgency level correlates well with resource utilization
Prospective triage, retrospective chart review
Admission rate, medical interventions, PRISA score, comparison with used triage tool
ESI score predicts resource use, length of stay, and admission to hospital
The Netherlands 
Reference standard for urgency *
The Netherlands 
Reference standard for urgency *
The ESI has a moderate (actual simultaneous triage) to good (written case scenarios) reliability when applied to triage children. ESI urgency levels are correlated to resource use, length of stay at the ED.  The paedCTAS has a moderate inter-rater agreement using actual simultaneous triage. [9, 10]
Several validity studies of triage systems in children show a correlation of urgency levels with admission. A large study on the validity of the paedCTAS showed that 90% of the patients admitted to the PICU, were triaged as urgency level 1 or 2.
3 patients out of the total 58,529 were 'incorrectly' triaged as level 4 or 5.  Patients triaged as level 3–5 were admitted in 6% (out of 400 patients) using the ESI. , and in 7% (out of 510 patients) and 6% (out of 53,846 patients) using the paedCTAS. [8, 11]
Patients triaged as level 1 or 2 were admitted in 36% (out of 110 patients) using the ESI , and in 30% (out of 27 patients).  and 41% (out of 4683 patients) using the paedCTAS.  Percentage admission per urgency level is comparable between triage systems.
Furthermore, paedCTAS urgency levels are related to resource use and length of stay, although length of stay was shorter for level 1 patients compared to level 2 patients (191 minutes versus 250 minutes). [11, 33] The ATS showed a poor to moderate reliability. [34, 35] We did not find studies on the validity of the ATS for children.
The inter-rater agreement of the MTS in adults and children was studied in the Netherlands and showed a good to excellent reliability. [15, 17] For children the inter-rater agreement of the MTS is good (simultaneous triage of actual patients) to excellent (written case scenarios). Validity, expressed in agreement between the MTS and reference standard for urgency, shows 34% correct triage, 54% were over-triaged and 12% under-triaged. Sensitivity was 63% (95% CI 59–66) and specificity 79% (95% CI 79–80). 
Several triage systems are extensively used to triage children at the emergency department. Several studies are performed to assess the reliability and validity of these systems in children.
The aim of triage is to identify high urgent patients. Triage systems that show a large proportion of under-triage or perform a low sensitivity (real high urgent patients are triaged as low urgent) are therefore unsafe.
Since it will be difficult for a triage system to reach 100% sensitivity and specificity, a good balance between over- and under-triage is important. A high sensitivity may result in a low specificity resulting in many patients with real low urgent problems who will be treated as high urgent. This may result in long waiting times for real high urgent patients.
Since outcome measures used for validity studies are different, a comparison between triage systems cannot be made on how they predict 'true' urgency. However, from the available studies and the design of the triage systems, some points can be made. The ESI performs a moderate to good inter-rater agreement.  Inter-rater agreement for the paedCTAS is moderate when written case scenarios are used. When the paedCTAS is studied using real life scenarios, results are similar to the inter-rater agreement of the ESI. Reliability is good for the MTS [15, 17] and poor to moderate for the ATS.(Table 1)
Validity is confirmed for the MTS and paedCTAS. Validity of the paediatric fever criteria of the ESI was not studied. Since patients presenting with fever are 15% of the paediatric population , it is important to study these fever criteria as well. (Table 2) The MTS is both detailed and objective and discriminators are organized in flowcharts of presenting problems. The system contains several specific flowcharts for children. .
From a methodological view triage can be seen as a diagnostic test; predicting 'true' urgency. In that way sensitivity and specificity must be used as measures of performance.  A disadvantage of this method is that urgency levels following from a 5 level triage system should be dichotomised. When one chooses to combine the two highest levels of a triage system as 'high urgency' and the three lowest as 'low urgency', a distinction between the two highest levels and between the three lowest levels is not made anymore. However, the aim of triage is to identify true high urgent patients. A misclassification in the two highest urgency levels (level 1 or level 2) is clinically less important than a misclassification from level 2 to level 3, 4 or even 5. By dichotomising the 5 urgency levels and calculating sensitivity and specificity, weights are incorporated. Moreover sensitivity and specificity are very commonly used in diagnostic research and therefore easily interpretable by most users. 
Implementation includes application of the system to all patients and compliance to the advice for urgency by the ED nurses. The implementation of the triage system in practise is important for the triage process. Patients who enter the emergency department should be triaged as soon as possible. If children are sitting in a waiting room without being triaged, potentially dangerous delay in treatment can occur for potentially serious diseases.
Especially in a crowded emergency department it is important that there is a triage nurse whose primarily role is triage. She will perform a rapid assessment (30–60 seconds) and long conversations with patients should be avoided.  The founders of the ESI and the MTS claim that a complete assessment does not need to be done at the initial triage station, although sufficient information should be gained to be able to determine the correct triage category. [4, 5] Vital signs should be completed on all paediatric patients at some time during their emergency visit.  The triage nurse will take care that that all patients entering are directly triaged (within 10 minutes of arrival)  while other nurses take care of further observation and treatment of patients.
As for implementation of clinical prediction rules, certain criteria should be met for successful implementation. At first predictions of the triage system should be better than that of the users. Secondly, users should feel that the system is valid (face validity). Since wide validation of triage system is often lacking, this is a point for improvement. Thirdly the system should be user friendly. The best predictors of a rule to be used in practice are the familiarity acquired during training, the confidence in the usefulness of the rule, and the user-friendliness of the rule. [36, 37]
Computerized triage showed a better agreement in correct triage outcome, compared to triage without the support of a computerized application.  Application of the paedCTAS using a computerized application (Staturg) resulted in a better reliability of the system.  Therefore, a computerized application of a triage system should be used.  Especially the MTS and the CTAS are complex systems for which several questions should be answered before a triage advice is suggested.
Several systems are available for triage in paediatric emergency care. The MTS, ESI and CTAS contain parts specific for children. Evaluation of a triage system concerns research of reliability and validity. The MTS and paedCTAS both seem valid to triage children in paediatric emergency care. Available studies show that reliability of the MTS is good, is moderate to good for the ESI, moderate for the paedCTAS and poor to moderate for the ATS. More research is needed on the reliability and validity of triage systems when applied to children especially if they are used to identify low urgent patient for referral to another caregiver.
Manchester Triage System
Emergency Severity Index
Paediatric Canadian Triage and Acuity Scale
Australasian Triage Scale
- Mackway-Jones K: Emergency Triage, Manchester Triage Group. 1997, London: BMJ Publishing GroupGoogle Scholar
- Australian College for Emergency Medicine. Guidelines on the implementation of the australasian triage scale in emergency departments. [http://www.acem.org.au/media/policies_and_guidelines/G24_Implementation__ATS.pdf]
- Canadian Paediatric Triage and Acuity Scale: Implementation Guidelines for Emergency departments. Can J Emerg Med. 2001, 3 (4 Suppl): 10.1111/j.1553-2712.2005.tb00872.x.
- Gilboy N, Tanabe P, Travers D, Rosenau A, Eitel D: Emergency Severity Index, version 4: Implementation Handbook. 2001, Rockville: Agency for healthcare Research and Quality, [http://www.ahrq.gov/research/esi/esihandbk.pdf]Google Scholar
- Mackway-Jones K, Marsden J, Windle J: Emergency Triage, Manchester Triage Group. 2006, Oxford: Blackwell Publishing Ltd, SecondGoogle Scholar
- Baumann MR, Strout TD: Evaluation of the Emergency Severity Index (version 3) triage algorithm in pediatric patients. Acad Emerg Med. 2005, 12 (3): 219-224. 10.1097/01.pec.0000136067.07081.ae.View ArticlePubMedGoogle Scholar
- Bergeron S, Gouin S, Bailey B, Amre DK, Patel H: Agreement among pediatric health care professionals with the pediatric Canadian triage and acuity scale guidelines. Pediatr Emerg Care. 2004, 20 (8): 514-518. 10.1016/j.ajem.2004.02.046.View ArticlePubMedGoogle Scholar
- Gouin S, Gravel J, Amre DK, Bergeron S: Evaluation of the Paediatric Canadian Triage and Acuity Scale in a pediatric ED. Am J Emerg Med. 2005, 23 (3): 243-247. 10.1016/j.ajem.2004.02.046.View ArticlePubMedGoogle Scholar
- Gravel J, Gouin S, Bailey B, Roy M, Bergeron S, Amre D: Reliability of a computerized version of the Pediatric Canadian Triage and Acuity Scale. Acad Emerg Med. 2007, 14 (10): 864-869. 10.1111/j.1553-2712.2008.00268.x.View ArticlePubMedGoogle Scholar
- Gravel J, Gouin S, Manzano S, Arsenault M, Amre D: Interrater Agreement between Nurses for the Pediatric Canadian Triage and Acuity Scale in a Tertiary Care Center. Acad Emerg Med. 2008, 15 (12): 1262-1267. 10.1111/j.1553-2712.2008.00268.x.View ArticlePubMedGoogle Scholar
- Gravel J, Manzano S, Arsenault M: Validity of the Canadian Paediatric Triage and Acuity Scale in a tertiary care hospital. Cjem. 2009, 11 (1): 23-28. 10.1542/peds.114.2.356.PubMedGoogle Scholar
- Maldonado T, Avner JR: Triage of the Pediatric Patient in the Emergency Department: Are We All in Agreement?. Pediatrics. 2004, 114 (2): 356-360. 10.1016/j.jemermed.2006.01.011.View ArticlePubMedGoogle Scholar
- Maningas PA, Hime DA, Parker DE: The use of the Soterion Rapid Triage System in children presenting to the Emergency Department. J Emerg Med. 2006, 31 (4): 353-359. 10.1136/emj.2006.038877.View ArticlePubMedGoogle Scholar
- Roukema J, Steyerberg EW, van Meurs A, Ruige M, Lei van der J, Moll HA: Validity of the Manchester Triage System in paediatric emergency care. Emerg Med J. 2006, 23 (12): 906-910. 10.1136/emj.2007.055228.PubMed CentralView ArticlePubMedGoogle Scholar
- Wulp van der I, van Baar ME, Schrijvers AJ: Reliability and validity of the Manchester Triage System in a general emergency department patient population in the Netherlands: results of a simulation study. Emerg Med J. 2008, 25 (7): 431-434. 10.1136/bmj.a1501.View ArticlePubMedGoogle Scholar
- van Veen M, Steyerberg EW, Ruige M, van Meurs AH, Roukema J, Lei van der J, Moll HA: Manchester triage system in paediatric emergency care: prospective observational study. BMJ. 2008, 337: a1501-10.1136/bmj.a1501.PubMed CentralView ArticlePubMedGoogle Scholar
- van Veen M, Walle van der V, Steyerberg E, van Meurs A, Ruige M, Strout T, Lei van der J, Moll H: Repeatability of the Manchester Triage System for children. Emergency Medicine Journal. 2009, 10.1016/j.ienj.2008.11.008.Google Scholar
- Olofsson P, Gellerstedt M, Carlström ED: Manchester Triage in Sweden – Interrater reliability and accuracy. International Emergency Nursing. 2009, 17 (3): 143-8. 10.1136/emj.2006.039883.View ArticlePubMedGoogle Scholar
- Elshove-Bolk J, Mencl F, van Rijswijck BT, Simons MP, van Vugt AB: Validation of the Emergency Severity Index (ESI) in self-referred patients in a European emergency department. Emerg Med J. 2007, 24 (3): 170-174. 10.1136/emj.2008.059378.PubMed CentralView ArticlePubMedGoogle Scholar
- Storm-Versloot MN, Ubbink DT, Chin a Choi V, Luitse JS: Observer agreement of the Manchester Triage System and the Emergency Severity Index: a simulation study. Emerg Med J. 2009, 26 (8): 556-60. 10.1136/emj.2008.059378.View ArticlePubMedGoogle Scholar
- Canadian Association of Emergency Physicians: Canadian Paediatric Triage and Acuity Scale: Implementation Guidelines for Emergency Departments. Can J Emerg Med. 2001, 3 (4): 1-32.Google Scholar
- Triage in the emergency department. [http://wacebnm.curtin.edu.au/workshops/Triage.pdf]
- Lyon F, Boyd R, Mackway-Jones K: The convergent validity of the Manchester Pain Scale. Emerg Nurse. 2005, 13 (1): 34-38.View ArticlePubMedGoogle Scholar
- Vertesi L: Does the Canadian Emergency Department Triage and Acuity Scale identify non-urgent patients who can be triaged away from the emergency department?. Cjem. 2004, 6 (5): 337-342. 10.1111/j.1553-2712.1999.tb00121.x.PubMedGoogle Scholar
- Hardern RD: Critical appraisal of papers describing triage systems. Acad Emerg Med. 1999, 6 (11): 1166-1171. 10.2307/2529310.View ArticlePubMedGoogle Scholar
- Landis JR, Koch GG: The measurement of observer agreement for categorical data. Biometrics. 1977, 33 (1): 159-174. 10.1037/h0026256.View ArticlePubMedGoogle Scholar
- Cohen J: Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological bulletin. 1968, 70 (4): 213-220. 10.1177/001316447303300309.View ArticlePubMedGoogle Scholar
- Fleiss JL, Cohen J: The equivalence of weighted kappa and the intraclass correlation coeficient as measures of reliability. Educational and Psychological Measurement. 1973, 33: 613-619. 10.1097/00003246-199605000-00004.View ArticleGoogle Scholar
- Pollack MM, Patel KM, Ruttimann UE: PRISM III: an updated Pediatric Risk of Mortality score. Crit Care Med. 1996, 24 (5): 743-752. 10.1097/00003246-199605000-00004.View ArticlePubMedGoogle Scholar
- Hunink M, Glasziou P, Siegel P, Weeks J, Pliskin J, Elstein A, Weinstein M: Decision making in health and medicine: integrating evidence and values. vol. Fourth printing, 2005. 2001, Cambridge University PressGoogle Scholar
- Wulp van der I, van Stel HF: Adjusting weighted kappa for severity of mistriage decreases reported reliability of emergency department triage systems: a comparative study. J Clin Epidemiol. 2009, 10.1016/S0168-8510(03)00023-X.Google Scholar
- Lee A, Hazlett CB, Chow S, Lau F-l, Kam C-w, Wong P, Wong T-w: How to minimize inappropriate utilization of Accident and Emergency Departments: improve the validity of classifying the general practice cases amongst the A&E attendees. Health Policy. 2003, 66 (2): 159-168. 10.1016/j.ajem.2008.02.024.View ArticlePubMedGoogle Scholar
- Ma W, Gafni A, Goldman RD: Correlation of the Canadian Pediatric Emergency Triage and Acuity Scale to ED resource utilization. Am J Emerg Med. 2008, 26 (8): 893-897. 10.1016/j.annemergmed.2004.04.007.View ArticlePubMedGoogle Scholar
- Considine J, LeVasseur SA, Villanueva E: The Australasian Triage Scale: examining emergency department nurses' performance using computer and paper scenarios. Ann Emerg Med. 2004, 44 (5): 516-523. 10.1016/j.annemergmed.2004.04.007.View ArticlePubMedGoogle Scholar
- Crellin DJ, Johnston L: Poor agreement in application of the Australasian Triage Scale to paediatric emergency department presentations. Contemp Nurse. 2003, 15 (1–2): 48-60.View ArticlePubMedGoogle Scholar
- Brehaut JC, Stiell IG, Graham ID: Will a new clinical decision rule be widely used? The case of the Canadian C-spine rule. Acad Emerg Med. 2006, 13 (4): 413-420. 10.1016/j.jclinepi.2008.04.008.View ArticlePubMedGoogle Scholar
- Toll DB, Janssen KJ, Vergouwe Y, Moons KG: Validation, updating and impact of clinical prediction rules: a review. J Clin Epidemiol. 2008, 61 (11): 1085-1094. 10.1111/j.1553-2712.2005.tb00889.x.View ArticlePubMedGoogle Scholar
- Dong SL, Bullard MJ, Meurer DP, Colman I, Blitz S, Holroyd BR, Rowe BH: Emergency triage: comparing a novel computer triage program with standard triage. Acad Emerg Med. 2005, 12 (6): 502-507. 10.1136/bmj.38398.500764.8F.View ArticlePubMedGoogle Scholar
- Kawamoto K, Houlihan CA, Balas EA, Lobach DF: Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005, 330 (7494): 765-10.1136/bmj.38398.500764.8F.PubMed CentralView ArticlePubMedGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.