Simulation-based randomized trial of medical emergency cognitive aids

Abstract

Background

Medical emergencies are complex and stressful, especially for the young and inexperienced. Cognitive aids (CA) have been shown to facilitate management of simulated medical emergencies by experienced teams. In this randomized trial we evaluated guideline adherence and treatment efficacy in simulated medical emergencies managed by residents with and without CA.

Methods

Physicians attending educational courses executed simulated medical emergencies. Teams were randomly assigned to manage emergencies with or without CA. Primary outcome was risk reduction of essential working steps. Secondary outcomes included prior experience in emergency medicine and CA, perceptions of usefulness, clinical relevance, acceptability, and accuracy in CA selection. Participants were grouped as “medical” (internal medicine and neurology) and “perioperative” (anesthesia and surgery) regarding their specialty. The study was designed as a prospective randomized single-blind study that was approved by the ethical committee of the University Duisburg-Essen (19-8966-BO). Trial registration: DRKS, DRKS00024781. Registered 16 March 2021—Retrospectively registered, http://www.drks.de/DRKS00024781.

Results

Eighty teams participated in 240 simulated medical emergencies. Cognitive aid usage led to 9% absolute and 15% relative risk reduction. Per protocol analysis showed 17% absolute and 28% relative risk reduction. Wrong CA were used in 4%. Cognitive aids were judged as helpful by 94% of the participants. Teams performed significantly better when emergency CA were available (p < 0.05 for successful completion of critical work steps). Stress reduction using CA was more likely in “medical” than in “perioperative” subspecialties (3.7 ± 1.2 vs. 2.9 ± 1.2, p < 0.05).

Conclusions

In a high-fidelity simulation study, CA usage was associated with significant reduction of incorrect working steps in medical emergencies management and was characterized by high acceptance. These findings suggest that CA for medical emergencies may have the potential to improve emergency care.

Background

Medical emergencies are stressful and complex events that require rapid and coordinated care in a time-critical setting. Guideline-adherent management has been recognized as an important outcome-relevant factor (1;2). Data concerning unexperienced physician’s management of medical emergencies showed potential deficits in the work-up (3) and thus, unexperienced teams often fail to properly manage medical emergencies. Room for improvement was identified during emergencies (e.g. cardiac life support) as well as in the early phase after a critical event (2;4). Checklists and protocols have long been accepted in high-risk industries (e.g., aviation and nuclear power) as a tool to aid performance during rare and unpredictable critical events and as components of the safety culture translated to medical practice (3;5). The use of surgery-specific safety checklists during routine operative care have been associated with significant reductions in morbidity and mortality (3) and has thus become a standard of care (6;7). In medicine, the use of checklists, amongst others, has been associated with improved patient outcomes in circumstances where the checklist can be used for largely linear processes such as the preparation for central venous catheter insertion (5). While the usage of checklists is thought to improve care, they simultaneously bear the risk that clinicians may feel restricted (8), and, if wrong checklist are selected they may influence overall team-performance as well. In addition to checklists, there are a number of other tools and instruments designed to work with in medicine. As the text-based algorithms used in this study do not contain any obvious actions or criteria to "check off", we chose the term “medical emergency cognitive aid” (MECA) instead in order to ensure transparency with readers.

In the current study, we sought to analyze the effect of unknown MECA for multi-professional teams of physicians in a simulated setting, using a high-fidelity medical simulation to facilitate a structured observation of the medical emergencies. We tested the hypothesis that MECA would significantly reduce the number of incorrect process steps in the work-up of medical emergencies, expressed as absolute and relative risk reduction in an intention to treat and per protocol analysis of a randomized, controlled trial involving physicians (trainees) from different professions (e.g., anesthetists, surgeons, neurologists, internists) as well as different levels of experience.

Methods

Study design

The current analysis was performed during five educational critical care medicine courses from end of 2019 to mid of 2020. The study was designed as a prospective randomized single-blind study that was approved by the ethical committee of the University Duisburg-Essen (19–8966-BO) and published in a national clinical trial data register (DRKS00024781; www.drks.de).

Participants and setting

On a regular basis, the “Arbeitsgemeinschaft Intensivmedizin”, Arnsberg, Germany (http://www.aim-arnsberg.de) organizes educational courses for physicians from all over Germany working in intensive care medicine. For most participants (usually residents in their 2nd to 3rd year of clinical practice in internal medicine, anesthesia, surgery, or neurology) this course represents their first post-graduate training course related to intensive care. Interdisciplinary plenary lectures about medical emergencies of all kinds were held prior to the start of the workshops as part of the course content. Course participants were offered to take part in voluntary simulator-based emergency training. Teams of three to six physicians each were equally randomized (single blinding) with research randomizer (www.randomizer.org) to perform simulated emergency scenarios with or without a dedicated MECA folder. Structure and contents of the MECA were briefed immediately prior to the emergency training. We used checklists adapted by JM and TS in their institution from a work by Arriaga et al.in 2013 (3). The MECA folder contained a total of 10 life-threatening medical emergencies, including cardiac arrest, bradycardia, difficult airway, hypoxia, bleeding, pulmonary embolism, anaphylaxis, in German language, with three determined to select by the course format (see Additional file 1: Appendix A). The degree of implementation of critical process steps was determined (CONSORT flow chart; Fig. 1). Written informed consent was obtained for study participation and video recording of the emergency scenarios. Additionally, the participants were asked to fill in a questionnaire on the topic.

Fig. 1

CONSORT Flow chart showing the test set-up. MECA = medical emergency cognitive aid. Due to the experimental procedure, neither an exact assignment of questionnaires/ participants (MECA used?) nor an exact assignment of participants/ MECA (when used?) is possible

Scenario and checklists

All participants received a standardized introduction to the simulation environment and were made familiar with the manikins used and the medical equipment available. For only those groups allocated to use checklists, the folder including all MECA was handed over directly before starting emergency simulation. Teams consisted of three to six physicians each. Team members were then informed that their role during the following scenarios was that of an in-hospital rapid response team (RRT) called to the hospital’s emergency department (ED). The core RRT consisted of at least three doctors (anesthesiologist, internist, and surgeon, all random and not necessarily corresponding to the real profession) and a nurse (played by another doctor). The allocation of roles between MECA lecturer leader and team members was open free and up to the teams themselves.

The patient introduced by an ED physician was either suffering from a myocardial infarction deteriorating into ventricular fibrillation, suffering from hemodynamic unstable ventricular tachycardia (VT) or bradycardia deteriorating into asystole caused by hyperkalemia. All rhythm disorders could be diagnosed on the display of the defibrillator simulator once attached. The study period ended with return of spontaneous circulation after the third defibrillation in the first setting, after termination of ventricular tachycardia by cardioversion in the second setting and after treatment of hyperkalemia in the third setting. Manikins were operated by trained tutors who were instructed to refrain from any intervention until the end of the study period. Except for this study-specific instruction, scenarios with or without MECA were identical. “Critical action items” (labelled red and asterisked for a better traceability in Additional file 1: Appendix A) were predetermined by interdisciplinary consensus on the basis of binary assessability (performed/ not performed). In case of lengthy scenarios it was up to the tutor to schedule them on time if necessary.

Structured survey assessment

Following the scenario, participants were asked to complete a structured questionnaire. Since checklists or cognitive aids are already more or less widely used in medicine in Germany, all participants (i.e., also those randomized to the non-MECA group) were asked about any previous experience with checklists. In addition to basic demographic data (age, gender, medical specialty) and prior experience (general, intensive care and emergency medicine work experience), prior use of checklists or related tools as well as statements about the usefulness of MECA were evaluated. The latter parameters were stratified by using a 5-scale Likert scale. Participants were also asked to indicate their position within the scenarios (team member or leader, checklist lecturer). To allow better comparability, specialties were added up into "medical" (internists and neurologists) and “perioperative” (surgeons and anesthesiologists), a division that has been done comparably before [9, 10].

Video analysis of checklist usage

Interrater reliability was assessed in the beginning of the analysis and randomly by two independent reviewers (SA, FB). In case of disagreement or uncertainty among reviewers regarding adherence by the team to a key process, these were resolved by bringing in external additional experts (assessment by a senior anesthesiologist, or an emergency physician; FB, TS, SA) and concluded by a joint consensus. Due to structural and technical requirements, video analysists could not be blinded. The use of MECA by a participating team was not blinded either. Start for the timing of all events was defined to be the start of handover to the team. The correct choice of MECA within the intervention arm was evaluated binarily (yes or no). Each scenario was basically evaluated with eight critical work steps. These were derived from evidence-based guidelines and also scored in a binary fashion. Usage of wrong MECA resulted in zero points. In case of premature termination of a scenario (e.g., prior third defibrillation before application of amiodarone or after first cardioversion before increase of cardioversion energy), the total number of critical steps to be achieved was reduced accordingly. Teams committing relevant protocol violation (i. e. mandatory interference by the tutor) were excluded.

In an additional per protocol analysis, the following scenarios were excluded from evaluation: teams choosing the false MECA, teams using no MECA even though provided, and teams committing any protocol violation (e.g., interference with the tutor).

Endpoints

The primary outcome measure was the error rate reduction of critical work processes for each emergency. In the overall evaluation, all critical steps (including misapplication of MECA) were assessed and thus the relative and absolute risk reduction was calculated. Risk reduction of each individual scenario (myocardial infarction, hemodynamically unstable VT, hyperkalemia-induced asystole) was determined separately. All analyses were performed for the 240 scenarios and for the additional per protocol analysis.

Secondary endpoints included basic demographic data, a survey regarding participants` perceptions of the usefulness and clinical relevance of the MECA, prior medical and emergency care knowledge, pre-existing experience with MECA, influence of experience and specialization, quality of the MECA and the emergency scenarios and overall acceptance as well as accuracy of checklists usage. Additionally, encouragement of the participants` subjective sense of security and the influence of the MECA on the interaction during the scenario were evaluated.

Statistics

All data were analyzed on an intention to treat as well as per protocol basis. Following testing for standard deviation (SD), data were expressed as means ± SD unless otherwise stated. Student’s t-test and Chi square test were applied as appropriate. All reported p-values are two-sided, and a p < 0.05 was considered significant. Statistical analysis was performed using SPSS (version 22). Due to the purely observational character of the study, no power analysis was performed to determine a sample size.

Results

Basic demographic data

Overall 520 physicians took part in our study (52% female, 48% male) and 95% were less than 40 years of age. Previous experience was available in 48% for ICU work and in 17% for MECA use.

The participants, all from specialties related to emergency medicine, had a wide range of years of experience in their specialty (Table 1), with about a half (52%) being in the 2nd or 3rd year of their specialist training (residency). The relation “medical” (335 internists, 28 neurologists) to “perioperative” (anesthetists 85, surgeons 46) was 363 to 131. Seven participants belonged to “other” specialties (not explicitly specified). During the study about a third (34%) of the participants acted as team leader, 61% as team members and 5% as MECA presenters.

Table 1 Previous overall work and emergency care medicine knowledge of the participants

A total of 80 rapid response teams participated in three directly consecutive medical emergency simulation scenarios each, adding up to 240 simulated events. Forty teams executing 120 scenarios received MECA (intervention group), whereas another forty teams served as control group without MECA (control group). In all scenarios, a maximum of 1920 critical process steps for analysis were available.

Analysis of checklist usage

In those scenarios with MECA, usage resulted in 9% absolute and 15% relative risk reduction of failure to adhere to guideline-adherent critical process steps. All teams had a lower failure rate for adherence to key processes when MECA were available. In those groups using MECA, failure of adherence was observed in 43% of critical process steps, in those groups without MECA critical process step were wrong or missing in 60% (p < 0.05, Fig. 2). The significant effect of MECA usage was seen for scenario one (CPR after MI) and three (hyperkalemia induced asystole) when stratifying the results according to the different scenarios.

Fig. 2

Completion of critical process steps – the use of MECA (medical emergency cognitive aids) resulted in 9% absolute and 15% relative risk reduction in the intention to treat and 17% absolute and 28% relative risk reduction in failure to adhere to critical work steps by per protocol analysis respectively (p < 0.05). Data are shown as median of failure rate, delineate the 5th and 95th percentile respectively. Dots displayed horizontally indicate values outside the 5th and 95th percentile. Left bar = intervention group (MECA available), right bar = control group (no MECA available)

Per protocol analysis of checklist usage

Medical emergency cognitive aids were used in 63 scenarios. Forty-one scenarios were performed without MECA usage even though provided. Eleven scenarios were excluded due to protocol violation. In five scenarios, teams chose the wrong MECA. Using an incorrect MECA resulted in worse performance than groups that did not use a MECA at all, mainly based on a premature misinterpretation of the underlying cardiac rhythm. In the control group, seven scenarios had to be excluded due to protocol violation. All exclusions in both groups were due to the need for active interference by the tutor.

Per protocol checklist usage resulted in a 17% absolute and 28% relative risk reduction of failure to adhere to guideline-adherent critical process steps. The significant effect of MECA usage was also seen when stratifying the results according to the different scenarios (Table 2). Here, we saw a significant reduction of the failure rate per scenario for each checklist (p < 0.05).

Table 2 Effect of MECA use according to the scenarios (per protocol analysis)

Analysis of structured survey assessment responses

A total of 501 participants (96%) completed the questionnaire. All participants rated the overall quality of the session as above average or excellent (score of 4 or 5, respectively). A total of 94% of the participants found the use of MECA reasonable. Participants reported that the MECA were easy to use, that the MECA helped them feel better prepared, and that they would use the MECA if presented with these medical emergencies in real life. The only significant difference was found in the evaluation of the checklist regarding the reduction of the stress level: “medical” rated a higher score on the 5-scale-likert range than “perioperative” residents (3.7 ± 1.2 vs. 2.9 ± 1.2, p < 0.05). Detailed information is given in Table 3.

Table 3 Participants’ perceptions of medical emergency cognitive aids (MECA)

Discussion

In medicine, different data exist on the effectiveness of checklists and MECA: whereas prior research in the perioperative setting showed a highly significant reduction of missed critical process steps (6% when checklists were available vs. 23% when they were unavailable) [3], the implementation of a pediatric sedation safety checklist failed to show a significant reduction in sedation-related adverse events [11]. Additionally, there is conflicting evidence for the effectiveness of checklists to improve perioperative outcomes in some populations [12]. In intensive and emergency care medicine, the implementation of a multidisciplinary safety checklist during bedside bronchoscopy-guided percutaneous tracheostomy was independently associated with a 580% reduction in adverse procedural events [13] and the implementation of a preintubation checklist for ED intubation of trauma patients was associated with a 7.7% absolute risk reduction [14]. Implementation of a multifaceted quality improvement intervention with daily checklists, goal setting, and clinician prompting did not reduce in-hospital mortality among critically ill patients treated in ICUs in Brazil [15]. Since data on risk reduction using checklists in intensive care and emergency medicine still seem to be very limited, it is difficult to classify our results precisely [5]. In order to correctly classify the observed effect, it depends on the intention to treat population. If MECA were not applied, or only partially applied, or in the worst case, the wrong MECA was chosen (expectably leading to worse results than in the control group), this is a very crucial consideration in the evaluation of the system. A per protocol analysis may be helpful to further break down the observed effect. Additionally, checklists may be designed as diagnostic (Scriven schema) or problem-solving (Higgins and Boorman schema), but, as the text-based algorithms used in this study do not contain any obvious actions or criteria to “check off”, we chose the term “medical emergency cognitive aid” (MECA) instead of checklist in order to ensure transparency with readers [16]. In our simulations, MECA were subjectively judged as helpful by 94% of the participants. Overall, our results suggest that in a high-fidelity simulation MECA use by residents of different specialties led to a relevant risk reduction in the management of medical emergencies. Teams performed significantly better when MECA were available even though the MECA used were previously unknown to the study participants. In case of insufficient performance using MECA, those teams have simply refrained from using the checklists to the extent provided. Interestingly the use of wrong MECA led to an even worse performance than not using MECA at all, which resulted from a consecutive fault after misinterpretation of the underlying cardiac rhythm. Our study participants, mostly 2nd or 3rd year residents, were quite equally distributed in terms of age, gender and overall work experience. “Perioperative” residents had significantly more pre-experience in ED medicine and pre-hospital trauma life support (PHTLS), which is in line with previous data from Germany [17]. Participants’ perceptions of MECA revealed a high level of acceptance across all specialties with a single exception regarding potential stress reduction which was significantly more obvious in “medical” versus “perioperative” residents. The underlying cause mainly remains uncertain, and we can only speculate on the reasons. With the introduction of the WHO surgical checklist in 2008 [18], checklists have become an indispensable part of daily surgical and anesthesiologic routine. Thus, maybe “perioperative” residents were more experienced in the overall checklist or MECA use. Additionally, “perioperative” residents had significantly higher preexisting emergency medicine experience, which could also contribute to the fact that the stress-reducing effect in this group was not as pronounced as in the group of "medical" residents.

Participants with prior experience using checklists who were not randomized to the MECA group also rated them mostly favorably.Although our results are promising and inspiring concerning a broader introduction of MECA in hospitals, their use can also be problematic when applied to clinical problems that require nonlinear responses [5] and there may be a risk of therapeutic misalignment [5,6,7,8] with the delivery of excess or inappropriate interventions, like in sepsis. To minimize this bias, we deliberately selected established guidelines that can easily be described by checklists, algorithms, MECA or protocols [16,17,18,19,20]. If teamwork-training initiatives are combined with the implementation of MECA, this may confound the results [7]. In our study, MECA—although available—were generally not used in 34% of cases. In the per protocol analysis, MECA were used in 63 of the 120 scenarios. We can only speculate why this was the case despite the broad fundamental agreement in the survey conducted afterwards. With regard to the approximately 50% usage in the intervention group, it is hard to reconcile the 94% of the 501 respondents who subjectively judged the checklists to be useful. We cannot exclude a bias in this context. One has to bear in mind that self-reported perceptions are weaker compared with other more objective forms of evidence related to the impact and efficacy of MECA. However, we anticipate that training the teams with their individual roles within the teamwork before the emergency and, most importantly, training the teams in the handling of the MECA would have improved MECA utilization and may have led to a further reduction in the failure of critical work steps so that our results are potentially underestimated. In fact, the only information our participants received before the start of the scenario was that a total of 10 MECA would be available. In order not to distort the results, the issue of specific MECA content was not broached in advance. In our analysis of the performances of the two groups, we excluded teams who selected to use checklists, but nonetheless, selected the wrong ones during the simulation case scenarios. By doing so, there was a slight chance that this omission of data would inadvertently inflate the ratings of performances of the intervention group. Indeed, a recalculation showed only negligible differences. One might discuss how far the “critical steps” selection process contributes to the construct validity of the ratings based on the scores generated. We aimed to provide a scoring system as transparent and comprehensible as possible and thus chose a binary technique as this method seemed to be most appropriate. Additional limiting factors for MECA usage are lack of time and high stress levels in an emergency. This might explain why most groups used the MECA only brief at the beginning, whereas other groups did not use MECA until they were stuck. Only 10 groups systematically worked through the MECA from the beginning to the end, resulting in > 90% execution of correct work steps.

Simulator-based studies nearly always lack real patients. However, high-fidelity simulation has become an accepted part of medical training and evaluation [3]. Overlooking other fields, like aviation, show that simulation is an established and efficient part of testing and assessing (a) the value of safety protocols and (b) possible consequences in case of deferring from such protocols without the often-deleterious results seen in real life. In case of medical simulation, high agreement rates with findings in real cases have been reported [21, 22]. Simulation also helps to make rare events, like the emergencies used in this study, trainable and enabling to investigate topics that for a variety of practical and ethical reasons would be very difficult to investigate in real cases. Further strengths include the number of participants and the identical conditions for all teams which would have been impossible to achieve outside of a simulated setting. Video-analysis and the high number of checkpoints enabled us to perform a detailed analysis. We are sure about a benefit for trainees when using MECA, however with the numbers as low as they are, that is impossible to say with certainty. Although this may not really be generalizable by now, our study contributes its part to literature, and paves the way for further investigations with additional data and analysis.

Limitations

One might fear that the use of MECA is potentially contrary to the physician's "therapeutic freedom" and sometimes considered as criticism of one's own competence. Clinicians may feel restricted in their available responses [8]. Strategies to overcome this and further barriers have already been introduced [19]. In the design of our research, we chose to place the MECA in the hands of the frontline providers, where cognitive load is greatest. Another approach might be considering the effect or impact of placing this tool in the hands of another team members (e.g., a nurse in the role of recorder). Many RRT are comprised of interprofessional team members; removing this tool from the "front-line" healthcare provider and placing it in the hands of someone engaged in the response but tasked with a different role. This might be an interesting next direction to consider. Again, this approach might yield an even better demonstration of outcomes associated with MECA use than observed in this study.

It would have been interesting to examine both correctness and efficiency. For this reason, response (processing) time could have been explored between groups to better reflect possible benefits of MECA as well. Due to strict timey regulations we have chosen against temporal measures in the presented study, since no differences were expected.

The error rate of critical work processes, our primary endpoint, may be discussable in the context whether the absolute or relative risk reduction is of primary interest or likewise whether it is the effect in the intention to treat or in the per protocol population. The latter point can actually be answered clearly, since according to all guidelines the ITT population should be the primary evaluation population in a superiority trial. Considering this, still an effect, though less pronounced, was observable. Not being able to blind the video analysist could have affected the review process. However, as only critical care process steps were counted, the influence of a non-blinded analyst should be considered to be lower at most.

Finally, it would have been desirable to be able to test not only the occupational groups but also the two study arms among themselves for equal distribution. However, due to the anonymization requirements, an exact assignment of the respondents to the study arms was unfortunately impossible.

Conclusions

In our simulation, the use of MECA in medical emergency situations significantly reduced failure rates. The use of MECA was widely accepted, and MECA were easy to use. In a high percentage, stress level of the participants was diminished. Based on our data, an introduction of MECA at least for the ED, the intensive care unit and the in-hospital emergency response teams could be considered. Additionally, MECAs could be used beforehand as teaching aids to avoid potentially fatal misinterpretations in an emergency. Further studies should focus on the transferability to the pre-clinical sector or high-risk in-hospital settings (e.g., catheter laboratory) and their impact on the performance in key aspects (e.g., more benefit and less harm for patients). MECA may lead to a destructive outcome. Therefore, one of the priorities in MECA/checklist implementations in the future should be how to secure the correct cognitive aid selection for a critical incident.