Impact of peer feedback on the performance of lecturers in emergency medicine: a prospective observational study

The lecture represents an economical and efficient method of conveying both factual and experiential knowledge to a large group of students and thus remains a fundamental part of the learning experiences of students during their medical education [1]-[5]. In their AMEE Medical Education Guide No. 22, Brown and Monogue conclude based on a review of the research on lecturing over the past 70 years that lectures are at least as effective for presenting and explaining conceptual and systematic knowledge and fostering enthusiasm and motivation for learning as other teaching methods [1]. Lectures offer a real-time, human presence with spoken communication, which for most people, is easy to learn from [2].

However, lectures, like all teaching methods, have their limitations and if administered incorrectly, can be boring and or even worse, useless. Although this didactic format is widely used and familiar to audiences, the skills required to prepare and to deliver an effective and structured lecture are mostly passed along through experiential learning and only seldom acquired by specific instruction in teaching techniques [1]. However, many studies exist that describe strategies for improving lecture presentation, ultimately increasing student learning [1],[4],[6],[7].

Reflection on practice is the cornerstone and most powerful source of continuing professional development in all teaching environments, but reflection on practice and change requires insight, effort, and a willingness to change [1],[8]. Although an educator’s teaching is mostly assessed by students, there is growing consent that effective assessment of teaching must emerge from multiple sources, especially peers, to provide essential data [9]-[13]. Feedback from peers and professional staff (faculty) developers is increasingly recognized as a valuable adjunct to surveys of student opinion. Such feedback can provide insights not possible based on student opinion alone. Effective peer assessment of teaching should be criteria-based, emphasize teaching excellence and use instruments that produce highly reliable measures [9],[14].

The aim of this study is to evaluate the effect of a standardized written peer assessment on the quality of a lecture series in emergency medicine for undergraduate medical students. Several studies have reported the development of instruments for peer assessment and assessed their feasibility and reliability in pilot runs [7],[12],[14],[15]; however, only McLeod et al. have described in a recent qualitative study the perceptions, benefits and shortcomings of peer assessment of reviewers and individuals reviewed [16]. However, this is the first study we are aware of with the intention to analyse the impact of peer assessment in a lecture series on the lecture itself.

In the first lecture series, the lecturers with no didactic training received the worst ratings for the items ‘provides a conclusion for the talk’, ‘encourages audience interaction’ and ‘linkage to previous knowledge’; the lecturers with didactic training received the worst ratings in the categories ‘provides a brief outline’, ‘clearly states goal of the talk’ and ‘provides a conclusion for the talk’. All of these items are included in the category ‘content and organisation’. The items that received the highest ratings and the most significant improvements after the intervention for both groups, ‘provides a brief outline’, ‘provides a conclusion for the talk’ and ‘clearly states goal of the talk’, were also in this category (p < 0.001).

The inter-rater reliability for the pair of raters who observed the same lecture was assessed using the Kappa coefficient. The raters were in good agreement for all of the criteria (0.82-0.9).

Lecturing has been criticised as ineffective compared with other methods of teaching that involve students as active participants in the learning process rather than passive observers. This is unfortunate because lecturing is often indispensable, especially for large classes with hundreds of students. Furthermore, when done effectively, lecturing can transmit new information in an efficient manner, explain or clarify difficult concepts, organize ideas and thoughts, challenge beliefs, model problem-solving, and foster enthusiasm and a motivation for learning [2],[3]. Didactic lecture will continue to be a mainstay in all parts of medical training [1]. As such, it is important to maintain and improve the quality of lectures.

To our knowledge, this is the first study analyzing the effect of a standardized written feedback on the performance of lecturers in a lecture series. Our results demonstrate that even with a ‘simple’ written feedback, lecturers effectively integrate their newly gained knowledge in future lectures, improving their teaching. Although we hypothesized that positive changes in the lecturers’ performances would occur after written peer feedback, we were extremely surprised by the extent of the improvement since only written feedback was provided, and no additional training occurred. The improvements made were independent of the lecturers’ experience as medical teachers and their prior didactic training.

Our findings are consistent with the existing literature. Providing feedback to faculty members has been shown to clarify performance quality and provide a formative assessment [11],[24],[25]. It facilitates self-reflection of teaching practices and encourages faculty to discuss their teaching skills and effective instruction [13],[25]. In our study, we were able to affirm the existing literature, demonstrating that feedback can help to close the gap between current performance levels and the desired goals of curriculum designers [24],[25]. The responses and reactions from the lecturers regarding the feedback were very positive with several lecturers completely revising their lecture (e.g. new slides, figures, and/or videos). These findings are consistent with the recent qualitative study of McLeod et al., demonstrating that all participants receiving peer review enthusiastically endorsed the benefits of peer assessment [16].

However, effective peer assessment of teaching should be criteria-based and use instruments that produce highly reliable measures [9],[14]. For this reason, in addition to a validated assessment instrument, adequate rater training is essential to ensure that all of the raters have internalized the rating standards and are committed to giving the necessary time and effort [14],[22]. In addition to using the assessment instrument during their postgraduate training, the raters discussed the rating standards during the rater training, using videos of former lectures. They focused on those items identified as difficult in both the literature and training to reach a consensus. The clear, descriptive benchmarks for the excellent, adequate and poor performance rating levels provided by Newman et al. [14],[22] helped in this area, facilitating powerful feedback by the raters. Thus, the raters were in agreement on all of the criteria in our study.

During the first peer assessment, the main deficits were in the items of the category ‘content & organisation’. After written feedback, this category showed the greatest improvement, with highly significant improvement in 6 of the 10 items. The items ‘provides a brief outline’ and ‘provides a conclusion for the talk’ showed the greatest improvement.

The smallest improvements were found in the category ‘delivery’. This category includes items such as ‘speech flow’ and ‘enthusiasm for the topic’. These items are innate to an individual and cannot be changed as easily without additional training compared with items in the other categories. We hypothesize that practical training e.g. with video feedback and clear information about the expected behaviour would have a higher impact on these items than written feedback. With respect to the raters, we were able to demonstrate that an agreement on a definition of ‘expected behaviour’ for these items can be achieved. To gain deeper insight into this specific area, further studies are needed.

Although the students were blinded towards the study, they noticed and appreciated the efforts of those lecturers who changed their lecture habits; several added under the “comments” section of their teacher evaluation that they appreciated the inclusion of clear learning objectives, an outline of the purpose and contents of the lectures and the manner in which questions were encouraged and addressed.

Both the students in lecture series 1 and lecture series 2 rated those lecturers with didactical training higher than those without training despite being blinded to the training experience of their lecturer. Thus, students are able to differentiate between lecturers with different teaching abilities.

We have regularly asked all students to evaluate each lecture and have provided the detailed results to all lecturers since 2005. However, we found only small improvements in the quality of the lectures following the students’ feedback. These findings are consistent with the statement of Newman et al. [14] arguing that the faculty undergoing a review needs to trust that the ratings are not idiosyncratic scores but reflect their actual performance. This means that the assessment instrument must be reliable as measured by inter-rater agreement, the rater must be respected and the feedback must be as specific as possible in its items and provide points of action [14].

This study has some limitations because it was conducted at a single medical school with only one study sample of lecturers in emergency medicine, which might restrict its explanatory power and its transferability to other medical schools. However, this limitation does not diminish the significance of the results and the pronounced impact of the written peer feedback on the performance of most of the lecturers. It may serve as a model for the development of similar programs in all levels of medical training to improve instructional effectiveness.

The fact that the reviewers rated a lecturer based on a videotape of the lecture rather than a live presentation is also a limitation. Providing feedback based on videotaped teaching sessions can be criticised because the real environment and atmosphere cannot be completely captured [22]. Furthermore, it cannot be guaranteed that the raters are not disturbed (e.g. answering a phone call) or watch the video in discontinuous segments. However, our results regarding inter-rater reliability have acceptable high levels for all of the items.

This study does not investigate the lecturers’ self-assessments and compare self-assessments to peer ratings. To gain an insight into this area and the effect of peer feedback on self-assessment, future research is needed.

This study demonstrates the significant impact of a single standardized written peer feedback on lecture quality. Based on this study, the assessment instrument and study design will be used as a basis to evaluate and improve additional lecture series in other disciplines at our medical school.