Developing a tool for measuring the disaster resilience of healthcare rescuers: a modified Delphi study

Abstract

Background

Disaster resilience is an essential personal characteristics of health rescue workers to respond to disasters in an effective manner, and maintain a state of adaptation after deployment. It is essential for disaster managers to recruit, assess, and prepare healthcare rescuers with this characteristic. A specific tool for measuring the disaster resilience of healthcare rescuers has yet to be devised.

Objective

The purpose of this study was to establish the content validity of a tool for measuring the disaster resilience of disaster rescue workers.

Methods

A modified Delphi approach was employed. Experts in disaster work and research were invited to rate the domains and items of a prototype tool for measuring disaster resilience in healthcare rescue workers. The panel of experts rated the relevance of the items using a 4-point Likert scale. The median and interquartile range, as well as the level of agreement, were calculated for each item using the Kendall coefficient W, to assess the consensus of the experts. The content validity index (CVI) was calculated to assess the content validity of this tool.

Results

A total of 22 and 21 experts were involved in the first and second rounds of this modified Delphi study (response rate of 91.7 and 95.5%), respectively. After two rounds of expert query, an eight-domain and 27-item disaster resilience measuring tool was established. The median range of all of the included items was 3.50 to 4.00 and the interquartile range was 0.00 to 1.00, and all items achieved ≥85% agreement. The Kendall coordination coefficient W was 0.21 and 0.33 in the first and second rounds, respectively, with P < 0.01. The I-CVI ranged from 0.85 to 1.0, while the S-CVI/UA and S-CVI /Ave were 0.69 and 0.97, respectively.

Conclusion

Consensus was reached on a disaster resilience measuring tool covering 27 items. The content validity of this tool for measuring the disaster resilience of healthcare rescuers was excellent. This tool is validated and ready to be tested in a pilot study to assess its psychometric properties.

Background

Resilience is regarded as the ability to “bounce back” from disaster, sustaining well-being and life satisfaction without negative psychological symptoms over time [1]. Resilience is also considered one of protective factors against occupational burnout [2, 3]. It has been suggested that healthcare rescue workers who have a high level of disaster resilience are not only less likely to suffer from negative psychological problems such as anxiety and depression, and posttraumatic stress disorder (PTSD), but also work more effectively [4,5,6]. Thus, disaster resilience is essential for the health and well-being of both disaster rescuers and the survivors of a disaster. It is desirable for disaster rescue workers to be recruited from among those with a high level of resilience.

An annual average of 77,144 deaths due to disasters were recorded between 2000 and 2017 [7]. Recent data show that 10,373 lives were lost in 2018 because of catastrophic events such as earthquakes, tsunamis, and volcanic activity, which is a demonstrable decline. The prevalence of Post-traumatic stress disorder PTSD among healthcare disaster rescuers was reported to be as high as 28.6% at 8 months after the Yushu earthquake in China [8]. Nurses who responded to the 2008 Wenchuan earthquake were at higher risk of suffering from PTSD (30%), compared to other healthcare rescuers [9].

Studies have also suggested that factors protective of resilience, such as social support and coping strategies, can be modified, learned, or cultivated through intervention programs [10, 11]. Thus, it is possible to design and develop interventions to foster the resilience of rescuers who are at a high risk of suffering from negative psychological consequences. There is also a need to have a valid and reliable tool for measuring disaster resilience, for use in recruiting disaster rescue workers and in evaluating the effectiveness of interventions that have been developed to enhance the resilience of individuals.

Existing instruments, such as the Connor-Davidson Resilience Scale (CD-RISC) [12, 13] and the Resilience Scale [14] have been used in studies to measure resilience among disaster rescue workers. However, these instruments were originally developed based on the general population or on patients with psychological disorders rather than specifically on rescue workers. Instruments that are “borrowed” from other populations or contexts may not be appropriate for the specific population or context of interest [15]. It is thus inappropriate to use an existing non-specific measuring tool to screen rescue workers for resilience in the recruitment process, or to evaluate the effectiveness of intervention programs aimed at fostering resilience in disaster rescue workers. As there is no specific resilience scale that can serve as a “gold standard,” and no specific instrument to measure the disaster resilience of rescue workers, the use of “borrowed” instruments on resilience has led to confusion in disaster management and research. It is therefore imperative to develop a valid and reliable instrument specifically for assessing the disaster resilience of disaster rescue workers in the context of disaster deployment.

Validation of a prototype tool

To our knowledge, there is no consensus on a framework for assessing disaster resilience in healthcare rescuers. A prototype tool for measuring the disaster resilience of rescue workers was developed by the research team. The tool was developed based on an extensive review of the literature on the characteristics of resilience among disaster rescue workers, a concept analysis of the concept “disaster resilience” and a focus group interview study of disaster healthcare rescuers, who were asked to give their views on disaster resilience [16]. Based on the results of these works, a scoping review of the tools for measuring the resilience of adults was conducted, and a prototype disaster resilience tool for healthcare rescuers was developed. The scale consists of eight domains: optimism, altruism, preparations for disaster, social support, perceived control, self-efficacy, coping strategies, and positive growth.

Methods

This study adopted a modified Delphi method to validate the instrument. A modified Delphi was a kind of technique of establishing consensus among a panel of experts on a topic of interest [17]. A traditional Delphi process begins with an open-ended questionnaire, which is time-consuming and usually leads to a low response rate [18, 19]. In a modified Delphi study approach, experts are consulted in the very first round using a structured questionnaire developed based on extensive reviews of the literature and / or on a focus group interview study [20]. The use of a modified Delphi process is appropriate when basic information concerning the target issue / topic is available and usable [17].

An online modified Delphi approach [19, 21] was the approach adopted in this study to obtain the judgment of a panel of independent experts on this specific issue, on which there is insufficient knowledge and research evidence to provide guidance on practice [22].

The aim of this study was to refine the domains and items of the prototype tool for measuring disaster resilience among healthcare rescuers and to establish the content validity of the items in that tool.

Panel selection

A purposive and criterion-based sampling method [23, 24] was adopted for selecting the members of the panel.

In a Delphi study, the experts represent those from various geographical locations [25], who are knowledgeable [19, 24, 26], possess professional and special expertise [27], have attained a certain level of educational status [28], and are willing to participate in the survey [26, 28, 29].

It has been suggested that a sample of experts can be identified through conferences [30] and published literature [31]. The potential experts for this study were acquaintances from international conferences / workshops on disaster nursing/management, such as the World Society of Disaster Nursing (WSDN, Germany, October 2018) and the Asia Pacific Emergency and Disaster Nursing Network (APEDNN, Cambodia, November 2018), as well as internationally known experts identified from published research studies / books on topics related to disaster healthcare.

The experts who were involved in academic and / or empirical work on disasters were selected in accordance with the purpose of this project [32]. They are from various geographical locations: the United States of America, the United Kingdom, Australia, Japan, South Korea, Taiwan, Hong Kong, and mainland China. The members of the panel of experts in the present study were selected from different countries based on the following criteria for inclusion: [1] the possession of a bachelor’s degree or above [2]; relevant experience / significant contributions in disaster management, disaster nursing / medicine / healthcare, or disaster-related research; and [3] at least 5 years of disaster-related clinical or academic experience. Those who could not read English, or could not be reached by electronic means via computer / email were excluded.

In the literature on Delphi studies, it is suggested that ten to fifteen subjects could be sufficient [17, 26]. It is also common for three experts to be considered sufficient for assessing the content validity of an instrument that has been developed [33]. Our aim was to recruit at least 15 international experts to take part in this study.

Format of the prototype tool for validation

The experts were invited to provide comments on the domains / components of the tool for measuring the disaster resilience of healthcare rescuers by rating the relevance of each item of the prototype tool on a 4-point Likert scale, with 1 = not relevant, 2 = somewhat relevant, 3 = quite relevant, 4 = highly relevant [34]. Although a 3-or 5-point rating scale is the commonly adopted format for validation, a 4-point Likert scale was adopted to obtain an even number of possible responses to avoid a neutral and ambivalent result at mid-point [33].

A pilot test of the validation form of the prototype tool was conducted among three experts who were not included in the panel of expert in the Delphi rounds. The pilot was to estimate the time required to complete the form and to ensure the clarity of the items. The experts suggested that information on the background and aims of this study should be provided, and information the background and demographics of the experts in the Delphi survey should be collected. Some minor clarifications were made to the wording of the items.

Data collection procedure

At the commencement of the Delphi expert query, an invitation letter, information sheet with an explanation of the background and aim of the Delphi survey, together with the prototype tool survey form, were sent to the experts via email. These experts were those whom the researchers had approached while attending various conferences or whom they had identified from the literature and contacted by email. All of them agreed to take part in this Delphi query.

The panelists were asked to rate the relevance of each item on a 4-point Likert scale. The experts were also given the opportunity to suggest additional domains and items that might not have been included in the tool, and to give comments on the tool at the end of the survey form. As it has been suggested that ten to fourteen days should be a sufficient interval between rounds of assessment for expert query [35], the experts were given 2 weeks to return their ratings and comments. An email of reminder was sent to those in the panel who had not given their feedback after 2 weeks. If there was still no response within the next 2 weeks (4 weeks in total), it was concluded that the expert was not available or no longer interested in taking part in the study, and no further attempts were made to contact that person.

There was a two-week interval between the rounds of the Delphi survey. During this period, the feedback from the experts was summarized, scrutinized, and studied to refine the prototype tool. This feedback was also sent to the panel of experts in the next round. Only those who took part in the first round were invited to take part in the subsequent round(s) of the Delphi survey. In those subsequent round(s), the panelists were asked to rate the items using the same criteria for assessment described earlier.

The number of rounds depended on the level of consensus reached [19, 36], and the amount of time available [24]. Recent evidence has shown that two to three rounds are sufficient in a modified Delphi study [37,38,39,40]. The number of rounds of surveys in this study ceased when a consensus was reached on all items, as indicated when 70% of the experts reached an agreement [41,42,43].

Statistical analysis

After the completion of each round of the Delphi survey, the data were inputted for statistical analyses into SPSS (Statistical Package for the Social Scientists) software version 25.0 for Windows (IBM Corp., New York, NY, USA). Consensus is one of the most contentious components of the Delphi method [36]. The median and interquartile range, as well as the level of agreement, were calculated to evaluate the consensus for each item in this Delphi research [44]. A consensus was considered to have been reached on the inclusion of an item in the disaster resilience scale if the median of the item was up to 3.25 on a 4-point scale [17], the interquartile range was less than 1 [45, 46], and the level of agreement was at least 70% [19]. The Kendall coefficient W test was adopted to evaluate the consensus on agreement among the panel of experts [47]. A two-tailed p-value of < 0.05 was considered statistically significant.

The content validity was calculated using the content validity index (CVI) [33]. Content validity was computed for each item (I-CVI) as well as for the overall scale (S-CVI), including the universal agreement (S-CVI/UA) and average (S-CVI/Ave). In this study, both I-CVI and S-CVI were calculated, and the values of statistical significance for I-CVI, S-CVI/UA, and S-CVI /Ave were set at ≥0.78, 0.8, and 0.9, respectively [48].

Ethical considerations

Ethical approval for this study was obtained from the Human Research Ethics Committee of the School of Nursing, the Hong Kong Polytechnic University (HSEARS20190102004), and the West China Hospital, Sichuan University (2019#65). The experts were informed that their participation in this Delphi study was voluntary. Experts who returned their ratings of the tool were considered to have given their implied consent to participate in this study. In our study, only the researchers knew the name of the experts, and no individuals are identified in the report.

Results

This Delphi survey took place from 4th February to 20th April 2019. A consensus on the items was achieved after two rounds of the survey. A flow diagram of the Delphi process is given in Fig. 1.

Fig. 1

Flow diagram of panel selection and Delphi process

Before commencing the first round of the Delphi survey, a total of 38 experts were invited to take part in this modified Delphi study. Twenty-eight of them responded, of whom 24 had the willingness and time to become involved in this study. Emails were then sent to these 24 experts. A total of 22 of them gave feedback on the prototype tool, although a few required email reminders before doing so (for a response rate of 91.7%). In the second round, emails were sent to these 22 experts. In the end, a total of 21 experts completed the Delphi survey (for a response rate of 95.5%). The demographic characteristics of the panel experts are presented in Table 1.

Table 1 The characteristics of the experts in panel

Table 2 shows the median, interquartile range, and level of agreement of all items in the prototype tool in the first round of the Delphi study. Regarding the 66 items in the first draft of the prototype tool, a total of 81 comments were received from experts. The researchers held a meeting to discuss these comments. As a result of the discussion, a total of 17 items were accepted and their wording was revised as suggested, 25 items were merged into 11 items, and 3 items were added. Another 19 items were regarded as irrelevant by 8 experts or did not meet the criteria for a consensus and were deleted as suggested. The Kendall’s coefficient of concordance (W) of the first round of the Delphi survey was calculated to be 0.21 (P < 0.01). The prototype tool was reduced from 66 to 36 items after the first round of the survey.

Table 2 The median, interquartile range, and the level of agreement of items in the first-round query

Table 3 shows the median, interquartile range, and level of agreement of all items in the disaster resilience tool following the second round. Another meeting was held among the researchers to discuss the comments and suggestions that were received. A total of 8 items were deleted since a consensus was not reached on them. A total of 12 items were accepted but the wording of these items was changed to achieve more precision in proper English, 4 items were merged into 2 items due to overlap, and 1 item was added after the discussion and approval from all researchers. The Kendall’s coefficient of concordance (W) in the second round of queries was 0.33 (P < 0.01). The final version of a 27-item tool for measuring the disaster resilience of healthcare rescue workers was established (Table 4).

Table 3 The median, interquartile range, and the level of agreement of items in the second-round query

Table 4 The final version of the disaster resilience tool for healthcare rescuers after a two-round Delphi survey

After two rounds of the modified Delphi survey, the I-CVI for the disaster resilience tool for healthcare rescuers ranged from 0.85 to 1.0. The S-CVI/UA and S-CVI /Ave were 0.69 and 0.97, respectively. The consensus of all items for the disaster resilience tool was reached. Therefore, the Delphi experts survey was completed after two rounds.

Discussion

To the best of our knowledge, this is the first study to validate a tool for measuring the disaster resilience of healthcare rescuers through the use of a Delphi survey to gauge the views of experts in the field of disaster work and research. After two rounds of a web-based modified Delphi survey, a 27-item tool for screening the disaster resilience of rescuers was identified (Table 4). The outcome of this study, the measuring tool, can be used as a reference to recruit and identify disaster rescue workers who have the characteristics of disaster resilience, or as a tool to evaluate the effectiveness of resilience training programs for healthcare disaster rescue workers [49]. The modified approach adopted in this Delphi survey is considered superior to the original approach because it is highly effective and less time consuming [50, 51].

Having an “expert panel” is central to the process of the Delphi technique, although there are no standard criteria for determining expertise [52]. In the current study, the panel of experts comprised people from seven countries/cities who are in diverse professions, such as university academics, physicians, and nurses. They are from the fields of disaster nursing, disaster medicine, disaster education, disaster management, and disaster research. A panel consisting of experts from different geographical locations [25] and areas of professional expertise [27, 53] will produce better results than a panel comprised of those from the same field [54].

Among the experts in our panel, some had taken part in national and / or international disaster rescue work, and the panel as a whole reflected the full range of stakeholders with a common interest [55]. These experts can also be regarded as “consumers” with lived experience of disaster rescue [23]. Therefore, the measuring tool from this Delphi study can be applied in all countries by disaster practitioners, educators, researchers, and management personnel.

Through their active and very timely responses, the expert panel in this study showed strong motivation and interest in taking part in the Delphi survey. Although there is no strict rule for what is considered an acceptable response rate for Delphi studies, a response rate of 70% is suggested necessary for each round [19]. The response rates of the two rounds of Delphi surveys in this study were higher than 90%, and a great number of constructive suggestions and comments were received, indicating the experts’ considerable enthusiasm and interest in this topic. This may be related to the fact that most of the experts also considered it is important to have such measuring tool, and that they were approached in disaster-related international conferences and invited in person. The short timeframe between the two rounds of surveys (2 to 3 weeks) also served to keep the subject fresh in their minds and prevent fatigue. This also helped to enhance the content validity of this modified Delphi study, ultimately strengthening the validity of the results [24].

The Content Validity of this study was good. The I-CVI of the included items varied from 0.85 to 1.00, which is higher than the recommended level of 0.78, suggesting that the content of each item of the disaster resilience tool is excellent [48]. The S-CVI/UA was 0.69, which did not reach the acceptable value of 0.8. This can be explained by the larger sample size of experts in this study for the calculation of S-CVI/AV, and the consequent risk of disagreements [48]. The S-CVI /Ave was as high as 0.97, indicating that the content validity of the whole scale is excellent. Thus, the content validity of this disaster resilience tool can be regarded as excellent, although some of the items may need to be slightly revised based on the comments or suggestions of the expert panel.

After two rounds of surveys, the experts reached a consensus on all of the items that were finally included in the disaster resilience tool. The median range of all of the included items was 3.50 to 4.00 and the interquartile range was 0.00 to 1.00, while the statement of all items demonstrated ≥85% agreement, indicating good consensus [41,42,43]. The Kendall coordination coefficient W, used to assess the agreement among several expert evaluators [56], was 0.207 in the first round and 0.33 in the second round, with P < 0.01. This suggests a highly significant level of consensus among the experts in the panel.

As with any research, there are some limitations to this study that need to be acknowledged. First, to prevent peer influence in a Delphi study, each member of the panel of experts should not know about the others [57]. However, the members of the expert panel in this study were mainly approached during international conferences. Thus, it was inevitable that some of these experts would know about each other, which means that absolute anonymity could not be achieved in this study. Nevertheless, all of the experts rated the items of this tool independently at their own location, so the results of the rating of each item were anonymous. Second, because the experts lived in different countries/cities, no face-to-face meetings were held among them in the process of carrying out this modified technique. Finally, the opinion of the panel of experts in this study may not be representative of all experts within the field of disaster studies, as experts from other countries that are frequently affected by disasters, such as India and Indonesia, were not involved in this study.

In spite of its limitations, this study provides significant information for disaster management. In the next step, this developed disaster resilience tool is to be validated for its psychometric quality, including its reliability and validity, in a cross-sectional study on healthcare disaster rescue workers. Researchers, management, and policy makers will then have a validated tool to use in recruiting or assessing disaster resilience among healthcare rescuers.

The validation process would include the following: translating the tool into languages other than English if it is to be validated in countries where English is not the main language, conducting a pilot test among a sample of disaster healthcare rescuers to assess the clarity and pertinence of the items in the language of the country where the tool is being tested and, finally, conducting a cross-sectional survey among a large sample of disaster healthcare rescuers to test the reliability and construct validity of the tool.

Conclusion

This study has established a tool for assessing the disaster resilience of healthcare rescuers, using a modified Delphi technique. The tool is a scale made up of a total of eight domains and 27 items. The panel of experts reached a consensus on all of the items in this scale, and the items and the overall scale were found to have excellent content validity. A study to establish the psychometric properties of this scale is needed in the next step before it can be used as a tool in the recruitment and management of disaster healthcare rescue workers.