Skip to main content

Development and evaluation of an appraisal form to assess clinical effectiveness of adult invasive mechanical ventilation systems



Rapid developments in intensive care medicine have made mechanical ventilation an essential method in the resuscitation and comprehensive treatment of critical care patients. This study aimed to develop and evaluate an appraisal form assessing the clinical effectiveness of adult invasive mechanical ventilation systems.


An appraisal form was designed according to the effectiveness evaluation theory of the American Weapons Systems Effectiveness Industry Advisory Committee (WSEIAC) along with literature review and expert panel review. Content validity of the preliminary form was analyzed in a cohort of 200 patients. Exploratory and confirmatory factor analysis was used to assess appraisal form validity. Discriminate validity of different ventilation outcomes was analyzed by t test. Test/retest reliability and inter-scorer reliability were evaluated with 30 patients after a 2-week interval by Cronbach's alpha.


Exploratory factor analysis showed eigenvalues for 3 dimensions (availability, dependability, capability) to be 7.85, 4.43, and 4.22, respectively. Cronbach’s α for internal consistency of the appraisal form was 0.957, and 0.922, 0.961 and 0.937, respectively, for the 3 dimensions. Test-retest reliability of 3 dimensions was 0.976, and 0.862, 0.857, 0.885, respectively. Intra-class correlation coefficient verified test-retest reliability; ICC 0.976 and 0.862, 0.857, 0.885 for 3 dimensions, respectively.


The appraisal form for clinical effectiveness of adult invasive mechanical ventilation systems has high reliability and validity and may be used in clinical setting.


Rapid developments in intensive care medicine have made mechanical ventilation an essential method in the resuscitation and comprehensive treatment of critical care patients [1]. About 80% of patients in intensive care units are reported to require mechanical ventilation, and nursing care of patients receiving mechanical ventilation has become increasingly important, including nurse-led weaning of ventilator patients [2]. While hospitals in other countries may rely on trained respiratory care therapists to assess and care for mechanical ventilation systems [3], in the intensive care units of China, nursing staff especially trained in basic knowledge and techniques of intensive care medicine provide the main care for critically ill patients, including airway care for patients requiring mechanical ventilation. An important role of clinical nursing staff is to correctly assess the effectiveness of mechanical ventilation systems during respiratory intensive care in order to guarantee safe and effective treatment of patients [35]. While mechanical ventilation potentially saves patients’ lives, the complexity and diversity of tracheostomy tubes, specific care of surgical sites and other nursing issues such as unplanned intubation and complications may have an impact on its clinical effectiveness [5]. In North American hospitals, while care of the mechanical ventilation system may be in the hands of trained respiratory therapist, airway assessment and management remains a primary responsibility of nurses caring for critically ill adults [5]. Serious concerns of clinical nursing staff and respiratory therapists about mechanical ventilation have led to investigations of the relative effectiveness of ventilation machines, endotracheal tubes and tracheostomy tubes [35], as well as the critical care environment [6, 7]; personnel issues [8, 9]; management of artificial airways [10, 11]; and prevention of complications [1214]. However, current research about nursing evaluation of mechanical ventilation mainly focuses on analyzing and discussing single influential factors, while studies of comprehensive nursing interventions for mechanical ventilation systems are lacking. This may be due in part to the fact that nursing staff may not be responsible for mechanical ventilation systems in some countries, while in China, ICU nurses are solely responsible for mechanical ventilation systems and related patient care. Therefore, the design of an appraisal form to evaluate the comprehensive clinical effectiveness of adult mechanical ventilation systems may provide a valuable assessment tool for critical care nurses responsible for mechanical ventilation administered to patients in intensive care units in China.

Mechanical ventilation systems comprise positive airway pressure ventilation machines connected to patients via artificial airways (i.e., endotracheal tubes or tracheotomy) [15]. The Joint Commission (TJC) (formerly the Joint Commission on Accreditation of Healthcare Organizations [JCAHO]), is a US-based hospital accreditation organization that demands regular evaluation of the effectiveness of mechanical ventilation systems along with other equipment used for patient care or diagnostic testing; TJC also requires that regulations and operating procedures must be maintained in written form for professional staff [16]. System effectiveness can be measured by any of several models and one of the most accepted is the effectiveness model advanced by the American Weapons Systems Effectiveness Industry Advisory Committee (WSEIAC) [17]. The WSEIAC model has been used successfully to evaluate the effectiveness of various systems, including radar jamming [18], weapons systems [19], reconnaissance satellites [20], and preventive maintenance of equipment [21]. According to the theory of effectiveness evaluation of the WSEIAC model, there are three dimensions that define system effectiveness [17]. The first dimension of system effectiveness is availability. For mechanical ventilation systems, this means ensuring proper mechanical ventilation by conducting regular system checks, including testing of machine function, setting up proper alarms, maintaining the machine circuit and humidification device and selecting proper endotracheal or tracheotomy tubes [35]. Personnel training is also necessary to ensure that all nursing staff members or respiratory therapists possess the essential related knowledge and techniques for machine operation and patient care [2124]. In addition, the working environment itself can have an impact on nursing quality and the mortality of intensive care patients [1, 6, 7]. Dependability is the second dimension of system effectiveness in the WSEIAC model, which includes maintaining patency of the airway as an essential nursing intervention [10, 11, 25, 26], securing endotracheal and tracheotomy tubes in a safe position without pressure on the skin [5], reducing intratracheal damage by maintaining the lowest pressure of the endotracheal balloon and preventing air leaks by frequent checking and adjusting as needed and avoiding twisting or pulling on the endotracheal or tracheostomy tubes [27, 28]. The airway also must be maintained when patients are being transferred or their position changed [11, 29, 30]. Patients under mechanical ventilation may also develop physical and mental reactions such as anxiety, fear, pain, sleeping disorders, depression and hallucinations [31, 32], requiring that every patient with mechanical ventilation is assessed by nurses to establish an individual nurse-patient communication method and plan [15]. The third dimension of system effectiveness is capability, which is determined by the assessment of complications (e.g., ventilator-associated pneumonia and airway damage), unplanned extubation [33], aspiration [34, 35] and patients’ physical and mental reactions [36, 37].

Based on the WSEIAC model of system effectiveness, this research aimed to develop and evaluate an effectiveness appraisal form for assessing adult invasive mechanical ventilation systems (EAP-AIMVS), and to apply it for measuring the effectiveness of nursing staff in establishing artificial airways and helping to wean patients from mechanical ventilation. To our knowledge, this is the first system effectiveness appraisal form to be developed for assessing mechanical ventilation systems based on the WSEIAC model.


We designed a cross-sectional study to develop and test an appraisal form for evaluating the effectiveness of adult mechanical ventilation systems. This study is divided into two stages: development of the appraisal form and an evaluation stage, including clinical validation of the appraisal form. The Internal Review Board of the Nursing School of Fujian Medical University reviewed the study protocol and approved the study.

Stage 1: Development stage

Items development

The effectiveness appraisal form for mechanical ventilation, including 3 dimensions and 45 items, was formulated based on the WSEIAC model [17], integrated literature review to formulate the item pool, and expert review by a panel of five clinical nurses. Literature review was conducted by the research team; previously published reports regarding scale development and establishing mechanical ventilation system evaluation indexes helped to guide development of the items of the appraisal form [3846]. The three dimensions of the scale are described as follows: (1) The first dimension is system availability (A), representing the state of the mechanical ventilation system availability at all times (13 items). (2) The second dimension is system dependability (D), representing the correct operation of the mechanical ventilation by nursing staff (19 items). (3) The third dimension is system capability (C), representing the ability of the mechanical ventilation system to achieve goals of respiratory support (13 items). Therefore, system effectiveness (E) is defined as the comprehensive effectiveness of mechanical ventilation system to meet the mission of ventilation within a given time and under intensive care environment.

Content validity test

Content validity of the appraisal form was tested by content validity index (CVI), and evaluated by experts as previously described [39]. The panel of experts had 25 members, including 13 experts in clinical intensive care medicine or research and 12 registered nurses, each with more than 10 years of working experience. Four scores were used to evaluate the appraisal form items: 1 = irrelevant, 2 = some relevance, 3 = relevant, 4 = very relevant. Experts assessed every item and the relevance and feasibility of its dimensions, and advanced suggestion and opinions. The appraisal form was modified based on review and suggestions of the experts after deleting six items and integrating six items into other items, a preliminary appraisal form has 33 items in total. CVI of the appraisal form was 0.87 on average, indicating that every item reflects the theme concept of mechanical ventilation system effectiveness evaluation and can be studied further as previously described [40].

Preliminary examination

To determine readability and practicability of the appraisal form, 30 patients were assessed by three clinical nurses using the appraisal form. Two items were deleted and three items only partially understood as expressed were revised. As a result, the final EAP-AIMVS had 31 items deemed appropriate for further testing and analysis in the clinical setting.

Stage 2: Clinical validation of the appraisal form

Study subjects

The sample size for multiple factors analysis research should be more than 200 or 10 times the number of items according to a previously established formula [41]. Sample size for this study was established at 200 according to the number of items, progress of this study and feasibility. Patients receiving mechanical ventilation were selected by convenience sampling and 210 ICU patients were enrolled. Inclusion criteria were: patients age ≥18 years with invasive artificial airway (orotracheal or nasotracheal intubation, or tracheotomy.) and assisted mechanical ventilation; conscious patients willing to participate in this study, or unconscious patients with permission of family members. Exclusion criteria were: patients with noninvasive mechanical ventilation, or unwilling to participate in this study; patients without important indicators and patients transferred to other hospitals during ventilation. Ten intensive care units in 8 hospital in Fujian were selected as study locations. Inclusion criteria for ICU nursing staff were: permanent staff with nursing practice qualification certification and having had training for basic concepts, knowledge and techniques of ICU medicine. Nurses who were rotating ICU nursing staff and not permanent staff were excluded. The data collection period was from September 2010 to March 2011. Signed informed consent was received from all participants or their responsible family members.

Assessment tool

Demographic data: We recorded gender, age, APACHE II score, type of ventilation machine and artificial airway, time of mechanical ventilation and successful weaning of each patient from mechanical ventilation.

Assessment form scoring: The effectiveness appraisal form for assessing adult invasive mechanical ventilation systems is a nurse-administered assessment tool with three dimensions and 31 items. (Additional file 1: Table S1: Original EAP-AIMVS questionnaire) Four grades were recorded for each item: fully achieved, mostly achieved, hardly achieved and not at all achieved, and scores were 4, 3, 2, and 1, respectively. The higher scores are, the higher the level of clinical effectiveness will be.

Data collection method

Before collecting data, the effectiveness appraisal form for adult invasive mechanical ventilation system was audited and approved by the hospital internal review committee. Researchers then used the effectiveness appraisal form to assess patients with nonparticipant observation, meaning that except for researchers and coordinators of each study location, nursing staff were not aware of researchers’ identity. Researchers observed every index of the mechanical ventilation system as trainees, collecting data by referring to medical records, and assessed every item of the appraisal form. A total of 210 appraisal forms were filled out, and 210 appraisal forms were returned. Return rate was 100%, and 200 appraisal form were effective, attaining an effectiveness rate of 95.2%.

Data analysis

All statistical analyses were performed using SPSS 15.0 for Windows statistics software (SPSS Inc, Chicago, IL, USA). Subjects’ demographics and characteristics were summarized as mean with range (min. to max.) for continuous variables and n (%) for categorical variables. Demography statistics used descriptive analysis (frequency, mean, standard deviation and median). Statistical significance was established as P < 0.05.

Item analysis

Item analysis assesses relativity between each item and total scores. When the correlation coefficient of each assessment item and total scores was more than 0.9, it was considered redundant. When less than 0.4, the item was deleted as not reflecting the concept to be assessed.

Construct validation

Construct validity of the appraisal form was assessed by exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). Exploratory factor analysis evaluates the structure of factors and the consistency between each item and factor structure. Kaiser-Meyer-Olkin Measure of Sampling Adequacy and Bartlett’s Test of Sphericity were used to judge whether factor analysis can be employed. Principle Component Methods was employed to select varimax orthogonal rotation. During factor analysis, guidelines for filtration of assessed items were: (1) each factor’s eigenvalue >1; (2) loading capacity of items with related factor >0.4; (3) each factor at least covers 3 items; (4) tested by scree plot.

To validate matching level of sample data and factor structure of the appraisal form, linear structural equation modeling software [AMOS 17.0, Chicago, SPSS Inc] was employed for confirmatory factor analysis of the appraisal form items that were attained by exploratory factor analysis. Fitting degree was evaluated with χ2/df, comparative fit index (CFI), normed fit index (NFI), root mean square error of approximation (RMSEA). Model fitting effectiveness is acceptable when χ2/df < 3, CFI > 0.9, NFI > 0.9, RMSEA < 0.08, according to a previous report [43].

Congruent validity

Congruent validity was tested by comparing scores of different groups to see whether there were significant differences, as described in a previous report [44]. This study divided mechanical ventilation patients into a ‘weaned from ventilator successfully’ group and a ‘deceased died during ventilation’ group, and compared the scores of these two groups to determine differences, using discriminate analysis to evaluate the descriptive validity. Results were shown for Wilk’s Lambda test and, and respective sensitivity, specificity.

Reliability testing

Internal consistency reliability, test-retest reliability and inter-scorer reliability were selected to evaluate the appraisal form. Cronbach's alpha was calculated to analyze internal consistency of each dimension and the form as a whole. Thirty patients were randomly selected within an interval of 2 weeks to evaluate test-retest reliability by calculating weight coincidence coefficients of these two assessments. Since the appraisal form is a nurse-administered evaluation form, intraclass correlation coefficients were done by 2 scorers evaluating the same 30 patients in the same period to test inter-scorer reliability.

Ethical considerations

This research was approved by the thesis defense committee for masters degree and by the internal review boards of 8 hospitals in Fujian province whose ICUs were included in the study. Before data collection, conscious patients and family members of unconscious patients were fully informed of the purpose and method of this study, and all provided signed informed consent. All appraisal forms were completed anonymously.


A total of 200 subjects (142 males/58 females) were enrolled in this study. The mean age was 58.3 years (Range: 16 to 99), with 42 subjects younger than 45 years (yrs), 35 aged between 45 to 55 year, 46 aged between 55 to 65 yrs, 38 aged between 65 to 75 yrs, and 39 older than 75 yrs. The mean time in duration of mechanical ventilation was 216.8 hours (range: 2 to 2880 hours). Ventilation outcomes for the 200 subjects included 123 (61.5%) weaning patients and 77 (38.5%) deceased patients during ventilation. (Table 1)

Table 1 Subjects’ demographics and characteristics

Items analysis

The EAP-AIMVS analysis result for 31 items was: 9 items deleted whose relativity with total score was <0.3, and 22 items were reserved for construct validity test.

Construct validity

The exploratory factor analysis (EFA) used the principle component method for selection and performing varimax rotation for the 200 subjects. EFA results are summarized in Table 2. The KMO index = 0.946 and χ² = 4359 from the Bartlett’s Test of Sphericity (P < 0.01) indicated that this dataset could be applied into EFA. (Results are not shown in tables.) Three factors were selected according to the criteria of Eigenvalue > 1, the three observed eigenvalues greater than 1 were 7.85, 4.43, and 4.22, respectively (with green background). The 22 items were classified into three factors representing the three dimensions of system assessment, as follows: availability (factor 1), dependability (factor 2), and capability (factor 3). Factor loadings were observed from 0.74 to 0.87 in factor 1, from 0.43 to 0.85 in factor 2, and from 0.76 to 0.87 in factor 3. The cumulative variance explained by factors were 75.01%, including 35.68% for factor 1, 20.15% for factor 2, and 19.18% for factor 3 (Table 2).

Table 2 Re-arranged EAP-AIMVS form via Exploratory Factor analysis

Based on the constructed model, the CFA shows the final modified model yielded Chi-square = 34.79 (p < .001), df = 13,χ2/df = 2.68, GFI = 0.98, AGFI = 0.94, NFI = 0.95, PNFI = 0.44, PGFI = 0.35, RMSE = 0.07 and exhibited good fit indices. (Figure 1)

Figure 1
figure 1

This figure presents the measurement model evaluated outside the structural equation model (SEM) with correlation coefficients for the relationships among latent variables and standardized betas for the factor loadings. Availability, dependability, and capability are used as defined from EFA approach. Availability is presented by five indicators (item 1 to 5), dependability is presented by 12 indicators (item 6 to 17), and capability is presented by five indicators (item 18 to 22) The final modified model yielded Chi-square = 34.79 (p < .001), df = 13, χ 2/df = 2.68, GFI = 0.98, AGFI = 0.94, NFI = 0.95, PNFI = 0.44, PGFI = 0.35, RMSE = 0.07.

Congruent validity

Regarding ventilation outcomes, 123 subjects (61.5%) were weaning and 77 (38.5%) died during mechanical ventilation, and the total EAP-AIMVS score was observed as 80.41 (SD = 6.29) for weaning subjects and 63.13 (SD = 10.19) for those who died during ventilation. (Table 3) The Wilk’s Lambda test from discriminate analysis shows significant discriminate analysis (Wilk’s Lambda = 0.432, P < 0.05), with sensitivity and specificity equal to 91.1% and 77.9%, respectively. (Results are not shown in tables.)

Table 3 Summary of distribution of EAP-AIMVS re-arranged after EFA for 200 subjects by ventilation status (N = 200)


Internal consistency reliability of the appraisal form by Cronbach’s α was 0.957 and that for availability, dependability and capability were 0.922, 0.961 and 0.937, respectively. Test-retest reliability of the appraisal form by Kappa is 0.976, and for the three dimensions were 0.862, 0.857, 0.885, respectively. Inter-scorer reliability of the appraisal form, intra-class correlation coefficient verified test-retest reliability; ICC of the appraisal form was 0.976, and for the three dimensions, respectively, 0.862, 0.857, 0.885.


In this 15-month study, we designed, developed and tested an appraisal form for assessing the effectiveness of adult invasive mechanical ventilation systems, the EAP-AIMVS. Our design was based on the WSEIAC model of system effectiveness [17], which had been successfully applied to measure effectiveness of other types of systems [1821]. To our knowledge, this is the first appraisal form developed for mechanical ventilation systems using the WSEIAC model. Testing of the appraisal form with a convenience sample of 200 critical care patients on mechanical ventilation revealed that the three dimensions of effectiveness according to the WSEIAC model, namely system availability, dependability, and capability, had acceptable internal consistency (overall 0.957, and for the three dimensions, 0.922, 0.961 and 0.937, respectively), reliability, and test-retest reliability. As such, this indicates that the appraisal form can feasibly be applied to the assessment of mechanical ventilation systems by the responsible staff in clinical settings.

Briefly, we must note possible differences in the intensive care clinical setting in China compared to those in other countries. To ensure providing optimum therapeutic requirements for critically ill patients, patients are placed together in a dedicated area in single units of about 15-square meters with necessary equipment (e.g., bedside and central monitors, multi-function breathing and electrocardiograph machines, defibrillators, pacemakers, infusion pumps and equipment for tracheal intubation and tracheotomy). Nurses specially trained in basic knowledge and techniques of intensive care medicine are the primary care staff for critically ill patients. Medical and nursing staff members working independently are responsible for allocation of all equipment and nurses are directly responsible for airway care for patients requiring mechanical ventilation, including the system, all related interventions and weaning from the ventilator system.

The appraisal form has a solid theoretical basis

The design and development of a research tool requires a powerful theoretical basis [45]. The appraisal form for effectiveness of mechanical ventilation systems as advanced in this research, and which relied on the WSEIAC model as a basic framework complemented by integrated literature review and expert evaluation, combined the three dimensions of availability, dependability and capability to evaluate essential connections and related factors of mechanical ventilation systems, including pre-use state, functioning procedure and results, reflecting the quality control of nursing from pre-use until end-stage. The appraisal form is intended to assist clinical nurses in recognizing key indicators of mechanical ventilation effectiveness. Effectiveness of nursing intervention for the whole procedure or different stages can be assessed by using either the whole appraisal form or its subscales, as suggested by Noar (2003) [47].

The appraisal form has good construct validity

As evaluated in the second part of this study, the appraisal form exhibited good construct validity. The three common factors, corresponding to the three dimensions of system effectiveness in the WSEIAC model, were extracted via exploratory factor analysis, and its accumulated variance contribution rate was 75.01%. Factor classification was followed with the varimax orthogonal rotation of each item, which was even and clear, and the loading capacity of items with related factors was more than 0.4. This indicates good construct validity of the newly designed appraisal form. Limitations of estimating relationships between exploratory factor analysis and variables can indicate uncertainty about the relationships. Therefore, after using exploratory factor analysis as a base, this research then used confirmatory factor analysis to test the level of fitting of the three dimensions model structure with the structural equation model as a foundation. Results of confirmatory factor analysis proved that the three dimensions structure of the appraisal form is highly consistent with the WSEIAC model theoretical framework. This result also reinforced the construct validity of the appraisal form.

The appraisal form has good congruent validity

Congruent validity demonstrates how a scale differentiates characteristics of different groups of subjects [44]. Esteban and Anzueto (2002) conducted an extensive, foresighted and multi-center research study, finding that patients’ survival rate and prognosis were influenced by different factors of pre-use of mechanical ventilation, and complications and management of patients in mechanical ventilation [47]. As an example of classical test theory, results of that study demonstrated that the assessment index, including management of mechanical ventilation, nursing management of airways and development of complications, influences the prognosis of patients [47]. Therefore, it is commonly understood that the poorer the prognosis of ventilation in a given patient, the lower the effectiveness score would be. Congruent validity demonstrates whether or not the appraisal form reflects differences between factors. In the present study, results showed that patients weaned from the ventilator successfully had higher effectiveness scores than the patients who died during ventilation (t = 13.38,P < 0.05), indicating acceptable congruent validity of the appraisal form.

The appraisal form demonstrated good reliability

Reliability refers to the level of credibility and stability of a scale and its evaluation results. Reliability can be evaluated by various means. This study reviewed internal consistency reliability, test-retest reliability and inter-scorer reliability. Cronbach’s α is an estimate of internal consistency and will typically increase as relationships between the test items increase. In the present study, Cronbach’s α of the whole appraisal form and its different dimensions achieved the measurement model of more than 0.7, which indicates internal consistency reliability of the appraisal form, in keeping with previously published results [48]. Our result indicates good reliability of the effectiveness appraisal form and the homology of items that form each dimension. Test-retest reliability and inter-scorer reliability also achieved the measurement model of more than 0.75 [49]. It is shown that the variance of these two tests of the appraisal form is relatively low and stable, which indicates that the appraisal form is suitable for use by different clinical assessors.

Prior to actually designing the EAP-AIMVS, we theorized that availability of a standardized form for assessing the effectiveness of adult invasive mechanical ventilation would be a valuable adjunct for nurses in Chinese hospitals caring for ventilated critical care patients, not only to help address nursing issues but to provide safer care for patients. This was the rationale for developing the EAP-AIMVS as a practical, theory-based form for appraising effectiveness of mechanical ventilation systems, and during development we considered exactly how an effectiveness appraisal form might help to ensure patient safety and reduce complications, morbidity and mortality. Other nursing interventions reported in the literature provided encouragement. In a study documenting the need for education of nurses and other medical professionals who care for ventilated patients, the need for additional education and training regarding clinical practice in intensive care was expressed by 62% of medical staff, including intensivists and 42% of nurses [1]. Although having an effectiveness appraisal form will not take the place of education and training, the effectiveness evaluation can serve as a readily available guide or checklist that can be followed by ICU caregivers to help ensure safe, effective ventilation and efficient monitoring. Studies of weaning from mechanical ventilation illustrate another aspect of standardizing care. Because prolonged mechanical ventilation is associated with high morbidity, especially ventilator associated pneumonia and lung injury, and increased mortality, patients receiving mechanical ventilation may actually spend 40% of the time being weaned from ventilation; prolonged ventilation may also preclude patients’ recovery from their primary critical illness [50]. The investigators in that study recommended surveying practitioners and multidisciplinary teams to collect the best ways to put optimal weaning methods into practice and then apply them [50]. Relative to weaning methods, routine application of a nurses’ protocol-directed weaning procedure applied in a French hospital resulted in improved outcomes and clinical benefits in patients requiring more than 48 hours of mechanical ventilation [2]. In our study, routinely following a simple, step-by-step appraisal form for mechanical ventilation effectiveness allowed more than 60% of patients to be weaned successfully, helped to prevent complications and reassuring staff that proper procedures had been carried out for system effectiveness, which implies patient safety. We made sure that effectiveness criteria included items such as ventilator circuits and secretion management, which have been shown to influence rates of ventilator-associated pneumonia [13]. We also included the intensive care physical environment as a factor in the appraisal (Item 5, Table 3) since a healthy environment is needed to protect against airborne infection and prevent cross-infection; hand washing and aseptic technique must be reinforced, air quality must be monitored, visitors restricted and ventilation equipment needs disinfecting on a regular basis. Also, a 28-day international study of ventilated patient characteristics and outcomes suggested that survival was associated not only with baseline factors present at the inception of mechanical ventilation but the development of complications and the management of patients [47]. Thus, future research on mechanical ventilation effectiveness must evaluate patient data, including complications, unexpected developments and outcomes, at baseline and during the course of mechanical ventilation.

Implications for practice

Results of the present study have provided an evaluation form by which nursing staff or other responsible staff such as respiratory therapists can evaluate the effectiveness of mechanical ventilation systems and related care aspects of nursing interventions for mechanical ventilation patients. Specifically, the developed form can help clinical nursing staff to differentiate key indexes and increase work efficiency. Future research should focus on progressive evaluation of mechanical ventilation efficiency in China and abroad, and collect clinical practice data for ventilation technique and management of artificial airways. Guiding principles for evaluation of mechnical ventilation systems can be improved and progressive development of appraisal forms may ultimately include software applications that evaluate each item according to individual patients’ clinical situation.


First, the appraisal form developed in this study for assessing the effectiveness of mechanical ventilation was not developed on the basis of an existing scale but was an initial attempt to develop an original appraisal form and evaluate its suitability as an assessment tool. The sample was also relatively small. Assessment items of the appraisal form therefore require further evaluation with a larger clinical sample. Secondly, convenience sampling method was used and its representativeness is limited. Sampling range and quantity should be broadened in future research, and the appraisal form will be further developed and improved through clinical evidence and progressive mechanical ventilation research.


The theory-based appraisal instrument to assess nursing interventions for mechanical ventilation systems demonstrates acceptable internal consistency, inter-scorer reliability, and test-retest reliability and can reasonably guide clinical nursing staff to evaluate essential monitoring techniques and factors related to mechanical ventilation. The appraisal form is appropriate for use in the clinical setting and its application may help to improve the effectiveness and performance of adult mechanical ventilation systems as well as helping to provide safe, effective treatment for patients under intensive care.


  1. Hong SB, Oh BJ, Kim YS, Kang EH, Kim CH, Park YB, Han MS, Shin C: Korean study group on respiratory failure (KOSREF): Characteristics of mechanical ventilation employed in intensive care units: a multicenter survey of hospitals. J Korean Med Sci. 2008, 23: 948-953. 10.3346/jkms.2008.23.6.948.

    Article  PubMed Central  PubMed  Google Scholar 

  2. Tonnelier J, Prat G, Le Gal G, Gut-Gobert C, Renault A, Boles JM, L'Her E: Impact of a nurses’ protocol-directed weaning procedure on outcomes in patients undergoing mechanical ventilation for longer than 48 hours: a prospective cohort study with a matched historical control group. Crit Care. 2005, 9: R83-R89. 10.1186/cc3030.

    Article  PubMed Central  PubMed  Google Scholar 

  3. Branson RD: Monitoring ventilator function. Crit Care Clin. 1995, 11: 127-143.

    CAS  PubMed  Google Scholar 

  4. Crimlisk JT, O'Donnell C, Grillone GA: Standardizing adult tracheostomy tube styles: what is the clinical and cost-effective impact?. Dimens Crit Care Nurs. 2006, 25: 35-43. 10.1097/00003465-200601000-00011.

    Article  PubMed  Google Scholar 

  5. St John RE, Malen JF: Contemporary issues in adult tracheostomy management. Crit Care Nurs Clin North Am. 2004, 16: 413-430. 10.1016/j.ccell.2004.04.002. ix-x

    Article  PubMed  Google Scholar 

  6. Halm EA, Lee C, Chassin MR: Is volume related to outcome in health care? A systematic review and methodologic critique of the literature. Ann Intern Med. 2002, 137: 511-520.

    Article  PubMed  Google Scholar 

  7. Kahn JM, Goss CH, Heagerty PJ, Kramer AA, O’Brien CR, Rubenfeld GD: Hospital volume and the outcomes of mechanical ventilation. N Engl J Med. 2006, 355: 41-50. 10.1056/NEJMsa053993.

    Article  CAS  PubMed  Google Scholar 

  8. Pronovost PJ, Jenckes MW, Dorman T, Garrett E, Breslow MJ, Rosenfeld BA, Lipsett PA, Bass E: Organizational characteristics of intensive care units related to outcomes of abdominal aortic surgery. JAMA. 1999, 281: 1310-1317. 10.1001/jama.281.14.1310.

    Article  CAS  PubMed  Google Scholar 

  9. Young MP, Birkmeyer JD: Potential reduction in mortality rates using an intensivist model to manage intensive care units. Eff Clin Pract. 2000, 3: 284-289.

    CAS  PubMed  Google Scholar 

  10. St John RE: Airway management. Crit Care Nurse. 2004, 24: 93-96.

    PubMed  Google Scholar 

  11. Edgtton WM, Wright K: Tracheostomy: a guide to nursing care. Aust Nurs J. 2005, 13: 17-20.

    Google Scholar 

  12. Turner P, Glass C, Grap MJ: Care of the patient requiring mechanical ventilation. Medsurg Nurs. 1997, V6N2: 68-73,76,94.

    Google Scholar 

  13. Cook D, De Jonghe B, Brochard L, Brun-Buisson C: Influence of airway management on ventilator-associated pneumonia. JAMA. 1998, 279: 781-787. 10.1001/jama.279.10.781.

    Article  CAS  PubMed  Google Scholar 

  14. Hubmayr RD: Statement of the 4th International Consensus Conference in Critical Care on ICU-Acquired Pneumonia, Chicago. Illinois, May 2002. Intensive Care Med. 2002, 28: 1521-1536. 10.1007/s00134-002-1514-0.

    Article  CAS  PubMed  Google Scholar 

  15. Lindgren VA, Ames NJ: Caring for patients on mechanical ventilation: what research indicates is best practice. Am J Nurs. 2005, 105: 50-60.

    Article  PubMed  Google Scholar 

  16. Provonost PJ, Jenckes M, To M, Dorman T, Lipsett PA, Berenholtz S, Bass EB: Reducing failed extubations in the intensive care unit. Jt Comm J Qual Improv. 2002, 28: 595-604.

    Google Scholar 

  17. Tillman FA, Hwang CL, Kuo W: System effectiveness models: an annotated bibliography. Reliability: IEEE Transactions. 2009, R-29: 295-304.

    Article  Google Scholar 

  18. Li W, Mao N, Pan W: The effectiveness evaluation model of radar jamming system based on WSEIAC model[C]. Control and Decision Conference 2008, Yantai, China: IEEE. 2008, 1848-1852.

    Chapter  Google Scholar 

  19. Mullins R: Visualization of Weapon System Effectiveness Measures of Performance. AIAA. 2001, 98: 1062-

    Google Scholar 

  20. Meng J, Li QM, Zhang Liu FU: Effectiveness Evaluation of Reconnaissance Satellite Based on the ADC Model. Computer Science (China). 2009, 36: 41-67.

    Google Scholar 

  21. Bao M, Dai YW, Kong JS, ZOU Y: ADC-effectiveness-evaluation model with preventive maintenance. Control Theory & Applications. (China). 2010, 27: 1166-1170.

    Google Scholar 

  22. Birdsall C, Gutekunst M: Preparing staff for opening of a new ventilator unit in long-term care. J Gerontol Nurs. 1995, 21: 39-43.

    Article  CAS  PubMed  Google Scholar 

  23. Dougherty JM, Parrish JM, Hock-long L: Developing a competency-based curriculum for tracheostomy and ventilator care. Pediatr Nurs. 1995, 21: 581-584.

    CAS  PubMed  Google Scholar 

  24. Serra A: Tracheostomy care. Nurs Stand. 2000, 14: 42-52.

    Article  Google Scholar 

  25. Pedersen CM, Rosendahl-Nielsen M, Hjermind J: Endotracheal suctioning of the adult intubated patient–what is the evidence?. Intensive Crit Care Nurs. 2009, 25: 21-30. 10.1016/j.iccn.2008.05.004.

    Article  PubMed  Google Scholar 

  26. Russell C: Providing the nurse with a guide to tracheostomy care and management. Br J Nurs. 2005, 14: 428-433.

    Article  PubMed  Google Scholar 

  27. Kollef MH: The prevention of ventilator-associated pneumonia. N Engl J Med. 1999, 340: 627-634. 10.1056/NEJM199902253400807.

    Article  CAS  PubMed  Google Scholar 

  28. De Leyn P, Bedert L, Delcroix M: Tracheotomy: clinical review and guidelines. Eur J Cardiothorac Surg. 2007, 32: 412-421. 10.1016/j.ejcts.2007.05.018.

    Article  PubMed  Google Scholar 

  29. Yeh SH, Lee LN, Ho TH: Implications of nursing care in the occurrence and consequences of unplanned extubation in adult intensive care units. Int J Nurs Stud. 2004, 41: 255-262. 10.1016/S0020-7489(03)00136-6.

    Article  PubMed  Google Scholar 

  30. Richmond AL, Jarog DL, Hanson VM: Unplanned extubation in adult critical care. Quality improvement and education payoff. Crit Care Nurse. 2004, 24: 32-37.

    PubMed  Google Scholar 

  31. Hofhuis JG, Spronk PE, van Stel HF, Schrijvers AJ, Rommes JH, Bakker J: Experiences of critically ill patients in the ICU. Intensive Crit Care Nurs. 2008, 24: 300-313. 10.1016/j.iccn.2008.03.004.

    Article  PubMed  Google Scholar 

  32. Arbour R: Sedation and pain management in critically ill adults. Crit Care Nurse. 2000, 20: 39-56.

    CAS  PubMed  Google Scholar 

  33. Birkett KM, Southerland KA, Leslie GD: Reporting unplanned extubation. Intensive Crit Care Nurs. 2005, 21 (2): 65-75. 10.1016/j.iccn.2004.07.012.

    Article  PubMed  Google Scholar 

  34. McClave SA, DeMeo MT, DeLegge MH, DiSario JA, Heyland DK, Maloney JP, Metheny NA, Moore FA, Scolapio JS, Spain DA, Zaloga GP: North American summit on aspiration in the ciritically ill patient: consensus statement. J Parenter Enteral Nutr. 2002, 26 (Suppl): S80-S85. 10.1177/014860710202600613.

    Article  Google Scholar 

  35. Metheny NA: Risk factors for aspiration. J Parenteral Enteral Nutr. 2002, 26 (Suppl): S26-S31. 10.1177/014860710202600605.

    Article  Google Scholar 

  36. White JM: State of the science of music interventions: critical care and perioperative practice. Crit Care Nurs Clin North Am. 2000, 12: 219-225.

    CAS  PubMed  Google Scholar 

  37. Happ MB, Tuite P, Dobbin K, DiVirgilio-Fhomas D, Kitutu J: Communication ability, method, and content among nonspeaking nonsurviving patients treated with mechanical ventilation in the intensive care unit. Am J Crit Care. 2004, 13: 210-218.

    PubMed  Google Scholar 

  38. Chen LL, Li H, Li N: The establishment of adult invasive mechanical ventilation system evaluation indexes. J Nurs Sci. 2011, 26: 25-28.

    CAS  Google Scholar 

  39. Lynn MR: Determination and quantification of content validity. Nurs Res. 1986, 36: 382-386.

    Google Scholar 

  40. Waltz CF, Strickland OL, Lenz ER: Measurement in Nursing Research. 1991, FA Davis Company, Philadelphia

    Google Scholar 

  41. DeVellis R: Scale development: theory and applications. 1991, Sage Publications, Thousand Oaks, CA

    Google Scholar 

  42. Bentle PM, Chou CP: Practical Issues in Structural Modeling. Sociological Methods & Research. 1986, 16: 78-117.

    Article  Google Scholar 

  43. Hu LT, Bentler PM: Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Structural Equation Modeling. 1999, 6: 1-55. 10.1080/10705519909540118.

    Article  Google Scholar 

  44. Streiner DL, Norman GR: Health Measurement Scales: A Practical Guide to their Development and Use. 2006, Oxford University Press, Oxford, UK, 3

    Google Scholar 

  45. Bear M: Theoretical issues in measurement. Advancing Nursing Science Through Research, Vol. 2. Edited by: Moody LE. 1990, Sage, Newbury Park, CA, 249-274.

    Google Scholar 

  46. Noar SM: The role of structural equation modeling in scale development. Structural Equation Modeling. 2003, 10: 622-647. 10.1207/S15328007SEM1004_8.

    Article  Google Scholar 

  47. Esteban A, Anzueto A, Frutos F, Alía I, Brochard L, Stewart TE, Benito S, Epstein SK, Apezteguía C, Nightingale P, Arroliga AC, Tobin MJ: Mechanical Ventilation International Study Group: Characteristics and outcomes in adult patients receiving mechanical ventilation: a 28-day international study. JAMA. 2002, 287: 345-355. 10.1001/jama.287.3.345.

    Article  PubMed  Google Scholar 

  48. DeVellis R: Classical test theory. Medical Care. 2006, 44: S50-S59. 10.1097/01.mlr.0000245426.10853.30.

    Article  PubMed  Google Scholar 

  49. Rosner B: Fundamentals of Biostatistics. 2006, Toronto, Canada, Thomson, 6

    Google Scholar 

  50. Meade M, Guyatt G, Griffith L, Booker L, Randall J, Cook D: Introduction to a series of systematic reviews of weaning from mechanical ventilation. Chest. 2001, 120 (suppl): 396S-399S.

    Article  CAS  PubMed  Google Scholar 

Download references


We are grateful for the support of the 200 patients and the nursing staff of the intensive care units. We acknowledge and thank the 25 experts for their hard work and valuable advice.


National key project of nursing specialists of Ministry of Health, P.R.C. No. 2011. National key project of critical care medicine of Ministry of Health, P.R.C. No. 2011.Key Project of Fujian Provincial Department of Science and Technology, P.R.C. NO.2012Y0013.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Hong Li.

Additional information

Competing interests

The authors declare there is no competing interests.

Authors’ contributions

LH was responsible for conception, overall design, technical support and content revision. CL was responsible for project implementation, data collection and data analysis. LN was responsible for data organization and analysis. All authors read and approved the final manuscript.

Electronic supplementary material

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and Permissions

About this article

Cite this article

Li, H., Chen, LL. & Li, N. Development and evaluation of an appraisal form to assess clinical effectiveness of adult invasive mechanical ventilation systems. Scand J Trauma Resusc Emerg Med 20, 45 (2012).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: