Table 2 Validity evidence results by category
Validity evidence source | Question related to the source of evidence | Validity evidence for the assessment | |
|---|---|---|---|
PBLS | FBAOM | ||
Content | Is the content measuring the intended construct (skill levels of laypersons)? | International resuscitation experts identified the assessment items as essential for laypersons | |
Response process | Are bias sources reduced? | One item was not applicable to the layperson training and excluded. | Pilot testing of the rating procedure revealed 4 FBAOM items could not be scored. |
The raters participated in rater training and participants’ skill levels were blinded for the raters. | |||
Internal structure | Are the test scores reliable? | The generalizability analysis and the d-study identified the number of tests and raters needed for different levels of reliability. | |
Pearson’s correlations above 0.93 (p < 0.001) between global ratings scores and assessment scores support the construct of the test. | |||
The high Cronbach’s alpha supports the match of items and the intended construct | The questionable Cronbach’s alpha suggests internal inconsistency in the test items. | ||
Relation to other variables | Does the score correlate with other measures of skills? | The assessment scores increased with increasing duration of training and significantly differentiated all the three groups. | The assessment scores increased with training and discriminated untrained laypersons from all other groups. The assessment scores were not able to discriminate trained laypersons from lifeguards. |
Consequences | What is the consequences of the pass/fail score | All untrained laypersons and one lifeguard failed. Theoretical false positives and negative with the contrasting groups method was 1.0% and 0.5%, respectively. | Eight untrained laypersons and three lifeguards failed. Theoretical false positives and negative with the contrasting groups method was 22% and 29%, respectively |
Unintended consequences of the pass score could be low self-efficacy and reluctance to intervene in real resuscitation attempts | |||