External Validity and Multi-Organization Samples: Levels-of-Analysis Implications of Crowdsourcing and College Student Samples

Here, we expand on Landers and Behrend's (2015) discussion of the external validity of convenience samples. In particular, we note that their focal article failed to mention one important limitation of multi-organization convenience samples (e.g., MTurk samples, student samples): Multi-organization convenience samples tend to confound levels of analysis, which affects the external validity of these samples. Specifically, between-organizations phenomena (i.e., organization-level) and within-organizations phenomena (i.e., individual-level) are distinct and separable (Ostroff, 1993; Robinson, 1950). Unfortunately, multi-organization samples such as those found in MTurk or MBA student samples can confound these two sets of phenomena. The current commentary uses a levels-of-analysis framework to expand on Landers and Behrend's discussion of what external validity is, and then the commentary illustrates how the diversity of convenience samples can actually harm external validity under some common circumstances.