Too Soon to Dismiss? A Reanalysis of Forster and Neugebauer (2024) Highlighting the Role of Effect Heterogeneity in Evaluating Factorial Surveys

This replication study critically examines the findings and conclusions of Forster and Neugebauer (2024), who question the external validity of factorial surveys in capturing real-world behavior. Using the publicly available data from the original study, we reassess the effects of educational signals and ethnic origin. Our analyses show that the original results do not withstand all robustness checks and that theoretically derived effect heterogeneities across occupational fields provide more nuanced explanations. These findings challenge the general claim that factorial surveys are unsuitable for studying hiring processes and instead underscore the importance of context-sensitive modeling. We argue that factorial surveys remain a valuable methodological tool, particularly when heterogeneity is explicitly addressed in both design and analysis.