Experiments in systems engineering and design research

Controlled experiments ranging from laboratory experiments to field experiments are becoming increasingly important within systems engineering and design (SE&D) research. Controlled experiments provide unique capabilities for testing hypotheses, and guiding the development of scientific theories, but their potential is not being fully exploited. As the systems engineering research community embraces different research methods commonly adopted in other domains, there is a need for understanding the role and appropriateness of those methods within the research process. While the design of experiments in SE&D research is not unique per se, we contend that aspects of representativeness in the SE&D context pose different experimental design challenges than do the problems typically studied by experimental social scientists. There is therefore a need to translate established practices to the SE&D context, and also belay widely held concerns about the value of experiments in SE&D research. To address this need, the goals of the paper are to discuss the potential role of controlled experiments in systems engineering and design research, to review the benefits and limitations of different types of experiments, and to highlight the key issues in validating experimental studies. The taxonomy and the discussion on validation build on the rich history in social and behavioral sciences, with specific adaptation to the SE&D context. The discussion is anchored in specific examples from the authors’ own research on open innovation in systems engineering.