How to Compare Performance in Program Design Activities: Towards an Empirical Evaluation of CoExist

We present the design of an empirical experiment to compare programmers’ performance in program design tasks. The experiment is targeted to empirically examine the benefits of CoExist, a set of extensions to programming environments. CoExist supports programmers in dealing with unexpected and undesired consequences of making changes to their code base. Changing source code involves the risk of making errors. For example, a promising idea to simplify the code can suddenly turn out inappropriate, a situation that, if not prepared, requires programmers to manually withdraw recent changes. Traditionally, programmers have to strictly follow a structured and disciplined approach to reduce the costs of making errors. However, this traditional approach requires planning for upcoming but still uncertain changes in advance, which is time-consuming and also error prone. In addition, it requires significant effort to not forget the regular execution of the required activities, in particular in situations full of uncertainty. In contrast to this, CoExist offers dedicated tool support to recover fast and easily from undesired consequences. We believe that the presence of such tools encourages programmers to make source code changes at the moment they think of them, independent of whether or not the implications of such changes are already apparent. The presented experiment design to compare performance in program design tasks will help to examine this hypothesis.