What difference does it make? Impact of peer-reviewed scholarships on scientific production

We investigated the extent to which different selection mechanisms for awarding scholarships varied in their short- and longer-term consequences in the performance of awardees in terms of scientific production. We conducted an impact evaluation study on undergraduate, master’s, and PhD research scholarships and compared two different financial sources in Brazil: in one, the selection mechanism was based on a peer review system; the other was based on an institutional system other than peer review. Over 8,500 questionnaires were successfully completed, covering the period 1995–2009. The two groups were compared in terms of their scientific performance using a propensity score approach. We found that the peer-reviewed scholarship awardees showed better performance: they published more often and in journals with higher impact factors than scholarship awardees from the other group. However, two other results indicate a different situation. First, over the long-term, awardees under the peer review system continued to increase their publication rate and published in higher-quality journals; however, the differences with the control group tended to diminish after PhD graduation. Second, the better performance of peer-reviewed scholarships was not observed in all subject areas. The main policy implications of this study relate to a better understanding of selection mechanisms and the heterogeneity regarding the relation between selection processes and scientific and academic output.