Evaluating the quality of research publications: A pilot study of school librarianship

Evaluation of research quality is becoming more important in the field of library and information science, as in other fields. This pilot study is a preliminary attempt to address issues associated with determining the quality of the published research in one area of library and information science, specifically school librarianship. The main aims were (1) to test the extent to which experienced evaluators agreed in their rankings of published research articles on the basis of quality and (2) to investigate the approaches to evaluation used by the experienced evaluators. A qualitative, naturalistic research design was used. On the basis of a comprehensive literature review, four approaches to evaluation were identified; they were generally supported through an analysis of the responses of the experienced evaluators. However, although the majority of the evaluators agreed on the article ranked lowest, basic statistical analyses showed less agreement about the other articles. Although subject knowledge (of the field of school librarianship in this case) may have some influence on the evaluations, cluster analysis suggests that there may be differences in the value perceptions of the evaluators that also carry weight. More research would be needed to gain a better understanding of these value perceptions and their relationship, if any, to the four approaches to evaluation that were identified through the literature.