‘Please explain your response’: A guide to uncovering cognitive processes from open-text box data using pragmatic and reflexive content analysis

This guide provides a philosophical framework and practical advice for gathering, analyzing, and reporting a particular type of qualitative data. These data are obtained from including an open-text box following the key quantitative question in survey-style studies with the request to ‘Please explain your response’. While many studies currently collect such data, they often either fail to report or analyze it, or they conduct unstructured analyses with limited detail, often mistakenly referring to it as ‘thematic analysis’. Content analysis provides a well-established framework for analyzing such data, and the simplicity of the data form allows for a highly pragmatic and flexible approach. The guide integrates the concept of reflexivity from qualitative research to navigate the large number of researcher degrees of freedom involved in the process, particularly in working with the second coder. It begins by arguing for the value of this data, before outlining the guide’s philosophy, offering advice on maximizing the validity of your data, and addressing the common concern of confabulation. It then provides advice on developing a coding scheme, recruiting and collaborating with a second coder, and writing your report, considering the potential role of large language models at these various stages. Additionally, it provides a checklist for reviewers to evaluate the quality of a given analysis. Throughout the guide, a running example is used to demonstrate the implementation of the provided advice, accompanied by extensive example materials in the online repository, which can be used to practice the method.