Comparative analysis of concept mapping: human participants vs. ChatGPT

ChatGPT has great potential in academia, but exploring AI’s impact on social sciences research has been neglected. This paper uses concept mapping, a systematic way to organize and represent group ideas, to evaluate ChatGPT and compare it to a human study. As ChatGPT collects ideas from a variety of human generated training data and presents them to the user. Thus, could ChatGPT falsify social science survey, questionnaire, and sorting results? The study employs a methodology centered around concept mapping, a bottom-up exploratory research design widely used in various fields, comprising five stages: statement generation, statement grouping, multi-dimensional scaling (MDS), hierarchical cluster analysis (HCA), and cluster labelling. However, to compare ChatGPT’s performance, adjustments are made. ChatGPT is prompted to thematically group provided statements and generate outputs. Despite being instructed to include all statements, ChatGPT’s iterations consistently miss some. These outputs are then entered into software for MDS and HCA analysis. Results demonstrate disparities in bridging values, indicating statement groupings’ coherence, reveal significant differences between ChatGPT and human-generated maps. These methodological differences reflect challenges in integrating AI technologies like ChatGPT into research methodologies. While ChatGPT’s natural language processing facilitates accessibility, its inability to consistently include all data and accurately replicate human cognitive processes poses limitations. Thus, rigorous evaluation and quality control procedures are imperative to ensure the reliability and accuracy of research findings when employing AI technologies in social science research.