IDEAS home Printed from
   My bibliography  Save this article

Quality and rigor of the concept mapping methodology: A pooled study analysis


  • Rosas, Scott R.
  • Kane, Mary


The use of concept mapping in research and evaluation has expanded dramatically over the past 20 years. Researchers in academic, organizational, and community-based settings have applied concept mapping successfully without the benefit of systematic analyses across studies to identify the features of a methodologically sound study. Quantitative characteristics and estimates of quality and rigor that may guide for future studies are lacking. To address this gap, we conducted a pooled analysis of 69 concept mapping studies to describe characteristics across study phases, generate specific indicators of validity and reliability, and examine the relationship between select study characteristics and quality indicators. Individual study characteristics and estimates were pooled and quantitatively summarized, describing the distribution, variation and parameters for each. In addition, variation in the concept mapping data collection in relation to characteristics and estimates was examined. Overall, results suggest concept mapping yields strong internal representational validity and very strong sorting and rating reliability estimates. Validity and reliability were consistently high despite variation in participation and task completion percentages across data collection modes. The implications of these findings as a practical reference to assess the quality and rigor for future concept mapping studies are discussed.

Suggested Citation

  • Rosas, Scott R. & Kane, Mary, 2012. "Quality and rigor of the concept mapping methodology: A pooled study analysis," Evaluation and Program Planning, Elsevier, vol. 35(2), pages 236-245.
  • Handle: RePEc:eee:epplan:v:35:y:2012:i:2:p:236-245
    DOI: 10.1016/j.evalprogplan.2011.10.003

    Download full text from publisher

    File URL:
    Download Restriction: Full text for ScienceDirect subscribers only

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    1. Joanna Sale & Kevin Brazil, 2004. "A Strategy to Identify Critical Appraisal Criteria for Primary Mixed-Method Studies," Quality & Quantity: International Journal of Methodology, Springer, vol. 38(4), pages 351-365, August.
    2. Dumont, Jeanne, 1989. "Validity of multidimensional scaling in the context of structured conceptualization," Evaluation and Program Planning, Elsevier, vol. 12(1), pages 81-86, January.
    3. J. Kruskal, 1964. "Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis," Psychometrika, Springer;The Psychometric Society, vol. 29(1), pages 1-27, March.
    4. Keith, Douglas, 1989. "Refining concept maps: Methodological issues and an example," Evaluation and Program Planning, Elsevier, vol. 12(1), pages 75-80, January.
    5. repec:aph:ajpbhl:10.2105/ajph.2005.066001_4 is not listed on IDEAS
    6. Joanna Sale & Lynne Lohfeld & Kevin Brazil, 2002. "Revisiting the Quantitative-Qualitative Debate: Implications for Mixed-Methods Research," Quality & Quantity: International Journal of Methodology, Springer, vol. 36(1), pages 43-53, February.
    7. Trochim, William M. K., 1989. "Concept mapping : Soft science or hard art?," Evaluation and Program Planning, Elsevier, vol. 12(1), pages 87-110, January.
    Full references (including those not matched with items on IDEAS)


    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.

    Cited by:

    1. Miller, J. Jay & Grise-Owens, Erlene & Addison, Donia & Marshall, Midaya & Trabue, Donna & Escobar-Ratliff, Laura, 2016. "Planning an organizational wellness initiative at a multi-state social service agency," Evaluation and Program Planning, Elsevier, vol. 56(C), pages 1-10.
    2. Jay Miller, J. & Benner, Kalea & Pope, Natalie & Dumas, Tamikia & Damron, Larry J. & Segress, Melissa & Slone, Melissa & Thrasher, Shawndaya & Niu, Chunling, 2017. "Conceptualizing effective foster parent mentor programs: A participatory planning process," Children and Youth Services Review, Elsevier, vol. 73(C), pages 411-418.
    3. Donnelly, James P., 2017. "A systematic review of concept mapping dissertations," Evaluation and Program Planning, Elsevier, vol. 60(C), pages 186-193.
    4. Stoyanov, Slavi & Jablokow, Kathryn & Rosas, Scott R. & Wopereis, Iwan G.J.H. & Kirschner, Paul A., 2017. "Concept mapping—An effective method for identifying diversity and congruity in cognitive style," Evaluation and Program Planning, Elsevier, vol. 60(C), pages 238-244.
    5. repec:eee:epplan:v:66:y:2018:i:c:p:70-78 is not listed on IDEAS
    6. Lisa M. Vaughn & Farrah Jacquez & Daniel Marschner & Daniel McLinden, 2016. "See what we say: using concept mapping to visualize Latino immigrant’s strategies for health interventions," International Journal of Public Health, Springer;Swiss School of Public Health (SSPH+), vol. 61(7), pages 837-845, September.
    7. repec:eee:cysrev:v:85:y:2018:i:c:p:127-136 is not listed on IDEAS
    8. Orsi, Rebecca, 2017. "Use of multiple cluster analysis methods to explore the validity of a community outcomes concept map," Evaluation and Program Planning, Elsevier, vol. 60(C), pages 277-283.
    9. Nabitz, Udo & van Randeraad-van der Zee, Carlijn & Kok, Ineke & van Bon-Martens, Marja & Serverens, Peter, 2017. "An overview of concept mapping in Dutch mental health care," Evaluation and Program Planning, Elsevier, vol. 60(C), pages 202-212.
    10. Tubbing, Luuk & Harting, Janneke & Stronks, Karien, 2015. "Unravelling the concept of integrated public health policy: Concept mapping with Dutch experts from science, policy, and practice," Health Policy, Elsevier, vol. 119(6), pages 749-759.
    11. McLinden, Daniel, 2017. "And then the internet happened: Thoughts on the future of concept mapping," Evaluation and Program Planning, Elsevier, vol. 60(C), pages 293-300.
    12. Schalock, Robert L. & Lee, Tim & Verdugo, Miguel & Swart, Kees & Claes, Claudia & van Loon, Jos & Lee, Chun-Shin, 2014. "An evidence-based approach to organization evaluation and change in human service organizations evaluation and program planning," Evaluation and Program Planning, Elsevier, vol. 45(C), pages 110-118.
    13. Miller, J. Jay & Jones, Blake, 2015. "Using concept mapping as a planning tool: Child welfare citizen review panels," Evaluation and Program Planning, Elsevier, vol. 53(C), pages 99-106.
    14. repec:eee:epplan:v:64:y:2017:i:c:p:78-84 is not listed on IDEAS
    15. Rosas, Scott R. & Ridings, John W., 2017. "The use of concept mapping in measurement development and evaluation: Application and future directions," Evaluation and Program Planning, Elsevier, vol. 60(C), pages 265-276.
    16. Soellner, Renate & Lenartz, Norbert & Rudinger, Georg, 2017. "Concept mapping as an approach for expert-guided model building: The example of health literacy," Evaluation and Program Planning, Elsevier, vol. 60(C), pages 245-253.


    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:epplan:v:35:y:2012:i:2:p:236-245. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Dana Niculescu). General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.