IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v36y2012i4p243-271.html
   My bibliography  Save this article

Measuring Stakeholder Participation in Evaluation

Author

Listed:
  • Pierre-Marc Daigneault
  • Steve Jacob
  • Joël Tremblay

Abstract

Background: Stakeholder participation is an important trend in the field of program evaluation. Although a few measurement instruments have been proposed, they either have not been empirically validated or do not cover the full content of the concept. Objectives: This study consists of a first empirical validation of a measurement instrument that fully covers the content of participation, namely the Participatory Evaluation Measurement Instrument (PEMI). It specifically examines (1) the intercoder reliability of scores derived by two research assistants on published evaluation cases; (2) the convergence between the scores of coders and those of key respondents (i.e., authors); and (3) the convergence between the authors’ scores on the PEMI and the Evaluation Involvement Scale (EIS). Sample: A purposive sample of 40 cases drawn from the evaluation literature was used to assess reliability. One author per case in this sample was then invited to participate in a survey; 25 fully usable questionnaires were received. Measures: Stakeholder participation was measured on nominal and ordinal scales. Cohen’s κ, the intraclass correlation coefficient, and Spearman’s Ï were used to assess reliability and convergence. Results: Reliability results ranged from fair to excellent. Convergence between coders’ and authors’ scores ranged from poor to good. Scores derived from the PEMI and the EIS were moderately associated. Conclusions: Evidence from this study is strong in the case of intercoder reliability and ranges from weak to strong in the case of convergent validation. Globally, this suggests that the PEMI can produce scores that are both reliable and valid.

Suggested Citation

  • Pierre-Marc Daigneault & Steve Jacob & Joël Tremblay, 2012. "Measuring Stakeholder Participation in Evaluation," Evaluation Review, , vol. 36(4), pages 243-271, August.
    • Handle: RePEc:sae:evarev:v:36:y:2012:i:4:p:243-271
    DOI: 10.1177/0193841X12458103
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X12458103
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X12458103?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Jacob, Steve & Ouvrard, Laurence & Bélanger, Jean-François, 2011. "Participatory evaluation and process use within a social aid organization for at-risk families and youth," Evaluation and Program Planning, Elsevier, vol. 34(2), pages 113-123, May.
    2. Connors, Susan C. & Magilvy, Joan K., 2011. "Assessing vital signs: Applying two participatory evaluation frameworks to the evaluation of a college of nursing," Evaluation and Program Planning, Elsevier, vol. 34(2), pages 79-86, May.
    3. Adcock, Robert & Collier, David, 2001. "Measurement Validity: A Shared Standard for Qualitative and Quantitative Research," American Political Science Review, Cambridge University Press, vol. 95(3), pages 529-546, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Schakel, Arjan Hille, 2009. "A Postfunctionalist Theory of Regional Government," MPRA Paper 21596, University Library of Munich, Germany.
    2. Zim Nwokora & Riccardo Pelizzo, 2017. "Measuring Party System Change: A Systems Perspective," Research Africa Network Working Papers 17/048, Research Africa Network (RAN).
    3. Gustav Lidén, 2013. "What about theory? The consequences on a widened perspective of social theory," Quality & Quantity: International Journal of Methodology, Springer, vol. 47(1), pages 213-225, January.
    4. Malte Luebker, 2019. "Can the Structure of Inequality Explain Fiscal Redistribution? Revisiting the Social Affinity Hypothesis," LIS Working papers 762, LIS Cross-National Data Center in Luxembourg.
    5. J. C. Sharman, 2007. "Rationalist and Constructivist Perspectives on Reputation," Political Studies, Political Studies Association, vol. 55(1), pages 20-37, March.
    6. Jørgen Møller, 2016. "Composite and Loose Concepts, Historical Analogies, and the Logic of Control in Comparative Historical Analysis," Sociological Methods & Research, , vol. 45(4), pages 651-677, November.
    7. Nora Lustig, 2013. "Commitment to Equity: Diagnostic Questionnaire," Commitment to Equity (CEQ) Working Paper Series 02, Tulane University, Department of Economics.
    8. Muhammad Nabeel Siddiqui, 2013. "Impact Of Work Life Conflict On Employee Performance," Far East Journal of Psychology and Business, Far East Research Centre, vol. 12(3), pages 26-40, September.
    9. Liesbet Hooghe & Gary Marks, 2015. "Delegation and pooling in international organizations," The Review of International Organizations, Springer, vol. 10(3), pages 305-328, September.
    10. Jose Antonio Alonso & Ana Luiza Cortez & Stephan Klasen, 2014. "LDC and other country groupings: How useful are current approaches to classify countries in a more hetergeneous developing world?," CDP Background Papers 021, United Nations, Department of Economics and Social Affairs.
    11. Jacint Jordana & Xavier Fernández‐i‐Marín & Andrea C. Bianculli, 2018. "Agency proliferation and the globalization of the regulatory state: Introducing a data set on the institutional features of regulatory agencies," Regulation & Governance, John Wiley & Sons, vol. 12(4), pages 524-540, December.
    12. Vanessa Alexandra Boese & Matthew Charles Wilson, 2023. "Contestation and participation: Concepts, measurement, and inference," International Area Studies Review, Center for International Area Studies, Hankuk University of Foreign Studies, vol. 26(2), pages 89-106, June.
    13. Willem E. Saris & André Pirralha & Diana Zavala-Rojas, 2018. "Testing the Comparability of Different Types of Social Indicators Across Groups," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 135(3), pages 927-939, February.
    14. Anete Veidemane & Frans Kaiser & Daniela Craciun, 2021. "Inclusive Higher Education Access for Underrepresented Groups: It Matters, But How Can Universities Measure It?," Social Inclusion, Cogitatio Press, vol. 9(3), pages 44-57.
    15. Svend-Erik Skaaning, 2018. "Different Types of Data and the Validity of Democracy Measures," Politics and Governance, Cogitatio Press, vol. 6(1), pages 105-116.
    16. Maria J. Debre & Hylke Dijkstra, 2023. "Are international organisations in decline? An absolute and relative perspective on institutional change," Global Policy, London School of Economics and Political Science, vol. 14(1), pages 16-30, February.
    17. Peter Mascini & Yannis Bacharias, 2012. "Integrating a Top‐Down and a Bottom‐Up Approach: Formal and Informal Risk‐Handling Strategies in a Utility Company," Risk Analysis, John Wiley & Sons, vol. 32(9), pages 1547-1560, September.
    18. Todd Landman, 2018. "Democracy and Human Rights: Concepts, Measures, and Relationships," Politics and Governance, Cogitatio Press, vol. 6(1), pages 48-59.
    19. Pierre-Marc Daigneault & Dominic Duval & Louis M. Imbeau, 2018. "Supervised scaling of semi-structured interview transcripts to characterize the ideology of a social policy reform," Quality & Quantity: International Journal of Methodology, Springer, vol. 52(5), pages 2151-2162, September.
    20. Daigneault, Pierre-Marc, 2014. "Taking stock of four decades of quantitative research on stakeholder participation and evaluation use: A systematic map," Evaluation and Program Planning, Elsevier, vol. 45(C), pages 171-181.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:36:y:2012:i:4:p:243-271. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.