IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0265312.html
   My bibliography  Save this article

Does it work? Using a Meta-Impact score to examine global effects in quasi-experimental intervention studies

Author

Listed:
  • Nancy Elizabeth Doyle
  • Almuth McDowall
  • Raymond Randall
  • Kate Knight

Abstract

The evaluation of applied psychological interventions in the workplace or elsewhere is challenging. Randomisation and matching are difficult to achieve and this often results in substantial heterogeneity within intervention and control groups. As a result, traditional comparison of group means using null hypothesis significance testing may mask effects experienced by some participants. Using longitudinal studies of coaching interventions designed to provide support for dyslexic employees, this study describes and evaluates a different approach using a Meta-Impact score. We offer a conceptual rationale for our method, illustrate how this score is calculated and analysed, and show how it highlights person-specific variations in how participants react and respond to interventions. We argue that Meta-Impact is an incremental supplement to traditional variable-centric group-wise comparisons and can more accurately demonstrate in practice the extent to which an intervention worked. Such methods are needed for applied research, where personalized intervention protocols may require impact analysis for policy, legal and ethical purposes, despite modest sample sizes.

Suggested Citation

  • Nancy Elizabeth Doyle & Almuth McDowall & Raymond Randall & Kate Knight, 2022. "Does it work? Using a Meta-Impact score to examine global effects in quasi-experimental intervention studies," PLOS ONE, Public Library of Science, vol. 17(3), pages 1-21, March.
    • Handle: RePEc:plo:pone00:0265312
    DOI: 10.1371/journal.pone.0265312
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0265312
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0265312&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0265312?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Hoenig J. M. & Heisey D. M., 2001. "The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis," The American Statistician, American Statistical Association, vol. 55, pages 19-24, February.
    2. Özbilgin, Mustafa F. & Beauregard, T. A. & Tatli, Ahu & Bell, Myrtle P., 2011. "Work-life, diversity and intersectionality: a critical review and research agenda," LSE Research Online Documents on Economics 36557, London School of Economics and Political Science, LSE Library.
    3. Briner, Rob B. & Walshe, Neil D., 2013. "The Causes and Consequences of a Scientific Literature We Cannot Trust: An Evidence-Based Practice Perspective," Industrial and Organizational Psychology, Cambridge University Press, vol. 6(3), pages 269-272, September.
    4. Santuzzi, Alecia M. & Waltz, Pamela R. & Finkelstein, Lisa M. & Rupp, Deborah E., 2014. "Invisible Disabilities: Unique Challenges for Employees and Organizations," Industrial and Organizational Psychology, Cambridge University Press, vol. 7(2), pages 204-219, June.
    5. Matthew C. Saleh & Susanne M. Bruyère, 2018. "Leveraging Employer Practices in Global Regulatory Frameworks to Improve Employment Outcomes for People with Disabilities," Social Inclusion, Cogitatio Press, vol. 6(1), pages 18-28.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Nancy Elizabeth Doyle & Almuth McDowall, 2019. "Context matters: A review to formulate a conceptual framework for coaching as a disability accommodation," PLOS ONE, Public Library of Science, vol. 14(8), pages 1-30, August.
    2. Tom Coupé & W. Robert Reed, 2021. "Do Negative Replications Affect Citations?," Working Papers in Economics 21/14, University of Canterbury, Department of Economics and Finance.
    3. Lavanya Vijayasingham & Uma Jogulu & Pascale Allotey, 2018. "Enriching the Organizational Context of Chronic Illness Experience Through an Ethics of Care Perspective," Journal of Business Ethics, Springer, vol. 153(1), pages 29-40, November.
    4. Mary Wickenden & Philip Mader & Stephen Thompson & Jackie Shaw, 2022. "Mainstreaming disability inclusive employment in international development," Journal of International Development, John Wiley & Sons, Ltd., vol. 34(5), pages 933-941, July.
    5. Weili Ding, 2020. "Laboratory experiments can pre-design to address power and selection issues," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 6(2), pages 125-138, December.
    6. Jiarui Tian, 2021. "A Replication of “The effect of the conservation reserve program on rural economies: Deriving a statistical verdict from a null finding” (American Journal of Agricultural Economics, 2019)," Working Papers in Economics 21/12, University of Canterbury, Department of Economics and Finance.
    7. Thomas Aichner, 2021. "The economic argument for hiring people with disabilities," Palgrave Communications, Palgrave Macmillan, vol. 8(1), pages 1-4, December.
    8. Jason W. Beckstead, 2007. "A note on determining the number of cues used in judgment analysis studies: The issue of type II error," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 2, pages 317-325, October.
    9. Siti Khadijah Zainal, 2013. "Job Characteristics as the Antecedents of Work-to-family Enrichment: A Literature Review," Journal of Social and Development Sciences, AMH International, vol. 4(8), pages 394-401.
    10. Kimball Chapman & Michael Drake & Joseph H. Schroeder & Timothy Seidel, 2023. "Earnings announcement delays and implications for the auditor-client relationship," Review of Accounting Studies, Springer, vol. 28(1), pages 45-90, March.
    11. Shae Wissell & Leila Karimi & Tanya Serry & Lisa Furlong & Judith Hudson, 2022. "Leading Diverse Workforces: Perspectives from Managers and Employers about Dyslexic Employees in Australian Workplaces," IJERPH, MDPI, vol. 19(19), pages 1-18, September.
    12. Jeffrey C. Valentine & Therese D. Pigott & Hannah R. Rothstein, 2010. "How Many Studies Do You Need?," Journal of Educational and Behavioral Statistics, , vol. 35(2), pages 215-247, April.
    13. Ana M. Morales-Hernández & Carlos Fernández-Hernández & Flora M. Díaz-Pérez & Carlos G. García-González, 2022. "Rural tourism networking and covid-19 crisis: a gender perspective," Service Business, Springer;Pan-Pacific Business Association, vol. 16(4), pages 1111-1137, December.
    14. Andrea Gragnano & Silvia Simbula & Massimo Miglioretti, 2020. "Work–Life Balance: Weighing the Importance of Work–Family and Work–Health Balance," IJERPH, MDPI, vol. 17(3), pages 1-20, February.
    15. van Koten, Silvester, 2021. "The forward premium in electricity markets: An experimental study," Energy Economics, Elsevier, vol. 94(C).
    16. Ernest Nkansah‐Dwamena, 2022. "How can we create a diverse, equitable and inclusive workplace in society without the voice of disability? Lessons from Ghana," Journal of International Development, John Wiley & Sons, Ltd., vol. 34(5), pages 1028-1047, July.
    17. Markku Maula & Wouter Stam, 2020. "Enhancing Rigor in Quantitative Entrepreneurship Research," Entrepreneurship Theory and Practice, , vol. 44(6), pages 1059-1090, November.
    18. Samantha Evans & Madeleine Wyatt, 2023. "A Bridge over Troubled Borders: Social Class and the Interplay between Work and Life," Work, Employment & Society, British Sociological Association, vol. 37(1), pages 137-156, February.
    19. Nigel Golden & Kadambari Devarajan & Cathleen Balantic & Joseph Drake & Michael T Hallworth & Toni Lyn Morelli, 2021. "Ten simple rules for productive lab meetings," PLOS Computational Biology, Public Library of Science, vol. 17(5), pages 1-13, May.
    20. Irina Surdu & Kamel Mellahi & Keith Glaister, 2017. "Once bitten, not necessarily shy? Organisational learning prior experience effects on foreign market re-entry commitment decisions," John H Dunning Centre for International Business Discussion Papers jhd-dp2017-04, Henley Business School, University of Reading.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0265312. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.