IDEAS home Printed from https://ideas.repec.org/a/spr/annopr/v333y2024i2d10.1007_s10479-022-04874-2.html
   My bibliography  Save this article

Data-driven subjective performance evaluation: An attentive deep neural networks model based on a call centre case

Author

Listed:
  • Abdelrahman Ahmed

    (University of Bradford)

  • Uthayasankar Sivarajah

    (University of Bradford)

  • Zahir Irani

    (University of Bradford)

  • Kamran Mahroof

    (University of Bradford)

  • Vincent Charles

    (University of Bradford)

Abstract

Every contact centre engages in some form of Call Quality Monitoring in order to improve agent performance and customer satisfaction. Call centres have traditionally used a manual process to sort, select, and analyse a representative sample of interactions for evaluation purposes. Unfortunately, such a process is marked by subjectivity, which in turn results in a distorted picture of agent performance. To address the challenge of identifying and removing subjectivity, empirical research is required. In this paper, we introduce an evidence-based, machine learning-driven framework for the automatic detection of subjective calls. We analyse a corpus of seven hours of recorded calls from a real-estate call centre using Deep Neural Network (DNN) for a multi-classification problem. The study establishes the first baseline for subjectivity detection, with an accuracy of 75%, which is comparable to relevant speech studies in emotional recognition and performance classification. We conclude, among other things, that in order to achieve the best performance evaluation, subjective calls should be removed from the evaluation process or subjective scores deducted from the overall results.

Suggested Citation

  • Abdelrahman Ahmed & Uthayasankar Sivarajah & Zahir Irani & Kamran Mahroof & Vincent Charles, 2024. "Data-driven subjective performance evaluation: An attentive deep neural networks model based on a call centre case," Annals of Operations Research, Springer, vol. 333(2), pages 939-970, February.
  • Handle: RePEc:spr:annopr:v:333:y:2024:i:2:d:10.1007_s10479-022-04874-2
    DOI: 10.1007/s10479-022-04874-2
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s10479-022-04874-2
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    File URL: https://libkey.io/10.1007/s10479-022-04874-2?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Milgrom, Paul R, 1988. "Employment Contracts, Influence Activities, and Efficient Organization Design," Journal of Political Economy, University of Chicago Press, vol. 96(1), pages 42-60, February.
    2. Bentley W. MacLeod, 2003. "Optimal Contracting with Subjective Evaluation," American Economic Review, American Economic Association, vol. 93(1), pages 216-240, March.
    3. Luria Gil & Gal Iddo & Yagil Dana, 2015. "Spending more time with the customer: service-providers’ behavioral discretion and call-center operations," Service Business, Springer;Pan-Pacific Business Association, vol. 9(3), pages 427-443, September.
    4. Angelovski, Andrej & Brandts, Jordi & Sola, Carles, 2016. "Hiring and escalation bias in subjective performance evaluations: A laboratory experiment," Journal of Economic Behavior & Organization, Elsevier, vol. 121(C), pages 114-129.
    5. Frederiksen, Anders & Lange, Fabian & Kriechel, Ben, 2017. "Subjective performance evaluations and employee careers," Journal of Economic Behavior & Organization, Elsevier, vol. 134(C), pages 408-429.
    6. Gunasekaran, Angappa & Ngai, Eric W.T., 2012. "The future of operations management: An outlook and analysis," International Journal of Production Economics, Elsevier, vol. 135(2), pages 687-701.
    7. Siti Nur Hidayah Ibrahim & Choo Ling Suan & Osman M. Karatepe, 2019. "The effects of supervisor support and self-efficacy on call center employees’ work engagement and quitting intentions," International Journal of Manpower, Emerald Group Publishing Limited, vol. 40(4), pages 688-703, February.
    8. Wamba, Samuel Fosso & Gunasekaran, Angappa & Akter, Shahriar & Ren, Steven Ji-fan & Dubey, Rameshwar & Childe, Stephen J., 2017. "Big data analytics and firm performance: Effects of dynamic capabilities," Journal of Business Research, Elsevier, vol. 70(C), pages 356-365.
    9. Charles, Vincent & Aparicio, Juan & Zhu, Joe, 2019. "The curse of dimensionality of decision-making units: A simple approach to increase the discriminatory power of data envelopment analysis," European Journal of Operational Research, Elsevier, vol. 279(3), pages 929-940.
    10. Prendergast, Canice & Topel, Robert H, 1996. "Favoritism in Organizations," Journal of Political Economy, University of Chicago Press, vol. 104(5), pages 958-978, October.
    11. Stephen Deery & Roderick Iverson & Janet Walsh, 2002. "Work Relationships in Telephone Call Centres: Understanding Emotional Exhaustion and Employee Withdrawal," Journal of Management Studies, Wiley Blackwell, vol. 39(4), pages 471-496, June.
    12. Akter, Shahriar & Wamba, Samuel Fosso & Gunasekaran, Angappa & Dubey, Rameshwar & Childe, Stephen J., 2016. "How to improve firm performance using big data analytics capability and business strategy alignment?," International Journal of Production Economics, Elsevier, vol. 182(C), pages 113-131.
    13. Sarah Hudson & Helena González-Gómez & Aude Rychalski, 2017. "Call centers: is there an upside to the dissatisfied customer experience?," Post-Print hal-02006507, HAL.
    14. W. Bentley MacLeod & Teck Yong Tan, 2016. "Optimal Contracting with Subjective Evaluation: The Effects of Timing, Malfeasance and Guile," NBER Working Papers 22156, National Bureau of Economic Research, Inc.
    15. Elaine Mosakowski, 1997. "Strategy Making Under Causal Ambiguity: Conceptual Issues and Empirical Evidence," Organization Science, INFORMS, vol. 8(4), pages 414-442, August.
    16. Prendergast, Canice & Topel, Robert, 1993. "Discretion and bias in performance evaluation," European Economic Review, Elsevier, vol. 37(2-3), pages 355-365, April.
    17. Tirole, Jean, 1986. "Hierarchies and Bureaucracies: On the Role of Collusion in Organizations," The Journal of Law, Economics, and Organization, Oxford University Press, vol. 2(2), pages 181-214, Fall.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Frederiksen, Anders & Lange, Fabian & Kriechel, Ben, 2017. "Subjective performance evaluations and employee careers," Journal of Economic Behavior & Organization, Elsevier, vol. 134(C), pages 408-429.
    2. Ockenfels, Axel & Sliwka, Dirk & Werner, Peter, 2024. "Multi-Rater Performance Evaluations and Incentives," IZA Discussion Papers 16812, Institute of Labor Economics (IZA).
    3. Lucia Marchegiani & Tommaso Reggiani & Matteo Rizzolli, 2013. "Severity vs. Leniency Bias in Performance Appraisal: Experimental evidence," BEMPS - Bozen Economics & Management Paper Series BEMPS01, Faculty of Economics and Management at the Free University of Bozen.
    4. Marchegiani, Lucia & Reggiani, Tommaso & Rizzolli, Matteo, 2016. "Loss averse agents and lenient supervisors in performance appraisal," Journal of Economic Behavior & Organization, Elsevier, vol. 131(PA), pages 183-197.
    5. Giebe, Thomas & Gürtler, Oliver, 2012. "Optimal contracts for lenient supervisors," Journal of Economic Behavior & Organization, Elsevier, vol. 81(2), pages 403-420.
    6. Robin, Stéphane & Rusinowska, Agnieszka & Villeval, Marie Claire, 2014. "Ingratiation: Experimental evidence," European Economic Review, Elsevier, vol. 66(C), pages 16-38.
    7. repec:eee:labchp:v:3:y:1999:i:pb:p:2373-2437 is not listed on IDEAS
    8. Anders Frederiksen & Lisa B. Kahn & Fabian Lange, 2020. "Supervisors and Performance Management Systems," Journal of Political Economy, University of Chicago Press, vol. 128(6), pages 2123-2187.
    9. Agnieszka Rusinowska & Vassili Vergopoulos, 2020. "Ingratiation and Favoritism in Organizations," Journal of Institutional and Theoretical Economics (JITE), Mohr Siebeck, Tübingen, vol. 176(3), pages 413-445.
    10. Corgnet, Brice & Rodriguez-Lara, Ismael, 2013. "Are you a good employee or simply a good guy? Influence costs and contract design," Journal of Economic Behavior & Organization, Elsevier, vol. 92(C), pages 259-272.
    11. Eren, Ozkan, 2023. "Potential in-group bias at work: Evidence from performance evaluations," Journal of Economic Behavior & Organization, Elsevier, vol. 206(C), pages 296-312.
    12. Oriana Bandiera & Iwan Barankay & Imran Rasul, 2009. "Social Connections and Incentives in the Workplace: Evidence From Personnel Data," Econometrica, Econometric Society, vol. 77(4), pages 1047-1094, July.
    13. Alan Benson & Danielle Li & Kelly Shue, 2019. "Promotions and the Peter Principle," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 134(4), pages 2085-2134.
    14. Rocio Bonet & Tor Eriksson & Jaime Ortega, 2019. "Up for Review: Unravelling the Link between Formal Evaluations and Performance‐Based Rewards," Industrial Relations: A Journal of Economy and Society, Wiley Blackwell, vol. 58(1), pages 108-137, January.
    15. Juho Jokinen & Jaakko Pehkonen, 2021. "The role of personal and relative job performance in promotion decisions," LABOUR, CEIS, vol. 35(4), pages 485-499, December.
    16. Bandiera, Oriana & Barankay, Iwan & Rasul, Imran, 2008. "Social capital in the workplace: Evidence on its formation and consequences," Labour Economics, Elsevier, vol. 15(4), pages 724-748, August.
    17. Pierre Fleckinger & David Martimort & Nicolas Roux, 2024. "Should They Compete or Should They Cooperate? The View of Agency Theory," Journal of Economic Literature, American Economic Association, vol. 62(4), pages 1589-1646, December.
    18. Marianne Bertrand & Robin Burgess & Arunish Chawla & Guo Xu, 2020. "The Glittering Prizes: Career Incentives and Bureaucrat Performance," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 87(2), pages 626-655.
    19. De Chiara, Alessandro & Livio, Luca, 2017. "The threat of corruption and the optimal supervisory task," Journal of Economic Behavior & Organization, Elsevier, vol. 133(C), pages 172-186.
    20. Kouroche Vafaï, 1999. "A Theory of Abuse of Authority in Hierarchies," CIRANO Working Papers 99s-07, CIRANO.
    21. Peter Cappelli & Martin J. Conyon, 2018. "What Do Performance Appraisals Do?," ILR Review, Cornell University, ILR School, vol. 71(1), pages 88-116, January.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:annopr:v:333:y:2024:i:2:d:10.1007_s10479-022-04874-2. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.