IDEAS home Printed from https://ideas.repec.org/a/sae/metjou/v22y2023i1p18-27.html
   My bibliography  Save this article

Validating an Automated System of Evaluation for English Drafting Skill: A Case of Large-scale, High-stakes Selection of Entry-level Managerial Positions

Author

Listed:
  • A. K. Konar
  • Ayesha Martin
  • Sombala Ningthoujam

Abstract

The effectiveness of using a constructed response measure for assessing drafting ability is proven and has been used extensively in managerial selection in state-owned organizations. With the advent of online recruitment trends and technology-enhanced assessments, automated scoring has been conceived as a replacement for human scoring with the purpose of emulating the human scoring system. In the context of large-scale writing assessments, automated scoring could provide superior results to human scoring in terms of validity and reliability. In the present study, an attempt has been made to validate an automated essay scoring (AES) algorithm. The study was conducted on a sample of 11,497 randomly selected from a population of 54,392 shortlisted candidates for a national-level examination for the selection of entry-level executives in managerial positions in state-owned banks and a state-owned insurance company. The evaluation of the descriptive (constructed response) component of the examination (English composition) was carried out parallelly by four expert human raters and the AES algorithm. The parameters for evaluation were devised and made available to raters and the algorithm. Data were analysed using mean SD and Pearson correlation coefficient. Results show that the mean scores of human expert raters ( M = 14.648) and automated algorithm method ( M = 15.804) were similar. Further analysis was also undertaken to check the convergent validity of the features used in the algorithm by examining the relation of algorithm scores with sub scores from the objective test of the same construct. Results indicate a significant correlation. It can thus be said that the algorithm scoring method developed can be considered a replacement in that it can complement human expert raters in the evaluation of descriptive papers with consistent scoring and fairness, without inherent biases of inter-rater and intra-rater variation, in addition to practical benefits of speed and cost.

Suggested Citation

  • A. K. Konar & Ayesha Martin & Sombala Ningthoujam, 2023. "Validating an Automated System of Evaluation for English Drafting Skill: A Case of Large-scale, High-stakes Selection of Entry-level Managerial Positions," Metamorphosis: A Journal of Management Research, , vol. 22(1), pages 18-27, June.
    • Handle: RePEc:sae:metjou:v:22:y:2023:i:1:p:18-27
    DOI: 10.1177/09726225231173054
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/09726225231173054
    Download Restriction: no
    File URL: https://libkey.io/10.1177/09726225231173054?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:metjou:v:22:y:2023:i:1:p:18-27. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.