IDEAS home Printed from https://ideas.repec.org/a/eee/ejores/v228y2013i1p148-157.html
   My bibliography  Save this article

Comparing the validity of numerical judgements elicited by direct rating and point allocation: Insights from objectively verifiable perceptual tasks

Author

Listed:
  • Bottomley, Paul A.
  • Doyle, John R.

Abstract

Two popular methods for assigning numerical values to a set of to-be-judged objects in order to capture their relative standing are Direct Rating (DR) and Point Allocation (PA). People using PA distribute a fixed sum of 100 points among the objects, while people using DR rate each object on a fixed scale, typically 0–10, later rescaled to sum to 100. Prior research shows that these methods exhibit distinct profiles when values are ranked from largest to smallest, with DR being more test–retest reliable. But which method best translates people’s inner judgments into outer numerical values (is more valid)? Instead of examining subjective or abstract stimuli, we use objectively verifiable perceptual tasks, namely judgments of line length presented using bar charts. We show that (i) DR is more inter-rater reliable than PA; (ii) DR is more accurate than PA at the individual level; (iii) but there is no difference in accuracy when individual judgments are combined to form group-level estimates; and (iv) DR judgments were improved by using prior knowledge of method bias, whereas PA judgments were not.

Suggested Citation

  • Bottomley, Paul A. & Doyle, John R., 2013. "Comparing the validity of numerical judgements elicited by direct rating and point allocation: Insights from objectively verifiable perceptual tasks," European Journal of Operational Research, Elsevier, vol. 228(1), pages 148-157.
  • Handle: RePEc:eee:ejores:v:228:y:2013:i:1:p:148-157
    DOI: 10.1016/j.ejor.2013.01.005
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0377221713000143
    Download Restriction: Full text for ScienceDirect subscribers only

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Simonson, Itamar, 1989. " Choice Based on Reasons: The Case of Attraction and Compromise Effects," Journal of Consumer Research, Oxford University Press, vol. 16(2), pages 158-174, September.
    2. Van Ittersum, Koert & Pennings, Joost M.E. & Wansink, Brian & van Trijp, Hans C.M., 2007. "The validity of attribute-importance measurement: A review," Journal of Business Research, Elsevier, vol. 60(11), pages 1177-1190, November.
    3. Yeh, Chung-Hsing & Chang, Yu-Hern, 2009. "Modeling subjective evaluation for fuzzy group multicriteria decision making," European Journal of Operational Research, Elsevier, vol. 194(2), pages 464-473, April.
    4. F. Hutton Barron & Bruce E. Barrett, 1996. "Decision Quality Using Ranked Attribute Weights," Management Science, INFORMS, vol. 42(11), pages 1515-1523, November.
    5. John R. Doyle, 1999. "Elicitation and Context Effects in Judgments: Fixed Sum Versus Fixed Scale Frames," Management Science, INFORMS, vol. 45(7), pages 972-979, July.
    6. Conde, Eduardo & de la Paz Rivera Pérez, María, 2010. "A linear optimization problem to derive relative weights using an interval judgement matrix," European Journal of Operational Research, Elsevier, vol. 201(2), pages 537-544, March.
    7. Wansink, Brian & van Ittersum, Koert, 2003. " Bottoms Up! The Influence of Elongation on Pouring and Consumption Volume," Journal of Consumer Research, Oxford University Press, vol. 30(3), pages 455-463, December.
    8. Arrow, Kenneth J, 1982. "Risk Perception in Psychology and Economics," Economic Inquiry, Western Economic Association International, vol. 20(1), pages 1-9, January.
    9. Bous, Géraldine & Fortemps, Philippe & Glineur, François & Pirlot, Marc, 2010. "ACUTA: A novel method for eliciting additive value functions on the basis of holistic preference statements," European Journal of Operational Research, Elsevier, vol. 206(2), pages 435-444, October.
    10. Kadziński, Miłosz & Greco, Salvatore & Słowiński, Roman, 2012. "Selection of a representative value function in robust multiple criteria ranking and choice," European Journal of Operational Research, Elsevier, vol. 217(3), pages 541-553.
    11. Jaccard, James & Brinberg, David & Ackerman, Lee J, 1986. " Assessing Attribute Importance: A Comparison of Six Methods," Journal of Consumer Research, Oxford University Press, vol. 12(4), pages 463-468, March.
    12. A Ishizaka & D Balkenborg & T Kaplan, 2011. "Does AHP help us make a choice? An experimental evaluation," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 62(10), pages 1801-1812, October.
    13. Siraj, S. & Mikhailov, L. & Keane, J.A., 2012. "Preference elicitation from inconsistent judgments using multi-objective optimization," European Journal of Operational Research, Elsevier, vol. 220(2), pages 461-471.
    14. Poyhonen, Mari & Vrolijk, Hans & Hamalainen, Raimo P., 2001. "Behavioral and procedural consequences of structural variation in value trees," European Journal of Operational Research, Elsevier, vol. 134(1), pages 216-227, October.
    15. Srivastava, Joydeep & Connolly, Terry & Beach, Lee Roy, 1995. "Do Ranks Suffice? A Comparison of Alternative Weighting Approaches in Value Elicitation," Organizational Behavior and Human Decision Processes, Elsevier, vol. 63(1), pages 112-116, July.
    16. Doyle, John R. & Green, Rodney H. & Bottomley, Paul A., 1997. "Judging Relative Importance: Direct Rating and Point Allocation Are Not Equivalent," Organizational Behavior and Human Decision Processes, Elsevier, vol. 70(1), pages 65-72, April.
    17. Slovic, Paul & Finucane, Melissa L. & Peters, Ellen & MacGregor, Donald G., 2007. "The affect heuristic," European Journal of Operational Research, Elsevier, vol. 177(3), pages 1333-1352, March.
    18. Musal, R. Muzaffer & Soyer, Refik & McCabe, Christopher & Kharroubi, Samer A., 2012. "Estimating the population utility function: A parametric Bayesian approach," European Journal of Operational Research, Elsevier, vol. 218(2), pages 538-547.
    19. Jessop, Alan, 2004. "Minimally biased weight determination in personnel selection," European Journal of Operational Research, Elsevier, vol. 153(2), pages 433-444, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Lienert, Judit & Duygan, Mert & Zheng, Jun, 2016. "Preference stability over time with multiple elicitation methods to support wastewater infrastructure decision-making," European Journal of Operational Research, Elsevier, vol. 253(3), pages 746-760.
    2. Fasolo, Barbara & Bana e Costa, Carlos A., 2014. "Tailoring value elicitation to decision makers' numeracy and fluency: Expressing value judgments in numbers or words," Omega, Elsevier, vol. 44(C), pages 83-90.
    3. Jessop, Alan, 2014. "IMP: A decision aid for multiattribute evaluation using imprecise weight estimates," Omega, Elsevier, vol. 49(C), pages 18-29.
    4. Ülengin, Füsun & Önsel, Şule & Aktas, Emel & Kabak, Özgür & Özaydın, Özay, 2014. "A decision support methodology to enhance the competitiveness of the Turkish automotive industry," European Journal of Operational Research, Elsevier, vol. 234(3), pages 789-801.
    5. Hongdan Li & Wenjiao Shi & Bing Wang & Tingting An & Shuang Li & Shuangyi Li & Jingkuan Wang, 2017. "Comparison of the modeled potential yield versus the actual yield of maize in Northeast China and the implications for national food security," Food Security: The Science, Sociology and Economics of Food Production and Access to Food, Springer;The International Society for Plant Pathology, vol. 9(1), pages 99-114, February.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:ejores:v:228:y:2013:i:1:p:148-157. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Dana Niculescu). General contact details of provider: http://www.elsevier.com/locate/eor .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.