IDEAS home Printed from https://ideas.repec.org/a/sae/somere/v46y2017i3p490-539.html
   My bibliography  Save this article

First Equals Most Important? Order Effects in Vignette-Based Measurement

Author

Listed:
  • Katrin Auspurg
  • Annette Jäckle

Abstract

To measure what determines people’s attitudes, definitions, or decisions, surveys increasingly ask respondents to judge vignettes. A vignette typically describes a hypothetical situation or object as having various attributes (dimensions). In factorial surveys, the values (levels) of dimensions are experimentally varied, so that their impact on respondents’ judgments can be estimated. Drawing on the literature in cognitive psychology and survey methodology, we examine two research questions: Does the order in which dimensions are presented impact the vignette evaluations and change substantive conclusions? Under which conditions are order effects mostly likely to occur? Using data from a web survey of 300 students, we analyze several possible moderators: features of the vignette design, characteristics of respondents, and interactions between these features. Results show that strong order effects can occur, but only when the vignettes are of a minimum complexity or respondents show a low attitude certainty.

Suggested Citation

  • Katrin Auspurg & Annette Jäckle, 2017. "First Equals Most Important? Order Effects in Vignette-Based Measurement," Sociological Methods & Research, , vol. 46(3), pages 490-539, August.
    • Handle: RePEc:sae:somere:v:46:y:2017:i:3:p:490-539
    DOI: 10.1177/0049124115591016
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0049124115591016
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0049124115591016?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Guillermina Jasso, 2012. "Safeguarding Justice Research," Sociological Methods & Research, , vol. 41(1), pages 217-239, February.
    2. Barry Markovsky & Kimmo Eriksson, 2012. "Comparing Direct and Indirect Measures of Just Rewards," Sociological Methods & Research, , vol. 41(1), pages 199-216, February.
    3. Borghans, Lex & Romans, Margo & Sauermann, Jan, 2010. "What makes a good conference? Analysing the preferences of labour economists," Labour Economics, Elsevier, vol. 17(5), pages 868-874, October.
    4. McKEE J. McCLENDON, 1991. "Acquiescence and Recency Response-Order Effects in Interview Surveys," Sociological Methods & Research, , vol. 20(1), pages 60-103, August.
    5. Denise Bijlenga & Gouke J. Bonsel & Erwin Birnie, 2011. "Eliciting willingness to pay in obstetrics: comparing a direct and an indirect valuation method for complex health outcomes," Health Economics, John Wiley & Sons, Ltd., vol. 20(11), pages 1392-1406, November.
    6. Hermann Dülmer, 2007. "Experimental Plans in Factorial Surveys," Sociological Methods & Research, , vol. 35(3), pages 382-409, February.
    7. Louviere,Jordan J. & Hensher,David A. & Swait,Joffre D. With contributions by-Name:Adamowicz,Wiktor, 2000. "Stated Choice Methods," Cambridge Books, Cambridge University Press, number 9780521788304.
    8. Heike Diefenbach & Karl-Dieter Opp, 2007. "When and Why Do People Think There Should Be a Divorce?," Rationality and Society, , vol. 19(4), pages 485-517, November.
    9. Joop J. Hox & Ita G. G. Kreft & Piet L. J. Hermkens, 1991. "The Analysis of Factorial Surveys," Sociological Methods & Research, , vol. 19(4), pages 493-510, May.
    10. Sauer, Carsten & Liebig, Stefan & Auspurg, Katrin & Hinz, Thomas & Donaubauer, Andy & Schupp, Jürgen, 2009. "A Factorial Survey on the Justice of Earnings within the SOEP-Pretest 2008," IZA Discussion Papers 4663, Institute of Labor Economics (IZA).
    11. Scott, Anthony & Vick, Sandra, 1999. "Patients, Doctors and Contracts: An Application of Principal-Agent Theory to the Doctor-Patient Relationship," Scottish Journal of Political Economy, Scottish Economic Society, vol. 46(2), pages 111-134, May.
    12. Jan Fidrmuc & Peter Huber, 2007. "Introduction," Empirica, Springer;Austrian Institute for Economic Research;Austrian Economic Association, vol. 34(4), pages 281-286, September.
    13. Shelley Farrar & Mandy Ryan, 1999. "Response‐ordering effects: a methodological issue in conjoint analysis," Health Economics, John Wiley & Sons, Ltd., vol. 8(1), pages 75-79, February.
    14. Trine Kjær & Mickael Bech & Dorte Gyrd‐Hansen & Kristian Hart‐Hansen, 2006. "Ordering effect and price sensitivity in discrete choice experiments: need we worry?," Health Economics, John Wiley & Sons, Ltd., vol. 15(11), pages 1217-1228, November.
    15. Anthony Scott & Sandra Vick, 1999. "Patients, Doctors and Contracts: An Application of Principal‐Agent Theory to the Doctor‐Patient Relationship," Scottish Journal of Political Economy, Scottish Economic Society, vol. 46(2), pages 111-134, May.
    16. Vincent Buskens & Jeroen Weesie, 2000. "An Experiment On The Effects Of Embeddedness In Trust Situations," Rationality and Society, , vol. 12(2), pages 227-253, May.
    17. Liebig, Stefan & Mau, Steffen, 2005. "Wann ist ein Steuersystem gerecht? Einstellungen zu allgemeinen Prinzipien der Besteuerung und zur Gerechtigkeit der eigenen Steuerlast," Duisburger Beiträge zur soziologischen Forschung 1/2005, University of Duisburg-Essen, Institute of Sociology.
    18. Jeff Bennett & Russell Blamey (ed.), 2001. "The Choice Modelling Approach to Environmental Valuation," Books, Edward Elgar Publishing, number 2028.
    19. Carsten Sauer & Katrin Auspurg & Thomas Hinz & Stefan Liebig & Jürgen Schupp, 2014. "Method Effects in Factorial Surveys: An Analysis of Respondents' Comments, Interviewers' Assessments, and Response Behavior," SOEPpapers on Multidisciplinary Panel Data Research 629, DIW Berlin, The German Socio-Economic Panel (SOEP).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Krug, Gerhard & Bähr, Sebastian & Diener, Katharina & Abraham, Martin, 2020. "Do parents' flexible working hours affect fathers' contribution to domestic work? : evidence from a factorial survey," IAB-Discussion Paper 202012, Institut für Arbeitsmarkt- und Berufsforschung (IAB), Nürnberg [Institute for Employment Research, Nuremberg, Germany].
    2. Blaine G. Robbins, 2017. "Status, identity, and ability in the formation of trust," Rationality and Society, , vol. 29(4), pages 408-448, November.
    3. Tamara Gutfleisch & Robin Samuel & Stefan Sacchi, 2021. "The application of factorial surveys to study recruiters’ hiring intentions: comparing designs based on hypothetical and real vacancies," Quality & Quantity: International Journal of Methodology, Springer, vol. 55(3), pages 775-804, June.
    4. Carsten Sauer & Katrin Auspurg & Thomas Hinz & Stefan Liebig & Jürgen Schupp, 2014. "Method Effects in Factorial Surveys: An Analysis of Respondents' Comments, Interviewers' Assessments, and Response Behavior," SOEPpapers on Multidisciplinary Panel Data Research 629, DIW Berlin, The German Socio-Economic Panel (SOEP).
    5. Di Falco, Salvatore & Sharma, Sindra, 2018. "Investing in Climate Change Adaptation: Motivations and Green Incentives in the Fiji Islands," Ecological Economics, Elsevier, vol. 154(C), pages 394-408.
    6. Talia Goren & Itai Beeri & Dana R. Vashdi, 2023. "Framing policies to mobilize citizens' behavior during a crisis: Examining the effects of positive and negative vaccination incentivizing policies," Regulation & Governance, John Wiley & Sons, vol. 17(2), pages 570-591, April.
    7. Jonathan Sicsic & Serge Blondel & Sandra Chyderiotis & François Langot & Judith E. Mueller, 2023. "Preferences for COVID-19 epidemic control measures among French adults: a discrete choice experiment," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 24(1), pages 81-98, February.
    8. Fabian Hattke & Janne Kalucza, 2019. "What influences the willingness of citizens to coproduce public services? Results from a vignette experiment," Journal of Behavioral Public Administration, Center for Experimental and Behavioral Public Administration, vol. 2(1).
    9. Knut Petzold, 2020. "Migration, Commuting, or a Second Home? Insights from an Experiment Among Academics," European Journal of Population, Springer;European Association for Population Studies, vol. 36(2), pages 277-315, April.
    10. Blaine Robbins & Edgar Kiser, 2018. "Legitimate authorities and rational taxpayers: An investigation of voluntary compliance and method effects in a survey experiment of income tax evasion," Rationality and Society, , vol. 30(2), pages 247-301, May.
    11. Volker Lingnau & Florian Fuchs & Florian Beham, 2019. "The impact of sustainability in coffee production on consumers’ willingness to pay–new evidence from the field of ethical consumption," Journal of Management Control: Zeitschrift für Planung und Unternehmenssteuerung, Springer, vol. 30(1), pages 65-93, April.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Taro Ohdoko & Kentaro Yoshida, 2012. "Public preferences for forest ecosystem management in Japan with emphasis on species diversity," Environmental Economics and Policy Studies, Springer;Society for Environmental Economics and Policy Studies - SEEPS, vol. 14(2), pages 147-169, April.
    2. Nick Hanley & Mandy Ryan & Robert Wright, 2003. "Estimating the monetary value of health care: lessons from environmental economics," Health Economics, John Wiley & Sons, Ltd., vol. 12(1), pages 3-16, January.
    3. Logar, Ivana & Brouwer, Roy & Campbell, Danny, 2020. "Does attribute order influence attribute-information processing in discrete choice experiments?," Resource and Energy Economics, Elsevier, vol. 60(C).
    4. Trine Kjær & Mickael Bech & Dorte Gyrd‐Hansen & Kristian Hart‐Hansen, 2006. "Ordering effect and price sensitivity in discrete choice experiments: need we worry?," Health Economics, John Wiley & Sons, Ltd., vol. 15(11), pages 1217-1228, November.
    5. Nguyen, Thanh Cong & Le, Hoa Thu & Nguyen, Hang Dieu & Ngo, Mai Thanh & Nguyen, Hong Quang, 2021. "Examining ordering effects and strategic behaviour in a discrete choice experiment," Economic Analysis and Policy, Elsevier, vol. 70(C), pages 394-413.
    6. Alessandro Mengoni & Chiara Seghieri & Sabina Nuti, 2013. "The application of discrete choice experiments in health economics: a systematic review of the literature," Working Papers 201301, Scuola Superiore Sant'Anna of Pisa, Istituto di Management.
    7. Mandy Ryan & Nicolas Krucien & Frouke Hermens, 2018. "The eyes have it: Using eye tracking to inform information processing strategies in multi‐attributes choices," Health Economics, John Wiley & Sons, Ltd., vol. 27(4), pages 709-721, April.
    8. Axel Mühlbacher & Uwe Junker & Christin Juhnke & Edgar Stemmler & Thomas Kohlmann & Friedhelm Leverkus & Matthias Nübling, 2015. "Chronic pain patients’ treatment preferences: a discrete-choice experiment," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 16(6), pages 613-628, July.
    9. Ulf Liebe & Veronika A. Andorfer & Patricia A. Gwartney & Jürgen Meyerhoff, 2014. "Ethical Consumption and Social Context: Experimental Evidence from Germany and the United States," University of Bern Social Sciences Working Papers 7, University of Bern, Department of Social Sciences.
    10. Kevin Boyle & Semra Özdemir, 2009. "Convergent Validity of Attribute-Based, Choice Questions in Stated-Preference Studies," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 42(2), pages 247-264, February.
    11. Debby van Helvoort‐Postulart & Benedict G. C. Dellaert & Trudy van der Weijden & Maarten F. von Meyenfeldt & Carmen D. Dirksen, 2009. "Discrete choice experiments for complex health‐care decisions: does hierarchical information integration offer a solution?," Health Economics, John Wiley & Sons, Ltd., vol. 18(8), pages 903-920, August.
    12. Harry Telser & Karolin Becker & Peter Zweifel, 2008. "Validity and Reliability of Willingness-to-Pay Estimates," The Patient: Patient-Centered Outcomes Research, Springer;International Academy of Health Preference Research, vol. 1(4), pages 283-298, October.
    13. Nicolas Krucien & Amiram Gafni & Nathalie Pelletier‐Fleury, 2015. "Empirical Testing of the External Validity of a Discrete Choice Experiment to Determine Preferred Treatment Option: The Case of Sleep Apnea," Health Economics, John Wiley & Sons, Ltd., vol. 24(8), pages 951-965, August.
    14. Robert J. Johnston & Kevin J. Boyle & Wiktor (Vic) Adamowicz & Jeff Bennett & Roy Brouwer & Trudy Ann Cameron & W. Michael Hanemann & Nick Hanley & Mandy Ryan & Riccardo Scarpa & Roger Tourangeau & Ch, 2017. "Contemporary Guidance for Stated Preference Studies," Journal of the Association of Environmental and Resource Economists, University of Chicago Press, vol. 4(2), pages 319-405.
    15. Choi, Andy S., 2013. "Nonmarket values of major resources in the Korean DMZ areas: A test of distance decay," Ecological Economics, Elsevier, vol. 88(C), pages 97-107.
    16. Mahieu, Pierre-Alexandre & Andersson, Henrik & Beaumais, Olivier & Crastes dit Sourd, Romain & Hess, François-Charles & Wolff, François-Charles, 2017. "Stated preferences: a unique database composed of 1657 recent published articles in journals related to agriculture, environment, or health," Review of Agricultural, Food and Environmental Studies, Institut National de la Recherche Agronomique (INRA), vol. 98(3), November.
    17. Joachim Marti, 2012. "Assessing preferences for improved smoking cessation medications: a discrete choice experiment," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 13(5), pages 533-548, October.
    18. de Ayala, Amaia & Hoyos, David & Mariel, Petr, 2015. "Suitability of discrete choice experiments for landscape management under the European Landscape Convention," Journal of Forest Economics, Elsevier, vol. 21(2), pages 79-96.
    19. Kallas, Z. & Gómez-Limón, J.A., 2007. "Valoración De La Multifuncionalidad Agraria: Una Aplicación A Través Del Método De Los Experimentos De Elección/Agricultural Multifunctionality Valuation: A Case Study Using The Choice Experiment," Estudios de Economia Aplicada, Estudios de Economia Aplicada, vol. 25, pages 107-144, Abril.
    20. Barr, Rhona F. & Mourato, Susana, 2014. "Investigating fishers' preferences for the design of marine Payments for Environmental Services schemes," Ecological Economics, Elsevier, vol. 108(C), pages 91-103.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:somere:v:46:y:2017:i:3:p:490-539. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.