IDEAS home Printed from https://ideas.repec.org/a/kap/enreec/v53y2012i3p389-407.html
   My bibliography  Save this article

Test–Retest Reliability of Choice Experiments in Environmental Valuation

Author

Listed:
  • Ulf Liebe
  • Jürgen Meyerhoff
  • Volkmar Hartje

Abstract

The paper presents the results of the first test–retest study on choice experiments in environmental valuation. In a survey concerning landscape externalities of onshore wind power in central Germany, respondents answered the same five choice sets at two different points in time. Each choice set comprised three alternatives described by five attributes, and the time interval between the test and the retest was eleven months. The analysis takes place at three different levels, investigating choice consistency at the choice task level and repeatability of the latent construct utility at the level of parametric models as well as at the level of willingness-to-pay estimates. At the choice task level we observed 59 % identical choices. The parametric analysis shows that the test and retest estimates are not equal, even when we control for scale, that is, differences in the error variance. However, comparing the marginal willingness-to-pay estimates among test and retest reveals only a statistically significant difference for one of the attributes. Overall, this indicates a moderate test–retest reliability taking into account that consistency at the choice task level overlooks the stochastic nature of the process underlying discrete choice experiments. Copyright The Author(s) 2012

Suggested Citation

  • Ulf Liebe & Jürgen Meyerhoff & Volkmar Hartje, 2012. "Test–Retest Reliability of Choice Experiments in Environmental Valuation," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 53(3), pages 389-407, November.
    • Handle: RePEc:kap:enreec:v:53:y:2012:i:3:p:389-407
    DOI: 10.1007/s10640-012-9567-1
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1007/s10640-012-9567-1
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1007/s10640-012-9567-1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Boxall, Peter C. & Adamowicz, Wiktor L. & Moon, Amanda, 2009. "Complexity in choice experiments: choice of the status quo alternative and implications for welfare measurement," Australian Journal of Agricultural and Resource Economics, Australian Agricultural and Resource Economics Society, vol. 53(4), pages 1-17.
    2. Christie, Mike & Gibbons, James, 2011. "The effect of individual ‘ability to choose’ (scale heterogeneity) on the valuation of environmental goods," Ecological Economics, Elsevier, vol. 70(12), pages 2250-2257.
    3. K. McConnell* & I. Strand & Sebastián Valdés, 1998. "Testing Temporal Reliability and Carry-over Effect: The Role of Correlated Responses in Test-retest Reliability Studies," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 12(3), pages 357-374, October.
    4. Danny Campbell & W. Hutchinson & Riccardo Scarpa, 2008. "Incorporating Discontinuous Preferences into the Analysis of Discrete Choice Experiments," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 41(3), pages 401-417, November.
    5. William S. Breffle & Robert D. Rowe, 2002. "Comparing Choice Question Formats for Evaluating Natural Resource Tradeoffs," Land Economics, University of Wisconsin Press, vol. 78(2), pages 298-314.
    6. Louis Guttman, 1945. "A basis for analyzing test-retest reliability," Psychometrika, Springer;The Psychometric Society, vol. 10(4), pages 255-282, December.
    7. David A. Hensher & Stewart Jones & William H. Greene, 2007. "An Error Component Logit Analysis of Corporate Bankruptcy and Insolvency Risk in Australia," The Economic Record, The Economic Society of Australia, vol. 83(260), pages 86-103, March.
    8. Gregory L. Poe & Kelly L. Giraud & John B. Loomis, 2005. "Computational Methods for Measuring the Difference of Empirical Distributions," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 87(2), pages 353-365.
    9. Stirling Bryan & Lisa Gold & Rob Sheldon & Martin Buxton, 2000. "Preference measurement using conjoint methods: an empirical investigation of reliability," Health Economics, John Wiley & Sons, Ltd., vol. 9(5), pages 385-395, July.
    10. Meyerhoff, Jürgen & Ohl, Cornelia & Hartje, Volkmar, 2010. "Landscape externalities from onshore wind power," Energy Policy, Elsevier, vol. 38(1), pages 82-92, January.
    11. Ferrini, Silvia & Scarpa, Riccardo, 2007. "Designs with a priori information for nonmarket valuation with choice experiments: A Monte Carlo study," Journal of Environmental Economics and Management, Elsevier, vol. 53(3), pages 342-363, May.
    12. DeShazo, J. R. & Fermo, German, 2002. "Designing Choice Sets for Stated Preference Methods: The Effects of Complexity on Choice Consistency," Journal of Environmental Economics and Management, Elsevier, vol. 44(1), pages 123-143, July.
    13. Jorgensen, Bradley S. & Syme, Geoffrey J. & Smith, Leigh M. & Bishop, Brian J., 2004. "Random error in willingness to pay measurement: A multiple indicators, latent variable approach to the reliability of contingent values," Journal of Economic Psychology, Elsevier, vol. 25(1), pages 41-59, February.
    14. Thomas P. Holmes & Kevin J. Boyle, 2005. "Dynamic Learning and Context-Dependence in Sequential, Attribute-Based, Stated-Preference Valuation Questions," Land Economics, University of Wisconsin Press, vol. 81(1).
    15. Denzil G. Fiebig & Michael P. Keane & Jordan Louviere & Nada Wasi, 2010. "The Generalized Multinomial Logit Model: Accounting for Scale and Coefficient Heterogeneity," Marketing Science, INFORMS, vol. 29(3), pages 393-421, 05-06.
    16. Ryan, Mandy & Netten, Ann & Skatun, Diane & Smith, Paul, 2006. "Using discrete choice experiments to estimate a preference-based measure of outcome--An application to social care for older people," Journal of Health Economics, Elsevier, vol. 25(5), pages 927-944, September.
    17. Søren Olsen, 2009. "Choosing Between Internet and Mail Survey Modes for Choice Experiment Surveys Considering Non-Market Goods," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 44(4), pages 591-610, December.
    18. David Hensher & John Rose & William Greene, 2005. "The implications on willingness to pay of respondents ignoring specific attributes," Transportation, Springer, vol. 32(3), pages 203-222, May.
    19. Krinsky, Itzhak & Robb, A Leslie, 1986. "On Approximating the Statistical Properties of Elasticities," The Review of Economics and Statistics, MIT Press, vol. 68(4), pages 715-719, November.
    20. Jayson L. Lusk & F. Bailey Norwood, 2005. "Effect of Experimental Design on Choice-Based Conjoint Valuation Estimates," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 87(3), pages 771-785.
    21. Søren Olsen & Thomas Lundhede & Jette Jacobsen & Bo Thorsen, 2011. "Tough and Easy Choices: Testing the Influence of Utility Difference on Stated Certainty-in-Choice in Choice Experiments," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 49(4), pages 491-510, August.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Colombo, Sergio & Christie, Michael & Hanley, Nick, 2013. "What are the consequences of ignoring attributes in choice experiments? Implications for ecosystem service valuation," Ecological Economics, Elsevier, vol. 96(C), pages 25-35.
    2. Dan Rigby & Michael Burton & Jo Pluske, 2016. "Preference Stability and Choice Consistency in Discrete Choice Experiments," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 65(2), pages 441-461, October.
    3. repec:sss:wpaper:201404 is not listed on IDEAS
    4. Enni Roukamo & Mikolaj Czajkowski & Nick Hanley & A. Juutinen & R. Svento, 2016. "Linking perceived choice complexity with scale heterogeneity in discrete choice experiments: home heating in Finland," Discussion Papers in Environment and Development Economics 2016-16, University of St. Andrews, School of Geography and Sustainable Development.
    5. Schaafsma, Marije & Brouwer, Roy & Liekens, Inge & De Nocker, Leo, 2014. "Temporal stability of preferences and willingness to pay for natural areas in choice experiments: A test–retest," Resource and Energy Economics, Elsevier, vol. 38(C), pages 243-260.
    6. Mikołaj Czajkowski & Nick Hanley & Jacob LaRiviere, 2016. "Controlling for the Effects of Information in a Public Goods Discrete Choice Model," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 63(3), pages 523-544, March.
    7. Mara Thiene & Riccardo Scarpa & Jordan Louviere, 2015. "Addressing Preference Heterogeneity, Multiple Scales and Attribute Attendance with a Correlated Finite Mixing Model of Tap Water Choice," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 62(3), pages 637-656, November.
    8. Robert J. Johnston & Kevin J. Boyle & Wiktor (Vic) Adamowicz & Jeff Bennett & Roy Brouwer & Trudy Ann Cameron & W. Michael Hanemann & Nick Hanley & Mandy Ryan & Riccardo Scarpa & Roger Tourangeau & Ch, 2017. "Contemporary Guidance for Stated Preference Studies," Journal of the Association of Environmental and Resource Economists, University of Chicago Press, vol. 4(2), pages 319-405.
    9. Logar, Ivana & Brouwer, Roy & Campbell, Danny, 2020. "Does attribute order influence attribute-information processing in discrete choice experiments?," Resource and Energy Economics, Elsevier, vol. 60(C).
    10. Riccardo Scarpa & Mara Thiene & David A. Hensher, 2010. "Monitoring Choice Task Attribute Attendance in Nonmarket Valuation of Multiple Park Management Services: Does It Matter?," Land Economics, University of Wisconsin Press, vol. 86(4), pages 817-839.
    11. Kountouris, Yiannis & Nakic, Zoran & Sauer, Johannes, 2015. "Political instability and non-market valuation: Evidence from Croatia," Resource and Energy Economics, Elsevier, vol. 41(C), pages 19-39.
    12. Dan Marsh & Lena Mkwara & Riccardo Scarpa, 2011. "Do Respondents’ Perceptions of the Status Quo Matter in Non-Market Valuation with Choice Experiments? An Application to New Zealand Freshwater Streams," Sustainability, MDPI, vol. 3(9), pages 1-23, September.
    13. Jürgen Meyerhoff & Ulf Liebe, 2009. "Status Quo Effect in Choice Experiments: Empirical Evidence on Attitudes and Choice Task Complexity," Land Economics, University of Wisconsin Press, vol. 85(3), pages 515-528.
    14. Lindhjem, Henrik & Navrud, Ståle, 2011. "Using Internet in Stated Preference Surveys: A Review and Comparison of Survey Modes," International Review of Environmental and Resource Economics, now publishers, vol. 5(4), pages 309-351, September.
    15. Hoyos, David, 2010. "The state of the art of environmental valuation with discrete choice experiments," Ecological Economics, Elsevier, vol. 69(8), pages 1595-1603, June.
    16. Sergio Colombo & Nick Hanley & Jordan Louviere, 2009. "Modeling preference heterogeneity in stated choice data: an analysis for public goods generated by agriculture," Agricultural Economics, International Association of Agricultural Economists, vol. 40(3), pages 307-322, May.
    17. Haghani, Milad & Bliemer, Michiel C.J. & Hensher, David A., 2021. "The landscape of econometric discrete choice modelling research," Journal of choice modelling, Elsevier, vol. 40(C).
    18. Campbell, Danny, 2007. "Combining mixed logit models and random effects models to identify the determinants of willingness to pay for rural landscape improvements," 81st Annual Conference, April 2-4, 2007, Reading University, UK 7975, Agricultural Economics Society.
    19. Scarpa, Riccardo & Notaro, Sandra & Raffaelli, Roberta & Louviere, Jordan, 2011. "Modelling attribute non-attendance in best-worst rank ordered choice data to estimate tourism benefits from Alpine pasture heritage," 2011 International Congress, August 30-September 2, 2011, Zurich, Switzerland 115990, European Association of Agricultural Economists.
    20. Scarpa, Riccardo & Rose, John M., 2008. "Design efficiency for non-market valuation with choice modelling: how to measure it, what to report and why," Australian Journal of Agricultural and Resource Economics, Australian Agricultural and Resource Economics Society, vol. 52(3), pages 1-30.
    21. Espinosa-Goded, María & Rodriguez-Entrena, Macario & Salazar-Ordóñez, Melania, 2021. "A straightforward diagnostic tool to identify attribute non-attendance in discrete choice experiments," Economic Analysis and Policy, Elsevier, vol. 71(C), pages 211-226.

    More about this item

    Keywords

    Choice experiment; Environmental valuation; Test–retest reliability; Wind power; C8; Q0; Q5;
    All these keywords.

    JEL classification:

    • C8 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs
    • Q0 - Agricultural and Natural Resource Economics; Environmental and Ecological Economics - - General
    • Q5 - Agricultural and Natural Resource Economics; Environmental and Ecological Economics - - Environmental Economics

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:kap:enreec:v:53:y:2012:i:3:p:389-407. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.