IDEAS home Printed from https://ideas.repec.org/p/vie/viennp/vie1512.html
   My bibliography  Save this paper

Who gives Direction to Statistical Testing? Best Practice meets Mathematically Correct Tests

Author

Abstract

We are interested in statistical tests that are able to uncover that one method is better than another one. The Wilcoxon-Mann-Whitney rank-sum and the Wilcoxon sign-rank test are the most popular tests for showing that two methods are different. Yet all of the 32 papers in Economics we surveyed misused them to claim evidence that one method is better, without making any additional assumptions. We present eight nonparametric tests that can correctly identify which method is better in terms of a stochastic inequality, median difference and difference in medians or means, without adding any assumptions. We show that they perform very well in the data sets from the surveyed papers. The two tests for comparing medians are novel, constructed in the spirit of Mood's test.

Suggested Citation

  • Karl H.Schlag, 2015. "Who gives Direction to Statistical Testing? Best Practice meets Mathematically Correct Tests," Vienna Economics Papers vie1512, University of Vienna, Department of Economics.
    • Handle: RePEc:vie:viennp:vie1512
    as

    Download full text from publisher

    File URL: https://papersecon.univie.ac.at/RePEc/vie/viennp/vie1512.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Masella, Paolo & Meier, Stephan & Zahn, Philipp, 2014. "Incentives and group identity," Games and Economic Behavior, Elsevier, vol. 86(C), pages 12-25.
    2. Elena Asparouhova & Peter Bossaerts & Jon Eguia & William Zame, 2015. "Asset Pricing and Asymmetric Reasoning," Journal of Political Economy, University of Chicago Press, vol. 123(1), pages 66-122.
    3. Bin Miao & Songfa Zhong, 2015. "Risk Preferences Are Not Time Preferences: Separating Risk and Time Preference: Comment," American Economic Review, American Economic Association, vol. 105(7), pages 2272-2286, July.
    4. Charles N. Noussair & Stefan T. Trautmann & Gijs van de Kuilen, 2014. "Higher Order Risk Attitudes, Demographics, and Financial Decisions," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 81(1), pages 325-355.
    5. Dittmann, Ingolf & Kübler, Dorothea & Maug, Ernst & Mechtenberg, Lydia, 2014. "Why votes have value: Instrumental voting with overconfidence and overestimation of others' errors," Games and Economic Behavior, Elsevier, vol. 84(C), pages 17-38.
    6. Friedman, Daniel & Huck, Steffen & Oprea, Ryan & Weidenholzer, Simon, 2015. "From imitation to collusion: Long-run learning in a low-information environment," Journal of Economic Theory, Elsevier, vol. 155(C), pages 185-205.
    7. Regner, Tobias, 2014. "Social preferences? Google Answers!," Games and Economic Behavior, Elsevier, vol. 85(C), pages 188-209.
    8. Forsythe Robert & Horowitz Joel L. & Savin N. E. & Sefton Martin, 1994. "Fairness in Simple Bargaining Experiments," Games and Economic Behavior, Elsevier, vol. 6(3), pages 347-369, May.
    9. Alain Cohn & Jan Engelmann & Ernst Fehr & Michel André Maréchal, 2015. "Evidence for Countercyclical Risk Aversion: An Experiment with Financial Professionals," American Economic Review, American Economic Association, vol. 105(2), pages 860-885, February.
    10. Dufour, J-M. & Hallin, M., 1990. "Improved Eaton Bounds for Linear Combinations of Bounded Random Variables , with Statistical Applications," Papers 9104, Universite Libre de Bruxelles - C.E.M.E..
    11. Hugh-Jones, David & Kurino, Morimitsu & Vanberg, Christoph, 2014. "An experimental study on the incentives of the probabilistic serial mechanism," Games and Economic Behavior, Elsevier, vol. 87(C), pages 367-380.
    12. Karl Schlag, 2008. "A new method for constructing exact tests without making any assumptions," Economics Working Papers 1109, Department of Economics and Business, Universitat Pompeu Fabra.
    13. Andrea Isoni & Anders Poulsen & Robert Sugden & Kei Tsutsui, 2014. "Efficiency, Equality, and Labeling: An Experimental Investigation of Focal Points in Explicit Bargaining," American Economic Review, American Economic Association, vol. 104(10), pages 3256-3287, October.
    14. Filiz-Ozbay, Emel & Lopez-Vargas, Kristian & Ozbay, Erkut Y., 2015. "Multi-object auctions with resale: Theory and experiment," Games and Economic Behavior, Elsevier, vol. 89(C), pages 1-16.
    15. Charness, Gary & Cobo-Reyes, Ramón & Jiménez, Natalia, 2014. "Identities, selection, and contributions in a public-goods game," Games and Economic Behavior, Elsevier, vol. 87(C), pages 322-338.
    16. Joseph P. Romano, 2004. "On Non‐parametric Testing, the Uniform Behaviour of the t‐test, and Related Problems," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 31(4), pages 567-584, December.
    17. Jessica Cohen & Pascaline Dupas, 2010. "Free Distribution or Cost-Sharing? Evidence from a Randomized Malaria Prevention Experiment," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 125(1), pages 1-45.
    18. Leonard Wantchekon & Marko Klašnja & Natalija Novta, 2015. "Education and Human Capital Externalities: Evidence from Colonial Benin," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 130(2), pages 703-757.
    19. Tsang, Eric W. K., 2014. "Old and New," Management and Organization Review, Cambridge University Press, vol. 10(03), pages 390-390, November.
    20. Bhattacharya, Sourav & Duffy, John & Kim, Sun-Tak, 2014. "Compulsory versus voluntary voting: An experimental study," Games and Economic Behavior, Elsevier, vol. 84(C), pages 111-131.
    21. Cabral, Luis & Ozbay, Erkut Y. & Schotter, Andrew, 2014. "Intrinsic and instrumental reciprocity: An experimental study," Games and Economic Behavior, Elsevier, vol. 87(C), pages 100-121.
    22. Timothy N. Cason & Charles R. Plott, 2014. "Misconceptions and Game Form Recognition: Challenges to Theories of Revealed Preference and Framing," Journal of Political Economy, University of Chicago Press, vol. 122(6), pages 1235-1270.
    23. Luba Petersen & Abel Winn, 2014. "Does Money Illusion Matter? Comment," American Economic Review, American Economic Association, vol. 104(3), pages 1047-1062, March.
    24. Chen, Yan & Li, Sherry Xin & Liu, Tracy Xiao & Shih, Margaret, 2014. "Which hat to wear? Impact of natural identities on coordination and cooperation," Games and Economic Behavior, Elsevier, vol. 84(C), pages 58-86.
    25. Catherine C. Eckel & Sascha C. Füllbrunn, 2015. "Thar SHE Blows? Gender, Competition, and Bubbles in Experimental Asset Markets," American Economic Review, American Economic Association, vol. 105(2), pages 906-920, February.
    26. Michael Kosfeld & Devesh Rustagi, 2015. "Leader Punishment and Cooperation in Groups: Experimental Field Evidence from Commons Management in Ethiopia," American Economic Review, American Economic Association, vol. 105(2), pages 747-783, February.
    27. John Duffy & Daniela Puzzello, 2014. "Gift Exchange versus Monetary Exchange: Theory and Evidence," American Economic Review, American Economic Association, vol. 104(6), pages 1735-1776, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Rizzolli, Matteo & Tremewan, James, 2018. "Hard labor in the lab: Deterrence, non-monetary sanctions, and severe procedures," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 77(C), pages 107-121.
    2. Karl Schlag & James Tremewan, 2021. "Simple belief elicitation: An experimental evaluation," Journal of Risk and Uncertainty, Springer, vol. 62(2), pages 137-155, April.
    3. Montag, Josef & Tremewan, James, 2020. "Let the punishment fit the criminal: An experimental study," Journal of Economic Behavior & Organization, Elsevier, vol. 175(C), pages 423-438.
    4. Eryk Krysowski & James Tremewan, 2021. "Why Does Anonymity Make Us Misbehave: Different Norms Or Less Compliance?," Economic Inquiry, Western Economic Association International, vol. 59(2), pages 776-789, April.
    5. Alcocer, Christian Diego & Jeitschko, Thomas D. & Shupp, Robert, 2020. "Naive and sophisticated mixing: Experimental evidence," Journal of Economic Behavior & Organization, Elsevier, vol. 170(C), pages 157-173.
    6. Francesco De Sinopoli & Giovanna Iannantuoni & Valeria Maggian & Stefania Ottone, 2018. "A Two-Party System under the Proportional Rule is Possible: Strategic Voting in the Lab," Working Papers 381, University of Milano-Bicocca, Department of Economics, revised 16 May 2018.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Karl H.Schlag, 2015. "Who gives Direction to Statistical Testing? Best Practice meets Mathematically Correct Tests," Vienna Economics Papers 1512, University of Vienna, Department of Economics.
    2. Hett, Florian & Kröll, Markus & Mechtel, Mario, 2017. "Choosing Who You Are: The Structure and Behavioral Effects of Revealed Identification Preferences," VfS Annual Conference 2017 (Vienna): Alternative Structures for Money and Banking 168223, Verein für Socialpolitik / German Economic Association.
    3. Florian Hett & Markus Kröll & Mario Mechtel, 2019. "Choosing Who You Are: The Structure and Behavioral Effects of Revealed Identification Preferences," Working Papers 1903, Gutenberg School of Management and Economics, Johannes Gutenberg-Universität Mainz.
    4. Tom Lane, 2023. "The strategic use of social identity," Discussion Papers 2023-01, The Centre for Decision Research and Experimental Economics, School of Economics, University of Nottingham.
    5. Schnellenbach, Jan & Schubert, Christian, 2015. "Behavioral political economy: A survey," European Journal of Political Economy, Elsevier, vol. 40(PB), pages 395-417.
    6. Ciril Bosch-Rosa & Thomas Meissner & Antoni Bosch-Domènech, 2018. "Cognitive bubbles," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 132-153, March.
    7. Drouvelis, Michalis & Malaeb, Bilal & Vlassopoulos, Michael & Wahba, Jackline, 2021. "Cooperation in a fragmented society: Experimental evidence on Syrian refugees and natives in Lebanon," Journal of Economic Behavior & Organization, Elsevier, vol. 187(C), pages 176-191.
    8. Andrea Essl & Frauke von Bieberstein & Michael Kosfeld & Markus Kröll, 2018. "Sales Performance and Social Preferences," CESifo Working Paper Series 7030, CESifo.
    9. Wladislaw Mill & Cornelius Schneider, 2023. "The Bright Side of Tax Evasion," CESifo Working Paper Series 10615, CESifo.
    10. Jetter, Michael & Magnusson, Leandro M. & Roth, Sebastian, 2020. "Becoming sensitive: Males’ risk and time preferences after the 2008 financial crisis," European Economic Review, Elsevier, vol. 128(C).
    11. Engler, Daniel & Groh, Elke D. & Ziegler, Andreas, 2019. "The causal effect of religious and environmental identity on green preferences: A combined priming and stated choice experiment," VfS Annual Conference 2019 (Leipzig): 30 Years after the Fall of the Berlin Wall - Democracy and Market Economy 203610, Verein für Socialpolitik / German Economic Association.
    12. Fuhai Hong & Yohanes E. Riyanto & Ruike Zhang, 2022. "Multidimensional social identity and redistributive preferences: an experimental study," Theory and Decision, Springer, vol. 93(1), pages 151-184, July.
    13. Cariappa, A.G. Adeeth & Chandel, B.S. & Sendhil, R. & Dixit, Anil Kumar & Sankhala, Gopal & Mani, Veena & Meena, B.S., 2022. "Do the prices of a preventive animal health product affect dairy farmers’ willingness to pay and product use? Evidence from an experimental study," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 100(C).
    14. Jonathan Chapman & Mark Dean & Pietro Ortoleva & Erik Snowberg & Colin Camerer, 2018. "Econographics," CESifo Working Paper Series 7202, CESifo.
      • Jonathan Chapman & Mark Dean & Pietro Ortoleva & Erik Snowberg & Colin Camerer, 2018. "Econographics," NBER Working Papers 24931, National Bureau of Economic Research, Inc.
    15. Galbiati, Roberto & Schlag, Karl H. & van der Weele, Joël J., 2013. "Sanctions that signal: An experiment," Journal of Economic Behavior & Organization, Elsevier, vol. 94(C), pages 34-51.
    16. Luhan, Wolfgang J. & Poulsen, Anders U. & Roos, Michael W.M., 2017. "Real-time tacit bargaining, payoff focality, and coordination complexity: Experimental evidence," Games and Economic Behavior, Elsevier, vol. 102(C), pages 687-699.
    17. Dominique Cappelletti & Luigi Mittone & Matteo Ploner, 2015. "Language and intergroup discrimination. Evidence from an experiment," CEEL Working Papers 1504, Cognitive and Experimental Economics Laboratory, Department of Economics, University of Trento, Italia.
    18. Hyndman, Kyle & Müller, Rudolf, 2020. "The role of incentives in dynamic favour exchange: An experimental investigation," Journal of Economic Behavior & Organization, Elsevier, vol. 172(C), pages 83-96.
    19. Heinke, Steve & Olschewski, Sebastian & Rieskamp, Jörg, 2022. "Experiences and Asset Price Dynamics," VfS Annual Conference 2022 (Basel): Big Data in Economics 264017, Verein für Socialpolitik / German Economic Association.
    20. Roberto Galbiati & Karl Schlag & Joël van der Weele, 2009. "Can Sanctions Induce Pessimism? An Experiment," Labsi Experimental Economics Laboratory University of Siena 024, University of Siena.

    More about this item

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C14 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Semiparametric and Nonparametric Methods: General
    • C90 - Mathematical and Quantitative Methods - - Design of Experiments - - - General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:vie:viennp:vie1512. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Paper Administrator (email available below). General contact details of provider: https://econ.univie.ac.at/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.