IDEAS home Printed from https://ideas.repec.org/a/ejw/journl/v19y2022i1p4-29.html
   My bibliography  Save this article

Invalid Methods and False Answers: Physics Education Research and the Use of GREs

Author

Listed:
  • Michael B. Weissman

Abstract

Finding good educational policies requires sound estimates of their potential effects. Methods for making such estimates, i.e., finding causal estimands, have made great progress in the last few decades. Nevertheless, serious errors in causal reasoning have been found previously in papers in a leading physics education journal, Physical Review Physics Education Research. Here I examine three papers from that journal that present explicit methods of causal inference. The methods offered involve major errors, including errors in identifying causal mediation, choosing variables to control for, and imputing missing data. The erroneous methods lead to major underestimation of the predictive power of Graduate Record Exams.

Suggested Citation

  • Michael B. Weissman, 2022. "Invalid Methods and False Answers: Physics Education Research and the Use of GREs," Econ Journal Watch, Econ Journal Watch, vol. 19(1), pages 1-4–29, March.
    • Handle: RePEc:ejw:journl:v:19:y:2022:i:1:p:4-29
    as

    Download full text from publisher

    File URL: https://econjwatch.org/File+download/1226/WeissmanMar2022.pdf?mimetype=pdf
    Download Restriction: no
    File URL: https://econjwatch.org/1267
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Keele, Luke, 2015. "The Statistics of Causal Inference: A View from Political Methodology," Political Analysis, Cambridge University Press, vol. 23(3), pages 313-335, July.
    2. Casey Miller & Keivan Stassun, 2014. "A test that fails," Nature, Nature, vol. 510(7504), pages 303-304, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Linda Sealy & Christina Saunders & Jeffrey Blume & Roger Chalkley, 2019. "The GRE over the entire range of scores lacks predictive ability for PhD outcomes in the biomedical sciences," PLOS ONE, Public Library of Science, vol. 14(3), pages 1-17, March.
    2. Hartwell, Christopher A., 2021. "What Drove the First Response to the COVID-19 Pandemic? The Role of Institutions and Leader Attributes," MPRA Paper 110563, University Library of Munich, Germany.
    3. Thang Dang, 2019. "Quasi-experimental evidence on the political impacts of education in Vietnam," Education Economics, Taylor & Francis Journals, vol. 27(2), pages 207-221, March.
    4. Heijs, Joost & Cruz-Calderón, Selene Cruz, 2023. "A novel research strategy of measuring housing disadvantages of vulnerable populations for all income levels: the Propensity Score Matching approach," MPRA Paper 117212, University Library of Munich, Germany, revised 04 May 2023.
    5. Sandra L Petersen & Evelyn S Erenrich & Dovev L Levine & Jim Vigoreaux & Krista Gile, 2018. "Multi-institutional study of GRE scores as predictors of STEM PhD degree completion: GRE gets a low mark," PLOS ONE, Public Library of Science, vol. 13(10), pages 1-15, October.
    6. Wen, Xiao & Ranjbari, Andisheh & Qi, Fan & Clewlow, Regina R. & MacKenzie, Don, 2021. "Challenges in credibly estimating the travel demand effects of mobility services," Transport Policy, Elsevier, vol. 103(C), pages 224-235.
    7. Fervers, Lukas, 2018. "Can public employment schemes break the negative spiral of long-term unemployment, social exclusion and loss of skills? Evidence from Germany," Journal of Economic Psychology, Elsevier, vol. 67(C), pages 18-33.
    8. Esterling, Kevin M. & Brady, David & Schwitzgebel, Eric, 2023. "The Necessity of Construct and External Validity for Generalized Causal Claims," I4R Discussion Paper Series 18, The Institute for Replication (I4R).
    9. Liane Moneta-Koehler & Abigail M Brown & Kimberly A Petrie & Brent J Evans & Roger Chalkley, 2017. "The Limitations of the GRE in Predicting Success in Biomedical Graduate School," PLOS ONE, Public Library of Science, vol. 12(1), pages 1-17, January.
    10. Cai, Yunhao & Jing, Peng & Wang, Baihui & Jiang, Chengxi & Wang, Yuan, 2023. "How does “over-hype” lead to public misconceptions about autonomous vehicles? A new insight applying causal inference," Transportation Research Part A: Policy and Practice, Elsevier, vol. 175(C).
    11. Ke Zhu & Hanzhong Liu, 2023. "Pair‐switching rerandomization," Biometrics, The International Biometric Society, vol. 79(3), pages 2127-2142, September.
    12. Lukas Fervers, 2016. "Fast track to the labour market or highway to hell? The effect of activation policies on quantity and quality of labour market integration," IAW Discussion Papers 125, Institut für Angewandte Wirtschaftsforschung (IAW).
    13. Glazer Amanda K. & Pimentel Samuel D., 2023. "Robust inference for matching under rolling enrollment," Journal of Causal Inference, De Gruyter, vol. 11(1), pages 1-19, January.
    14. David A. Bateman & Dawn Langan Teele, 2020. "A developmental approach to historical causal inference," Public Choice, Springer, vol. 185(3), pages 253-279, December.
    15. Brathwaite, Timothy & Walker, Joan L., 2018. "Causal inference in travel demand modeling (and the lack thereof)," Journal of choice modelling, Elsevier, vol. 26(C), pages 1-18.
    16. Andrew Hussey & Sheena Murray & Wendy Stock, 2022. "Gender, coauthorship, and academic outcomes in economics," Economic Inquiry, Western Economic Association International, vol. 60(2), pages 465-484, April.
    17. Bodendorf, Frank & Sauter, Maximilian & Franke, Jörg, 2023. "A mixed methods approach to analyze and predict supply disruptions by combining causal inference and deep learning," International Journal of Production Economics, Elsevier, vol. 256(C).
    18. Ye Steven Siwei & Chen Yanzhen & Padilla Oscar Hernan Madrid, 2023. "2D score-based estimation of heterogeneous treatment effects," Journal of Causal Inference, De Gruyter, vol. 11(1), pages 1-25.

    More about this item

    Keywords

    Causal inference; GRE; physics education research; education; statistics;
    All these keywords.

    JEL classification:

    • A2 - General Economics and Teaching - - Economic Education and Teaching of Economics
    • B4 - Schools of Economic Thought and Methodology - - Economic Methodology
    • C1 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General
    • I2 - Health, Education, and Welfare - - Education

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ejw:journl:v:19:y:2022:i:1:p:4-29. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Jason Briggeman (email available below). General contact details of provider: https://edirc.repec.org/data/edgmuus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.