IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2409.10750.html
   My bibliography  Save this paper

GPT takes the SAT: Tracing changes in Test Difficulty and Math Performance of Students

Author

Listed:
  • Vikram Krishnaveti
  • Saannidhya Rawat

Abstract

Scholastic Aptitude Test (SAT) is crucial for college admissions but its effectiveness and relevance are increasingly questioned. This paper enhances Synthetic Control methods by introducing "Transformed Control", a novel method that employs Large Language Models (LLMs) powered by Artificial Intelligence to generate control groups. We utilize OpenAI's API to generate a control group where GPT-4, or ChatGPT, takes multiple SATs annually from 2008 to 2023. This control group helps analyze shifts in SAT math difficulty over time, starting from the baseline year of 2008. Using parallel trends, we calculate the Average Difference in Scores (ADS) to assess changes in high school students' math performance. Our results indicate a significant decrease in the difficulty of the SAT math section over time, alongside a decline in students' math performance. The analysis shows a 71-point drop in the rigor of SAT math from 2008 to 2023, with student performance decreasing by 36 points, resulting in a 107-point total divergence in average student math performance. We investigate possible mechanisms for this decline in math proficiency, such as changing university selection criteria, increased screen time, grade inflation, and worsening adolescent mental health. Disparities among demographic groups show a 104-point drop for White students, 84 points for Black students, and 53 points for Asian students. Male students saw a 117-point reduction, while female students had a 100-point decrease.

Suggested Citation

  • Vikram Krishnaveti & Saannidhya Rawat, 2024. "GPT takes the SAT: Tracing changes in Test Difficulty and Math Performance of Students," Papers 2409.10750, arXiv.org.
    • Handle: RePEc:arx:papers:2409.10750
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2409.10750
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Anton Korinek, 2023. "Generative AI for Economic Research: Use Cases and Implications for Economists," Journal of Economic Literature, American Economic Association, vol. 61(4), pages 1281-1317, December.
    2. Caroline M. Hoxby, 2009. "The Changing Selectivity of American Colleges," Journal of Economic Perspectives, American Economic Association, vol. 23(4), pages 95-118, Fall.
    3. Jack Mountjoy & Brent Hickman, 2020. "The Returns to College(s): Estimating Value-Added and Match Effects in Higher Education," Working Papers 2020-08, Becker Friedman Institute for Research In Economics.
    4. Raj Chetty & David J. Deming & John N. Friedman, 2023. "Diversifying Society’s Leaders? The Determinants and Causal Effects of Admission to Highly Selective Private Colleges," NBER Working Papers 31492, National Bureau of Economic Research, Inc.
    5. Alberto Abadie & Javier Gardeazabal, 2003. "The Economic Costs of Conflict: A Case Study of the Basque Country," American Economic Review, American Economic Association, vol. 93(1), pages 113-132, March.
    6. Mark Hoekstra, 2009. "The Effect of Attending the Flagship State University on Earnings: A Discontinuity-Based Approach," The Review of Economics and Statistics, MIT Press, vol. 91(4), pages 717-724, November.
    7. Saboe, Matt & Terrizzi, Sabrina, 2019. "SAT optional policies: Do they influence graduate quality, selectivity or diversity?," Economics Letters, Elsevier, vol. 174(C), pages 13-17.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sandra E. Black & Jeffrey T. Denning & Jesse Rothstein, 2023. "Winners and Losers? The Effect of Gaining and Losing Access to Selective Colleges on Education and Labor Market Outcomes," American Economic Journal: Applied Economics, American Economic Association, vol. 15(1), pages 26-67, January.
    2. Bacalhau, Priscilla & Mattos, Enlinson & Ponczek, Vladimir Pinheiro, 2019. "College quality signaling and individual performance: effects on labor market outcomes after graduation," Textos para discussão 502, FGV EESP - Escola de Economia de São Paulo, Fundação Getulio Vargas (Brazil).
    3. Stacy Dale & Alan B. Krueger, "undated". "Estimating the Return to College Selectivity Over the Career Using Administrative Earning Data," Mathematica Policy Research Reports d76ec29a0bbb4b1bb9d285b5a, Mathematica Policy Research.
    4. Molly Schnell & Janet Currie, 2018. "Addressing the Opioid Epidemic: Is There a Role for Physician Education?," American Journal of Health Economics, MIT Press, vol. 4(3), pages 383-410, Summer.
    5. Qiong Zhu & Junghee Choi & Yi Meng, 2021. "The Impact of No-Loan Policies on Student Economic Diversity at Public Colleges and Universities," Research in Higher Education, Springer;Association for Institutional Research, vol. 62(6), pages 733-764, September.
    6. Suqin Ge & Elliott Isaac & Amalia Miller, 2022. "Elite Schools and Opting In: Effects of College Selectivity on Career and Family Outcomes," Journal of Labor Economics, University of Chicago Press, vol. 40(S1), pages 383-427.
    7. Ashby, Nathan J., 2023. "An examination of peer effects using high school competition realignments," Journal of Economic Behavior & Organization, Elsevier, vol. 206(C), pages 122-135.
    8. Rodney J. Andrews & Kevin M. Stange, 2019. "Price Regulation, Price Discrimination, and Equality of Opportunity in Higher Education: Evidence from Texas," American Economic Journal: Economic Policy, American Economic Association, vol. 11(4), pages 31-65, November.
    9. Amanda L. Griffith & Kevin N. Rask, 2016. "The Effect Of Institutional Expenditures On Employment Outcomes And Earnings," Economic Inquiry, Western Economic Association International, vol. 54(4), pages 1931-1945, October.
    10. Peter Arcidiacono & Michael Lovenheim, 2016. "Affirmative Action and the Quality-Fit Trade-Off," Journal of Economic Literature, American Economic Association, vol. 54(1), pages 3-51, March.
    11. Sarena Goodman, 2013. "Learning from the test: raising selective college enrollment by providing information," Finance and Economics Discussion Series 2013-69, Board of Governors of the Federal Reserve System (U.S.).
    12. Armona, Luis & Chakrabarti, Rajashri & Lovenheim, Michael F., 2022. "Student debt and default: The role of for-profit colleges," Journal of Financial Economics, Elsevier, vol. 144(1), pages 67-92.
    13. Nguyen, Hieu, 2020. "Free college? Assessing enrollment responses to the Tennessee Promise program," Labour Economics, Elsevier, vol. 66(C).
    14. Cecilia Machado & Germ'an Reyes & Evan Riehl, 2023. "The Direct and Spillover Effects of Large-scale Affirmative Action at an Elite Brazilian University," Papers 2305.02513, arXiv.org, revised Jun 2023.
    15. Eleanor Wiske Dillon & Jeffrey Andrew Smith, 2020. "The Consequences of Academic Match between Students and Colleges," Journal of Human Resources, University of Wisconsin Press, vol. 55(3), pages 767-808.
    16. Josh Kinsler & Ronni Pavan, 2011. "Family Income and Higher Education Choices: The Importance of Accounting for College Quality," Journal of Human Capital, University of Chicago Press, vol. 5(4), pages 453-477.
    17. W. Bentley MacLeod & Evan Riehl & Juan E. Saavedra & Miguel Urquiola, 2017. "The Big Sort: College Reputation and Labor Market Outcomes," American Economic Journal: Applied Economics, American Economic Association, vol. 9(3), pages 223-261, July.
    18. Jia, Ruixue & Li, Hongbin, 2021. "Just above the exam cutoff score: Elite college admission and wages in China," Journal of Public Economics, Elsevier, vol. 196(C).
    19. Clingingsmith, David & Sheremeta, Roman M, 2017. "Status and Economic Rent: Experimental Evidence on the Matthew Effect," SocArXiv evwpa, Center for Open Science.
    20. Caroline M. Hoxby, 2018. "The Productivity of US Postsecondary Institutions," NBER Chapters, in: Productivity in Higher Education, pages 31-66, National Bureau of Economic Research, Inc.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2409.10750. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.