IDEAS home Printed from https://ideas.repec.org/a/wly/empleg/v17y2020i2p224-261.html
   My bibliography  Save this article

The Problem of Data Bias in the Pool of Published U.S. Appellate Court Opinions

Author

Listed:
  • Keith Carlson
  • Michael A. Livermore
  • Daniel N. Rockmore

Abstract

For decades, researchers have studied the relationship between the political leanings of judges and the outcomes of appellate litigation in the United States. The primary source of data for this research has been published judicial opinions that describe cases and their outcomes. However, only a relatively small number of cases result in published opinions, and this sample of cases may be subject to serious biases. Based on computational text analysis of over 150,000 published opinions issued by federal appellate courts in the years 1970–2010, we find strong evidence of data bias based on relationships between the party affiliations of judges on appellate court panels and the characteristics of cases that result in published opinions. These relationships imply that the inferential model that underlies much of the judicial politics literature can lead to biased or spurious findings concerning the causal influence of judicial attributes on case outcomes.

Suggested Citation

  • Keith Carlson & Michael A. Livermore & Daniel N. Rockmore, 2020. "The Problem of Data Bias in the Pool of Published U.S. Appellate Court Opinions," Journal of Empirical Legal Studies, John Wiley & Sons, vol. 17(2), pages 224-261, June.
    • Handle: RePEc:wly:empleg:v:17:y:2020:i:2:p:224-261
    DOI: 10.1111/jels.12253
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/jels.12253
    Download Restriction: no
    File URL: https://libkey.io/10.1111/jels.12253?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Grimmer, Justin & Stewart, Brandon M., 2013. "Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts," Political Analysis, Cambridge University Press, vol. 21(3), pages 267-297, July.
    2. Margaret Roberts & Brandon Stewart & Tingley, Dustin & Edoardo Airoldi, 2013. "The structural topic model and applied social science," Working Paper 132666, Harvard University OpenScholar.
    3. Kornhauser, Lewis A., 1992. "Modeling collegial courts I: Path-dependence," International Review of Law and Economics, Elsevier, vol. 12(2), pages 169-185, June.
    4. Joshua B. Fischman, 2014. "Measuring Inconsistency, Indeterminacy, and Error in Adjudication," American Law and Economics Review, American Law and Economics Association, vol. 16(1), pages 40-85.
    5. Deborah Beim & Alexander V. Hirsch & Jonathan P. Kastellec, 2016. "Signaling and Counter‐Signaling in the Judicial Hierarchy: An Empirical Analysis of En Banc Review," American Journal of Political Science, John Wiley & Sons, vol. 60(2), pages 490-508, April.
    6. Erin B. Kaheny & Susan Brodie Haire & Sara C. Benesh, 2008. "Change over Tenure: Voting, Variance, and Decision Making on the U.S. Courts of Appeals," American Journal of Political Science, John Wiley & Sons, vol. 52(3), pages 490-503, July.
    7. Christina L. Boyd & Lee Epstein & Andrew D. Martin, 2010. "Untangling the Causal Effects of Sex on Judging," American Journal of Political Science, John Wiley & Sons, vol. 54(2), pages 389-411, April.
    8. Benjamin E. Lauderdale & Tom S. Clark, 2014. "Scaling Politically Meaningful Dimensions Using Texts and Votes," American Journal of Political Science, John Wiley & Sons, vol. 58(3), pages 754-771, July.
    9. Joshua B. Fischman, 2015. "Interpreting Circuit Court Voting Patterns: A Social Interactions Framework," The Journal of Law, Economics, and Organization, Oxford University Press, vol. 31(4), pages 808-842.
    10. Paul Brace & Brent D. Boyea, 2008. "State Public Opinion, the Death Penalty, and the Practice of Electing Judges," American Journal of Political Science, John Wiley & Sons, vol. 52(2), pages 360-372, April.
    11. Matthew Spitzer & Eric Talley, 2013. "Left, Right, and Center: Strategic Information Acquisition and Diversity in Judicial Panels," The Journal of Law, Economics, and Organization, Oxford University Press, vol. 29(3), pages 638-680, June.
    12. Goldman, Sheldon, 1975. "Voting Behavior on the United States Courts of Appeals Revisited," American Political Science Review, Cambridge University Press, vol. 69(2), pages 491-506, June.
    13. Margaret E. Roberts & Brandon M. Stewart & Dustin Tingley & Christopher Lucas & Jetson Leder‐Luis & Shana Kushner Gadarian & Bethany Albertson & David G. Rand, 2014. "Structural Topic Models for Open‐Ended Survey Responses," American Journal of Political Science, John Wiley & Sons, vol. 58(4), pages 1064-1082, October.
    14. Sean Farhang & Jonathan P. Kastellec & Gregory J. Wawro, 2015. "The Politics of Opinion Assignment and Authorship on the US Court of Appeals: Evidence from Sexual Harassment Cases," The Journal of Legal Studies, University of Chicago Press, vol. 44(S1), pages 59-85.
    15. Kornhauser, Lewis A, 1992. "Modeling Collegial Courts. II. Legal Doctrine," The Journal of Law, Economics, and Organization, Oxford University Press, vol. 8(3), pages 441-470, October.
    16. Joshua B. Fischman, 2011. "Estimating Preferences of Circuit Judges: A Model of Consensus Voting," Journal of Law and Economics, University of Chicago Press, vol. 54(4), pages 781-809.
    17. Kevin M. Quinn & Burt L. Monroe & Michael Colaresi & Michael H. Crespin & Dragomir R. Radev, 2010. "How to Analyze Political Attention with Minimal Assumptions and Costs," American Journal of Political Science, John Wiley & Sons, vol. 54(1), pages 209-228, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Greg Goelzhauser, 2024. "Constitutional accountability for police shootings," Journal of Empirical Legal Studies, John Wiley & Sons, vol. 21(1), pages 92-108, March.
    2. David A. Hyman & Jing Liu & Bernard S. Black, 2022. "Should patients use online reviews to pick their doctors and hospitals?," Journal of Empirical Legal Studies, John Wiley & Sons, vol. 19(4), pages 897-935, December.
    3. Christina L. Boyd & Pauline T. Kim & Margo Schlanger, 2020. "Mapping the Iceberg: The Impact of Data Sources on the Study of District Courts," Journal of Empirical Legal Studies, John Wiley & Sons, vol. 17(3), pages 466-492, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Charles M. Cameron & Lewis A. Kornhauser, 2017. "Rational choice attitudinalism?," European Journal of Law and Economics, Springer, vol. 43(3), pages 535-554, June.
    2. Bernardo Guimaraes & Bruno Meyerhof Salama, 2023. "Permitting Prohibitions," The Journal of Legal Studies, University of Chicago Press, vol. 52(1), pages 241-271.
    3. Guimaraesy, Bernardo & Meyerhof Salama, Bruno, 2017. "Contingent judicial deference: theory and application to usury laws," LSE Research Online Documents on Economics 86146, London School of Economics and Political Science, LSE Library.
    4. Guimarães, Bernardo de Vasconcellos & Salama, Bruno Meyerhof, 2017. "Contingent judicial deference: theory and application to usury laws," Textos para discussão 440, FGV EESP - Escola de Economia de São Paulo, Fundação Getulio Vargas (Brazil).
    5. Christoph Engel, 2021. "Lucky You: Your Case is Heard by a Seasoned Panel – Panel Effects in the German Constitutional Court," Discussion Paper Series of the Max Planck Institute for Research on Collective Goods 2021_05, Max Planck Institute for Research on Collective Goods, revised 01 Jun 2022.
    6. Dehler-Holland, Joris & Schumacher, Kira & Fichtner, Wolf, 2021. "Topic Modeling Uncovers Shifts in Media Framing of the German Renewable Energy Act," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 2(1).
    7. Xiaohong Yu & Zhaoyang Sun, 2022. "The company they keep: When and why Chinese judges engage in collegiality," Journal of Empirical Legal Studies, John Wiley & Sons, vol. 19(4), pages 936-1002, December.
    8. Shamena Anwar & Patrick Bayer & Randi Hjalmarsson, 2019. "Politics in the Courtroom: Political Ideology and Jury Decision Making," Journal of the European Economic Association, European Economic Association, vol. 17(3), pages 834-875.
    9. Mohamed M. Mostafa, 2023. "A one-hundred-year structural topic modeling analysis of the knowledge structure of international management research," Quality & Quantity: International Journal of Methodology, Springer, vol. 57(4), pages 3905-3935, August.
    10. Zhang, Han, 2021. "How Using Machine Learning Classification as a Variable in Regression Leads to Attenuation Bias and What to Do About It," SocArXiv 453jk, Center for Open Science.
    11. Dehler-Holland, Joris & Okoh, Marvin & Keles, Dogan, 2022. "Assessing technology legitimacy with topic models and sentiment analysis – The case of wind power in Germany," Technological Forecasting and Social Change, Elsevier, vol. 175(C).
    12. David Ardia & Keven Bluteau & Mohammad Abbas Meghani, 2021. "Thirty Years of Academic Finance," Papers 2112.14902, arXiv.org, revised Aug 2022.
    13. Karell, Daniel & Freedman, Michael Raphael, 2019. "Rhetorics of Radicalism," SocArXiv yfzsh, Center for Open Science.
    14. Matthew Gentzkow & Bryan T. Kelly & Matt Taddy, 2017. "Text as Data," NBER Working Papers 23276, National Bureau of Economic Research, Inc.
    15. Mourtgos, Scott M. & Adams, Ian T., 2019. "The rhetoric of de-policing: Evaluating open-ended survey responses from police officers with machine learning-based structural topic modeling," Journal of Criminal Justice, Elsevier, vol. 64(C), pages 1-1.
    16. Michal Ovádek & Nicolas Lampach & Arthur Dyevre, 2020. "What’s the talk in Brussels? Leveraging daily news coverage to measure issue attention in the European Union," European Union Politics, , vol. 21(2), pages 204-232, June.
    17. Sanders, James & Lisi, Giulio & Schonhardt-Bailey, Cheryl, 2018. "Themes and topics in parliamentary oversight hearings: a new direction in textual data analysis," LSE Research Online Documents on Economics 87624, London School of Economics and Political Science, LSE Library.
    18. Sandra Wankmüller, 2023. "A comparison of approaches for imbalanced classification problems in the context of retrieving relevant documents for an analysis," Journal of Computational Social Science, Springer, vol. 6(1), pages 91-163, April.
    19. List, Christian & Polak, Ben, 2010. "Introduction to judgment aggregation," Journal of Economic Theory, Elsevier, vol. 145(2), pages 441-466, March.
    20. Grajzl, Peter & Murrell, Peter, 2021. "A machine-learning history of English caselaw and legal ideas prior to the Industrial Revolution I: generating and interpreting the estimates," Journal of Institutional Economics, Cambridge University Press, vol. 17(1), pages 1-19, February.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:empleg:v:17:y:2020:i:2:p:224-261. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://doi.org/10.1111/(ISSN)1740-1461 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.