IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0156527.html

Crowdsourcing the Measurement of Interstate Conflict

Author

Listed:
  • Vito D’Orazio
  • Michael Kenwick
  • Matthew Lane
  • Glenn Palmer
  • David Reitter

Abstract

Much of the data used to measure conflict is extracted from news reports. This is typically accomplished using either expert coders to quantify the relevant information or machine coders to automatically extract data from documents. Although expert coding is costly, it produces quality data. Machine coding is fast and inexpensive, but the data are noisy. To diminish the severity of this tradeoff, we introduce a method for analyzing news documents that uses crowdsourcing, supplemented with computational approaches. The new method is tested on documents about Militarized Interstate Disputes, and its accuracy ranges between about 68 and 76 percent. This is shown to be a considerable improvement over automated coding, and to cost less and be much faster than expert coding.

Suggested Citation

  • Vito D’Orazio & Michael Kenwick & Matthew Lane & Glenn Palmer & David Reitter, 2016. "Crowdsourcing the Measurement of Interstate Conflict," PLOS ONE, Public Library of Science, vol. 11(6), pages 1-21, June.
    • Handle: RePEc:plo:pone00:0156527
    DOI: 10.1371/journal.pone.0156527
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0156527
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0156527&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0156527?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Seth Cooper & Firas Khatib & Adrien Treuille & Janos Barbero & Jeehyung Lee & Michael Beenen & Andrew Leaver-Fay & David Baker & Zoran Popović & Foldit players, 2010. "Predicting protein structures with a multiplayer online game," Nature, Nature, vol. 466(7307), pages 756-760, August.
    2. Gabriele Paolacci & Jesse Chandler & Panagiotis G. Ipeirotis, 2010. "Running experiments on Amazon Mechanical Turk," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 5(5), pages 411-419, August.
    3. D'Orazio, Vito & Landis, Steven T. & Palmer, Glenn & Schrodt, Philip, 2014. "Separating the Wheat from the Chaff: Applications of Automated Document Classification Using Support Vector Machines," Political Analysis, Cambridge University Press, vol. 22(2), pages 224-242, April.
    4. King, Gary & Lowe, Will, 2003. "An Automated Information Extraction Tool for International Conflict Data with Performance as Good as Human Coders: A Rare Events Evaluation Design," International Organization, Cambridge University Press, vol. 57(3), pages 617-642, July.
    5. Berinsky, Adam J. & Huber, Gregory A. & Lenz, Gabriel S., 2012. "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk," Political Analysis, Cambridge University Press, vol. 20(3), pages 351-368, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Michael R Kenwick & Beth A Simmons & Richard J McAlexander, 2024. "Infrastructure and authority at the state’s edge: The Border Crossings of the World dataset," Journal of Peace Research, Peace Research Institute Oslo, vol. 61(3), pages 500-510, May.
    2. Glenn Palmer & Roseanne W McManus & Vito D’Orazio & Michael R Kenwick & Mikaela Karstens & Chase Bloch & Nick Dietrich & Kayla Kahn & Kellan Ritter & Michael J Soules, 2022. "The MID5 Dataset, 2011–2014: Procedures, coding rules, and description," Conflict Management and Peace Science, Peace Science Society (International), vol. 39(4), pages 470-482, July.
    3. Zhanna Terechshenko, 2020. "Hot under the collar: A latent measure of interstate hostility," Journal of Peace Research, Peace Research Institute Oslo, vol. 57(6), pages 764-776, November.
    4. Juan D Botero & Weisi Guo & Guillem Mosquera & Alan Wilson & Samuel Johnson & Gicela A Aguirre-Garcia & Leonardo A Pachon, 2019. "Gang confrontation: The case of Medellin (Colombia)," PLOS ONE, Public Library of Science, vol. 14(12), pages 1-19, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Cantarella, Michele & Strozzi, Chiara, 2019. "Workers in the Crowd: The Labour Market Impact of the Online Platform Economy," IZA Discussion Papers 12327, IZA Network @ LISER.
    2. Azzam, Tarek & Harman, Elena, 2016. "Crowdsourcing for quantifying transcripts: An exploratory study," Evaluation and Program Planning, Elsevier, vol. 54(C), pages 63-73.
    3. Gandullia, Luca & Lezzi, Emanuela & Parciasepe, Paolo, 2020. "Replication with MTurk of the experimental design by Gangadharan, Grossman, Jones & Leister (2018): Charitable giving across donor types," Journal of Economic Psychology, Elsevier, vol. 78(C).
    4. Prissé, Benjamin & Jorrat, Diego, 2022. "Lab vs online experiments: No differences," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 100(C).
    5. Min Chung Han, 2021. "Thumbs down on “likes”? The impact of Facebook reactions on online consumers’ nonprofit engagement behavior," International Review on Public and Nonprofit Marketing, Springer;International Association of Public and Non-Profit Marketing, vol. 18(2), pages 255-272, June.
    6. Valerio Capraro & Hélène Barcelo, 2021. "Punishing defectors and rewarding cooperators: Do people discriminate between genders?," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 7(1), pages 19-32, September.
    7. Olenka Kacperczyk & Peter Younkin & Vera Rocha, 2023. "Do Employees Work Less for Female Leaders? A Multi-Method Study of Entrepreneurial Firms," Organization Science, INFORMS, vol. 34(3), pages 1111-1133, May.
    8. Farooq, Ali & Salminen, Joni & Dahabiyeh, Laila & Javed, Yousra & Jansen, Bernard J., 2026. "Does negative buzz result in social media discontinuation? Investigating the effects of negative word of mouth in the United States, India, and Finland," Technology in Society, Elsevier, vol. 84(C).
    9. Lefgren, Lars J. & Sims, David P. & Stoddard, Olga B., 2016. "Effort, luck, and voting for redistribution," Journal of Public Economics, Elsevier, vol. 143(C), pages 89-97.
    10. Aldijana Bunjak & Olivier Zehnder, 2026. "Quantifying the gig economy’s evolution: A bibliometric exploration of historical patterns and future projections," Electronic Markets, Springer;IIM University of St. Gallen, vol. 36(1), pages 1-26, December.
    11. Samir Mamadehussene & Francesco Sguera, 2023. "On the Reliability of the BDM Mechanism," Management Science, INFORMS, vol. 69(2), pages 1166-1179, February.
    12. Tim Straub & Henner Gimpel & Florian Teschner & Christof Weinhardt, 2015. "How (not) to Incent Crowd Workers," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 57(3), pages 167-179, June.
    13. Carpiano, Richard M. & Fitz, Nicholas S., 2017. "Public attitudes toward child undervaccination: A randomized experiment on evaluations, stigmatizing orientations, and support for policies," Social Science & Medicine, Elsevier, vol. 185(C), pages 127-136.
    14. Chandler, Dana & Kapelner, Adam, 2013. "Breaking monotony with meaning: Motivation in crowdsourcing markets," Journal of Economic Behavior & Organization, Elsevier, vol. 90(C), pages 123-133.
    15. Irene Scopelliti & Carey K. Morewedge & Erin McCormick & H. Lauren Min & Sophie Lebrecht & Karim S. Kassam, 2015. "Bias Blind Spot: Structure, Measurement, and Consequences," Management Science, INFORMS, vol. 61(10), pages 2468-2486, October.
    16. Thomas W. H. Ng & Lorenzo Lucianetti & Dennis Y. Hsu & Frederick H. K. Yim & Kelly L. Sorensen, 2021. "You Speak, I Speak: The Social‐Cognitive Mechanisms of Voice Contagion," Journal of Management Studies, Wiley Blackwell, vol. 58(6), pages 1569-1608, September.
    17. Mahmood, Ammara & Luffarelli, Jonathan & Mukesh, Mudra, 2019. "What's in a logo? The impact of complex visual cues in equity crowdfunding," Journal of Business Venturing, Elsevier, vol. 34(1), pages 41-62.
    18. Schwaiger, Rene & Hueber, Laura, 2021. "Do MTurkers exhibit myopic loss aversion?," Economics Letters, Elsevier, vol. 209(C).
    19. Harman, Elena & Azzam, Tarek, 2018. "Incorporating public values into evaluative criteria: Using crowdsourcing to identify criteria and standards," Evaluation and Program Planning, Elsevier, vol. 71(C), pages 68-82.
    20. Jasmijn Bol & Lisa Laviers & Jason Sandvik, 2023. "Creativity Contests: An Experimental Investigation of Eliciting Employee Creativity," Journal of Accounting Research, John Wiley & Sons, Ltd., vol. 61(1), pages 47-94, March.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0156527. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.