IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0156527.html

Crowdsourcing the Measurement of Interstate Conflict

Author

Listed:
  • Vito D’Orazio
  • Michael Kenwick
  • Matthew Lane
  • Glenn Palmer
  • David Reitter

Abstract

Much of the data used to measure conflict is extracted from news reports. This is typically accomplished using either expert coders to quantify the relevant information or machine coders to automatically extract data from documents. Although expert coding is costly, it produces quality data. Machine coding is fast and inexpensive, but the data are noisy. To diminish the severity of this tradeoff, we introduce a method for analyzing news documents that uses crowdsourcing, supplemented with computational approaches. The new method is tested on documents about Militarized Interstate Disputes, and its accuracy ranges between about 68 and 76 percent. This is shown to be a considerable improvement over automated coding, and to cost less and be much faster than expert coding.

Suggested Citation

  • Vito D’Orazio & Michael Kenwick & Matthew Lane & Glenn Palmer & David Reitter, 2016. "Crowdsourcing the Measurement of Interstate Conflict," PLOS ONE, Public Library of Science, vol. 11(6), pages 1-21, June.
    • Handle: RePEc:plo:pone00:0156527
    DOI: 10.1371/journal.pone.0156527
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0156527
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0156527&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0156527?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Seth Cooper & Firas Khatib & Adrien Treuille & Janos Barbero & Jeehyung Lee & Michael Beenen & Andrew Leaver-Fay & David Baker & Zoran Popović & Foldit players, 2010. "Predicting protein structures with a multiplayer online game," Nature, Nature, vol. 466(7307), pages 756-760, August.
    2. Gabriele Paolacci & Jesse Chandler & Panagiotis G. Ipeirotis, 2010. "Running experiments on Amazon Mechanical Turk," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 5(5), pages 411-419, August.
    3. D'Orazio, Vito & Landis, Steven T. & Palmer, Glenn & Schrodt, Philip, 2014. "Separating the Wheat from the Chaff: Applications of Automated Document Classification Using Support Vector Machines," Political Analysis, Cambridge University Press, vol. 22(2), pages 224-242, April.
    4. King, Gary & Lowe, Will, 2003. "An Automated Information Extraction Tool for International Conflict Data with Performance as Good as Human Coders: A Rare Events Evaluation Design," International Organization, Cambridge University Press, vol. 57(3), pages 617-642, July.
    5. Berinsky, Adam J. & Huber, Gregory A. & Lenz, Gabriel S., 2012. "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk," Political Analysis, Cambridge University Press, vol. 20(3), pages 351-368, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Michael R Kenwick & Beth A Simmons & Richard J McAlexander, 2024. "Infrastructure and authority at the state’s edge: The Border Crossings of the World dataset," Journal of Peace Research, Peace Research Institute Oslo, vol. 61(3), pages 500-510, May.
    2. Glenn Palmer & Roseanne W McManus & Vito D’Orazio & Michael R Kenwick & Mikaela Karstens & Chase Bloch & Nick Dietrich & Kayla Kahn & Kellan Ritter & Michael J Soules, 2022. "The MID5 Dataset, 2011–2014: Procedures, coding rules, and description," Conflict Management and Peace Science, Peace Science Society (International), vol. 39(4), pages 470-482, July.
    3. Zhanna Terechshenko, 2020. "Hot under the collar: A latent measure of interstate hostility," Journal of Peace Research, Peace Research Institute Oslo, vol. 57(6), pages 764-776, November.
    4. Juan D Botero & Weisi Guo & Guillem Mosquera & Alan Wilson & Samuel Johnson & Gicela A Aguirre-Garcia & Leonardo A Pachon, 2019. "Gang confrontation: The case of Medellin (Colombia)," PLOS ONE, Public Library of Science, vol. 14(12), pages 1-19, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Masha Shunko & Julie Niederhoff & Yaroslav Rosokha, 2018. "Humans Are Not Machines: The Behavioral Impact of Queueing Design on Service Time," Management Science, INFORMS, vol. 64(1), pages 453-473, January.
    2. Haas, Nicholas & Hassan, Mazen & Mansour, Sarah & Morton, Rebecca B., 2021. "Polarizing information and support for reform," Journal of Economic Behavior & Organization, Elsevier, vol. 185(C), pages 883-901.
    3. Cantarella, Michele & Strozzi, Chiara, 2019. "Workers in the Crowd: The Labour Market Impact of the Online Platform Economy," IZA Discussion Papers 12327, Institute of Labor Economics (IZA).
    4. O. Ashton Morgan & John C. Whitehead, 2018. "Willingness to Pay for Soccer Player Development in the United States," Journal of Sports Economics, , vol. 19(2), pages 279-296, February.
    5. Azzam, Tarek & Harman, Elena, 2016. "Crowdsourcing for quantifying transcripts: An exploratory study," Evaluation and Program Planning, Elsevier, vol. 54(C), pages 63-73.
    6. Wladislaw Mill & Cornelius Schneider, 2023. "The Bright Side of Tax Evasion," CESifo Working Paper Series 10615, CESifo.
    7. Gandullia, Luca & Lezzi, Emanuela & Parciasepe, Paolo, 2020. "Replication with MTurk of the experimental design by Gangadharan, Grossman, Jones & Leister (2018): Charitable giving across donor types," Journal of Economic Psychology, Elsevier, vol. 78(C).
    8. Prissé, Benjamin & Jorrat, Diego, 2022. "Lab vs online experiments: No differences," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 100(C).
    9. Min Chung Han, 2021. "Thumbs down on “likes”? The impact of Facebook reactions on online consumers’ nonprofit engagement behavior," International Review on Public and Nonprofit Marketing, Springer;International Association of Public and Non-Profit Marketing, vol. 18(2), pages 255-272, June.
    10. Valerio Capraro & Hélène Barcelo, 2021. "Punishing defectors and rewarding cooperators: Do people discriminate between genders?," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 7(1), pages 19-32, September.
    11. Kaitlynn Sandstrom‐Mistry & Frank Lupi & Hyunjung Kim & Joseph A. Herriges, 2023. "Comparing water quality valuation across probability and non‐probability samples," Applied Economic Perspectives and Policy, John Wiley & Sons, vol. 45(2), pages 744-761, June.
    12. Lefgren, Lars J. & Sims, David P. & Stoddard, Olga B., 2016. "Effort, luck, and voting for redistribution," Journal of Public Economics, Elsevier, vol. 143(C), pages 89-97.
    13. Bhatt, Vipul & Smith, Angela M., 2025. "Overconfidence and performance: Evidence from a simple real-effort task," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 114(C).
    14. Samir Mamadehussene & Francesco Sguera, 2023. "On the Reliability of the BDM Mechanism," Management Science, INFORMS, vol. 69(2), pages 1166-1179, February.
    15. Tim Straub & Henner Gimpel & Florian Teschner & Christof Weinhardt, 2015. "How (not) to Incent Crowd Workers," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 57(3), pages 167-179, June.
    16. Nicolas Jacquemet & Alexander G James & Stéphane Luchini & James J Murphy & Jason F Shogren, 2021. "Do truth-telling oaths improve honesty in crowd-working?," PLOS ONE, Public Library of Science, vol. 16(1), pages 1-18, January.
    17. Hiroki Ozono & Daisuke Nakama, 2022. "Effects of experimental situation on group cooperation and individual performance: Comparing laboratory and online experiments," PLOS ONE, Public Library of Science, vol. 17(4), pages 1-17, April.
    18. Brodeur, Abel & Cook, Nikolai & Heyes, Anthony, 2022. "We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell us about p-Hacking and Publication Bias in Online Experiments," I4R Discussion Paper Series 8, The Institute for Replication (I4R).
    19. Carpiano, Richard M. & Fitz, Nicholas S., 2017. "Public attitudes toward child undervaccination: A randomized experiment on evaluations, stigmatizing orientations, and support for policies," Social Science & Medicine, Elsevier, vol. 185(C), pages 127-136.
    20. Chandler, Dana & Kapelner, Adam, 2013. "Breaking monotony with meaning: Motivation in crowdsourcing markets," Journal of Economic Behavior & Organization, Elsevier, vol. 90(C), pages 123-133.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0156527. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.