IDEAS home Printed from https://ideas.repec.org/a/inm/oropre/v51y2003i2p240-254.html
   My bibliography  Save this article

Record Matching in Data Warehouses: A Decision Model for Data Consolidation

Author

Listed:
  • Debabrata Dey

    (University of Washington Business School, Seattle, Washington 98195--3200)

Abstract

The notion of a data warehouse for integrating operational data into a single repository is rapidly becoming popular in modern organizations. An important issue in the integration process is how to deal with the identifier mismatch problem when combining similar data from disparate sources. A real-world entity may be represented using different identifiers in different operational data sources, and matching them may often be difficult using simple database operations expressed, say, as an SQL query. A record-by-record manual matching is also not practical because the databases may be large. A decision model is presented that combines probability-based automated matching with manual matching in a cost minimization formulation. A heuristic approach is proposed for solving the decision model. Both the model and the heuristic solution approach have been tested on real data. The results from the testing indicate that the model can be effectively used in real-world situations.

Suggested Citation

  • Debabrata Dey, 2003. "Record Matching in Data Warehouses: A Decision Model for Data Consolidation," Operations Research, INFORMS, vol. 51(2), pages 240-254, April.
    • Handle: RePEc:inm:oropre:v:51:y:2003:i:2:p:240-254
    DOI: 10.1287/opre.51.2.240.12779
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/opre.51.2.240.12779
    Download Restriction: no
    File URL: https://libkey.io/10.1287/opre.51.2.240.12779?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. J. B. Copas & F. J. Hilton, 1990. "Record Linkage: Statistical Models for Matching Computer Records," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 153(3), pages 287-312, May.
    2. Debabrata Dey & Sumit Sarkar & Prabuddha De, 1998. "A Probabilistic Decision Model for Entity Matching in Heterogeneous Databases," Management Science, INFORMS, vol. 44(10), pages 1379-1395, October.
    3. Larsen M. D & Rubin D. B, 2001. "Iterative Automated Record Linkage Using Mixture Models," Journal of the American Statistical Association, American Statistical Association, vol. 96, pages 32-41, March.
    4. Debabrata Dey & Sumit Sarkar, 2000. "Modifications of Uncertain Data: A Bayesian Framework for Belief Revision," Information Systems Research, INFORMS, vol. 11(1), pages 1-16, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Kartik Hosanagar, 2011. "Usercentric Operational Decision Making in Distributed Information Retrieval," Information Systems Research, INFORMS, vol. 22(4), pages 739-755, December.
    2. Michael Scholz & Markus Franz & Oliver Hinz, 2016. "The Ambiguous Identifier Clustering Technique," Electronic Markets, Springer;IIM University of St. Gallen, vol. 26(2), pages 143-156, May.
    3. Shaobo Li & Matthew J. Schneider & Yan Yu & Sachin Gupta, 2023. "Reidentification Risk in Panel Data: Protecting for k -Anonymity," Information Systems Research, INFORMS, vol. 34(3), pages 1066-1088, September.
    4. Debabrata Dey & Subodha Kumar, 2013. "Data Quality of Query Results with Generalized Selection Conditions," Operations Research, INFORMS, vol. 61(1), pages 17-31, February.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Francesco D. d’Ovidio & Paola Perchinunno & Laura Antonucci, 2021. "Data Integration Techniques for the Identification of Poverty Profiles," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 156(2), pages 515-531, August.
    2. Kunpeng Zhang & Wendy Moe, 2021. "Measuring Brand Favorability Using Large-Scale Social Media Data," Information Systems Research, INFORMS, vol. 32(4), pages 1128-1139, December.
    3. Josef Schürle, 2005. "A method for consideration of conditional dependencies in the Fellegi and Sunter model of record linkage," Statistical Papers, Springer, vol. 46(3), pages 433-449, July.
    4. Lee, Gyumin & Lee, Sungjun & Lee, Changyong, 2023. "Inventor–licensee matchmaking for university technology licensing: A fastText approach," Technovation, Elsevier, vol. 125(C).
    5. Afshin Fallah & Mohsen Mohammadzadeh, 2010. "Bayesian regression analysis with linked data using mixture normal distributions," Statistical Papers, Springer, vol. 51(2), pages 421-430, June.
    6. Thomas Stringham, 2022. "Fast Bayesian Record Linkage With Record-Specific Disagreement Parameters," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 40(4), pages 1509-1522, October.
    7. Vo, Thanh Huan & Chauvet, Guillaume & Happe, André & Oger, Emmanuel & Paquelet, Stéphane & Garès, Valérie, 2023. "Extending the Fellegi-Sunter record linkage model for mixed-type data with application to the French national health data system," Computational Statistics & Data Analysis, Elsevier, vol. 179(C).
    8. Mary Layne & Deborah Wagner & Cynthia Rothhaas, 2014. "Estimating Record Linkage False Match Rate for the Person Identification Validation System," CARRA Working Papers 2014-02, Center for Economic Studies, U.S. Census Bureau.
    9. Kartik Hosanagar, 2011. "Usercentric Operational Decision Making in Distributed Information Retrieval," Information Systems Research, INFORMS, vol. 22(4), pages 739-755, December.
    10. Oleg Seleznjev & Bernhard Thalheim, 2010. "Random Databases with Approximate Record Matching," Methodology and Computing in Applied Probability, Springer, vol. 12(1), pages 63-89, March.
    11. Bartolini, Fabio & Brunori, Gianluca & Coli, Alessandra & Landi, Chiara & Pacini, Barbara, 2015. "Assessing the Causal Effect of Decoupled Payments on farm labour in Tuscany Using Propensity Score Methods," 2015 Conference, August 9-14, 2015, Milan, Italy 211200, International Association of Agricultural Economists.
    12. Shaobo Li & Matthew J. Schneider & Yan Yu & Sachin Gupta, 2023. "Reidentification Risk in Panel Data: Protecting for k -Anonymity," Information Systems Research, INFORMS, vol. 34(3), pages 1066-1088, September.
    13. Debabrata Dey & Subodha Kumar, 2013. "Data Quality of Query Results with Generalized Selection Conditions," Operations Research, INFORMS, vol. 61(1), pages 17-31, February.
    14. Sandeep Purao & Veda C. Storey & Taedong Han, 2003. "Improving Analysis Pattern Reuse in Conceptual Design: Augmenting Automated Processes with Supervised Learning," Information Systems Research, INFORMS, vol. 14(3), pages 269-290, September.
    15. Michael Scholz & Markus Franz & Oliver Hinz, 2016. "The Ambiguous Identifier Clustering Technique," Electronic Markets, Springer;IIM University of St. Gallen, vol. 26(2), pages 143-156, May.
    16. Bera Sabyasachi & Chatterjee Snigdhansu, 2020. "High dimensional, robust, unsupervised record linkage," Statistics in Transition New Series, Polish Statistical Association, vol. 21(4), pages 123-143, August.
    17. Jixian Wang & Peter Donnan, 2002. "Adjusting for missing record linkage in outcome studies," Journal of Applied Statistics, Taylor & Francis Journals, vol. 29(6), pages 873-884.
    18. Skinner, Chris J., 2007. "The probability of identification: applying ideas from forensic statistics to disclosure risk assessment," LSE Research Online Documents on Economics 39105, London School of Economics and Political Science, LSE Library.
    19. Jiexun Li & G. Alan Wang & Hsinchun Chen, 2011. "Identity matching using personal and social identity features," Information Systems Frontiers, Springer, vol. 13(1), pages 101-113, March.
    20. Donald Rubin, 2006. "Conceptual, computational and inferential benefits of the missing data perspective in applied and theoretical statistical problems," AStA Advances in Statistical Analysis, Springer;German Statistical Society, vol. 90(4), pages 501-513, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:oropre:v:51:y:2003:i:2:p:240-254. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.