Comparison of anonymization techniques regarding statistical reproducibility

Background: Anonymization opens up innovative ways of using secondary data without the requirements of the GDPR, as anonymized data does not affect anymore the privacy of data subjects. Anonymization requires data alteration, and this project aims to compare the ability of such privacy protection methods to maintain reliability and utility of scientific data for secondary research purposes. Methods: The French data protection authority (CNIL) defines anonymization as a processing activity that consists of using methods to make impossible any identification of people by any means in an irreversible manner. To answer project’s objective, a series of analyses were performed on a cohort, and reproduced on four sets of anonymized data for comparison. Four assessment levels were used to evaluate impact of anonymization: level 1 referred to the replication of statistical outputs, level 2 referred to accuracy of statistical results, level 3 assessed data alteration (using Hellinger distances) and level 4 assessed privacy risks (using WP29 criteria). Results: 87 items were produced on the raw cohort data and then reproduced on each of the four anonymized data. The overall level 1 replication score ranged from 67% to 100% depending on the anonymization solution. The most difficult analyses to replicate were regression models (sub-score ranging from 78% to 100%) and survival analysis (sub-score ranging from 0% to 100. The overall level 2 accuracy score ranged from 22% to 79% depending on the anonymization solution. For level 3, three methods had some variables with different probability distributions (Hellinger distance = 1). For level 4, all methods had reduced the privacy risk of singling out, with relative risk reductions ranging from 41% to 65%. Conclusion: None of the anonymization methods reproduced all outputs and results. A trade-off has to be find between context risk and the usefulness of data to answer the research question. Author summary: Anonymization is a processing of personal data to generate anonymous synthetic data. This project aimed to evaluate the capability of anonymization methods in maintaining reliability and utility of scientific data for research purposes, by comparing the impact of four different anonymization methods. The comparison of 4 anonymization methods is the novelty of this work and provides insight into the statistical and scientific value that remained. We found that no anonymized data reproduced accurately all results and these methods are not currently powerful enough for use in regulatory submissions. We also found that there is a lack of standardized privacy metrics to comprehensively evaluate and compare the effectiveness of anonymization techniques. As one size does not fit all, we strongly recommend the design of anonymization algorithms to be align with the data reuse objectives (e.g. open data, internal reuse or reuse by a unique external researcher).