Evaluating the quality of tabular synthetic data in health care

Machine Learning (ML) research in healthcare remains challenging as large, privacy-preserving open datasets are lacking. Synthetic data could offer a solution, but the value of synthetic data depends on diverse and conflicting criteria such as utility, fidelity, and privacy, which are rarely evaluated comprehensively. To close this gap, we explore the trade-off between these metrics in an empirical evaluation across a broad spectrum of generative models, datasets and metrics. In order to include as many metrics and models as possible and to ensure both applicability and comparability with other studies, we focus on the most widely available data modality and task setting: tabular data associated with a classification task. Extending prior work our results demonstrate that no single generative model excels across all metrics and datasets. Across 9 datasets and 11 generative models, the first principal variance direction of all metrics captures the dominant trade-off between fidelity and utility metrics on one side and the privacy metrics on the other side. Sensitivity analyses indicate that the privacy–fidelity/utility trade-off captured by the first principal variance direction remains consistent across several datasets and may support model selection. These insights highlight the potential of synthetic data for responsible data sharing in health care as well as the need for better tooling in synthetic data generation with a higher degree of automation when optimizing for metrics capturing fidelity, utility and privacy.Author summary: Artificial Intelligence (AI) has the potential to improve patient treatment with access to high-quality data. However, sharing such data is difficult under strict data protection privacy laws. Many generative AI models have been proposed to produce data that preserves both patients’ privacy as well as the statistical dependency structure and utility of the data. Yet comprehensive empirical evaluations and evaluation protocols that account for all relevant metrics remained scarce. In this study, we present evaluation protocols and experiments comparing generative models for tabular health care data. We assess 11 different generative model architectures for tabular data on 9 datasets to assess three key metrics used in research: fidelity - how good do the synthetic data resemble the original data, utility - how useful these data are, and privacy - how well they protect patient information. Our findings suggest that no single model outperforms all other models in all three metrics. We also find that dataset characteristics, or meta-data, cannot predict the best performing model for a given dataset. But our results also demonstrate that the complexity of the privacy utility trade-off is well captured by the first two principal components. These insights highlight the need for better using another approach - a reliable automation for selecting generative tabular models, especially when the ultimate goal is responsible data sharing in health care.