A modeling framework for analyzing repeated outcomes from high-dimensional complex big data using statistical learning models

The complex, high-dimensional big data collected repeatedly produce correlated outcomes that are common across all aspects of studies, and analysis of such data requires new statistical models. We propose Markov chain modeling techniques to accommodate repeated correlations. The covariate impact on outcomes is estimated using a logistic regression model for each follow-up, with a specific data configuration. Hence, overparameterization reduction compared to Markov regression models. We showed overparameterization reduction, dimensionality reduction, and trajectory risk prediction, among others. Dimensionality reduction was performed using Lasso within the regression modeling framework. The prediction accuracy with logistic regression ranges from 0.88 to 0.89 on the training data and from 0.89 to 0.91 on the test data, indicating good generalization. The Lasso prediction accuracy was closely followed by that from logistic regression, which proved useful. Finally, the future course of functional disability risks was illustrated graphically for two subjects using the conditional and joint probabilities.