Parameter-expanded ECME algorithms for logistic and penalized logistic regression

Parameter estimation in logistic regression is a well-studied problem with the Newton–Raphson method being one of the most prominent optimization techniques used in practice. A number of monotone optimization methods including minorization-maximization (MM) algorithms, expectation-maximization (EM) algorithms and related variational Bayes approaches offer useful alternatives guaranteed to increase the logistic regression likelihood at every iteration. In this article, we propose and evaluate an optimization procedure that is based on a straightforward modification of an EM algorithm for logistic regression. Our method can substantially improve the computational efficiency of the EM algorithm while preserving the monotonicity of EM and the simplicity of the EM parameter updates. By introducing an additional latent parameter and selecting this parameter to maximize the penalized observed-data log-likelihood at every iteration, our iterative algorithm can be interpreted as a parameter-expanded expectation-conditional maximization either (ECME) algorithm, and we demonstrate how to use the parameter-expanded ECME with an arbitrary choice of weights and penalty function. In addition, we describe a generalized version of our parameter-expanded ECME algorithm that can be tailored to the challenges encountered in specific high-dimensional problems, and we study several interesting connections between this generalized algorithm and other well-known methods. Performance comparisons between our method, the EM algorithm, Newton–Raphson, and several other optimization methods are presented using an extensive series of simulation studies based upon both real and synthetic datasets.