Policy gradient methods for optimal trade execution in limit order books

We discuss applications of policy gradient methods for the optimal execution of an asset position via limit orders. We study two examples in-depth: a parametric limit order book (LOB) model and a realistic generative adversarial neural network (GAN) LOB model. In the first case, we apply a zeroth-order gradient estimator to a suitable parameterization of candidate policies and propose modifications to lower the variance in the estimate, including conditional sampling and a backward-in-time recursion. In the second case, we adapt a recently published LOB-GAN model to obtain a differentiable map from the parameters to the objective. We then alter a standard policy gradient method with a pathwise gradient estimator to overcome issues with the nonconvexity and roughness of the loss landscape, studying different initializations using inexact dynamic programming and second-order optimization steps, as well as regularization of the learnt policies. In both cases, we are able to learn effective trading strategies.