Q-Learning and Algorithmic Market Making: Loss-free, Collusive, or Competitive Prices?

We study whether Algorithmic Market Makers using Q-learning produce competitive or supra-competitive prices in a quote-driven asset market. We show, through simulations and analytically, that the result depends on the way the algorithm is set up. A basic Q-learning algorithm leads to loss-free prices and is, therefore, not fit for trade. Carefully choosing the exploration and learning parameters leads to less extreme prices, but still far away from the competitive ones. When we endow the algorithm with a basic understanding of the market and basic information about outstanding quotes, the Q-learning algorithms produce competitive prices.