Steganographic Capacity of Transformer Models

As machine learning and deep learning models become ubiquitous, it is inevitable that there will be attempts to exploit such models in various attack scenarios. For example, in a steganographic-based attack, information could be hidden in a learning model, which might then be used to distribute malware, or for other malicious purposes. In this research, our focus is on the steganographic capacity a Transformer model, but for comparison we also consider a Multilayer Perceptron (MLP) and Convolutional Neural Network (CNN). All three models are trained on a challenging malware classification problem, and for each models, we determine the number of low-order bits of the trained parameters that can be altered without significantly affecting the classification accuracy. We find that the steganographic capacity of the learning models tested is surprisingly high, and that in each case, there is a clear threshold after which model performance rapidly degrades. Due to its large number of weights, we find that the Transformer model has a steganographic capacity that is orders of magnitude larger than that of either the MLP or CNN models.