Improving text classification: logistic regression makes small LLMs strong and explainable ‘tens-of-shot’ classifiers

Text classification tasks such as sentiment analysis are common in economics and finance. We demonstrate that smaller, local generative language models can be effectively used for these tasks. Compared to large commercial models, they offer key advantages in privacy, availability, cost, and explainability. We use 17 sentence classification tasks (each with 2 to 4 classes) to show that penalised logistic regression on embeddings from a small language model often matches or exceeds the performance of a large model, even when trained on just dozens of labelled examples per class – the same amount typically needed to validate a large model’s performance. Moreover, this embedding-based approach yields stable and interpretable explanations for classification decisions.