An Empirical Comparative Study of LoRA Fine-Tuning Hyperparameter Configurations on the GLUE Benchmark

Low-Rank Adaptation (LoRA) has become a dominant parameter-efficient fine-tuning paradigm for adapting pretrained Transformers to downstream tasks, yet practitioners still rely on rule-of-thumb settings for its central hyperparameters: the rank r, the scaling factor α, the choice of target modules, and the dropout probability. This paper presents an empirical comparative study that quantifies how these four hyperparameters jointly affect downstream behaviour on six representative tasks of the General Language Understanding Evaluation (GLUE) benchmark, using RoBERTa-base as the backbone. A controlled grid of 72 configurations is evaluated on SST-2, CoLA, MRPC, STS-B, QNLI, and RTE, with each cell averaged over three random seeds and reported along four axes: task-specific accuracy or correlation, trainable parameter count, peak GPU memory, and wall-clock training time. The results indicate that rank saturation occurs earlier than is commonly assumed at this scale, that the ratio α∕r is a stronger predictor of accuracy than α in isolation, and that expanding the adapted modules from the query--value pair to all linear projections yields task-dependent and modest gains while incurring a sizeable cost in memory and runtime. Practical configuration guidance is summarised by task-size category so that downstream users can pick a starting point with a clearer view of the trade-offs involved.