Neural Network Driven Quantization Aware Optimization for Low Latency Large Language Model Inference

Large Language Models (LLMs) have proven to be remarkably successful in natural language processing applications, but the cost of computation, large memory consumption and longer inference time represent limiting factors in their use. To overcome these challenges, this paper suggests a Neural Network Driven Quantization Aware Optimization framework. The approach integrates the concept of training with the understanding of quantization with a neural controller that dynamically selects the most effective quantification levels to apply in the different layers of the model. Adaptive quantization and latency-sensitive optimization can be employed to attain significantly reduced inference time and memory footprint without loss of model accuracy with the proposed technique. The experimental analysis demonstrates that the proposed framework is more effective and works better compared to the existing methods which incorporate GPTQ, LLM-MQ, and SVD-LLM. The results indicate the applicability of smart, learning-based optimization techniques to support real-time and resource-easy implementation of LLM.