Meta-Learned Adaptive Proximal Operators for Neural Network Optimization

We propose a novel approach to neural network optimization by replacing traditional proximal operators with meta-learned adaptive updates, thereby unifying the optimization of step sizes, regularization strengths, and sparsity thresholds into a single learned process. The proposed method introduces a bi-level optimization framework in which a recurrent meta-learner dynamically produces task- and architecture-specific proximal parameters during training, thereby removing the necessity for manual tuning. The system’s central mechanism relies on an LSTM-driven meta-learner handling optimization trajectories and architectural embeddings, with the latter derived from a graph neural network to support generalization across architectures. The proximal updates that emerge blend learned parameters with a continuous shrinkage operator, which prevents gradient discontinuities and preserves sparsity. The meta-learner is trained by optimizing a bi-level objective aimed at reducing the anticipated final loss over tasks, with gradients estimated by truncated backpropagation through time. The framework operates smoothly with traditional neural network training by substituting standard optimizer steps with updates derived from meta-learning. Experiments show that the method adjusts to various architectures and tasks, achieving better performance than fixed proximal approaches and diminishing the need for manual hyperparameter adjustment. Furthermore, the architectural embeddings support zero-shot generalization to novel network structures, which renders the approach especially appropriate for automated machine learning pipelines. This work is important because it moves away from strict optimization heuristics, adopting an approach that learns optimization strategies which inherently adjust to both task demands and architectural limitations.