Towards Stability of Parameter-free Optimization

arXiv:2405.04376v1 Announce Type: new
Abstract: Hyperparameter tuning, particularly the selection of an appropriate learning rate in adaptive gradient training methods, remains a challenge. To tackle this challenge, in this paper, we propose a novel parameter-free optimizer, AdamG (Adam with the golden step size), designed to automatically adapt to diverse optimization problems without manual tuning. The core technique underlying AdamG is our golden step size derived for the AdaGrad-Norm algorithm, which is expected to help AdaGrad-Norm preserve the tuning-free convergence and …

abstract adam adapt arxiv challenge cs.lg diverse free gradient hyperparameter novel optimization paper rate stability training type