Off-OAB: Off-Policy Policy Gradient Method with Optimal Action-Dependent Baseline

arXiv:2405.02572v1 Announce Type: new
Abstract: Policy-based methods have achieved remarkable success in solving challenging reinforcement learning problems. Among these methods, off-policy policy gradient methods are particularly important due to that they can benefit from off-policy data. However, these methods suffer from the high variance of the off-policy policy gradient (OPPG) estimator, which results in poor sample efficiency during training. In this paper, we propose an off-policy policy gradient method with the optimal action-dependent baseline (Off-OAB) to mitigate this variance issue. …

abstract arxiv benefit cs.ai cs.lg data estimator gradient however policy reinforcement reinforcement learning success type variance