Iterative Reasoning Preference Optimization

arXiv:2404.19733v1 Announce Type: new
Abstract: Iterative preference optimization methods have recently been shown to perform well for general instruction tuning tasks, but typically make little improvement on reasoning tasks (Yuan et al., 2024, Chen et al., 2024). In this work we develop an iterative approach that optimizes the preference between competing generated Chain-of-Thought (CoT) candidates by optimizing for winning vs. losing reasoning steps that lead to the correct answer. We train using a modified DPO loss (Rafailov et al., 2023) …

abstract arxiv chen cs.ai cs.cl general generated improvement instruction tuning iterative optimization reasoning tasks thought type work