Better & Faster Large Language Models via Multi-token Prediction

arXiv:2404.19737v1 Announce Type: new
Abstract: Large language models such as GPT and Llama are trained with a next-token prediction loss. In this work, we suggest that training language models to predict multiple future tokens at once results in higher sample efficiency. More specifically, at each position in the training corpus, we ask the model to predict the following n tokens using n independent output heads, operating on top of a shared model trunk. Considering multi-token prediction as an auxiliary training …

abstract arxiv cs.cl efficiency faster future gpt language language models large language large language models llama loss multiple next prediction results sample token tokens training type via work