Folded context condensation in Path Integral formalism for infinite context transformers

arXiv:2405.04620v1 Announce Type: cross
Abstract: This short note is written for rapid communication of long context training and to share the idea of how to train it with low memory usage. In the note, we generalize the attention algorithm and neural network of Generative Pre-Trained Transformers and reinterpret it in Path integral formalism. First, the role of the transformer is understood as the time evolution of the token state and second, it is suggested that the all key-token states in …

abstract algorithm arxiv attention communication context cs.ai cs.cl cs.lg cs.ne generative hep-ph integral low memory network neural network path train training transformers type usage