NOLA: Compressing LoRA using Linear Combination of Random Basis

arXiv:2310.02556v2 Announce Type: replace-cross
Abstract: Fine-tuning Large Language Models (LLMs) and storing them for each downstream task or domain is impractical because of the massive model size (e.g., 350GB in GPT-3). Current literature, such as LoRA, showcases the potential of low-rank modifications to the original weights of an LLM, enabling efficient adaptation and storage for task-specific models. These methods can reduce the number of parameters needed to fine-tune an LLM by several orders of magnitude. Yet, these methods face two …

arxiv combination cs.cl cs.cv linear lora random type