Rethinking the adaptive relationship between Encoder Layers and Decoder Layers

arXiv:2405.08570v1 Announce Type: new
Abstract: This article explores the adaptive relationship between Encoder Layers and Decoder Layers using the SOTA model Helsinki-NLP/opus-mt-de-en, which translates German to English. The specific method involves introducing a bias-free fully connected layer between the Encoder and Decoder, with different initializations of the layer's weights, and observing the outcomes of fine-tuning versus retraining. Four experiments were conducted in total. The results suggest that directly modifying the pre-trained model structure for fine-tuning yields suboptimal performance. However, upon …

abstract article arxiv bias cs.cl decoder encoder english free german helsinki layer nlp opus relationship sota type