原因1：过往研究证明decoder-only泛化化性能更好Google有两篇著名的发表于ICML’22的论文，一个是《Examining Scaling and Transfer of Language Model Architectures for Machine Translation》，另一个是《What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?》，两篇论文...
Despite the significant advancements in applying language models to the seq2seq task, there is still a lack of thorough analysis on the effectiveness of the decoder-only language model architecture. This paper aims to address this gap by conducting a detailed comparison between the encoder-decoder ...
Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, Furu Wei NeurIPS 2024|May 2024 Publication|Publication 下载BibTex We introduce a decoder-decoder architecture, YOCO, for large language models, which only caches key-value pairs once. It consists of two components, i.e...
Apart from the various interesting features of this model, one feature that catches the attention is its decoder-only architecture. In fact, not just PaLM, some of the most popular and widely used language models are decoder-only.
During the process, we mask out or hide the future tokens so that the model can’t have access to the future tokens. Because it is kind of cheating. We want the model itself to predict the future by only seeing the past tokens. That makes sense, right? That’s why we used a gray...
If GPT-1 makes predictions based solely on the previous token sequence, i.e., P(output | input), GPT-2 does so not only based on the sequence, but also based on the given task, i.e., P(output | input, task). With this property, the same prompt will cause the model to produce...
then it can model the distribution of any target vector sequence given the hidden stateccby simply multiplying all conditional probabilities. So how does the RNN-based decoder architecture modelpθdec(yi|Y0:i−1,c)pθdec(yi|Y0:i−1,c)?
seq2seq model: encoder-decoder 1.1. its probablistic model 1.2. RNN encoder-decoder model architecture context vector c = encoder’s final state i.e. fixed global representation of the input sequ... 查看原文 encoder-decoder框架和普通框架的区别在哪里?
pretraining and multilingual fine-tuning are both critical for facilitating cross-lingual transfer in zero-shot translation. Therefore the researchers present SixT+, a strong many-to-English NMT model that supports 100 source languages but is trained with...
In addition, we further probe the decoder-only architecture for speech-to-text tasks by training a smaller scale randomly initialized speech-LLaMA model from speech-text paired data alone. We conduct experiments on multilingual speech-to-text translation tasks and demonstrate a significant improvement ...