@article{Liu2024LetsLLS, title={Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learning}, author={Yinpeng Liu and Jiawei Liu and Xiang Shi and Qikai Cheng and Wei Lu}, year={2024
In-context learning是LLM最重要的emergent ability之一,它可以在模型inference阶段通过上下文的内容学习,无需梯度下降,可以在文本分类等任务达到很高的准确率。以图中的单词分类case为例,"foo"和"bar"是semantic-unrelated label,分别对应country和animal。图中的语句为"France : bar Cat : foo Dog :",demonstration是...
比起小模型,大模型有一个很重要的涌现能力(Emergent ability)就是In-Context Learning(ICL),也是一种新的范式,指在不进行参数更新的情况下,只在输入中加入几个示例就能让模型进行学习,如下图中用ICL做情感分析任务的栗子: 忽略大模型的贵,这个范式具备不少优势: 输入的形式是自然语言,可以让我们可以更好地跟语言...
(Pre-training for In-Context Learning), a framework to enhance the language models’ in-context learning ability by pre-training the model on a large collection of “intrinsic tasks” in the general plain-text corpus using the simple language modeling objective. PICL encourages the model to ...
🦦 Otter, a multi-modal model based on OpenFlamingo (open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and showcasing improved instruction-following and in-context learning ability. otter-ntu.github.io/ Topics machine-learning deep-learning multi-modality artificial-inteligence ...
learning efficiency, and task-conditional probabilities, which can help us understand how model behaviour changes under alignment. We use these to show how ICL ability varies at different model scales, understand how finetuning harms knowledge of disfavoured distributions, and compare base and instructi...
Large language models (LLMs) have demonstrated their ability to learn in-context, allowing them to perform various tasks based on a few input-output examples. However, the effectiveness of in-context learning is heavily reliant on the quality of the selected examples. In this paper...
In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Models. Yukun Huang, Yanda Chen, Zhou Yu, Kathleen McKeown. [pdf], 2022.12, Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step QuestionsProblems...
The predictions of Large Language Models (LLMs) on downstream tasks often improve significantly when including examples of the input--label relationship in the context. However, there is currently no consensus about how this in-context learning (ICL) ability of LLMs works. For example, while ...
In-context learning refers to the emerging ability of large language models (LLMs) to perform a target task without additional training, utilizing demonstrations of the task. Recent studies aim to enhance in-context learning performance by selecting more useful demonstrations. However, they overlook ...