ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models AuthorsIman Mirzadeh, Keivan Alizadeh Vahid, Sachin Mehta, Carlo C Del Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, Mehrdad Farajtabar View publication Copy Bibtex Large Language Models (LLMs) with billions of...
论文题目:ReLUStrikes Back: Exploiting Activation Sparsity in Large Language Models 论文链接:https://arxiv.org/abs/2310.04564 参数规模超过十亿(1B)的大型语言模型(LLM)已经彻底改变了现阶段人工智能领域的研究风向。越来越多的工业和学术研究者开始研究LLM领域中的难题,例如如何降低LLM在推理过程中的计算需求。
ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models Large Language Models (LLMs) with billions of parameters have drastically transformed AI applications. However, their demanding computation during inference has raised significant challenges for deployment on resource-constrained devices...
综合这些发现和设计,本文实现了基于ReLU的高效LLM计算方案,相比其他激活函数,将LLM的推理计算量大幅减少三倍。 论文题目: ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models 论文链接: https://arxiv.org/abs/2310.04564 一、引言 为了提高LLM的推理效率,研究者们提出了包括量化、推测解码、...