First introduced in 2019, Megatron (1, 2, and 3) sparked a wave of innovation in the AI community, enabling researchers and developers to utilize the underpinnings of this library to further LLM advancements. Today, many of the most popular LLM developer frameworks have been inspired by and ...
GitHub is where people build software. More than 100 million people use GitHub to discover, fork, and contribute to over 420 million projects.
First introduced in 2019, Megatron (1,2, and3) sparked a wave of innovation in the AI community, enabling researchers and developers to utilize the underpinnings of this library to further LLM advancements. Today, many of the most popular LLM developer frameworks have been inspired by and built...
Megatron-LLM This library enables pre-training and fine-tuning of large language models (LLMs) at scale. Our repository is a modification of the original Megatron-LM codebase by Nvidia. Added key features include: architectures supported: Llama, Llama 2, Code Llama, Falcon and Mistral support ...
sail-sg/Megatron-LLMPublic forked fromepfLLM/Megatron-LLM NotificationsYou must be signed in to change notification settings Fork0 Star3 Code Pull requests Actions Projects Security Insights Additional navigation options Files main .github docs
Megatron-LM serves as a research-oriented framework leveraging Megatron-Core for large language model (LLM) training. Megatron-Core, on the other hand, is a library of GPU optimized training techniques that comes with formal product support including versioned APIs and regular releases. You can ...
InstructRetro (Wang et al., 2023b) further scales up the size of Retro to 48B, featuring the largest LLM pretrained with retrieval (as of December 2023). The obtained foundation model, Retro 48B, largely outperforms the GPT counterpart in terms of perplexity. With instruction tuning on Retro...
In recent LLMs, distributed learning using multiple machine nodes has become common due to the large size of the models. When using distributed learning, libraries such as Megatron-LM and DeepSpeed are utilized. However, even in such cases, this dataset should be just as easy to handle....
InstructRetro (Wang et al., 2023b) further scales up the size of Retro to 48B, featuring the largest LLM pretrained with retrieval (as of December 2023). The obtained foundation model, Retro 48B, largely outperforms the GPT counterpart in terms of perplexity. With instruction tuning on Retro...
InstructRetro (Wang et al., 2023b) further scales up the size of Retro to 48B, featuring the largest LLM pretrained with retrieval (as of December 2023). The obtained foundation model, Retro 48B, largely outperforms the GPT counterpart in terms of perplexity. With instruction tuning on Retro...