Source code inbionemo/llm/utils/megatron_utils.py 2021222324252627282930313233343536 defis_only_data_parallel()->bool:"""Checks to see if you are in a distributed megatron environment with only data parallelism active.This is useful if you are working on a model, loss, etc and you know that ...
Ongoing research training transformer language models at scale, including: BERT - Megatron-LM/utils.py at master · sjm112/Megatron-LM
Update megatron/utils.py Browse files Loading branch information saforem2 committed Jun 25, 2024 1 parent 1d413c6 commit 93e4a51 Showing 1 changed file with 215 additions and 111 deletions. Whitespace Ignore whitespace Split Unified Loading Oops, something went wrong. Retry 0 comments...
Ongoing research training transformer models at scale - Megatron-LM/tasks/finetune_utils.py at main · xinyuegtxy/Megatron-LM
Ongoing research training transformer language models at scale, including: BERT & GPT-2 - History for megatron/data/dataset_utils.py - microsoft/Megatron-DeepSpeed
from megatron import print_rank_0 from megatron.core import mpu, tensor_parallel from megatron.data.dataset_utils import create_masked_lm_predictions, pad_and_convert_to_numpy from megatron import get_args, get_tokenizer, print_rank_0def get_one_epoch_dataloader(dataset, m...
Intel Gaudi's Megatron DeepSpeed Large Language Models for training - Megatron-DeepSpeed/tasks/eval_utils.py at main · HabanaAI/Megatron-DeepSpeed
An implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library. - gpt-neox/megatron/text_generation_utils.py at main · onlyone0001/gpt-neox
megatron scripts tasks eval_harness glue msdp orqa race vision zeroshot_gpt data_utils.py ensemble_classifier.py eval_utils.py finetune_utils.py main.py main_3d.py tests tools .coveragerc .gitignore .gitlab-ci.yml CODEOWNERS LICENSE MANIFEST.in README.md SECURITY.md finetune_llama.py pre...
data.dataset_utils import build_training_sample from megatron.data.indexed_dataset import make_dataset as make_indexed_dataset from megatron.data.realm_dataset import InverseClozeDataset from megatron import print_rank_0 DATASET_TYPES = ['standard_bert', 'ict', 'realm'] def build_train_valid_...