此外,研究者使用NLLB在自创的FLORES-200数据集上进行了任意语言的互译,并对比了不同资源级别语言的翻译性能。 由于翻译语料来自维基百科,研究者也在WMT等其他的数据集上进行了测试。研究者也在聊天、医疗、新闻和文本四个领域进行了测试,取得了一定的效果。 目前,NLLB模型及相关数据集已经开源,并集成到HuggingFace的...
The functionload_and_prune_for_lang_pairdownloads the NLLB-200 checkpoints fromHuggingFace(200G in total), if they are not in the HuggingFace cache already. Then it loads the model in CPU memory (100G), prunes it for a given language pair and moves it to GPU memory (approx. 28G)....
This project use the Meta NLLB-200 translation model through the Hugging Face transformers library. translationhuggingface-transformersllmsnllb200nllb UpdatedNov 2, 2023 Python Star5 Example application for the task of fine-tuning pretrained machine translation models on highly domain-specific, self-ext...
large model translator Xenova nllb-200-distilled-600M translator transformers transformers.js huggingface hugging face machine learning deep learning artificial intelligence AI ML machutpublished 1.0.7 • 10 months agopublished 1.0.7 10 months ago M Q PFooter...
The "research" I was referring to was entirely about how to properly use HuggingFace Transformers to prefix the target sentences with the lang code during decoding. I was not aware that source sentences were supposed to be prefixed with the lang code. We use NLLB pretty heavily, so I would...
E.g., /v1 (default: None) -mi MODEL_ID, --model_id MODEL_ID model ID; see https://huggingface.co/models?other=nllb (default: facebook/nllb-200-distilled-600M) -msl MAX_SRC_LEN, --max-src-len MAX_SRC_LEN max source len; longer seqs will be truncated (default: 250) REST...
E.g., /v1 (default: None) -mi MODEL_ID, --model_id MODEL_ID model ID; see https://huggingface.co/models?other=nllb (default: facebook/nllb-200-distilled-600M) -msl MAX_SRC_LEN, --max-src-len MAX_SRC_LEN max source len; longer seqs will be truncated (default: 250) REST...
Using Hugging Face and Metas NLLB Open-Source Model through the HuggingFace Transformers package you can easily build a machine translator shown in the code below. The code starts by installing the transformers and torch packages, to use pre-trained models for natural language processing tasks. ...
The function load_and_prune_for_lang_pair downloads the NLLB-200 checkpoints from HuggingFace (200G in total), if they are not in the HuggingFace cache already. Then it loads the model in CPU memory (100G), prunes it for a given language pair and moves it to GPU memory (approx. ...