Firefly(流萤): 中文对话式大语言模型(全量微调+QLoRA),支持微调Llma2、Llama、Qwen、Baichuan、ChatGLM2、InternLM、Ziya、Bloom等大模型 - 修复QWenTokenizer只有eod_id的问题,兼容所有tokenizer · googx/Firefly@67dd449
(self, token_ids): return self.tokenizer.decode(token_ids) @property def eod(self): return self.eod_id class _SentencePieceTokenizer(AbstractTokenizer): """SentencePieceTokenizer-Megatron wrapper""" def __init__(self, model_file, vocab_extra_ids=0): name = 'SentencePiece...