我按以下方式使用 Huggingface 的句子 BERT:from sentence_transformers import SentenceTransformer model = SentenceTransformer('all-MiniLM-L6-v2') model.max_seq_length = 512 model.encode(text) Run Code Online (Sandbox Code Playgroud) 当text很长且包含超过 512 个标记时,不会抛出异常。我假设它会自动...
HuggingFace's Trainer API, including the SFTrainer, by default pads all sequences to the maximum length within the batch, not to the max_seq_length argument. The max_seq_length argument serves as a hard limit to the sequence length, truncating any examples that are longer than that. The ...
fix: use st max_seq_length (huggingface#167) Browse files Loading branch information OlivierDehaene authored Feb 23, 2024 1 parent 00a17ea commit 9ca35a8 Showing 4 changed files with 63 additions and 7 deletions. Whitespace Ignore whitespace Split Unified ...
问题是否已解决？我遇到了相同的问题。