在本框架的语义内,additional_special_tokens 标志了除了 eos_token 以外的结束符 Originally posted by @hiyouga in #4203 (comment) 如果是这样的话,请问本框架是否提供增加普通token的接口呢? github-actions bot added the pending label Oct 12, 2024 Sign up for free to join this conversation on GitH...
Description This lets users specify "additional_special_tokens" of key of the add_special_tokens method. Motivation and Context Special tokens are treated different by the tokenizer, ensuring that ...
* Support for additional_special_tokens * Support for additional_special_tokens. Adjust whitespace. * Support for additional_special_tokens. Use correct quotes. * Support for additional_special_tokens. Safe pop. * Support for additional_special_tokens. nt. * Support for additional_special_tokens. ...