如果部署环境是Linux,推荐安装fast_tokenizer可以得到更极致的文本处理效率,进一步提升服务性能。目前暂不支持Windows设备安装,将会在下个版本支持。 5555 ```shell 56- pip installfast_tokenizer 56+ pip installfast-tokenizer-python 5757 ```--> 5858
目前由于人力问题，已经停止更新fast-tokenizer了，欢迎开发者贡献。https://github.com/PaddlePaddle/Paddle...
🚀 Feature request Tokenizer are provided with each model, some have a fast version of their tokenizer (Rust based), others like CamemBERT have only the slow version. Motivation Fast tokenizer improves inference times drastically (in real...