Add Tokenizer config file Browse files main (huggingface/blog#8) Cong authored and julien-c committed Feb 28, 2020 1 parent ce156da commit 63872d5 Showing 1 changed file with 1,828 additions and 116 deletions. Whitespace Ignore whitespace Split Unified 1,944 changes: 1,828 additions &...
when I use AutoTokenizer to load tokenizer,use the code below; tokenizer = transformers.AutoTokenizer.from_pretrained( args.pretrained_model_name_or_path, subfolder="tokenizer", revision=args.revision, use_fast=False, ) but I found can't get the right tokenizer_config.json file. Indeed the ...