When I inspect auto_tokenizer variable, then I get LlamaTokenizerFast: LlamaTokenizerFast(name_or_path='mistralai/Mistral-7B-Instruct-v0.3', vocab_size=32768, model_max_length=1000000000000000019884624838656, is_fast=True, padding_side='left', truncation_side='right', special_tokens={'bos_token'...
padding_side = padding_side, token = token, use_fast = False, ) return check_tokenizer( model = model, tokenizer = tokenizer, model_name = model_name, model_max_length = model_max_length, padding_side = padding_side, token = token, ...
consolereTrainedTokenizerFast(name_or_path='prajjwal1/bert-tiny',vocab_size=30522,model_max_len=1000000000000000019884624838656,is_fast=True,padding_side='right',truncation_side='right',special_tokens={'unk_token':'[UNK]','sep_token':'[SEP]','pad_token':'[PAD]','cls_token':'[CLS]','...