Describe the bug load_from_disk and save_to_disk are not compatible. When I use save_to_disk to save a dataset to disk it works perfectly but given the same directory load_from_disk throws an error that it can't find state.json. looks li...
I'm having the same issue, i've fine tuned a Llama 7b model using peft, and got satisfying results in inference, but when i try to use SFTTrainer.save_model, and load the model from the saved files using LlamaForCausalLM.from_pretrained, the inference result seem to just be of the ...
"dataset_filename_join_string": " ", "training_image_repeats_per_epoch": 1, "training_write_csv_every": 500, "training_xattention_optimizations": false, "training_enable_tensorboard": false, "training_tensorboard_save_images": false, "training_tensorboard_flush_every": 120, "sd_model_check...
There is a method to save tokenizer. Check this notebook: https://github.com/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb Author 008karan commented May 25, 2020 Thats what I am using. its saving it in the dataset variable not in any file. ByTokenize data I mean pretra...