git clone https://github.com/sylinrl/TruthfulQA cd TruthfulQA pip install -r requirements.txt pip install -e . To useGPT-J, download the HuggingFace-compatible model checkpoint provided by EleutherAI. Evaluation For supported models, answers and scores can be generated by runningtruthfulqa/evalu...
truthfulqa .gitignore LICENSE README.md TruthfulQA-demo.ipynb TruthfulQA.csv TruthfulQA_demo.csv requirements.txt setup.py Breadcrumbs TruthfulQA / TruthfulQA_demo.csv Latest commit sylinrl Updated README, full datasetAug 28, 2021 5fb9ef8· Aug 28, 2021 HistoryHistory File metadata and cont...
GitHub is where people build software. More than 100 million people use GitHub to discover, fork, and contribute to over 420 million projects.
https://github.com/open-compass/opencompass/blob/main/README_zh-CN.md#%EF%B8%8F-%E5%AE%89%E8%A3%85 # 下载数据集到 data/ 处 wget https://github.com/open-compass/opencompass/releases/download/0.1.1/OpenCompassData.zip unzip OpenCompassData.zip @BeyonderXX 你有下载这个数据集吗?./da...
GitHub:GitHub - sylinrl/TruthfulQA: TruthfulQA: Measuring How Models Imitate Human Falsehoods TL;DR 一个用来评判语言模型生成的答案是否真实的benchmark,精心设计了800+个问题,这些问题包含一些类似于流行的错误观念等,且容易被错误回答。为了表现得好,模型必须避免从人类文本中学到一些错误答案。
数据集位置:github.com/sylinrl/Trut 目标:在于引出大模型的模仿性谎言错误。 真实性标准:事实性,采用了严格的标准(类似科学文章和维基百科),认定是真实事实。eg:“天秤座的人性格友好“就是虚假的。数据集样例:每条数据集由问题、最好答案、真实答案、虚假答案和答案来源构成。 3、实验结论 在评估的时候,关注模...
https://github.com/sylinrl/TruthfulQAgithub.com/sylinrl/TruthfulQA Tasks TruthfulQA consists of two tasks that use the same sets of questions and reference answers. Generation (main task): Task: Given a question, generate a 1-2 sentence answer. Objective: 主要目标是总体的真实性,用模型...
Raw 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 exportCUDA_VISIBLE_DEVICES=0 ROOT=path_to_truthx_dir EXP_ROOT=$ROOT/results model_path=path_to_llm#e.g. Llama-2-7b-chat-hf #two-fold validation truthx_model1=truthx_models/Llama-2-7b-chat-hf...
然后下载alpaca_cleaned_dataset 以及truthfulQA数据集,他们都在github上有仓库,我们下载了json格式的数据集。 数据集加载 打开alpaca数据集,你会看到里面的形式类似于下面的代码块。 LLM 的微调一般指指令微调过程。所谓指令微调,是说我们使用的微调数据形如: { "instruction": "Give three tips for staying healthy...
Activity colesburyadded module: fsdp triagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate module on Jan 30, 2024 Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment...