version: llm-jp-evalのバージョン情報. basemodel_name: 評価実験を行なった言語モデルの情報. model_type: 評価実験を行なった言語モデルのカテゴリー情報. instruction_tuning_method_by_llm-jp: llm-jpでチューニングを行なった際,その手法の情報. ...
The bin/api/run_api_eval.sh script is used to run the evaluation against the vLLM API. To run evaluation against Liquid models: Launch on-prem stack. Run the following commands, one for each model: # run against lfm-3b-jp bin/api/run_api_eval.sh --config config_api.yaml \ --mode...
Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Cancel Create saved search Sign in Sign up Reseting focus {{ message }} llm-jp / llm-jp-eval Public Notifications You must be signed in to change notification settings Fork 39 Star 108 ...
おっしゃる通りllm-jp-evalは基本4-shotsでの評価をデフォにしており、 今の設定ではそれが不可能だと判断したのが今回の原因となります。 ただMBPPのように、例外的に0-shotをデフォにしているデータセットもあるにはありますので、
llm-jp / llm-jp-eval Public Notifications Fork 38 Star 100 Code Issues 4 Pull requests 7 Actions Projects Security Insights New issue llm-jp-evalで目的別のrequirements.txtを用意する #128 Closed hiroshi-matsuda-rit opened this issue Jul 4, 2024· 0 comments ...
# llm-jp-eval [ English | [**日本語**](./README.md) ] [](https://github.com/llm-jp/llm-jp-eval/actions/workflows/test.yml) [![lint](https://github.com/llm-jp/llm-jp-eval/actions/workflows...
What The WikiCorpus dataset is split inconsistently across different Python environments. Version llm-jp-eval: v1.3.0 To reproduce Clone the repo. $ git clone git@github.com:llm-jp/llm-jp-eval.git Install the dependencies. $ python3 -m v...
Actions: llm-jp/llm-jp-evalActions All workflows Generate requirements.txt Lint Test Management Caches Attestations Lint lint.yml 502 workflow runs Event Status Branch Actor do not need to assign eos/pad_token_id mandatorily
"offline_dir": "/model/takumi/working/temp4/llm-jp-eval/offline_inference/vllm/outputs/llm-jp--llm-jp-1.3b-v1.0_vllm_20240713_190142", にようにconfig/offline_dirにパスとして非明示的にgeneratorが埋め込まれているだけなので、config/generator属性を追加して明示的に推論ライブラリを残せ...
Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Cancel Create saved search Sign in Sign up Reseting focus {{ message }} llm-jp / llm-jp-eval Public Notifications You must be signed in to change notification settings Fork 39 Star 124 ...