To install the lm-eval package from the github repository, run: git clone https://github.com/EleutherAI/lm-evaluation-harness cd lm-evaluation-harness pip install -e . We also provide a number of optional dependencies for extended functionality. Extras can be installed via pip install -e "....
lm_eval --model vllm --model_args "pretrained=$model_identifier,tensor_parallel_size=$number_of_gpus,dtype=auto" --tasks $task_name --batch_size auto --log_samples --output_path "output/${model_identifier}_${task_name}" Footer © 2024 GitHub, Inc. Footer navigation Terms Privacy ...
在测评过程中,我们首先使用lm evaluation harness对C-Eval模型进行了基准测试。通过对比C-Eval与其他主流大语言模型在各项评估指标上的表现,我们发现C-Eval在文本生成和语言理解方面具有一定的优势。特别是在处理长文本和复杂语境时,C-Eval表现出了较强的泛化能力和鲁棒性。 接下来,我们使用vllm对C-Eval模型进行了更...
main Breadcrumbs lm-eval-harness/ requirements.txtLatest commit HistoryHistory File metadata and controls Code Blame 1 lines (1 loc) · 5 Bytes Raw 1 -e . Footer © 2024 GitHub, Inc. Footer navigation Terms Privacy Security Status Docs Contact Manage cookies Do not share my personal ...
Command:python eval/lm_eval_harness.py --checkpoint_dir checkpoints/stabilityai/stablelm-base-alpha-3b --precision "bf16-true" --eval_tasks "[gsm8k]" --batch_size 4 --save_filepath "results-stablelm-3b_gsm8k.json" Running greedy_until requests ...
I tried to test KIVI with the latest lm-eval-harness and found that it raise an error when _minmax_along_last_dim: File "/data2_7T/condy/KIVI/models/llama_kivi.py", line 635, in forward hidden_states, self_attn_weights, present_key_value = self.self_attn( File "/data2_7T/...
A high-throughput and memory-efficient inference and serving engine for LLMs - david-vllm/.buildkite/lm-eval-harness at main · vinayburugu/david-vllm
A framework for few-shot evaluation of language models. - lm-evaluation-harness/lm_eval/tasks/ifeval/README.md at main · EleutherAI/lm-evaluation-harness
Some-random / lm-eval-harness Public Notifications Fork 0 Star 0 Code Issues Pull requests Actions Projects Security Insights Footer © 2024 GitHub, Inc. Footer navigation Terms Privacy Security Status Docs Contact Manage cookies Do not share my personal information ...
11 changes: 11 additions & 0 deletions 11 .buildkite/lm-eval-harness/configs/Meta-Llama-3-8B-Instruct.yaml Original file line numberDiff line numberDiff line change @@ -0,0 +1,11 @@ # bash .buildkite/lm-eval-harness/run-lm-eval-gsm-hf-baseline.sh -m meta-llama/Meta-Llama-3-8B-...