Closed I have installed everything and it doesn't work. i have try everythings on internet... nothing work **Environment OS / Windows 11 Python version 3.11 Additional context Here my console CWindowsSystem32cmd.exe.txt Activity Sign up for freeto join this conversation on GitHub.Already have...
the pip install seems to work fine (not sure what it's doing differently) and gives the same "normal" ctx size (around 70KB) as running the model directly within vendor/llama.cpp and the -n 128 suggested for testing. Any suggestions for how to get a working libllama.so would be great...
Ignoring llama-cpp-python: markers 'platform_system != "Windows"' don't match your environment Collecting llama-cpp-python==0.1.36 Downloading https://github.com/abetlen/llama-cpp-python/releases/download/v0.1.36/llama_cpp_python-0.1.36-cp310-cp310-win_amd64.whl (128 kB)━━━ 128.7/12...
RUN pip install sentence-transformers --no-deps#This would be good in production but for debugging flexibility lets not add it right now#We need a more solid production ready entrypoint.sh anyway##ENTRYPOINT ["python", "-m", "llama_stack.distribution.server.server"]ADD ../../../../.....
Bug Description While running from llama_index import VectorStoreIndex, StorageContext, i got this error cannot import name 'VectorStoreIndex' from 'llama_index' (/usr/local/lib/python3.10/dist-packages/llama_index/init.py) Version 0.9.2...
llama-cpp-python-0.2.11 llama-cpp-python-cuda-0.2.11+cu117 lm-eval-0.3.0 markdown-3.4.4 markdown-it-py-2.2.0 matplotlib-3.8.0 mbstrdecoder-1.1.3 mdit-py-plugins-0.3.3 mdurl-0.1.2 multidict-6.0.4 multiprocess-0.70.15 ninja-1.11.1 numexpr-2.8.7 numpy-1.24.0 oauthlib-3.2.2 ...
Nuitka-Plugins:WARNING: anti-bloat: Undesirable import of 'setuptools' (intending to avoid 'setuptools') in 'torch.utils.cpp_extension' (at Nuitka-Plugins:WARNING: '/root/.pyenv/versions/3.11.6/lib/python3.11/site-packages/torch/utils/cpp_extension.py:27') encountered. It may slow down Nuitk...
To fix the issue with transformers not being okay with torch 2.1, i uninstalled it and did : pip install --pre xformers facebookresearch/xformers#870 (comment) the error is back: ImportError: DLL load failed while importing exllamav2_ext ...
Python Version (if applicable):3.10.12 PyTorch version:2.2.2 Container:nvcr.io/nim/meta/llama3-70b-instruct:1.0.0 Virtual server type (instance type) g5.24xlarge End Environment you can try installing libnvinfer10 usingsudo apt install libnvinfer10=${version} ...
File "/root/miniconda3/lib/python3.12/site-packages/vllm/model_executor/models/llama.py", line 345, in forward hidden_states, residual = layer(positions, hidden_states, ^^^ File "/root/miniconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_...