vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported3 Subscribers
Add a CodeTriage badge to vllm
Help out
- Issues
- [Installation]: ImportError: cannot import name 'LogicalTokenBlock' from 'vllm.block'
- [Bug]:vllm backed with triton server is not working
- [Bug]: RuntimeError: GET was unable to find an engine to execute this computation for llava-next model
- Update logits processor with tensor caching
- [Bug]: temperature=0 does not lead to Greedy Sampling
- [Feature]: vllm support for Ascend NPU
- [Bug]: vllm-0.5.3.post1部署Qwen2-72b-instruct-awq模型,刚开始服务正常,但是并发高的时候就报错
- [Bug]: custom docker Error
- [Doc]: Outdated docs on AutoAWQ
- [Usage]: How to disable logging
- Docs
- Python not yet supported