vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported4 Subscribers
Add a CodeTriage badge to vllm
Help out
- Issues
- [Bug]: InternVL bounding box prediction does not work
- [Feature]: Does vLLM support ONNX models?
- [Performance] In v0.6.2, when tp=1, TPOT becomes very slow for batch sizes of 10 or so. (not happened in v0.5.5)
- [Bug] BlockSpaceManagerV1.get_common_computed_block_ids returns empty string, causing msgspec decode failure
- [Usage]: Not getting the infrence metrics in the api response
- [Bugfix][Core] Handle empty ids_list in BlockSpaceManagerV1.get_common_computed_block_ids to prevent msgspec serialization errors
- [Core]: (2/N) Support prefill only models by Workflow Defined Engine - Prefill only attention
- [RFC]: hide continuous batching complexity through forward context
- [Doc]: Clear documentation about function / tool calling with examples
- Developed the PoC of dAttention support. It will utilize the similar idea of vAttention, but it introduces a new memory layout that overcomes the waste of memory of vAttention.
- Docs
- Python not yet supported