vllm
https://github.com/vllm-project/vllm
Python
A high-throughput and memory-efficient inference and serving engine for LLMs
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported4 Subscribers
Add a CodeTriage badge to vllm
Help out
- Issues
- [Feature]: Dockerfile.cpu for aarch64
- [Bug]: GPU Memory Utilization Lower Than Expected with --enable-prefix-caching
- [Feature]: BitsandBytes quantization with TP>1
- Reshape cache flash kernel to support HND layout
- Add VLLM_LOGGING_INTERVAL_SEC envvar to control logging rate
- [Bug]: vLLM 0.5.5 using prefix caching causing CUDA error: illegal memory access
- [Feature]: Allow partial context in speculative decoding when using draft models with smaller context than target model
- [Bug]: watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
- [Bug]: AssertionError: Logits Processors are not supported in multi-step decoding
- [Bug]: (OOM) Find two places that cause a significant increase in GPU memory usage (probably lead to memory leak)
- Docs
- Python not yet supported