⚡

vLLM

LLMs

High-throughput, memory-efficient LLM serving engine.

45kstars7kforksPython

About

vLLM is designed for fast LLM inference and serving using PagedAttention. Widely used for production-scale deployments with an OpenAI-compatible API and support for multi-GPU setups.

Key Features

PagedAttention
Fast inference
OpenAI-compatible API
Multi-GPU

Related Resources

🦜

LangChain

Framework for building LLM-powered applications using composable primitives.

🦙

llama.cpp

Run LLaMA-class models efficiently on consumer hardware.

🦙

Ollama

Simple local LLM runtime with a clean developer experience.