⚙️

SGLang

LLMs

High-performance serving framework for LLMs and multimodal models.

15kstars1.8kforksPython

About

SGLang pairs a fast runtime with a structured generation language, and has become a de-facto standard for high-throughput LLM inference — reportedly powering deployments across hundreds of thousands of GPUs.

Key Features

RadixAttention
Structured generation
High throughput
Multi-GPU

Related Resources

🦜

LangChain

Framework for building LLM-powered applications using composable primitives.

🦙

llama.cpp

Run LLaMA-class models efficiently on consumer hardware.

🦙

Ollama

Simple local LLM runtime with a clean developer experience.