Knowledge Share

Open Source Project #174: AirLLM — Run 70B Models on 4GB GPU, 405B on 8GB, and 2.8-Trillion-Parameter Kimi K3 on 3.7GB

AirLLM breaks the VRAM wall with layer-wise inference: splits transformer models into per-layer shards on disk, loads one layer at a time during inference, releases memory immediately after. No quantization, distillation, or pruning required. Runs 70B to 2.8T parameter models on consumer GPUs. Supports Llama 4, Qwen3, DeepSeek-V3/R1, Kimi K3. Optional 4bit/8bit block-wise compression for ~3x speedup. 25.8k Stars, Apache 2.0, pip install airllm.