AirLLM Runs 70B Inference on a Single 4GB GPU

AirLLM gained roughly 1,085 stars in a day and reached the Hacker News front page for running 70-billion-parameter inference on a single 4GB GPU through layered loading — streaming layers through limited memory rather than holding the model resident. It sits alongside a broader move this week toward systems-level local inference, including a C engine for DeepSeek 4 across Metal, CUDA and ROCm, and a LocalAI post arguing for hand-written C and C++ runtimes. Worth calibrating expectations: layered loading trades throughput for footprint, and the repository is primarily Jupyter notebooks rather than a serving stack.