How a 26B Model Runs in 2 GB of RAM
Yesterday’s most-discussed project on Hacker News was not a model or an agent framework. It was an inference engine that runs Gemma 4 26B on an 8 GB MacBook Air using about 2 gigabytes of RAM.
TurboFieldfare is Swift and Metal, Apache 2.0, and does one thing: it streams mixture-of-experts weights from SSD instead of holding them in memory. It reached 730 points and 256 comments in a day, which for a single-purpose local inference engine is unusual, and the reason is that it lands precisely on the constraint the industry spent this month talking about. Memory is the scarce resource. This is what a software answer to that looks like.
The engineering is worth reading closely, because the interesting part is not the headline number. It is the set of trade-offs underneath it, all of which the author states plainly.
Executive summary
- Gemma 4 26B-A4B has 26B total parameters but activates only ~3.9B per token. A conventional runtime still loads all of them; this one does not.
- The resident set is the shared core, embeddings, attention and KV cache — about 1.35 GB at 4-bit — plus a small cache of expert weights. The model remains ~14.3 GB on disk.
- Expert selection has strong temporal locality: ~40% of experts repeat on the next token, 57% within two, giving ~67% cache hit rate with 16 slots.
- Explicit
pread()beatsmmap()by roughly 3.5× on cold expert loads (2.8 ms vs 10 ms), which compounds to 4 tok/s vs 0.50 tok/s end to end. - Measured throughput: 5.1–6.3 tok/s on an 8 GB M2 Air; 31–35 tok/s on a 24 GB M5 Pro. MLX does ~75 tok/s on the same machine using ~14 GB.
- The honest framing is not “faster.” It is “runs at all on hardware that otherwise could not.”
The observation the design rests on
A mixture-of-experts layer contains many expert subnetworks and routes each token to a few of them. Gemma 4 26B-A4B carries 26 billion parameters and uses roughly 3.9 billion per token. That ratio is the entire opportunity.
Conventional runtimes cannot exploit it, and for a good reason: routing is data-dependent. You do not know which experts token 47 will need until you have computed token 47. The safe response is to hold everything resident, which is what llama.cpp, MLX and vLLM all do.
The alternative is to treat memory as a cache rather than as storage. Keep resident what every token needs — embeddings, attention, routers, the KV cache. Keep on disk what only some tokens need. Fetch on demand.
That reframing is not new; operating systems have done it since the 1960s. What is new is that MoE sparsity gives it a hit rate high enough to be practical for neural network inference, and that Apple Silicon’s unified memory removes the CPU-to-GPU copy that would otherwise dominate the fetch path.
Why the cache hit rate is the whole ballgame
If expert selection were uniformly random, this design would be dead on arrival. Every token would require several SSD reads with nothing to amortise them against.
It is not random. The author’s measurements:
| Metric | Value |
|---|---|
| Experts repeating on the next token | ~40% |
| Experts repeating within two tokens | 57% |
| Cache hit rate with 16 expert slots | ~67% |
| Resident cache size | ~2–3.5 GB |
Two thirds of expert lookups never touch the disk. That locality is presumably semantic — consecutive tokens in a passage are about the same thing, and routers learn to send similar content to similar experts — though nobody has published a proper characterisation of it, and it is a genuinely interesting open question how it varies by domain, language and prompt structure.
The practical consequence is that the hit rate, not the disk speed, is the primary lever. Doubling SSD bandwidth improves the 33% miss path. Improving locality removes work entirely.
The mmap result deserves its own section
The most transferable finding in the project has nothing to do with machine learning.
The obvious way to back weights with disk is mmap: map the file, touch the memory, let the kernel fault pages in. It is simple, it is what llama.cpp does, and here it was roughly 3.5× slower per fetch and 8× slower end to end.
| Approach | Cold 3.36 MB expert load | End-to-end throughput |
|---|---|---|
mmap | ~10 ms | 0.50 tok/s |
pread() | ~2.8 ms | 4 tok/s |
The reason is scheduling. A page fault is an invisible, blocking, unbatched I/O operation. The thread stops, the kernel fetches one page-sized region at a time on the fault path, and the runtime has no opportunity to issue other fetches in parallel or overlap the wait with GPU work. An explicit pread is an operation the engine owns: it can issue several concurrently, size them to the actual expert, and pipeline them against Metal command buffers.
The general lesson is one that applies well outside inference: when I/O is on the critical path, an abstraction that hides it also prevents you from scheduling it. mmap is excellent when access is unpredictable and latency is not the constraint. It is the wrong tool when you know exactly what you need and need it overlapped.
The third technique closes the loop: prefill runs in chunks of up to 128 tokens, so a single fetched expert serves many rows before eviction. That is straightforward batching, applied to I/O rather than to compute.
What it actually costs
The author publishes the comparison that a less honest project would omit:
| Configuration | Throughput | RAM |
|---|---|---|
| M2 Air, 8 GB — streaming | 5.1–6.3 tok/s | ~2 GB |
| M5 Pro, 24 GB — streaming | 31–35 tok/s | ~2 GB |
| M5 Pro, 24 GB — MLX | ~75 tok/s | ~14 GB |
On hardware with the memory to spare, MLX is roughly twice as fast. That comparison is real and should not be waved away — several commenters made it, and one noted that under 30 tok/s makes genuinely interactive use uncomfortable.
But it is also the wrong comparison for the case that matters. On the 8 GB M2 Air there is no MLX row, because a 14 GB resident model does not fit. The baseline there is not slower inference. It is no inference.
Read that way, the result is a capability unlock for the low end of the hardware distribution rather than an efficiency win at the high end — which happens to be where most laptops actually are.
Where this does and does not apply
Reasonable uses: single-user local inference on memory-constrained machines, offline and privacy-sensitive work, on-device assistants, and CI or test environments that need a model present but not fast.
Poor uses: anything with concurrency. Multiple simultaneous requests route to unrelated experts, the hit rate degrades toward random, and read amplification erases the advantage. Server inference wants the opposite trade — weights resident, batching amortising them across requests. This design optimises for a batch size of one.
Does not apply at all: dense models. With no routing sparsity there is no subset to stream. Every parameter participates in every token, and the resident set is the model.
The stated limitations are worth repeating because they are easy to skim past: text only, one specific checkpoint, no tool calling, inference only, and the usual caveat that the model can repeat itself or be wrong. This is a demonstration of a technique, not a product.
Why it matters beyond one repository
Zoom out and this is the same story as the memory shortage, viewed from the software side.
DRAM is scarce and expensive because AI demand redirected wafer capacity toward HBM, and that scarcity is now showing up in cloud bills and consumer device prices alike. The hardware response takes years — fabs, packaging, and now optical interconnect — because none of it can be conjured in a quarter.
The software response is available immediately, and it has a consistent shape: stop treating memory as free. Quantization reduced bytes per parameter. Sparse architectures reduced parameters per token. KV-cache compression reduced bytes per token of context. Expert streaming takes the next step and reduces resident parameters to the ones currently in use, spending SSD bandwidth — which is abundant and cheap — to buy DRAM, which is neither.
Two things follow for anyone building on local models. First, active parameters and resident footprint are becoming genuinely different numbers, and a model’s memory requirement is turning into a property of the runtime rather than of the checkpoint. Second, the storage hierarchy has quietly grown a new tier. We spent years reasoning about GPU memory versus system memory; SSD is now in that hierarchy for weights, with its own hit rates and locality patterns to design around.
That is a more durable idea than any single engine, and it is why a Swift project locked to one Gemma checkpoint was the most interesting thing on Hacker News yesterday.
Sources
- TurboFieldfare on GitHub — Apache 2.0, README benchmarks and design notes
- Show HN discussion (730 points, 256 comments) — author’s
preadvsmmapmeasurements and cache-hit figures
Related reading on this site: how to speed up LLM inference, how to run AI locally on your computer, and why memory, not GPUs, is the bottleneck.
Frequently asked questions
How can a 26B parameter model run in 2 GB of RAM?
By exploiting the fact that a mixture-of-experts model does not use most of its parameters on any given token. Gemma 4 26B-A4B has 26 billion total parameters but activates roughly 3.9 billion per token, because each layer routes to a small subset of its experts. A conventional runtime still loads all 26 billion into memory, since it cannot know in advance which experts a future token will need. The streaming approach inverts that: it keeps the always-needed parts resident — the shared core, embeddings, attention, and the KV cache, about 1.35 GB after 4-bit quantization — and treats the expert weights as a disk-backed cache, fetching only the experts each token actually routes to. The model still occupies about 14.3 GB on disk. What shrinks is the resident set, not the model.
Why is this fast enough to be usable at all?
Because expert selection is far from random between consecutive tokens. The author measured roughly 40% of experts repeating on the immediately following token and 57% repeating within two tokens, which yields about a 67% cache hit rate with 16 expert slots held in memory. That locality is what makes the design viable: two thirds of the time no disk read happens at all, and the remaining third overlaps with GPU compute. Without that temporal locality you would be paying an SSD round trip for every expert on every token and the throughput would collapse.
Why use pread() instead of mmap() for the weights?
Because mmap makes the fault path invisible and therefore unschedulable. With mmap, touching an unloaded page traps into the kernel and blocks the thread until the page arrives, so the runtime cannot overlap that wait with anything else or issue several fetches in parallel. Explicit pread calls make each fetch an operation the engine owns and can pipeline against GPU work. The measured difference is stark: a cold 3.36 MB expert load took roughly 10 ms via mmap against 2.8 ms via pread, which translated to 0.50 versus 4 tokens per second end to end.
What is the actual performance cost compared to a normal runtime?
Substantial, and openly stated. On the same hardware where MLX runs the model at about 75 tokens per second using roughly 14 GB of RAM, the streaming engine reports 31–35 tokens per second on an M5 Pro with 24 GB, and 5.1–6.3 tokens per second on an 8 GB M2 MacBook Air. So the trade is roughly half the throughput on a well-provisioned machine, and about a twelfth of it on a machine that could not have run the model at all. That last case is the point.
Does this technique generalize to other models?
The mechanism generalizes to any sparse mixture-of-experts model, which is most large open-weight releases in 2026, but this particular implementation does not. It is locked to a single Gemma 4 26B-A4B checkpoint, is text-only, exposes no tool calling, and does inference only. The underlying ideas — resident shared core, disk-backed expert cache, explicit pipelined reads, chunked prefill — are portable. What does not generalize is dense models: with no routing sparsity there is no subset to stream.
Is the bottleneck the SSD, and does this wear the drive out?
The bottleneck is SSD read latency and bandwidth on cache misses, which is why the hit rate and the read path get so much engineering attention. On modern NVMe storage a few megabytes read at microsecond-to-millisecond latency is affordable when it overlaps with compute; on a slow external drive this design would not work. Drive wear is not a serious concern because the workload is read-only and SSD endurance limits are governed by writes. The relevant costs are latency, bandwidth contention, and battery drain from sustained I/O.
Should I use this in production?
For a server workload, no. Concurrency is where this design breaks down: multiple simultaneous requests route to different experts, the cache hit rate falls toward random, and read amplification eats the gains. Server inference wants the opposite trade — weights resident, batching amortising them across many requests. Where it does make sense is single-user local inference on memory-constrained hardware: on-device assistants, offline use, privacy-sensitive work, and CI environments where a model must be present but throughput is not critical.