Run 70B LLM on 4GB GPU: AirLLM's Real Tradeoff

“Run 70B LLM on 4GB GPU” reads like a scam headline. It isn’t one. Not a quantized 70B, not a distilled one — the actual FP16 checkpoint, on a card that costs less than a nice dinner. AirLLM does it by keeping exactly one transformer layer in VRAM at a time, and the project hit the Hacker News front page again this week with 142 points and a comment section split cleanly between “impressive engineering feat” and “so what.”

Both camps are right, and the reason why is the most useful thing in the whole story. AirLLM doesn’t remove a constraint. It moves one — out of VRAM, where you were stuck, and into disk bandwidth, where most people have never measured their own hardware.

TL;DR

• The memory math is honest. Peak VRAM tracks your largest single layer (~1.7 GB for a 70B at FP16), not the ~140 GB of total weights.
• The bill converts, it doesn’t vanish. Every forward pass streams the whole model past the GPU — a hard floor near 20 s/token on a 7 GB/s Gen4 NVMe, before any compute.
• HN commenters reported far worse: ~292 s/token on an RTX 6000 Ada, and 416 days for a 108k-token job. Anecdotes, but directionally right.
• Your bottleneck is RAM and storage, not the GPU. A warm page cache moves the numbers more than a better card does.
• Use it for offline batch work, where the per-pass cost amortizes across the batch. For anything interactive, a 4-bit quant under llama.cpp wins easily.

What is layer-by-layer inference?

Layer-by-layer inference is the technique of holding exactly one transformer layer in GPU memory at a time — loading its weights, running the forward pass for that layer, freeing it, and moving to the next — so peak memory scales with the largest single layer instead of the whole model.

That’s the entire idea, and it works because a transformer forward pass is strictly sequential. Layer n needs only the activations produced by layer n−1. It does not need layer n−1’s weights still sitting in memory, and it does not need layer n+1’s weights yet. Standard inference keeps all of them resident anyway, because moving weights is expensive and everyone assumed you’d rather buy VRAM than pay that cost.

AirLLM takes the other side of that bet. On first use it decomposes a checkpoint into per-layer shards on disk, then streams them through the GPU one at a time:

from airllm import AutoModel

model = AutoModel.from_pretrained("Qwen/Qwen3-32B")

input_tokens = model.tokenizer(['What is the capital of United States?'],
    return_tensors="pt",
    return_attention_mask=False,
    truncation=True,
    max_length=128,
    padding=False)

generation_output = model.generate(
    input_tokens['input_ids'].cuda(),
    max_new_tokens=20,
    use_cache=True,
    return_dict_in_generate=True)

No custom serving stack, no model surgery. The README claims the same approach reaches 405B Llama 3.1 on 8 GB and 671B DeepSeek-V3 on roughly 12 GB — the larger sparse models fare better per parameter because mixture-of-experts layers let AirLLM stream one expert at a time instead of a full dense layer.

Why “run 70B LLM on 4GB GPU” is literally true

Run the arithmetic, because it’s the part people assume must be marketing.

A 70B model at FP16 is about 140 GB of weights. Llama-class 70B architectures use 80 decoder layers, so the per-layer share is roughly 1.7 GB. Add activations for a short sequence and a small KV cache and you’re still comfortably inside 4 GB. There’s no sleight of hand: the peak resident set really is one layer, and one layer really does fit.

💡 Key insight: VRAM requirements are a property of your scheduling, not of the model. The 140 GB figure everyone quotes is just the cost of the laziest possible schedule — keep everything, forever.

This is the same trick that shows up everywhere in systems engineering once you go looking. Streaming instead of buffering, working sets instead of totals. It’s the memory-layout thinking that let Node.js cut heap usage in half with pointer compression — different layer of the stack, identical insight: the bytes you must hold simultaneously are the only bytes that constrain you.

The disk-bandwidth wall

Here’s where the enthusiasm should stop.

Autoregressive generation runs one forward pass per output token. Under layer streaming, that means the entire model crosses the storage bus once per token. Not once per request — once per token.

That converts into a floor you can compute before installing anything:

Those are floors — pure transfer time, assuming zero overhead and zero compute. Reality is worse, and the HN thread is full of it: one commenter reported roughly 292 seconds per token on an RTX 6000 Ada, about 0.003 tokens/second, and another worked out that a 108k-token output would take 416 days at that rate. Those are unverified single reports rather than a benchmark suite, but nothing about the mechanism makes them implausible.

The project’s own numbers confirm the diagnosis from the other direction. AirLLM documents a 3× speedup from optional 4-bit block-wise compression and about 10% from prefetching that overlaps loading with computation. A speedup that tracks the reduction in bytes moved is the signature of a workload that is I/O-bound end to end. You are not running a GPU workload with a storage component. You are running a storage workload that occasionally touches a GPU.

Your real bottleneck is RAM, not VRAM

The most useful thing I can tell you about AirLLM’s performance is that the GPU is nearly irrelevant to it.

Those per-layer shards are ordinary files. Read them repeatedly and the operating system’s page cache starts serving them from RAM at tens of GB/s instead of from SSD at single digits. So two machines with the same 4 GB card produce wildly different results: 16 GB of system RAM means every token re-reads cold from disk, while 128 GB means most of a 4-bit-compressed 70B lives in cache and the streaming cost partially evaporates.

Which reframes the shopping list. If you’re determined to make this work, more system RAM buys you more than a bigger GPU does — the exact inverse of the advice that applies to every other local-inference setup, including the Qwen3-Coder desktop workflows I wrote about earlier.

Where layer streaming genuinely wins: batch

There is one regime where the economics stop being absurd, and the community discussion mostly skipped past it.

The streaming cost is paid per forward pass, not per sequence. Batch 32 prompts together and that same ~140 GB read produces 32 tokens instead of one. The amortized per-token cost drops by 32×, and suddenly the storage bus is being used the way bulk hardware is supposed to be used.

That makes AirLLM a defensible choice for a narrow, real set of jobs:

1. Offline classification or labelling over a large dataset, where you want the full-precision model’s judgment and don’t care if the run takes overnight.
2. Evaluation harnesses — scoring a fixed test set against an unquantized checkpoint to establish a fidelity baseline your quantized production model is measured against.
3. One-shot access to a model your hardware genuinely cannot host any other way, when renting a GPU isn’t an option for data-residency reasons.
4. Synthetic data generation where throughput matters and latency doesn’t at all.

The constraint on all four: your batch’s activations and KV cache still have to fit in that 4 GB alongside the resident layer. Long contexts eat the batch size that makes the whole scheme work, so short prompts batch beautifully and 32k-token documents don’t.

Four ways people misread this project

“It’s free 70B inference.” It’s traded, not free. You paid in VRAM before; now you pay in wall-clock time and SSD write endurance. Decomposition also needs full-model-sized disk space before you generate a single token.

“So it’s better than quantization.” Different axis entirely. Quantization trades accuracy for both memory and speed; layer streaming trades speed for memory while keeping accuracy exactly intact. If you want a local assistant, a 4-bit 70B under llama.cpp with hybrid GPU/CPU offload wins on every metric you’ll actually feel. AirLLM’s unique claim is zero quantization error — that’s the whole pitch, and it only matters when fidelity to the original weights is the point.

“The 292 s/token report proves it’s useless.” It proves it’s useless for chat, which was never a serious claim. Judge a batch tool on throughput per dollar, not latency.

“Bigger models are strictly worse here.” Sparse ones aren’t. MoE architectures activate a fraction of their parameters per token, and AirLLM streams experts individually — which is why 671B DeepSeek-V3 at ~12 GB is a less ridiculous proposition than dense 70B at 4 GB. The trend toward sparsity that made DeepSeek’s efficiency gains possible works in layer streaming’s favour too.

Should you use it?

Before you try to run 70B LLM on 4GB GPU hardware, decide with one question: is this workload interactive?

If yes, stop reading and go quantize. If no — if it’s a queue, a dataset, an eval run — then work down this list before you commit:

1. Enable 4-bit compression. It’s the single biggest lever, roughly 4× fewer bytes per token, and the project reports ~3× real speedup.
2. Measure your storage first. Run a sequential read benchmark and divide your model size by it. That number is your optimistic ceiling; if it’s unacceptable, nothing downstream will save you.
3. Max out system RAM before touching the GPU. Page-cache hits are the difference between “overnight” and “next week.”
4. Batch as hard as your context length allows. This is the only thing that moves the cost-per-token by an order of magnitude.
5. Compare against the boring option. A rented A100 for two hours often beats a week of local streaming on both cost and carbon. Do that math honestly before you fall in love with the constraint.

If you’re building a serving path rather than a one-off job, the architectural lesson generalizes past this one library — the same “what must be resident, and when?” discipline is what separates a RAG pipeline that scales from one that thrashes.

FAQ

Can you really run a 70B LLM on a 4GB GPU?

Yes, and the memory arithmetic is not a trick. AirLLM keeps exactly one transformer layer resident in VRAM at a time, so peak GPU memory scales with the largest single layer — roughly 1.7 GB for a 70B model at FP16 — rather than the full ~140 GB of weights. What it does not do is make that model fast: every forward pass has to stream the entire model past the GPU again.

How does AirLLM’s layer-by-layer inference work?

AirLLM decomposes a checkpoint into per-layer shards on disk, then during generation loads layer 1 into VRAM, runs it, frees it, loads layer 2, and so on to the end of the stack. Activations flow forward between layers while the weights are transient. For sparse mixture-of-experts models it goes finer still and streams individual experts rather than whole layers, which is why a 671B MoE can need less streaming bandwidth per token than a 70B dense model.

How slow is AirLLM in practice?

Slow enough that it changes what the tool is for. A dense 70B at FP16 means moving about 140 GB per forward pass, so even a Gen4 NVMe at 7 GB/s puts a hard floor near 20 seconds per token before any compute. Commenters on the Hacker News thread reported far worse in real runs — one measured roughly 292 seconds per token on an RTX 6000 Ada. It is a batch-processing tool, not a chat tool.

Is AirLLM better than quantization or llama.cpp offloading?

For interactive use, no — a 4-bit quantized 70B running under llama.cpp with layers split across GPU and CPU will beat it decisively on tokens per second. AirLLM wins on exactly one axis: it runs the unmodified FP16 weights, so there is no quantization error at all. Choose it when fidelity to the original checkpoint matters more than latency, not when you want a local assistant.

What actually determines AirLLM’s speed on my machine?

Not your VRAM — your storage bandwidth and your system RAM. Once the per-layer shards are warm in the OS page cache, the reads come from RAM instead of disk and throughput improves dramatically, so a box with 128 GB of RAM behaves very differently from one with 16 GB on the same GPU. Enabling 4-bit compression helps for the same reason: it cuts the number of bytes moved per token.

When is layer-by-layer inference actually the right choice?

When the work is offline, batchable, and latency-insensitive — bulk classification, dataset labelling, offline evaluation, or a one-off run against a model you cannot otherwise fit. Because the streaming cost is paid once per forward pass rather than once per sequence, large batches amortize it across many tokens, which is the only regime where the economics are defensible.

Final take

The dismissive HN read — 0.003 tokens/second, therefore a toy — measures the wrong thing. The interesting output of this project isn’t a chatbot. It’s a demonstration that the VRAM number printed next to every open-weights release is a scheduling artifact, and schedules are negotiable in a way silicon is not.

That matters right now, because the gap between what open weights offer and what consumer hardware can hold keeps widening. Every technique that trades a resource you can’t buy for one you can — RAM, disk, time — buys the local-inference community another release cycle of relevance. AirLLM trades badly for chat and reasonably for batch, and knowing precisely which is which is worth more than the library itself.

Measure your own disk before you form an opinion. Most people arguing about this have never checked the number that decides it.

Sources

• lyogavin, AirLLM — 70B inference with a single 4GB GPU — GitHub README (memory claims, compression and prefetching figures, quickstart code)
• AirLLM 70B inference with single 4GB GPU — Hacker News discussion, August 2026 (142 points; the 292 s/token and 416-day figures are commenter reports, not benchmarks)
• llama.cpp — the quantized-plus-offload baseline this approach should be compared against