LLM Engineering •
Qwen3.8 27B VRAM: how to fit 262K context in 16 GiB, not 64
Qwen3.8 27B VRAM math: 25.9 GiB of FP8 weights plus 16 GiB of KV cache at 262,144 tokens, not 64. The arithmetic, and where a 48 GB card breaks.
9 min read
Read more →
LLM Engineering •
Fix slow LLM inference in macOS VMs: 12.6 → 207 tok/s
LLM inference in macOS VMs collapses to 12.63 tok/s because the guest reports GPU family 5 and llama.cpp drops its matrix kernels. The check, and its limits.
9 min read
Read more →
LLM Engineering •
vLLM throughput tuning: configure these four flags, not a bigger GPU
vLLM throughput tuning starts with KV cache blocks, not a bigger GPU. The four flags that decide your tokens/sec, and the one that quietly backfires.
10 min read
Read more →
LLM Engineering •
Run 70B LLM on 4GB GPU: AirLLM's Real Tradeoff
Run 70B LLM on 4GB GPU hardware with AirLLM's layer-by-layer inference. The VRAM math is real — you just pay for it in disk bandwidth. The honest tradeoff.
11 min read
Read more →