LLM Engineering •
How to Decide: Loop Transformer Blocks or Add More Layers
Loop transformer blocks or add more layers? Reusing weights cuts training compute 6.8-18% at equal loss, but forward passes and KV cache never shrink.
10 min read
Read more →