4-bit vs 8-bit on a 7B model: what I actually lose
I wanted a number I could trust for my own tasks rather than a leaderboard delta, so I ran the same 7B instruction model at three precisions on one 12 GB GPU.
| Precision | VRAM | Decode tok/s | Perplexity | My task suite |
|---|---|---|---|---|
| FP16 | 14.2 GB* | — | 5.91 | baseline |
| 8-bit | 7.9 GB | 38 | 5.93 | −0.4 pts |
| 4-bit (group 128) | 4.6 GB | 61 | 6.12 | −2.1 pts |
* FP16 did not fit; measured with partial CPU offload, so no fair speed number.
For chat-style use the 4-bit model is fine. For structured extraction the 2-point drop showed up as more malformed JSON, so I keep 8-bit for that job.