Field notes on small models.
Running language and speech models on modest hardware, measuring what actually matters, and writing it down before I forget.
4-bit vs 8-bit on a 7B model: what I actually lose
Perplexity, a small task suite and tokens per second on a single consumer GPU.
Why my long prompts were slow: a KV-cache detour
Prefill vs decode, cache size arithmetic, and a 3× win from batching prompts that share a prefix.
Speech-to-text on a fanless mini PC
Real-time factor for a small speech model on CPU, and where the time goes.
Tiny evals that catch big regressions
Fifty hand-written test cases beat a leaderboard number for my use case.