The gap between the models being published and the machines most people own has been the whole problem with open weights this year. Two releases this week narrow it from opposite directions.
FreeToken, out Friday, is what its README calls an edge-native mixture-of-experts serving engine, from researchers at UC Berkeley and MIT with Ion Stoica, Matei Zaharia, Song Han and Kurt Keutzer among the authors, Apache 2.0 on GitHub as FlashML-org/FreeToken, paper at arXiv 2608.16157. The idea is scheduling rather than compression. Static expert offloading stalls the GPU whenever a needed expert is not resident, so FreeToken uses what the paper calls a q star policy, splitting each token's computation between CPU cores and GPU tensor cores based on measured PCIe throughput, with a fast weight format and full-layer double buffering so weight streaming overlaps computation, and an elastic memory manager that moves VRAM between KV cache and expert slots at runtime. InfoQ reports the numbers: Qwen3.6-35B at about 39 tokens per second on an 8GB RTX 4060 laptop, DeepSeek-V4-Flash at 284 billion parameters on an RTX 5090 desktop, GLM-5.2 at 753 billion on a single workstation GPU, and 3 to 4 times faster decode with 6 to 30 times faster prefill against Ollama and llama.cpp. Support covers RTX 30, 40 and 50 series on Linux and Windows, with MXFP4, NVFP4, FP8 and BF16 weights.
The other direction is quantization. Unsloth published a 3-bit GGUF of GLM-5.3-Flash on Thursday, the model Z.ai revealed this week as the Ox Alpha that sat at the top of OpenRouter for six days. AI Times reports the file at roughly 120GB, about 81 percent smaller than the full-precision release, running on a 128GB machine such as a Mac Studio or a workstation with no GPU required, and Unsloth claims 82 percent of the BF16 model's performance using layer-selective quantization with KL divergence monitoring to keep the output distribution close. That claim is the vendor's own, as is Z.ai's model card line that GLM-5.3-Flash approaches Claude Opus 4.8 on coding and agentic benchmarks.
Put the two together and the practical picture changes. A 320 billion parameter multimodal model with MIT weights, an engine that treats a consumer GPU and its host CPU as one scheduling problem, and a quantization that fits the whole thing in system RAM: the frontier tier is not local yet, but the tier below it now runs on a good desktop. What to watch is whether independent benchmarks reproduce the throughput claims, and how much quality the 3-bit path actually costs on real work rather than on perplexity.