Cerebras Works Out a 3-Trillion-Parameter Deployment: 59 Wafers

What happened

The budget starts from per-wafer capacity: one Cerebras WSE-3T wafer holds 44GB of on-chip memory, while a trillion-parameter model's weights alone need terabytes. Cerebras's answer is to split the model by layer and place whole layers directly in on-chip memory, so weights stop moving during inference.

At 4-bit quantization, the raw weight payload of 3 trillion parameters divided by 93 layers gives about 16GB per layer. Layer type decides how many layers fit per wafer: linear-attention layers store no key-value cache, so two fit on one wafer; dense layers need cache room for every request in flight, so one layer takes a whole wafer with the rest left for cache. The 69 linear layers take 35 wafers and the 24 dense layers take 24, for 59 total.

The wafers connect as a pipeline: each wafer is one stage that hands its output to the next, weights stay put, and only activations travel over RDMA over Ethernet at about 2 microseconds per hop. Different requests occupy different pipeline stages at the same time, and Cerebras says this layout supports 1,000 to 2,000-plus tokens per second for multi-trillion-parameter models.

Key facts

1. The worked example is Moonshot AI's open Kimi K3, roughly 3 trillion parameters and 93 layers — 69 linear-attention layers and 24 dense layers; the post calls it an illustrative example.
2. Calculation assumptions: 4-bit weight precision, an 8-bit KV cache, and an assumed average request size of 256k tokens; each layer is about 16GB and one WSE-3T wafer holds 44GB on-chip.
3. Layer mapping: linear-attention layers fit two per wafer (2×16GB=32GB), so 69 layers take 35 wafers; dense layers take one per wafer with room left for KV cache, so 24 layers take 24 wafers; total 59.
4. Pipeline mechanics: weights stay on-chip while only activations move between wafers over RDMA over Ethernet at about 2 microseconds per hop; multiple requests sit at different pipeline stages at once, and micro-batches can raise total throughput further.
5. The speed figure is a company claim: the post pitches 1,000 to 2,000-plus tokens per second for multi-trillion-parameter models, but it does not attach measured performance, power draw or concurrent-request counts for the 59-wafer layout, so the figure stands as a company claim.