A faster model does not always require a new model.
Google says it improved Qwen 3.5 397B inference on its Ironwood TPU platform by approximately 4.7x for prefill-heavy workloads and 3.1x for decode-heavy workloads between April and June 2026. The weights did not become smaller, and the system did not replace Qwen with a new frontier checkpoint. Engineers changed how the model was sharded, routed, fused, scheduled, and stored on the hardware.
That distinction matters. Training a new large model consumes enormous data and compute. Systems optimization can unlock more work from an existing checkpoint, improve latency, and reduce cost per token without repeating pretraining.
For teams serving stable, high-volume workloads, that can be more valuable than another model release.
Why Qwen 397B is difficult to serve
Qwen 3.5 397B is a sparse mixture-of-experts model with approximately 397 billion total parameters and 17 billion active parameters per token. Its 60 layers combine Gated DeltaNet linear attention with conventional grouped-query attention in a 3:1 pattern. Each MoE layer contains 512 routed experts and selects ten for each token, plus a shared expert.
The sparse activation reduces compute relative to a dense 397B model, but it creates a distributed-systems problem. The full weight footprint is still roughly 400GB. Tokens must reach the right experts, results must return to the correct request, recurrent state must be updated, and the KV cache must remain available under high concurrency.
Google tested the model on one host containing four physical Ironwood chips, exposed as eight logical devices. Its technical report separates two workloads:
| Workload | Input / output | Dominant limit |
|---|---|---|
| Prefill | 8K / 1K tokens | matrix compute and routing |
| Decode | 1K / 8K tokens | HBM memory bandwidth |
This separation is essential. An optimization that speeds prompt processing may do little for long token generation. Reporting one blended tokens-per-second number can hide the bottleneck a real application will encounter.
Figure: Long prompts stress matrix compute and routing, while token-by-token generation repeatedly streams model weights and becomes memory-bound.
The original sharding plan did not fit the model
The grouped-query attention layers use 32 query heads but only two key-value heads. A conventional eight-way tensor-parallel split cannot divide two KV heads evenly across eight devices. Replicating the heads on every device wastes KV-cache memory and limits concurrency.
Google instead used a hybrid topology:
- Attention data parallelism: each device keeps the complete attention heads and processes a different batch slice.
- Expert parallelism: the 512 routed experts are distributed across the eight logical devices, 64 per device.
Moving between those layouts requires communication. Rather than dynamically sending each token only to the device containing its selected experts, Google used an all-gather operation to replicate token information, performed local expert work, and then used reduce-scatter to return the combined result.
That choice performs some redundant data movement, but it produces more deterministic latency than a variable all-to-all exchange. The team then reduced the cost with packed routing metadata and a hierarchical reduce-scatter that uses faster local chiplet links before crossing physical chips.
Figure: The performance gain came from coordinated changes across topology, communication, kernels, quantization, and memory layout rather than one isolated trick.
Kernel fusion removed repeated trips to memory
Figure: Accelerators are only one constraint; power, cooling, networking, packaging, storage, and capital determine usable model capacity. Yield Signal Daily editorial diagram.
Qwen’s Gated DeltaNet layers update a recurrent state and use a causal 1D convolution. A naive implementation writes intermediate values to memory between operations. On a decode-heavy workload, those round trips add latency to every generated token.
Google created Pallas kernels that fused the convolution and recurrent update into one execution unit and kept intermediate state in local registers. The team also rearranged equations to skip an expensive matrix multiplication and changed recurrent state variables from Float32 to BFloat16, reporting no loss of numerical convergence or output quality in its verification path.
Other optimizations included:
- increasing KV page size to reduce indexing stalls;
- batching ragged page-attention operations;
- using SparseCore for irregular token routing and gather operations;
- fusing SwiGLU activation into grouped matrix multiplication;
- packing expert indices and routing weights into fewer collective operations;
- aligning GDN state and KV-cache layouts to reclaim HBM capacity.
This is why the result is a systems-engineering story rather than a TPU marketing benchmark. The architecture was profiled as a set of bottlenecks, and each bottleneck received a hardware-aware fix.
The reported performance needs precise interpretation
At concurrency 512, Google reports approximately 4.7x improvement for its prefill-heavy workload and 3.1x for decode-heavy work compared with its April baseline. At concurrency 64, the optimized stack reached 3,707 tokens per second per chip for prefill and 677 for decode.
Google estimates those results at 82.4% of its discounted prefill roofline and 79.6% of the decode roofline. Those percentages are based on Google’s own theoretical model, hardware, software stack, and workload assumptions. They should not be interpreted as universal Qwen performance or a direct price comparison with a GPU deployment.
The benchmark also tests a single-host topology with high concurrency. A small application with bursty traffic, short prompts, or strict per-user latency may see a different economic result. Cost depends on utilization, reservations, engineering time, model availability, and the ability to keep batches full.
Why a team might choose TPU for this workload
Figure: Lower infrastructure cost expands provider options, while competition, scarcity, segmentation, and data value determine customer pricing. Yield Signal Daily editorial diagram.
For a stable, high-volume Qwen serving deployment, TPU is an understandable choice. The hardware, high-speed interconnect, XLA compiler, JAX runtime, and Pallas kernel language are controlled as one platform. That vertical integration lets engineers reason about topology and write kernels around specific memory and communication behavior.
The strongest reasons are:
- Predictable large-scale topology. TPU pods and the accompanying compiler stack are designed around distributed matrix workloads rather than assembled from unrelated components.
- High HBM capacity and bandwidth. Large sparse models are often limited by moving weights and states, especially during decode.
- Compiler-level control. JAX and Pallas expose paths for custom fusion, layout, quantization, and collective communication.
- Managed operational surface. A team already on Google Cloud can standardize provisioning, telemetry, and deployment around one vendor.
That preference has conditions. GPU remains the safer default for many smaller teams because the CUDA ecosystem is broader, model implementations arrive quickly, and infrastructure is available across multiple providers. Choosing TPU means accepting a more specialized toolchain and some platform dependence.
The decision should follow the workload, not allegiance. For experimental models and heterogeneous tools, GPU portability may dominate. For a stable model serving millions of similar requests, the efficiency of a vertically optimized TPU stack may justify the lock-in.
Self-hosting is still a capital decision
An open-weight license does not make a 397B model cheap to operate. Even with 17B active parameters per token, the complete model must be placed across hundreds of gigabytes of accelerator memory. Production service also requires redundancy, networking, monitoring, secure storage, deployment automation, and staff able to debug distributed inference.
A company with sufficient capital may still prefer self-hosting for sensitive data, predictable volume, and control over retention. Sending every prompt to a third-party API can be operationally simpler, but the organization must accept the provider’s security model, pricing, rate limits, and product changes.
Smaller quantized models reduce the threshold but also change quality. Tool calling, instruction following, long-context stability, and domain-specific accuracy can degrade before an average benchmark makes the loss obvious. A serious self-hosting decision therefore needs an evaluation set built from the company’s own work, not only a model card.
Optimization can beat a new-model migration
Google’s Qwen result shows that model capability and product performance are different engineering layers. A checkpoint can remain unchanged while its useful throughput moves several times higher.
That is economically important. More efficient serving can reduce the accelerators required for the same traffic, make longer contexts practical, or free capacity for additional users. It also avoids the data and compute cost of creating a new base model merely to improve operational performance.
The next meaningful AI advantage may not always arrive as a new model name. It may arrive as a better sharding plan, a fused kernel, a smaller memory round trip, or an evaluation that finally reveals where the existing system was wasting most of its compute.
