i14 trains and deploys Laplace 3B linear attention model on AWS Trainium2 with native PyTorch

Laplace 3B is i14's 24-layer decoder model, each layer replacing softmax attention with Simple GLA linear attention that compresses history into a 96×192 matrix state per head. Sixteen heads use geometric decay schedules with half-lives from 1 to 2048 tokens. Training attention drops from O(T²) to O(T), inference is O(1) per token, and the per-sequence recurrent state stays fixed at 29.5 MB regardless of context length. To fit Trainium2, the team rewrote the GLA kernel with standard PyTorch ops, using 16 chunks of 128 tokens aligned to the 128 on-chip memory partitions of NeuronCore for a 14% throughput gain; under FSDP2 they replaced Muon orthogonalization with two all_to_all exchanges, cutting per-chip communication from 0.45 GB to 1/24 and raising throughput from 73,378 to 119,934 tokens/s; and statically unrolled the 16-step GLA recurrence to fit compiler graph limits. On one trn2.48xlarge instance (16 Trainium2 chips, 64 NeuronCores), Laplace 3B trains at ~120K tokens/s and serves 11,544 output tokens/s at 1280 concurrent sequences. Teams adopting this architecture should note that linear attention compresses rather than retains all key-values, so retrieval quality must be evaluated for their workload; developers deploying custom architectures on Trainium2 face tuning costs from software-managed on-chip memory and the static compiler.