Deepgram ports Nova-3 speech model to AWS Trainium2, boosting batch transcription audio-per-dollar by 11–70%

Deepgram first ran standard PyTorch Nova-3 on Trainium2 using TorchNeuron, confirming no operators fell back to CPU, then used torch.compile to compile the encoder and decoder into static graph shapes. They replaced fused operators in NKILib, including fused QKV projection, fused FFN, and depthwise separable 1D convolution, used Neuron Explorer to find bottlenecks and NAD tooling to rewrite kernels: the FFN kernel cut latency 2.0x at batch size 96, depthwise convolution sped up 1.5x, and a custom conv2d kernel let the model compile at batch sizes above 32. The final configuration runs 8 model replicas on a single trn2.3xlarge. The serving path bypasses Python and PyTorch: Deepgram wrote a work-stealing queue-driven data-parallel runtime in Rust that loads precompiled NEFFs directly via libnrt, measured at 7.5% higher throughput than a multiprocess Python baseline, with the biggest gains at low batch sizes. On throughput-bound workloads like offline batch transcription, this configuration processes 11–70% more audio per dollar than Deepgram's previous compute (based on instance pricing at evaluation time). Teams self-hosting Deepgram models on Amazon SageMaker AI in their own VPC see the same direct cost and deployment improvements.