NVIDIA Nemotron 3 Ultra: 550 billion parameters, 55 billion active per token

NVIDIA released its 550-billion-parameter model on Oct 19. Only 55 billion parameters are used for each token, and the rest do not participate in that token. The architecture is a hybrid of Mamba and Transformer with Latent MoE and MTP layers, and pretraining ran in NVFP4, a 4-bit number format.