K2-Horizon Opens Six Models: Flagship 375B MoE With Full Training Data
IFM's K2 Team published K2-Horizon on arXiv on Nov 16: six fully open language models, from a 0.9B dense model to a 375B-A23B mixture-of-experts, with training data, intermediate checkpoints and all code released alongside.

What happened
The fleet spans six sizes, from a 0.9B dense model to a 375B-A23B mixture-of-experts that activates 23B parameters per token; the models are built for general-purpose reasoning and agentic work across math and science, software engineering, deep research and office automation.
Two new techniques are introduced: MoVA, which applies mixture-of-experts to attention value projections, and softplus attention gating; the 36B-A4B model uses both and nearly matches a 32B dense model trained on the same data, with one-seventh the active parameters; Uno, a lossless diffusion adapter, speeds up decoding by up to 2.2x without a separate draft model.
The release follows the LLM360 definition of open source: the full development history, intermediate checkpoints from every training stage, pre-training data, training logs, and all training, evaluation and infrastructure code, including the production-scale xLLM training framework and the RL360 reinforcement-learning stack; four of the models share one 22T-token pre-training corpus and recipe, and roughly 10T synthetic tokens went into pre-training.
Key facts
- Six models: From a 0.9B dense model to a 375B-A23B mixture-of-experts, activating 23B parameters per token
- MoVA: Mixture-of-experts applied to attention value projections, plus softplus attention gating
- 36B-A4B: Nearly matches a 32B dense model trained on the same data, with about one-seventh the active parameters
- Uno: Lossless diffusion adapters speed up decoding by up to 2.2x without a separate draft model
- Corpus: Four models share one 22T-token pre-training corpus and recipe; roughly 10T synthetic tokens went into pre-training
- Open scope: Intermediate checkpoints, pre-training data, training logs, and the xLLM and RL360 framework code
- Release: arXiv:2611.21003, submitted Nov 16, 2026; artifacts at ifm.ai/k2 and on Hugging Face
Why it matters
all six models, from 0.9B to 375B, are fully open, four share a 22T-token corpus and one training recipe, and about 10T synthetic tokens went into pre-training.
Follow HIMIAO Horizon
No sign-up · Unsubscribe any timeNotifications
A ping on your phone or computer when the brief is out. No email needed.
Comments
Related
- Nov 16Nvidia Launches Nemotron for Science, Bringing AI Agents to the Lab
- Nov 17Ant's LingBot-World World Model Goes Open Source, Generating Nearly 10 Minutes of Video in One Take
- Nov 16Huawei Cloud Launches Ascend-Native Embodied World Model CloudRobo-WM1.2
- Nov 15Meituan Open-Sources LongCat-2.5 Weights: 1.6T-Parameter Agent Model Trained Without Nvidia Chips
- Nov 15Perplexity Open-Sources Decider 1.3, Adding Text-Span Extraction
Today
Nov 19 Thursday- Brief
- GitHubGitHub Weekly Vol.6: 7 Projects Worth a Look This Week
Tested or read — weeklyAll →
- Quiz
- Call
Latest news
All →- Nov 19MAI Code 2 vs Claude Opus 5.5 and GPT-6.1 Sol: $0.50 Input vs $4, Promo Price Until May 2027
- Nov 18Study: Frontier Models Can Be Tricked Into Marking Civilian Targets
- Nov 18Nvidia Reaffirms Open-Source Strategy, Hugging Face Deal on Track
- Nov 17Klarna Launches AI Assistant That Shops, Books and Runs Errands
- Nov 18Palo Alto to Buy Applied Compute for $4.5 Billion in All-Stock Deal
- Nov 18NGINX2Root Exposed: Five 9.2-Severity Flaws Let Attackers Run Code on Nginx Without Logging In
™