Flash

Flash·Models & Products·2026-11-18 20:08

K2-Horizon Opens Six Models: Flagship 375B MoE With Full Training Data

IFM's K2 Team published K2-Horizon on arXiv on Nov 16: six fully open language models, from a 0.9B dense model to a 375B-A23B mixture-of-experts, with training data, intermediate checkpoints and all code released alongside.

Layered mountain ridges on a vast desert horizon at sunrise, a modern glass research building in the foreground
AI-generated illustration

What happened

The fleet spans six sizes, from a 0.9B dense model to a 375B-A23B mixture-of-experts that activates 23B parameters per token; the models are built for general-purpose reasoning and agentic work across math and science, software engineering, deep research and office automation.

Two new techniques are introduced: MoVA, which applies mixture-of-experts to attention value projections, and softplus attention gating; the 36B-A4B model uses both and nearly matches a 32B dense model trained on the same data, with one-seventh the active parameters; Uno, a lossless diffusion adapter, speeds up decoding by up to 2.2x without a separate draft model.

The release follows the LLM360 definition of open source: the full development history, intermediate checkpoints from every training stage, pre-training data, training logs, and all training, evaluation and infrastructure code, including the production-scale xLLM training framework and the RL360 reinforcement-learning stack; four of the models share one 22T-token pre-training corpus and recipe, and roughly 10T synthetic tokens went into pre-training.

Key facts

  1. Six models: From a 0.9B dense model to a 375B-A23B mixture-of-experts, activating 23B parameters per token
  2. MoVA: Mixture-of-experts applied to attention value projections, plus softplus attention gating
  3. 36B-A4B: Nearly matches a 32B dense model trained on the same data, with about one-seventh the active parameters
  4. Uno: Lossless diffusion adapters speed up decoding by up to 2.2x without a separate draft model
  5. Corpus: Four models share one 22T-token pre-training corpus and recipe; roughly 10T synthetic tokens went into pre-training
  6. Open scope: Intermediate checkpoints, pre-training data, training logs, and the xLLM and RL360 framework code
  7. Release: arXiv:2611.21003, submitted Nov 16, 2026; artifacts at ifm.ai/k2 and on Hugging Face

Why it matters

all six models, from 0.9B to 375B, are fully open, four share a 22T-token corpus and one training recipe, and about 10T synthetic tokens went into pre-training.
Useful Tap if this story helped you
Send to a friendMakes a share image for X, WeChat, Telegram and more

SourcesarXiv:2611.21003 (Nov 16, 2026); ExplainX (Nov 18, 2026). This article is compiled from public information and does not constitute investment advice.

From HIMIAO Horizon · himiao.ai

Grab todayOpenAI Codex for Students: $100 in Free CreditsHIMIAO Picks →

Follow HIMIAO Horizon

No sign-up · Unsubscribe any time

Email

Notifications

A ping on your phone or computer when the brief is out. No email needed.

HIMIAO PicksWeekly freebies: one email every Friday, a 5-minute read

Comments

  1. Loading comments…
Ask the cat