Команда:

MERA

Название ML-модели:

DeepSeek-V4-Pro

Ссылка на ML-модель:

Размер модели

1,600.0B

Тип модели:

Открытая

SFT

MoE

Дополнительные ссылки:

Описание архитектуры:

61 Transformer layers, hidden dimension 7168. First 2 layers: HCA; subsequent layers: CSA and HCA interleaved. CSA: compression rate 4, 64 indexer heads (dim 128), top-k 1024. HCA: compression rate 128. Both: 128 query heads (dim 512), query compression dim 1536, 16 output groups (1024 each). Sliding window branch: size 128. All blocks use MoE (Hash routing in first 3 layers): 1 shared + 384 routed experts (dim 3072), 6 experts per token. MTP depth 1; mHC expansion factor 4, 20 Sinkhorn-Knopp iterations. 1.6T total parameters, 49B activated per token. Context: 1M tokens.

Описание обучения:

Pre-training: Muon (majority) + AdamW (embedding, head, RMSNorm). 33T tokens, max batch 94.4M. LR: warmup 2000 steps → peak 2.0×10⁻⁴ → cosine decay to 2.0×10⁻⁵. Sequence length 4K → 16K → 64K → 1M; longer dense-attention stage than Flash; sparse attention from 64K. Stability: Anticipatory Routing, SwiGLU clamping [−10, 10]. Post-training: SFT + GRPO RL for domain specialists (3 reasoning modes: Non-think / Think / Think Max) → On-Policy Distillation from 10+ teacher models. FP4 QAT for MoE weights and CSA indexer.

Данные претрейна:

Built on DeepSeek-V3 data: more diverse, higher-quality corpus with longer effective contexts. Web data filtered from auto-generated/templated content. Core: math, code; agentic data added in mid-training. Expanded multilingual data and long-document curation (scientific papers, technical reports). Corpus >32T tokens (Pro trained on 33T). Tokenizer: 128K vocab, FIM, document packing; sample-level attention masking (unlike V3).

Лицензия:

MIT License

Параметры инференса