Версия кодовой базы
d539716
Precision весов модели
auto
Версия transformers
4.57.6
Количество GPU и их тип
1 x NVIDIA A100-SXM4-80GB
Архитектура
openai-chat-completions
Название ML-модели:
MiniMax-M2.7
Тип модели:
Открытая
SFT
MoE
Дополнительные ссылки:
https://www.minimax.io/news/minimax-m27-en
Описание архитектуры:
62-layer decoder-only sparse MoE Transformer. 229.9B total parameters, ~9.8B activated per token (~4.3%). Hidden dim 3072, vocab 200,064. Full multi-head causal self-attention on all 62 layers with GQA (48 Q heads, 8 KV heads, head dim 128), RoPE (θ=5M), per-layer QK RMSNorm. MoE FFN: 256 fine-grained experts, top-8 per token via sigmoid gating with learnable per-expert bias; intermediate size 1536, SiLU. Multi-Token Prediction (MTP): 1 module in pre-training, expanded to 3 via weight copying for speculative decoding. Native context 192K (config: 204,800). FP8 quantization (float8_e4m3fn, dynamic, 128×128 blocks) on most weights; gate, routing bias, and lm_head excluded.
Описание обучения:
Pre-trained on 29.2T tokens (19.9T constant + 9.3T decay). Context curriculum: 8K→32K→192K. Post-training is agent-native: mixed-domain SFT + RL via Forge (decoupled training/inference/agent). RL uses CISPO with composite rewards (process, speed, performance), mixed-domain sampling across reasoning/coding/agent/general, trajectories up to 192K tokens. Forge adds windowed-FIFO scheduling and prefix-tree merging (up to ~40× speedup). Checkpoints evolve M2→M2.5→M2.7 with expanding agent data pipelines. M2.7 adds self-evolution: autonomously debugs training runs, edits agent harness/scaffold, ran 100-round self-improvement on programming scaffold (+30% internal gain). Interleaved thinking (Plan-Act-Reflect) with reasoning state persistence. Pre-training optimizer/LR schedule not disclosed in the technical report.
Данные претрейна:
Corpus: web documents, academic literature, books, programming code, structured Q&A. Quality filtering via model-based reward scoring and auxiliary classifiers; balanced sampling upweights high-quality content. Code, mathematics, and STEM significantly upsampled vs. natural distribution; remainder is general web, books, and domain-specific sources. Long-context extension uses code concatenation, long-form PDFs, and thematically related document packing. Post-training adds agentic coding (GitHub PRs, Docker environments, test-based rewards), app development, cowork/office tasks, and deep-search trajectories with executable verification. Tokenizer vocab 200,064.
Лицензия:
NON-COMMERCIAL LICENSE Non-commercial use permitted based on MIT-style terms; commercial use requires prior written authorization. Copyright (c) 2026 MiniMax
Параметры инференса
Параметры генерации:
rucodeeval - do_sample=true;temperature=0.6;until=["<|end▁of▁sentence|>"],max_gen_toks=10000; ruhumaneval - do_sample=true;temperature=0.6;until=["<|end▁of▁sentence|>"],max_gen_toks=10000; rutie_gen - do_sample=false,until=["<|end▁of▁sentence|>"],max_gen_toks=10000; bps_gen - do_sample=false,until=["<|end▁of▁sentence|>"],max_gen_toks=10000; chegeka - do_sample=false,until=["<|end▁of▁sentence|>"],max_gen_toks=10000; lcs_gen - do_sample=false,until=["<|end▁of▁sentence|>"],max_gen_toks=10000; mamuramu_gen - do_sample=false,until=["<|end▁of▁sentence|>"],max_gen_toks=10000; mathlogicqa_gen - do_sample=false,until=["<|end▁of▁sentence|>"],max_gen_toks=10000; multiq - do_sample=false,until=["<|end▁of▁sentence|>"],max_gen_toks=10000; parus_gen - do_sample=false,until=["<|end▁of▁sentence|>"],max_gen_toks=10000; rcb_gen - do_sample=false,until=["<|end▁of▁sentence|>"],max_gen_toks=10000; rudetox - do_sample=false,until=["<|end▁of▁sentence|>"],max_gen_toks=10000; ruethics_gen - do_sample=false,until=["<|end▁of▁sentence|>"],max_gen_toks=10000; ruhatespeech_gen - do_sample=false,until=["<|end▁of▁sentence|>"],max_gen_toks=10000; ruhhh_gen - do_sample=false,until=["<|end▁of▁sentence|>"],max_gen_toks=10000; rummlu_gen - do_sample=false,until=["<|end▁of▁sentence|>"],max_gen_toks=10000; rumodar - do_sample=false,until=["<|end▁of▁sentence|>"],max_gen_toks=10000; rumultiar - do_sample=false,until=["<|end▁of▁sentence|>"],max_gen_toks=10000; ruopenbookqa_gen - do_sample=false,until=["<|end▁of▁sentence|>"],max_gen_toks=10000; ruworldtree_gen - do_sample=false,until=["<|end▁of▁sentence|>"],max_gen_toks=10000; rwsd_gen - do_sample=false,until=["<|end▁of▁sentence|>"],max_gen_toks=10000; simplear - do_sample=false,until=["<|end▁of▁sentence|>"],max_gen_toks=10000; use - do_sample=false,until=["<|end▁of▁sentence|>"],max_gen_toks=10000;
Размер контекста:
204800