Kimi-K2-Instruct-0905

MERA Created at 28.10.2025 17:18
0.325
The overall result

Ratings for leaderboard tasks

The table will scroll to the left

Task name Result Metric
YABLoCo 0.077 / 0.029
EM pass@k
stRuCom 0.232
chrF
RealCode 0.188 / 0.986
pass@k execution_success
UnitTests 0.153
CodeBLEU
ruCodeEval 0.448 / 0.552 / 0.579
pass@k
JavaTestGen 0.233 / 0.507
pass@k compile@1
ruHumanEval 0.448 / 0.554 / 0.579
pass@k
RealCodeJava 0.342 / 0.987
pass@k execution_success
CodeLinterEval 0.423 / 0.548 / 0.573
pass@k
ruCodeReviewer 0.022 / 0.132 / 0.074 / 0.129 / 0.144
chrF BLEU judge@1 judge@5 judge@10
CodeCorrectness 0.571
EM

Information about the submission

Mera version
v1.0.0
Torch Version
2.7.1
The version of the codebase
c965818
CUDA version
12.6
Precision of the model weights
bf16
Seed
1234
Batch
1
Transformers version
4.56.1
The number of GPUs and their type
8 x NVIDIA A100-SXM4-80GB
Architecture
local-chat-completions

Team:

MERA

Name of the ML model:

Kimi-K2-Instruct-0905

Model size

1,000.0B

Model type:

Opened

SFT

MoE

Inference parameters

Generation Parameters:
rucodeeval - do_sample=true;temperature=0.6;max_gen_toks=1024;until=[" \nclass"," \ndef"," \n#"," \nif"," \nprint"]; \ncodelintereval - do_sample=true;temperature=0.6;max_gen_toks=1024;until=[" \n \n"]; \nrucodereviewer - temperature=0;do_sample=false;max_gen_toks=1000;until=[" \n \n"]; \nruhumaneval - do_sample=true;temperature=0.6;max_gen_toks=1024;until=[" \nclass"," \ndef"," \n#"," \nif"," \nprint"]; \nstrucom - do_sample=false;max_gen_toks=512;until=[" \n \n"]; \ncodecorrectness - until=[" \n \n"];do_sample=false;temperature=0; \nrealcode - do_sample=true;max_gen_toks=4096;temperature=0.7;repetition_penalty=1.05;top_p=0.8;until=["<|endoftext|>","<|im_end|>"]; \nrealcodejava - do_sample=true;max_gen_toks=4096;temperature=0.7;repetition_penalty=1.05;top_p=0.8;until=["<|endoftext|>","<|im_end|>"]; \njavatestgen - do_sample=true;max_gen_toks=4096;temperature=0.2;top_p=0.9;until=["<|endoftext|>","<|im_end|>"]; \nyabloco_oracle - max_gen_toks=2048;do_sample=false;until=["<|endoftext|>","<|im_end|>"," \n \n \n","\sclass\s","\sdef\s","^def\s","^class\s","^if\s","@","^#"]; \nunittests - do_sample=false;max_gen_toks=1024;until=[" \n \n"];

Description of the template:
{% macro render_content(msg) -%} {%- set c = msg.get('content') -%} {%- if c is string -%} {{ c }} {%- elif c is not none -%} {% for content in c -%} {% if content['type'] == 'image' or 'image' in content or 'image_url' in content -%} <|media_start|>image<|media_content|><|media_pad|><|media_end|> {% else -%} {{ content['text'] }} {%- endif -%} {%- endfor -%} {%- endif -%} {%- endmacro %} {%- if tools -%} <|im_system|>tool_declare<|im_middle|>{{ tools | tojson(separators=(',', ':')) }}<|im_end|> {%- endif -%} {% for message in messages %} {%- if loop.first and messages[0]['role'] != 'system' -%} <|im_system|>system<|im_middle|>You are Kimi, an AI assistant created by Moonshot AI.<|im_end|> {% endif %} {%- set role_name = message.get('name') or message['role'] -%} {%- if message['role'] == 'user' -%} <|im_user|>{{role_name}}<|im_middle|> {%- elif message['role'] == 'assistant' -%} <|im_assistant|>{{role_name}}<|im_middle|> {%- else -%} <|im_system|>{{role_name}}<|im_middle|> {%- endif -%} {%- if message['role'] == 'assistant' and message.get('tool_calls') -%} {{render_content(message)}}<|tool_calls_section_begin|> {%- for tool_call in message['tool_calls'] -%} {%- set formatted_id = tool_call['id'] -%} <|tool_call_begin|>{{ formatted_id }}<|tool_call_argument_begin|>{% if tool_call['function']['arguments'] is string %}{{ tool_call['function']['arguments'] }}{% else %}{{ tool_call['function']['arguments'] | tojson }}{% endif %}<|tool_call_end|> {%- endfor -%} <|tool_calls_section_end|> {%- elif message['role'] == 'tool' -%} {%- set tool_call_id = message.tool_call_id -%} ## Return of {{ tool_call_id }} {{render_content(message)}} {%- elif message['content'] is not none -%} {{render_content(message)}} {%- endif -%} <|im_end|> {%- endfor -%} {%- if add_generation_prompt -%} <|im_assistant|>assistant<|im_middle|> {%- endif -%}