GigaChat-3.1-Lightning

GigaChat Created at 28.03.2026 08:29
0.258
The overall result

Ratings for leaderboard tasks

The table will scroll to the left

Task name Result Metric
YABLoCo 0.014 / 0.005
EM pass@k
stRuCom 0.115
chrF
RealCode 0.091 / 0.956
pass@k execution_success
UnitTests 0.175
CodeBLEU
ruCodeEval 0.291 / 0.397 / 0.439
pass@k
JavaTestGen 0.057 / 0.33
pass@k compile@1
ruHumanEval 0.272 / 0.395 / 0.433
pass@k
RealCodeJava 0.188 / 0.977
pass@k execution_success
CodeLinterEval 0.482 / 0.491 / 0.5
pass@k
ruCodeReviewer 0.013 / 0.112 / 0.023 / 0.023 / 0.023
chrF BLEU judge@1 judge@5 judge@10
CodeCorrectness 0.794
EM

Information about the submission

Mera version
v1.0.0
Torch Version
2.9.0
The version of the codebase
0ac3a14
CUDA version
12.8
Precision of the model weights
auto
Seed
1234
Batch
1
Transformers version
4.57.1
The number of GPUs and their type
1 x NVIDIA A100-SXM4-80GB
Architecture
gigachat-completion

Team:

GigaChat

Name of the ML model:

GigaChat-3.1-Lightning

Model size

11.0B

Model type:

Opened

SFT

Inference parameters

Generation Parameters:
realcode - do_sample=true;max_gen_toks=4096;temperature=0.7;repetition_penalty=1.05;top_p=0.8;until=["<|endoftext|>","<|im_end|>"]; \nrealcodejava - do_sample=true;max_gen_toks=4096;temperature=0.7;repetition_penalty=1.05;top_p=0.8;until=["<|endoftext|>","<|im_end|>"]; \njavatestgen - do_sample=true;max_gen_toks=4096;temperature=0.2;top_p=0.9;until=["<|endoftext|>","<|im_end|>"]; \nyabloco_oracle - max_gen_toks=2048;do_sample=false;until=["<|endoftext|>","<|im_end|>","\n\n\n","\\sclass\\s","\\sdef\\s","^def\\s","^class\\s","^if\\s","@","^#"]; \nunittests - do_sample=false;max_gen_toks=1024;until=["\n\n"]; \ncodecorrectness - until=["\n\n"];do_sample=false;temperature=0; \ncodelintereval - do_sample=true;temperature=0.6;max_gen_toks=1024;until=["\n\n"]; \nrucodereviewer - temperature=0;do_sample=false;max_gen_toks=1000;until=["\n\n"]; \nstrucom - do_sample=false;max_gen_toks=512;until=["\n\n"]; \nrucodeeval_code - do_sample=true;temperature=0.6;max_gen_toks=1024;until=["\nclass","\ndef","\n#","\nif","\nprint"]; \nruhumaneval_code - do_sample=true;temperature=0.6;max_gen_toks=1024;until=["\nclass","\ndef","\n#","\nif","\nprint"];