Cotype Light 3

MWS AI Created at 27.03.2026 09:50
0.792
The overall result
10
Place in the rating
In the top by tasks:
3
MultiQ
The task is one of the main ones
10
ruWorldTree
The result on the task is higher than human
The task is one of the main ones
9
MathLogicQA
The result on the task is higher than human
The task is one of the main ones
5
SimpleAr
10
LCS
The result on the task is higher than human
The task is one of the main ones
+1
Hide
Weak tasks:
102
PARus
202
ruEthics
21
ruOpenBookQA
101
CheGeKa
89
ruMMLU
57
ruHateSpeech
85
ruDetox
105
ruHHH
69
ruTiE
21
USE
211
BPS
23
ruModAr
31
MaMuRAMu
+9
Hide

Ratings for leaderboard tasks

The table will scroll to the left

Task name Result Metric
LCS 0.71 Accuracy
RCB 0.6 / 0.598 Accuracy F1 macro
USE 0.591 Grade norm
RWSD 0.796 Accuracy
PARus 0.9 Accuracy
ruTiE 0.852 Accuracy
MultiQ 0.751 / 0.606 F1 Exact match
CheGeKa 0.325 / 0.257 F1 Exact match
ruModAr 0.992 Exact match
MaMuRAMu 0.863 Accuracy
ruMultiAr 0.998 Exact match
ruCodeEval 0.662 / 0.68 / 0.683 Pass@k
MathLogicQA 0.995 Accuracy
ruWorldTree 0.994 / 0.994 Accuracy F1 macro
ruOpenBookQA 0.95 / 0.95 Accuracy F1 macro

Evaluation on open tasks:

Go to the ratings by subcategory

The table will scroll to the left

Task name Result Metric
BPS 0.954 Accuracy
ruMMLU 0.732 Accuracy
SimpleAr 1.0 Exact match
ruHumanEval 0.649 / 0.662 / 0.665 Pass@k
ruHHH 0.826
ruHateSpeech 0.842
ruDetox 0.305
ruEthics
Correct God Ethical
Virtue 0.311 0.432 0.49
Law 0.358 0.433 0.497
Moral 0.357 0.477 0.532
Justice 0.303 0.39 0.472
Utilitarianism 0.27 0.381 0.419

Information about the submission

Mera version
v1.2.0
Torch Version
2.10.0
The version of the codebase
6efc50c3752813a87425eb29a2d8b06043a65b42
CUDA version
12.9
Precision of the model weights
bf16
Seed
4
Batch
32
Transformers version
5.3.0
The number of GPUs and their type
1 x NVIDIA A100-SXM4-40GB
Architecture
local-chat-completions

Team:

MWS AI

Name of the ML model:

Cotype Light 3

Model size

9.0B

Model type:

Closed

SFT

Architecture description:

Cotype Light 3 помогает извлекать главное из документов, понимать и суммаризировать тексты, вести диалог, создавать материалы и писать код. Модель работает сразу с текстом и изображениями: распознаёт разные форматы текста, учитывает визуальный контекст и помогает отвечать на содержательные вопросы по документам, скриншотам и графическим данным. Подходит для поиска по визуально насыщенным документам, клиентской поддержки и быстрой подготовки отчётов. На этих сценариях строится большинство корпоративных текстовых ассистентов и ИИ-агентов. Разработана MWS AI (входит в МТС Web Services).

License:

Проприетарная модель от MWS AI

Inference parameters

Generation Parameters:
chegeka - do_sample=false;until=<|endoftext|>;max_gen_toks=10240; \nrumultiar - do_sample=false;until=<|endoftext|>;max_gen_toks=10240; \nuse - do_sample=false;until=<|endoftext|>;max_gen_toks=10240; \nsimplear - do_sample=false;until=<|endoftext|>;max_gen_toks=2048;chat_template_kwargs={"enable_thinking":false}; \nrudetox - do_sample=false;until=<|endoftext|>;max_gen_toks=2048;chat_template_kwargs={"enable_thinking":false}; \nmultiq - do_sample=false;until=<|endoftext|>;max_gen_toks=2048;chat_template_kwargs={"enable_thinking":false}; \nrumodar - do_sample=false;until=<|endoftext|>;max_gen_toks=2048;chat_template_kwargs={"enable_thinking":false}; \nruhumaneval - do_sample=true;temperature=0.6;until=<|endoftext|>;max_gen_toks=2048;chat_template_kwargs={"enable_thinking":false}; \nrucodeeval - do_sample=true;temperature=0.6;until=<|endoftext|>;max_gen_toks=2048;chat_template_kwargs={"enable_thinking":false};

System prompt:
Решай задачу строго по инструкции. Только ответ, без объяснений. Числовой ответ — только число. Буква, цифра или слово — только их. Выбор варианта ответа — одна буква или цифра. Ответ должен быть точным, без лишних символов или слов. В случае, если нужно сгенерировать код на Python — твоим ответом должен быть только код (продолжения кода из инструкции), не повторяй имя функции, не давай пояснений, не пиши комментариев, не используй input, пиши код так, чтобы он дополнял функцию из инструкции (с соответствующими отступами), всегда обрамляй код вот так ```python \n *generation* \n```!

Description of the template:
{%- set image_count = namespace(value=0) %} \n{%- set video_count = namespace(value=0) %} \n{%- macro render_content(content, do_vision_count, is_system_content=false) %} \n {%- if content is string %} \n {{- content }} \n {%- elif content is iterable and content is not mapping %} \n {%- for item in content %} \n {%- if 'image' in item or 'image_url' in item or item.type == 'image' %} \n {%- if is_system_content %} \n {{- raise_exception('System message cannot contain images.') }} \n {%- endif %} \n {%- if do_vision_count %} \n {%- set image_count.value = image_count.value + 1 %} \n {%- endif %} \n {%- if add_vision_id %} \n {{- 'Picture ' ~ image_count.value ~ ': ' }} \n {%- endif %} \n {{- '<|vision_start|><|image_pad|><|vision_end|>' }} \n {%- elif 'video' in item or item.type == 'video' %} \n {%- if is_system_content %} \n {{- raise_exception('System message cannot contain videos.') }} \n {%- endif %} \n {%- if do_vision_count %} \n {%- set video_count.value = video_count.value + 1 %} \n {%- endif %} \n {%- if add_vision_id %} \n {{- 'Video ' ~ video_count.value ~ ': ' }} \n {%- endif %} \n {{- '<|vision_start|><|video_pad|><|vision_end|>' }} \n {%- elif 'text' in item %} \n {{- item.text }} \n {%- else %} \n {{- raise_exception('Unexpected item type in content.') }} \n {%- endif %} \n {%- endfor %} \n {%- elif content is none or content is undefined %} \n {{- '' }} \n {%- else %} \n {{- raise_exception('Unexpected content type.') }} \n {%- endif %} \n{%- endmacro %} \n{%- if not messages %} \n {{- raise_exception('No messages provided.') }} \n{%- endif %} \n{%- if tools and tools is iterable and tools is not mapping %} \n {{- '<|im_start|>system \n' }} \n {{- "# Tools \n \nYou have access to the following functions: \n \n<tools>" }} \n {%- for tool in tools %} \n {{- " \n" }} \n {{- tool | tojson }} \n {%- endfor %} \n {{- " \n</tools>" }} \n {{- ' \n \nIf you choose to call a function ONLY reply in the following format with NO suffix: \n \n<tool_call> \n<function=example_function_name> \n<parameter=example_parameter_1> \nvalue_1 \n</parameter> \n<parameter=example_parameter_2> \nThis is the value for the second parameter \nthat can span \nmultiple lines \n</parameter> \n</function> \n</tool_call> \n \n<IMPORTANT> \nReminder: \n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags \n- Required parameters MUST be specified \n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after \n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls \n</IMPORTANT>' }} \n {%- if messages[0].role == 'system' %} \n {%- set content = render_content(messages[0].content, false, true)|trim %} \n {%- if content %} \n {{- ' \n \n' + content }} \n {%- endif %} \n {%- endif %} \n {{- '<|im_end|> \n' }} \n{%- else %} \n {%- if messages[0].role == 'system' %} \n {%- set content = render_content(messages[0].content, false, true)|trim %} \n {{- '<|im_start|>system \n' + content + '<|im_end|> \n' }} \n {%- endif %} \n{%- endif %} \n{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %} \n{%- for message in messages[::-1] %} \n {%- set index = (messages|length - 1) - loop.index0 %} \n {%- if ns.multi_step_tool and message.role == "user" %} \n {%- set content = render_content(message.content, false)|trim %} \n {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %} \n {%- set ns.multi_step_tool = false %} \n {%- set ns.last_query_index = index %} \n {%- endif %} \n {%- endif %} \n{%- endfor %} \n{%- if ns.multi_step_tool %} \n {{- raise_exception('No user query found in messages.') }} \n{%- endif %} \n{%- for message in messages %} \n {%- set content = render_content(message.content, true)|trim %} \n {%- if message.role == "system" %} \n {%- if not loop.first %} \n {{- raise_exception('System message must be at the beginning.') }} \n {%- endif %} \n {%- elif message.role == "user" %} \n {{- '<|im_start|>' + message.role + ' \n' + content + '<|im_end|>' + ' \n' }} \n {%- elif message.role == "assistant" %} \n {%- set reasoning_content = '' %} \n {%- if message.reasoning_content is string %} \n {%- set reasoning_content = message.reasoning_content %} \n {%- else %} \n {%- if '</think>' in content %} \n {%- set reasoning_content = content.split('</think>')[0].rstrip(' \n').split('<think>')[-1].lstrip(' \n') %} \n {%- set content = content.split('</think>')[-1].lstrip(' \n') %} \n {%- endif %} \n {%- endif %} \n {%- set reasoning_content = reasoning_content|trim %} \n {%- if loop.index0 > ns.last_query_index %} \n {{- '<|im_start|>' + message.role + ' \n<think> \n' + reasoning_content + ' \n</think> \n \n' + content }} \n {%- else %} \n {{- '<|im_start|>' + message.role + ' \n' + content }} \n {%- endif %} \n {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %} \n {%- for tool_call in message.tool_calls %} \n {%- if tool_call.function is defined %} \n {%- set tool_call = tool_call.function %} \n {%- endif %} \n {%- if loop.first %} \n {%- if content|trim %} \n {{- ' \n \n<tool_call> \n<function=' + tool_call.name + '> \n' }} \n {%- else %} \n {{- '<tool_call> \n<function=' + tool_call.name + '> \n' }} \n {%- endif %} \n {%- else %} \n {{- ' \n<tool_call> \n<function=' + tool_call.name + '> \n' }} \n {%- endif %} \n {%- if tool_call.arguments is defined %} \n {%- for args_name, args_value in tool_call.arguments|items %} \n {{- '<parameter=' + args_name + '> \n' }} \n {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %} \n {{- args_value }} \n {{- ' \n</parameter> \n' }} \n {%- endfor %} \n {%- endif %} \n {{- '</function> \n</tool_call>' }} \n {%- endfor %} \n {%- endif %} \n {{- '<|im_end|> \n' }} \n {%- elif message.role == "tool" %} \n {%- if loop.previtem and loop.previtem.role != "tool" %} \n {{- '<|im_start|>user' }} \n {%- endif %} \n {{- ' \n<tool_response> \n' }} \n {{- content }} \n {{- ' \n</tool_response>' }} \n {%- if not loop.last and loop.nextitem.role != "tool" %} \n {{- '<|im_end|> \n' }} \n {%- elif loop.last %} \n {{- '<|im_end|> \n' }} \n {%- endif %} \n {%- else %} \n {{- raise_exception('Unexpected message role.') }} \n {%- endif %} \n{%- endfor %} \n{%- if add_generation_prompt %} \n {{- '<|im_start|>assistant \n' }} \n {%- if enable_thinking is defined and enable_thinking is false %} \n {{- '<think> \n \n</think> \n \n' }} \n {%- else %} \n {{- '<think> \n' }} \n {%- endif %} \n{%- endif %}

Ratings by subcategory

Metric: Grade Norm
Model, team 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 8_0 8_1 8_2 8_3 8_4
Cotype Light 3
MWS AI
0.633 0.6 0.833 0.167 0.667 0.867 0.467 - 0.167 0.067 0.267 0.3 0.6 0.467 0.367 0.617 0.7 0.433 0.833 0.4 0.6 0.667 0.667 0.733 0.7 0.867 0.367 0.6 0.7 0.867 0.667
Model, team Honest Helpful Harmless
Cotype Light 3
MWS AI
0.787 0.78 0.914
Model, team Anatomy Virology Astronomy Marketing Nutrition Sociology Management Philosophy Prehistory Human aging Econometrics Formal logic Global facts Jurisprudence Miscellaneous Moral disputes Business ethics Biology (college) Physics (college) Human Sexuality Moral scenarios World religions Abstract algebra Medicine (college) Machine learning Medical genetics Professional law PR Security studies Chemistry (школьная) Computer security International law Logical fallacies Politics Clinical knowledge Conceptual_physics Math (college) Biology (high school) Physics (high school) Chemistry (high school) Geography (high school) Professional medicine Electrical engineering Elementary mathematics Psychology (high school) Statistics (high school) History (high school) Math (high school) Professional accounting Professional psychology Computer science (college) World history (high school) Macroeconomics Microeconomics Computer science (high school) European history Government and politics
Cotype Light 3
MWS AI
0.667 0.5 0.875 0.855 0.837 0.866 0.816 0.714 0.762 0.709 0.684 0.675 0.46 0.778 0.787 0.708 0.77 0.882 0.644 0.832 0.611 0.807 0.62 0.786 0.652 0.89 0.535 0.648 0.796 0.56 0.75 0.843 0.785 0.848 0.781 0.833 0.61 0.906 0.682 0.754 0.843 0.801 0.752 0.735 0.871 0.718 0.877 0.526 0.539 0.718 0.71 0.869 0.813 0.891 0.91 0.836 0.886
Model, team SIM FL STA
Cotype Light 3
MWS AI
0.816 0.629 0.618
Model, team Anatomy Virology Astronomy Marketing Nutrition Sociology Managment Philosophy Pre-History Gerontology Econometrics Formal logic Global facts Jurisprudence Miscellaneous Moral disputes Business ethics Bilology (college) Physics (college) Human sexuality Moral scenarios World religions Abstract algebra Medicine (college) Machine Learning Genetics Professional law PR Security Chemistry (college) Computer security International law Logical fallacies Politics Clinical knowledge Conceptual physics Math (college) Biology (high school) Physics (high school) Chemistry (high school) Geography (high school) Professional medicine Electrical Engineering Elementary mathematics Psychology (high school) Statistics (high school) History (high school) Math (high school) Professional Accounting Professional psychology Computer science (college) World history (high school) Macroeconomics Microeconomics Computer science (high school) Europe History Government and politics
Cotype Light 3
MWS AI
0.778 0.931 0.833 0.787 0.934 0.793 0.776 0.737 0.904 0.8 0.808 0.833 0.6 0.884 0.836 0.815 0.757 0.8 0.895 0.842 0.825 0.915 0.956 0.929 0.889 0.909 0.885 0.719 0.93 0.956 0.844 0.923 0.866 0.912 0.773 0.929 0.933 0.889 0.93 0.954 0.927 0.937 0.844 1 0.897 0.911 0.931 0.909 0.938 0.947 0.911 0.884 0.873 0.792 0.837 0.801 0.911
Coorect
Good
Ethical
Model, team Virtue Law Moral Justice Utilitarianism
Cotype Light 3
MWS AI
0.311 0.358 0.357 0.303 0.27
Model, team Virtue Law Moral Justice Utilitarianism
Cotype Light 3
MWS AI
0.432 0.433 0.477 0.39 0.381
Model, team Virtue Law Moral Justice Utilitarianism
Cotype Light 3
MWS AI
0.49 0.497 0.532 0.472 0.419
Model, team Women Men LGBT Nationalities Migrants Other
Cotype Light 3
MWS AI
0.843 0.8 0.824 0.811 0.857 0.885