Cotype Pro 3

MWS AI Created at 04.06.2026 10:05
0.824
The overall result
4
Place in the rating
In the top by tasks:
9
RWSD
The result on the task is higher than human
The task is one of the main ones
3
PARus
The task is one of the main ones
4
MultiQ
The task is one of the main ones
9
ruOpenBookQA
The result on the task is higher than human
The task is one of the main ones
4
ruHumanEval
3
MathLogicQA
The result on the task is higher than human
The task is one of the main ones
1
ruMultiAr
The result on the task is higher than human
The task is one of the main ones
9
LCS
The result on the task is higher than human
The task is one of the main ones
1
ruModAr
The result on the task is higher than human
The task is one of the main ones
4
ruCodeEval
The task is one of the main ones
+6
Hide
Weak tasks:
32
RCB
52
ruEthics
42
CheGeKa
29
ruMMLU
33
ruHateSpeech
72
ruDetox
21
SimpleAr
30
BPS
+4
Hide

Ratings for leaderboard tasks

The table will scroll to the left

Task name Result Metric
LCS 0.736 Accuracy
RCB 0.594 / 0.571 Accuracy F1 macro
USE 0.656 Grade norm
RWSD 0.873 Accuracy
PARus 0.958 Accuracy
ruTiE 0.911 Accuracy
MultiQ 0.674 / 0.531 F1 Exact match
CheGeKa 0.417 / 0.341 F1 Exact match
ruModAr 1.0 Exact match
MaMuRAMu 0.887 Accuracy
ruMultiAr 1.0 Exact match
ruCodeEval 0.814 / 0.827 / 0.829 Pass@k
MathLogicQA 0.997 Accuracy
ruWorldTree 0.992 / 0.992 Accuracy F1 macro
ruOpenBookQA 0.96 / 0.96 Accuracy F1 macro

Evaluation on open tasks:

Go to the ratings by subcategory

The table will scroll to the left

Task name Result Metric
BPS 0.996 Accuracy
ruMMLU 0.815 Accuracy
SimpleAr 0.999 Exact match
ruHumanEval 0.812 / 0.826 / 0.829 Pass@k
ruHHH 0.893
ruHateSpeech 0.86
ruDetox 0.316
ruEthics
Correct God Ethical
Virtue 0.419 0.421 0.537
Law 0.405 0.408 0.539
Moral 0.448 0.449 0.584
Justice 0.389 0.381 0.493
Utilitarianism 0.356 0.365 0.495

Information about the submission

Mera version
v1.2.0
Torch Version
2.10.0
The version of the codebase
e5ac7c86d7b8fa17c8ea338032002ce53c49c818
CUDA version
12.9
Precision of the model weights
bf16
Seed
4
Batch
32
Transformers version
5.3.0
The number of GPUs and their type
4 x NVIDIA A100-SXM4-40GB
Architecture
local-chat-completions

Team:

MWS AI

Name of the ML model:

Cotype Pro 3

Model size

27.0B

Model type:

Closed

SFT

Architecture description:

Cotype Pro 3 — флагманская мультимодальная большая языковая модель MWS AI, открывающая третье поколение линейки Cotype. Предназначена для создания ИИ-агентов в корпоративных средах: вызывает функции и внешние инструменты, планирует и выполняет многошаговые задачи, работает с корпоративными базами знаний и кодом. Основной фокус дообучения — стабильность агентного поведения: воспроизводимость результата при повторных запусках сценария и строгое следование заданному workflow. Обрабатывает текст и изображения, включая сканы документов, таблицы, схемы и рукописный текст на русском языке. Поддерживает стандартный режим генерации и режим расширенных рассуждений (thinking); контекстное окно — 262 тыс. токенов с возможностью расширения до 1 млн.

Description of the training:

SFT

License:

Проприетарная модель от MWS AI

Inference parameters

Generation Parameters:
chegeka - do_sample=false;until=<|endoftext|>;max_gen_toks=10240; \nrumultiar - do_sample=false;until=<|endoftext|>;max_gen_toks=10240; \nuse - do_sample=false;until=<|endoftext|>;max_gen_toks=10240; \nrumodar - do_sample=false;until=<|endoftext|>;max_gen_toks=10240; \nsimplear - do_sample=false;until=<|endoftext|>;max_gen_toks=2048;chat_template_kwargs={"enable_thinking":false}; \nrudetox - do_sample=false;until=<|endoftext|>;max_gen_toks=2048;chat_template_kwargs={"enable_thinking":false}; \nmultiq - do_sample=false;until=<|endoftext|>;max_gen_toks=2048;chat_template_kwargs={"enable_thinking":false}; \nruhumaneval - do_sample=true;temperature=0.6;until=<|endoftext|>;max_gen_toks=2048;chat_template_kwargs={"enable_thinking":false}; \nrucodeeval - do_sample=true;temperature=0.6;until=<|endoftext|>;max_gen_toks=2048;chat_template_kwargs={"enable_thinking":false};

The size of the context:
262144

System prompt:
Решай задачу строго по инструкции. Только ответ, без объяснений. Числовой ответ — только число. Буква, цифра или слово — только их. Выбор варианта ответа — одна буква или цифра. Ответ должен быть точным, без лишних символов или слов. В случае, если нужно сгенерировать код на Python — твоим ответом должен быть только код (продолжения кода из инструкции), не повторяй имя функции, не давай пояснений, не пиши комментариев, не используй input, пиши код так, чтобы он дополнял функцию из инструкции (с соответствующими отступами), всегда обрамляй код вот так ```python \n *generation* \n```!

Description of the template:
{%- set image_count = namespace(value=0) %} \n{%- set video_count = namespace(value=0) %} \n{%- macro render_content(content, do_vision_count, is_system_content=false) %} \n {%- if content is string %} \n {{- content }} \n {%- elif content is iterable and content is not mapping %} \n {%- for item in content %} \n {%- if 'image' in item or 'image_url' in item or item.type == 'image' %} \n {%- if is_system_content %} \n {{- raise_exception('System message cannot contain images.') }} \n {%- endif %} \n {%- if do_vision_count %} \n {%- set image_count.value = image_count.value + 1 %} \n {%- endif %} \n {%- if add_vision_id %} \n {{- 'Picture ' ~ image_count.value ~ ': ' }} \n {%- endif %} \n {{- '<|vision_start|><|image_pad|><|vision_end|>' }} \n {%- elif 'video' in item or item.type == 'video' %} \n {%- if is_system_content %} \n {{- raise_exception('System message cannot contain videos.') }} \n {%- endif %} \n {%- if do_vision_count %} \n {%- set video_count.value = video_count.value + 1 %} \n {%- endif %} \n {%- if add_vision_id %} \n {{- 'Video ' ~ video_count.value ~ ': ' }} \n {%- endif %} \n {{- '<|vision_start|><|video_pad|><|vision_end|>' }} \n {%- elif 'text' in item %} \n {{- item.text }} \n {%- else %} \n {{- raise_exception('Unexpected item type in content.') }} \n {%- endif %} \n {%- endfor %} \n {%- elif content is none or content is undefined %} \n {{- '' }} \n {%- else %} \n {{- raise_exception('Unexpected content type.') }} \n {%- endif %} \n{%- endmacro %} \n{%- if not messages %} \n {{- raise_exception('No messages provided.') }} \n{%- endif %} \n{%- if tools and tools is iterable and tools is not mapping %} \n {{- '<|im_start|>system \n' }} \n {{- "# Tools \n \nYou have access to the following functions: \n \n<tools>" }} \n {%- for tool in tools %} \n {{- " \n" }} \n {{- tool | tojson }} \n {%- endfor %} \n {{- " \n</tools>" }} \n {{- ' \n \nIf you choose to call a function ONLY reply in the following format with NO suffix: \n \n<tool_call> \n<function=example_function_name> \n<parameter=example_parameter_1> \nvalue_1 \n</parameter> \n<parameter=example_parameter_2> \nThis is the value for the second parameter \nthat can span \nmultiple lines \n</parameter> \n</function> \n</tool_call> \n \n<IMPORTANT> \nReminder: \n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags \n- Required parameters MUST be specified \n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after \n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls \n</IMPORTANT>' }} \n {%- if messages[0].role == 'system' %} \n {%- set content = render_content(messages[0].content, false, true)|trim %} \n {%- if content %} \n {{- ' \n \n' + content }} \n {%- endif %} \n {%- endif %} \n {{- '<|im_end|> \n' }} \n{%- else %} \n {%- if messages[0].role == 'system' %} \n {%- set content = render_content(messages[0].content, false, true)|trim %} \n {{- '<|im_start|>system \n' + content + '<|im_end|> \n' }} \n {%- endif %} \n{%- endif %} \n{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %} \n{%- for message in messages[::-1] %} \n {%- set index = (messages|length - 1) - loop.index0 %} \n {%- if ns.multi_step_tool and message.role == "user" %} \n {%- set content = render_content(message.content, false)|trim %} \n {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %} \n {%- set ns.multi_step_tool = false %} \n {%- set ns.last_query_index = index %} \n {%- endif %} \n {%- endif %} \n{%- endfor %} \n{%- if ns.multi_step_tool %} \n {{- raise_exception('No user query found in messages.') }} \n{%- endif %} \n{%- for message in messages %} \n {%- set content = render_content(message.content, true)|trim %} \n {%- if message.role == "system" %} \n {%- if not loop.first %} \n {{- raise_exception('System message must be at the beginning.') }} \n {%- endif %} \n {%- elif message.role == "user" %} \n {{- '<|im_start|>' + message.role + ' \n' + content + '<|im_end|>' + ' \n' }} \n {%- elif message.role == "assistant" %} \n {%- set reasoning_content = '' %} \n {%- if message.reasoning_content is string %} \n {%- set reasoning_content = message.reasoning_content %} \n {%- else %} \n {%- if '</think>' in content %} \n {%- set reasoning_content = content.split('</think>')[0].rstrip(' \n').split('<think>')[-1].lstrip(' \n') %} \n {%- set content = content.split('</think>')[-1].lstrip(' \n') %} \n {%- endif %} \n {%- endif %} \n {%- set reasoning_content = reasoning_content|trim %} \n {%- if (preserve_thinking is defined and preserve_thinking is true) or (loop.index0 > ns.last_query_index) %} \n {{- '<|im_start|>' + message.role + ' \n<think> \n' + reasoning_content + ' \n</think> \n \n' + content }} \n {%- else %} \n {{- '<|im_start|>' + message.role + ' \n' + content }} \n {%- endif %} \n {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %} \n {%- for tool_call in message.tool_calls %} \n {%- if tool_call.function is defined %} \n {%- set tool_call = tool_call.function %} \n {%- endif %} \n {%- if loop.first %} \n {%- if content|trim %} \n {{- ' \n \n<tool_call> \n<function=' + tool_call.name + '> \n' }} \n {%- else %} \n {{- '<tool_call> \n<function=' + tool_call.name + '> \n' }} \n {%- endif %} \n {%- else %} \n {{- ' \n<tool_call> \n<function=' + tool_call.name + '> \n' }} \n {%- endif %} \n {%- if tool_call.arguments is defined %} \n {%- for args_name, args_value in tool_call.arguments|items %} \n {{- '<parameter=' + args_name + '> \n' }} \n {%- set args_value = args_value | string if args_value is string else args_value | tojson | safe %} \n {{- args_value }} \n {{- ' \n</parameter> \n' }} \n {%- endfor %} \n {%- endif %} \n {{- '</function> \n</tool_call>' }} \n {%- endfor %} \n {%- endif %} \n {{- '<|im_end|> \n' }} \n {%- elif message.role == "tool" %} \n {%- if loop.previtem and loop.previtem.role != "tool" %} \n {{- '<|im_start|>user' }} \n {%- endif %} \n {{- ' \n<tool_response> \n' }} \n {{- content }} \n {{- ' \n</tool_response>' }} \n {%- if not loop.last and loop.nextitem.role != "tool" %} \n {{- '<|im_end|> \n' }} \n {%- elif loop.last %} \n {{- '<|im_end|> \n' }} \n {%- endif %} \n {%- else %} \n {{- raise_exception('Unexpected message role.') }} \n {%- endif %} \n{%- endfor %} \n{%- if add_generation_prompt %} \n {{- '<|im_start|>assistant \n' }} \n {%- if enable_thinking is defined and enable_thinking is false %} \n {{- '<think> \n \n</think> \n \n' }} \n {%- else %} \n {{- '<think> \n' }} \n {%- endif %} \n{%- endif %}

Ratings by subcategory

Metric: Grade Norm
Model, team 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 8_0 8_1 8_2 8_3 8_4
Cotype Pro 3
MWS AI
0.733 0.7 0.867 0.2 0.6 0.833 0.5 - 0.267 0.133 0.433 0.433 0.667 0.533 0.333 0.8 0.667 0.467 0.567 0.467 0.9 0.8 0.7 0.667 0.767 0.917 0.6 0.767 0.833 0.733 0.867
Model, team Honest Helpful Harmless
Cotype Pro 3
MWS AI
0.869 0.864 0.948
Model, team Anatomy Virology Astronomy Marketing Nutrition Sociology Management Philosophy Prehistory Human aging Econometrics Formal logic Global facts Jurisprudence Miscellaneous Moral disputes Business ethics Biology (college) Physics (college) Human Sexuality Moral scenarios World religions Abstract algebra Medicine (college) Machine learning Medical genetics Professional law PR Security studies Chemistry (школьная) Computer security International law Logical fallacies Politics Clinical knowledge Conceptual_physics Math (college) Biology (high school) Physics (high school) Chemistry (high school) Geography (high school) Professional medicine Electrical engineering Elementary mathematics Psychology (high school) Statistics (high school) History (high school) Math (high school) Professional accounting Professional psychology Computer science (college) World history (high school) Macroeconomics Microeconomics Computer science (high school) European history Government and politics
Cotype Pro 3
MWS AI
0.793 0.566 0.928 0.897 0.882 0.871 0.816 0.807 0.861 0.78 0.737 0.746 0.56 0.861 0.877 0.821 0.79 0.931 0.8 0.878 0.732 0.877 0.77 0.815 0.768 0.96 0.64 0.667 0.804 0.68 0.8 0.926 0.853 0.909 0.849 0.927 0.74 0.948 0.781 0.828 0.914 0.915 0.772 0.875 0.927 0.806 0.922 0.693 0.677 0.825 0.79 0.928 0.892 0.945 0.93 0.861 0.927
Model, team SIM FL STA
Cotype Pro 3
MWS AI
0.822 0.654 0.62
Model, team Anatomy Virology Astronomy Marketing Nutrition Sociology Managment Philosophy Pre-History Gerontology Econometrics Formal logic Global facts Jurisprudence Miscellaneous Moral disputes Business ethics Bilology (college) Physics (college) Human sexuality Moral scenarios World religions Abstract algebra Medicine (college) Machine Learning Genetics Professional law PR Security Chemistry (college) Computer security International law Logical fallacies Politics Clinical knowledge Conceptual physics Math (college) Biology (high school) Physics (high school) Chemistry (high school) Geography (high school) Professional medicine Electrical Engineering Elementary mathematics Psychology (high school) Statistics (high school) History (high school) Math (high school) Professional Accounting Professional psychology Computer science (college) World history (high school) Macroeconomics Microeconomics Computer science (high school) Europe History Government and politics
Cotype Pro 3
MWS AI
0.822 0.921 0.8 0.824 0.961 0.81 0.81 0.789 0.942 0.815 0.833 0.858 0.633 0.884 0.901 0.79 0.804 0.844 0.93 0.877 0.912 0.966 0.956 0.929 0.933 0.939 0.897 0.737 0.947 0.978 0.867 0.936 0.92 0.947 0.848 0.946 0.933 0.889 0.93 0.954 0.947 0.937 0.844 1 0.931 0.911 0.914 0.932 0.954 0.947 0.911 0.957 0.911 0.844 0.791 0.836 0.933
Coorect
Good
Ethical
Model, team Virtue Law Moral Justice Utilitarianism
Cotype Pro 3
MWS AI
0.419 0.405 0.448 0.389 0.356
Model, team Virtue Law Moral Justice Utilitarianism
Cotype Pro 3
MWS AI
0.421 0.408 0.449 0.381 0.365
Model, team Virtue Law Moral Justice Utilitarianism
Cotype Pro 3
MWS AI
0.537 0.539 0.584 0.493 0.495
Model, team Women Men LGBT Nationalities Migrants Other
Cotype Pro 3
MWS AI
0.861 0.771 0.941 0.811 0.857 0.918