GPT4o-mini

MERA Created at 23.09.2024 14:22

0.57

The overall result

Place in the rating

In the top by tasks:

ruHumanEval

ruCodeEval

The task is one of the main ones

Weak tasks:

109

RWSD

PARus

RCB

128

ruEthics

MultiQ

ruWorldTree

ruOpenBookQA

100

CheGeKa

113

ruMMLU

ruHateSpeech

ruDetox

ruHHH

ruTiE

USE

149

MathLogicQA

121

ruMultiAr

SimpleAr

222

LCS

124

BPS

169

ruModAr

MaMuRAMu

+17

Hide

Ratings for leaderboard tasks

The table will scroll to the left

Task name	Result	Metric
LCS	0.108	Accuracy
RCB	0.571 / 0.507	Accuracy F1 macro
USE	0.303	Grade norm
RWSD	0.577	Accuracy
PARus	0.918	Accuracy
ruTiE	0.801	Accuracy
MultiQ	0.509 / 0.379	F1 Exact match
CheGeKa	0.293 / 0.233	F1 Exact match
ruModAr	0.495	Exact match
MaMuRAMu	0.779	Accuracy
ruMultiAr	0.301	Exact match
ruCodeEval	0.704 / 0.753 / 0.768	Pass@k
MathLogicQA	0.454	Accuracy
ruWorldTree	0.956 / 0.956	Accuracy F1 macro
ruOpenBookQA	0.875 / 0.874	Accuracy F1 macro

Evaluation on open tasks:

Go to the ratings by subcategory

The table will scroll to the left

Task name

Result

Metric

BPS

0.965

Accuracy

ruMMLU

0.652

Accuracy

SimpleAr

0.986

Exact match

ruHumanEval

0.704 / 0.733 / 0.744

Pass@k

ruHHH

0.809

ruHateSpeech

0.789

ruDetox

0.337

ruEthics

	Correct	God	Ethical
Virtue	0.385	0.32	0.372
Law	0.397	0.332	0.342
Moral	0.409	0.346	0.388
Justice	0.353	0.296	0.329
Utilitarianism	0.337	0.303	0.326

Information about the submission:

Mera version

v.1.2.0

Torch Version

2.0.1

The version of the codebase

1aa3a9aa

CUDA version

11.7

Precision of the model weights

Seed

1234

Butch

Transformers version

4.36.2

The number of GPUs and their type

Architecture

openai-chat-completions

Team:

MERA

Name of the ML model:

GPT4o-mini

Link to the ML model:

https://openai.com/api/pricing/

Model type:

Closed

API

SFT

Additional links:

Architecture description:

Version: gpt-4o-mini-2024-07-18 GPT-4o Mini is a simplified and more affordable version of the GPT-4O model from OpenAI. Starting from July 18, 2024, ChatGPT users at Free, Plus and Team tariffs can use GPT-4O Mini instead of GPT-3.5 Turbo.

Description of the training:

Pretrain data:

License:

https://openai.com/policies/terms-of-use/

Inference parameters

Generation Parameters:
simplear - do_sample=false;until=["\n"]; \nchegeka - do_sample=false;until=["\n"]; \nrudetox - do_sample=false;until=["\n"]; \nrumultiar - do_sample=false;until=["\n"]; \nuse - do_sample=false;until=["\n","."]; \nmultiq - do_sample=false;until=["\n"]; \nrumodar - do_sample=false;until=["\n"]; \nruhumaneval - do_sample=true;until=["\nclass","\ndef","\n#","\nif","\nprint"];temperature=0.6; \nrucodeeval - do_sample=true;until=["\nclass","\ndef","\n#","\nif","\nprint"];temperature=0.6;

System prompt:
Реши задачу по инструкции ниже. Не давай никаких объяснений и пояснений к своему ответу. Не пиши ничего лишнего. Пиши только то, что указано в инструкции. Если по инструкции нужно решить пример, то напиши только числовой ответ без хода решения и пояснений. Если по инструкции нужно вывести букву, цифру или слово, выведи только его. Если по инструкции нужно выбрать один из вариантов ответа и вывести букву или цифру, которая ему соответствует, то выведи только эту букву или цифру, не давай никаких пояснений, не добавляй знаки препинания, только 1 символ в ответе. Если по инструкции нужно дописать код функции на языке Python, пиши сразу код, соблюдая отступы так, будто ты продолжаешь функцию из инструкции, не давай пояснений, не пиши комментарии, используй только аргументы из сигнатуры функции в инструкции, не пробуй считывать данные через функцию input. Не извиняйся, не строй диалог. Выдавай только ответ и ничего больше.

Expand information

Ratings by subcategory

Metric: Grade Norm

Model, team	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	8_0	8_1	8_2	8_3	8_4
GPT4o-mini MERA	0.467	0.467	0.833	0.167	0.033	0.267	0.133	-	0	0.033	0.033	0.067	0.233	0.133	0.133	0.583	0.033	0.067	0	0.033	0.067	0.733	0.167	0.3	0.067	0.55	0.4	0.5	0.5	0.467	0.6

Model, team	Honest	Helpful	Harmless
GPT4o-mini MERA	0.836	0.763	0.828

Model, team	Anatomy	Virology	Astronomy	Marketing	Nutrition	Sociology	Management	Philosophy	Prehistory	Human aging	Econometrics	Formal logic	Global facts	Jurisprudence	Miscellaneous	Moral disputes	Business ethics	Biology (college)	Physics (college)	Human Sexuality	Moral scenarios	World religions	Abstract algebra	Medicine (college)	Machine learning	Medical genetics	Professional law	PR	Security studies	Chemistry (школьная)	Computer security	International law	Logical fallacies	Politics	Clinical knowledge	Conceptual_physics	Math (college)	Biology (high school)	Physics (high school)	Chemistry (high school)	Geography (high school)	Professional medicine	Electrical engineering	Elementary mathematics	Psychology (high school)	Statistics (high school)	History (high school)	Math (high school)	Professional accounting	Professional psychology	Computer science (college)	World history (high school)	Macroeconomics	Microeconomics	Computer science (high school)	European history	Government and politics
GPT4o-mini MERA	0.637	0.5	0.816	0.859	0.752	0.801	0.825	0.695	0.719	0.691	0.544	0.476	0.5	0.75	0.815	0.699	0.68	0.813	0.467	0.763	0.162	0.842	0.4	0.659	0.518	0.78	0.489	0.639	0.735	0.42	0.76	0.826	0.699	0.838	0.743	0.731	0.46	0.855	0.53	0.631	0.808	0.757	0.648	0.584	0.845	0.62	0.838	0.419	0.496	0.672	0.51	0.831	0.774	0.815	0.77	0.824	0.86

Model, team	SIM	FL	STA
GPT4o-mini MERA	0.702	0.707	0.711

Model, team	Anatomy	Virology	Astronomy	Marketing	Nutrition	Sociology	Managment	Philosophy	Pre-History	Gerontology	Econometrics	Formal logic	Global facts	Jurisprudence	Miscellaneous	Moral disputes	Business ethics	Bilology (college)	Physics (college)	Human sexuality	Moral scenarios	World religions	Abstract algebra	Medicine (college)	Machine Learning	Genetics	Professional law	PR	Security	Chemistry (college)	Computer security	International law	Logical fallacies	Politics	Clinical knowledge	Conceptual physics	Math (college)	Biology (high school)	Physics (high school)	Chemistry (high school)	Geography (high school)	Professional medicine	Electrical Engineering	Elementary mathematics	Psychology (high school)	Statistics (high school)	History (high school)	Math (high school)	Professional Accounting	Professional psychology	Computer science (college)	World history (high school)	Macroeconomics	Microeconomics	Computer science (high school)	Europe History	Government and politics
GPT4o-mini MERA	0.667	0.881	0.683	0.676	0.829	0.741	0.655	0.719	0.827	0.754	0.756	0.717	0.583	0.775	0.76	0.704	0.776	0.822	0.632	0.877	0.737	0.864	0.867	0.846	0.822	0.773	0.769	0.684	0.93	0.8	0.822	0.808	0.75	0.912	0.682	0.732	0.8	0.844	0.596	0.785	0.882	0.857	0.8	0.733	0.897	0.867	0.897	0.818	0.8	0.895	0.822	0.826	0.785	0.714	0.512	0.754	0.856

Coorect

Good

Ethical

Model, team	Virtue	Law	Moral	Justice	Utilitarianism
GPT4o-mini MERA	0.385	0.397	0.409	0.353	0.337

Model, team	Virtue	Law	Moral	Justice	Utilitarianism
GPT4o-mini MERA	0.32	0.332	0.346	0.296	0.303

Model, team	Virtue	Law	Moral	Justice	Utilitarianism
GPT4o-mini MERA	0.372	0.342	0.388	0.329	0.326

Model, team	Women	Men	LGBT	Nationalities	Migrants	Other
GPT4o-mini MERA	0.824	0.714	0.882	0.784	0.571	0.77

GPT4o-mini

Ratings for leaderboard tasks

Evaluation on open tasks:

Information about the submission:

Team:

Name of the ML model:

Link to the ML model:

Model type:

Additional links:

Architecture description:

Description of the training:

Pretrain data:

License:

Inference parameters

Ratings by subcategory

Confirm the deletion of the sub