Team:

MERA

Name of the ML model:

Human Benchmark

Link to the ML model:

Architecture description:

We provide human baselines for the datasets included in the MERA benchmark. Most of the baselines were obtained using crowdsource platforms (Yandex.Toloka & ABC Elementary) where the annotators were presented with the same instructions and tasks as it is intended for language models in MERA. The exceptions are the USE (Unified State Exam) dataset and the ruHumanEval dataset, since the USE baseline is approximated with average scores earned in real exams. The details about human assessment for Russian SuperGLUE and TAPE are covered in the corresponding papers (RSG paper, TAPE paper), we use the previous results of human evaluations on these two datasets.

Description of the training:

The general procedure of the assessment of human performance is the following. Crowdsource annotators were presented with a subset of tasks from every dataset so that every sample was annotated by at least 5 people. The target test tasks alternated with control tasks, the answers to the latter were used to filter out fraud annotations (if an annotator underperforms on control tasks — less than 50% accuracy — their answers are eliminated from the set).

Pretrain data:

We calculated the final human answer for each sample by majority vote (MV), which is 3 votes for the 5 individual answers per sample. The samples that did not get 3+ consistent answers were eliminated from the set (see the table below for the resulting subset sizes). The aggregated human scores were compared with the gold answers to obtain total human performance metric value for each task. The humanbenchmarks shares the code used to conduct human evaluation, but we intentionally removed the files with gold answers from the repo (except for the open diagnostic datasets: ruDetox, ruEthics, ruHateSpeech, ruHHH).

Training Details:

-

License:

MIT

Strategy, generation and parameters:

-