RUBIN

Task Description

RUBIN is a text dataset of Russian cultural-code items (HSE / Huawei). It evaluates whether models understand phrases and linguistic phenomena that are specific not only to the language itself, but to its cultural environment: memes, slang, songs, films, folklore, tongue twisters, sayings, and related material.

Evaluated skills: Culture code, Question Answering

Contributors: Polina Lazukova, Irina Piontkovskaya

Motivation

For a long time there was no publicly available cultural-code dataset for Russian. Even a narrowly specialized LLM can fail when culture-specific expressions appear in an otherwise specialized format. This matters for chatbots, document processing, machine translation, and localization of culture-bound references. The dataset checks how well a model detects and correctly interprets this specificity.

Limitations

The dataset is not suitable for evaluating literacy or other general language competencies. The current release contains questions and prompts in Russian only.

Intended Use of Results

The results are useful for businesses localizing into a Russian-speaking environment, machine-translation developers, LLM researchers, translators and adapters, marketers, linguists, and cultural studies specialists.

Evaluated Abilities

1. Up-to-date knowledge of cultural specifics — memes, folklore and slang constructions, quotes from songs and films, popular literary works, advertising slogans.
2. Ability to go beyond literal word meanings.
3. Matching synonyms at the level of cultural concepts rather than individual words.
4. Ability to interpret the use of cultural patterns.

The task format is multiple-choice with a single correct option (letters А, Б, В, Г). Many items follow the “How does the phrase end?” template. Distractors were generated by an LLM and additionally reviewed manually.

Dataset Description

The current MERA release contains 783 test examples and 5 few-shot examples (788 in total).

The original RUBIN pool has 5000 standardized questions. After HumanEval with native speakers (Yandex Tasks), items with human accuracy of at least 50% formed the RUBIN-Easy subset (4216 examples). This release mainly includes “hard” items (is_easy=false, human accuracy below 50%), plus 5 easy examples in shots.

Data Fields

Each example contains the following fields:

• instruction [str] — prompt template; curly-brace placeholders are filled from inputs.
• inputs — task inputs:
  • question [str] — question text;
  • option_a [str] — answer option A;
  • option_b [str] — answer option Б;
  • option_c [str] — answer option В;
  • option_d [str] — answer option Г;
• outputs [str] — the letter of the single correct answer: А, Б, В, or Г
• meta — metadata (hidden from the evaluated model):
  • id [int] — example identifier;
  • type [str] — cultural category (see below);
  • year [str] — year of appearance / peak cultural relevance; "Unknown" if unavailable;
  • is_phrase_ending [bool] — whether the item is a phrase-completion question;
  • is_easy [bool] — whether the item belonged to RUBIN-Easy according to HumanEval.

Categories (meta.type)

The release uses 10 values:

Song, Others, Proverb, Slang, Film, Riddle, Twister, Meme, Poems, Cartoon.

• Others

Data Example

{
    "instruction": "Помогите мне, пожалуйста.\n\nОтветьте на вопрос, выбрав один правильный вариант из предложенных.\n\nПоследняя строка ответа должна иметь вид:\n\nОтвет: <буква>\n\nДопустимые значения: А, Б, В, Г.\n\nВопрос:\n{question}\n\nВарианты ответа:\nА. {option_a}\nБ. {option_b}\nВ. {option_c}\nГ. {option_d}",
    "inputs": {
        "question": "Какая реакция из видео-ролика, где собака породы сиба-ину падает в реку, стала мемом о неудачах?",
        "option_a": "Это была классическая улыбка после падения, символизирующая оптимизм перед новыми возможностями",
        "option_b": "Дефлекторная манера собаки вылизывания себя после падения, как будто бы не обращая внимания на происходящее",
        "option_c": "Внезапное бегство собаки от воды, демонстрирующее ее умение быстро уходить от проблем",
        "option_d": "Это фиаско, братан"
    },
    "outputs": "Г",
    "meta": {
        "id": 137,
        "type": "Meme",
        "year": "2017",
        "is_phrase_ending": false,
        "is_easy": true
    }
}

Prompts

Five prompts were prepared. In test they are distributed evenly: 157 / 157 / 157 / 156 / 156. In shots each prompt appears once.

Example prompt:

Помогите мне, пожалуйста.

Ответьте на вопрос, выбрав один правильный вариант из предложенных.

Последняя строка ответа должна иметь вид:

Ответ: <буква>

Допустимые значения: А, Б, В, Г.

Вопрос:
{question}

Варианты ответа:
А. {option_a}
Б. {option_b}
В. {option_c}
Г. {option_d}

Few-shot examples use the same data format as the test set.

Dataset Construction

The original pool was built from folklore collections and songbooks, online meme encyclopedias, slang aggregators, and articles about popular films and cartoons. Plausible incorrect answers were generated with Qwen2.5-14B, then manually validated (topic, style, plausibility, no match with the correct answer, grammatical agreement).

Quality control included HumanEval with generation-aware annotation by native speakers (Yandex Tasks): participants aged 20+, native Russian, Russian citizenship and residence.

Evaluation

Metrics

• exact_match (Exact Match): the share of predictions whose extracted answer (after the Ответ: marker) exactly matches the gold letter.
• judge_score (LLM judge score): an LLM judge compares the model answer with the reference options and scores its correctness and completeness. A fully correct and complete answer gets 1; a partially correct or incomplete answer that still contains a substantial part of the solution gets 0.5; an incorrect, contradictory, or non-matching answer gets 0. The final metric is the mean score over all examples.