CheGeKa
Task Description
CheGeKa is a Jeopardy!-like the Russian QA dataset collected from the official Russian quiz database ChGK and belongs to the open-domain question-answering group of tasks. The dataset was created based on the corresponding dataset from the TAPE benchmark [1].
Keywords: Reasoning, World Knowledge, Logic, Question-Answering, Open-Domain QA
Authors: Ekaterina Taktasheva, Tatiana Shavrina, Alena Fenogenova, Denis Shevelev, Nadezhda Katricheva, Maria Tikhonova, Albina Akhmetgareeva, Oleg Zinkevich, Anastasiia Bashmakova, Svetlana Iordanskaia, Alena Spiridonova, Valentina Kurenshchikova, Ekaterina Artemova, Vladislav Mikhailov
Motivation
The task can be considered the most challenging in terms of reasoning, knowledge, and logic, as the task implies the QA pairs with a free response form (no answer choices); however, a long chain of causal relationships between facts and associations forms the correct answer.
Dataset Description
Data Fields
meta
is a dictionary containing meta-information about the example:id
is the task ID;author
is the author of the question;-
tour name
is the name of the game in which the question was used; -
tour_link
is a link to the game in which the question was used (None for the test set);
instruction
is an instructional prompt specified for the current task;inputs
is a dictionary containing the following input information:text
is a text fragment with a question from the game “What? Where? When?";topic
is a string containing the category of the question;
outputs
is a string containing the correct answer to the question.
Data Instances
Each instance in the dataset contains an instruction, a question, the topic of the question, the correct answer, and all the meta-information. Below is an example from the dataset:
{
"instruction": "Вы участвуете в викторине “Что? Где? Когда?”. Внимательно прочитайте вопрос из категории \\"{topic}\\" и ответьте на него.\\nВопрос: {text}\\nВ качестве ответа запишите только ваш вариант без дополнительных объяснений.\\nОтвет:",
"inputs": {
"text": "В корриде, кроме быка, он тоже играет одну из главных ролей.",
"topic": "\\"ТОР\\""
},
"outputs": [
"Тореадор"
],
"meta": {
"id": 7571,
"author": "Максим Стасюк",
"tour_name": "Своя игра. ШДК им. Рабиндраната Дебендранатовича Тагора",
"tour_link": ""
}
}
Data Splits
The dataset consists of 29376 training examples (train set) and 416 test examples (test set).
Prompts
We use four different prompts written in natural language for this task. An example of the prompt is given below:
"Вы участвуете в викторине “Что? Где? Когда?”. Категория вопроса: {topic}\\\\nВнимательно прочитайте вопрос и ответьте на него: {text}\\\\nОтвет:"
Dataset Creation
The dataset was created using the corresponding dataset from the TAPE benchmark [1], which is, in turn, based on the original corpus of the CheGeKa game introduced in [2].
Evaluation
Metrics
The dataset is evaluated via two metrics: F1-score and Exact Match (EM).
Human Benchmark
Human Benchmark was measured on a test set with Yandex.Toloka project with the overlap of 3 reviewers per task.
The F1-score / Exact Match results are 0.719
/ 0.645
, respectively.
References
[1] Taktasheva, Ekaterina, et al. "TAPE: Assessing Few-shot Russian Language Understanding." Findings of the Association for Computational Linguistics: EMNLP 2022. 2022.
[2] Mikhalkova, Elena, and Alexander A. Khlyupin. "Russian Jeopardy! Data Set for Question-Answering Systems." Proceedings of the Thirteenth Language Resources and Evaluation Conference. 2022.