MultiQ
Task Description
MultiQ is a multi-hop QA dataset for Russian, suitable for general open-domain question answering, information retrieval, and reading comprehension tasks. The dataset is based on the dataset of the same name from the TAPE benchmark [1].
Keywords: Multi-hop QA, World Knowledge, Logic, Question-Answering
Authors: Ekaterina Taktasheva, Tatiana Shavrina, Alena Fenogenova, Denis Shevelev, Nadezhda Katricheva, Maria Tikhonova, Albina Akhmetgareeva, Oleg Zinkevich, Anastasiia Bashmakova, Svetlana Iordanskaia, Alena Spiridonova, Valentina Kurenshchikova, Ekaterina Artemova, Vladislav Mikhailov
Motivation
Question-answering has been an essential task in natural language processing and information retrieval. However, certain areas in QA remain quite challenging for modern approaches, including the multi-hop one, which is traditionally considered an intersection of graph methods, knowledge representation, and SOTA language modeling.
Dataset Description
Data Fields
meta
is a dictionary containing meta-information about the example:id
is the task ID;bridge_answer
is a list of entities necessary to answer the question contained in theoutputs
field using two available texts;
instruction
is an instructional prompt specified for the current task;inputs
is a dictionary containing the following information:text
is the main text line;support_text
is a line with additional text;question
is the question, the answer to which is contained in these texts;
outputs
is the answer information:label
is the answer label;length
is the answer length;offset
is the answer start index;segment
is a string containing the answer.
Data Instances
Each dataset sample consists of two texts (the main and the supporting ones) and a question based on these two texts. Below is an example from the dataset:
{
"instruction": "Прочитайте два текста и ответьте на вопрос.\\\\nТекст 1: {support_text}\\\\nТекст 2: {text}\\\\nВопрос: {question}\\\\nОтвет:",
"inputs": {
"question": "В какую реку впадает река, притоком которой является Висвож?",
"support_text": "Висвож — река в России, протекает по Республике Коми. Устье реки находится в 6 км по левому берегу реки Кыбантывис. Длина реки составляет 24 км.",
"text": "Кыбантывис (Кабан-Тывис) — река в России, протекает по Республике Коми. Левый приток Айювы. Длина реки составляет 31 км. Система водного объекта: Айюва → Ижма → Печора → Баренцево море."
},
"outputs": [{
"label": "answer",
"length": 5,
"offset": 85,
"segment": "Айювы"
}],
"meta": {
"id": 9,
"bridge_answers": [{
"label": "passage",
"length": 10,
"offset": 104,
"segment": "Кыбантывис"
}]
}
}
Data Splits
The dataset consists of 1056 training examples (train set) and 900 test examples (test set).
Prompts
We prepared 5 different prompts of various difficulties for this task. An example of the prompt is given below:
"Прочитайте два текста и ответьте на вопрос.\\\\nТекст 1: {support_text}\\\\nТекст 2: {text}\\\\nВопрос: {question}\\\\nОтвет:"
.
Dataset Creation
The dataset was created using the corresponding dataset from the TAPE benchmark [1] and was initially sampled from Wikipedia and Wikidata. The whole pipeline of the data collection can be found here.
Evaluation
Metrics
To evaluate models on this dataset, two metrics are used: F1-score and complete match (Exact Match — EM).
Human Benchmark
The F1-score / EM results are 0.928
/ 0.91
, respectively.
References
[1] Taktasheva, Ekaterina, et al. "TAPE: Assessing Few-shot Russian Language Understanding." Findings of the Association for Computational Linguistics: EMNLP 2022. 2022.