SimpleAr

Task Description

Simple arithmetic is a mathematical task from BIG-Bench. The task itself tests language models' basic arithmetic capabilities by asking them to perform n-digit addition for a range of n.

Keywords: arithmetic, example task, free response, mathematics, numerical response, zero-shot

Motivation

The goal of the task is to analyze the ability of the model to solve simple mathematical addition tasks.

Dataset Description

Data Fields

instruction — is a string containing instructions for the task and information about the requirements for the model output format;
inputs — is the example of arithmetic expression;
outputs — is a string containing the correct answer of summation of two numbers;
meta — is a dictionary containing meta information:
- id — is an integer indicating the index of the example.

Data Instances

Below is an example from the dataset:

{
    "instruction": "Выполните арифметическую операцию.\\\\n{inputs}",
    "inputs": "901 + 164 = ",
    "outputs": "1065",
    "meta": {
        "id": 679
    }
}

Data Splits

The train set consists of 1000 examples of arithmetic expressions. The test set consists of 1000 examples of arithmetic expressions.

Prompts

The number of prompts used for the task: 6. The following prompts for the task are used:

    "Вычислите математическое выражение:\n{inputs}",
    "Выполните арифметическую операцию.\n{inputs}",
    "Напишите ответ для математического выражения.\n{inputs}",
    "Сложите два числа:\n{inputs}",
    "Сложите первое и второе слагаемые: {inputs} и напишите ответ.",
    "Выполните арифметическую операцию. В качестве ответа напишите число, которое получается после ее выполнения.\n{inputs}"

Dataset Creation

N-digit addition was created for n in the range [1;5] for both train and test sets.

Evaluation

Metrics

Exact Match (EM) is used for evaluation.

Human Benchmark

The human benchmark is measured on a subset of size 200 (sampled with the same original distribution). The final score for this task is 1.0.

Tasks