# Evaluator

The evaluator classes for automatic evaluation.

## Evaluator classes[[evaluate.evaluator]]

The main entry point for using the evaluator:

#### evaluate.evaluator[[evaluate.evaluator]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/__init__.py#L113)

Utility factory method to build an [Evaluator](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.Evaluator).
Evaluators encapsulate a task and a default metric name. They leverage `pipeline` functionality from `transformers`
to simplify the evaluation of multiple combinations of models, datasets and metrics for a given task.

Examples:
```python
>>> from evaluate import evaluator
>>> # Sentiment analysis evaluator
>>> evaluator("sentiment-analysis")
```

**Parameters:**

task (`str`) : The task defining which evaluator will be returned. Currently accepted tasks are: - `"image-classification"`: will return a [ImageClassificationEvaluator](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.ImageClassificationEvaluator). - `"question-answering"`: will return a [QuestionAnsweringEvaluator](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.QuestionAnsweringEvaluator). - `"text-classification"` (alias `"sentiment-analysis"` available): will return a [TextClassificationEvaluator](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.TextClassificationEvaluator). - `"token-classification"`: will return a [TokenClassificationEvaluator](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.TokenClassificationEvaluator).

**Returns:**

`[Evaluator](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.Evaluator)`

An evaluator suitable for the task.

The base class for all evaluator classes:

#### evaluate.Evaluator[[evaluate.Evaluator]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/base.py#L102)

The [Evaluator](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.Evaluator) class is the class from which all evaluators inherit. Refer to this class for methods shared across
different evaluators.
Base class implementing evaluator operations.

check_required_columnsevaluate.Evaluator.check_required_columnshttps://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/base.py#L295[{"name": "data", "val": ": typing.Union[str, datasets.arrow_dataset.Dataset]"}, {"name": "columns_names", "val": ": typing.Dict[str, str]"}]- **data** (`str` or `Dataset`) --
  Specifies the dataset we will run evaluation on.
- **columns_names** (`List[str]`) --
  List of column names to check in the dataset. The keys are the arguments to the [evaluate.EvaluationModule.compute()](/docs/evaluate/main/en/package_reference/main_classes#evaluate.EvaluationModule.compute) method,
  while the values are the column names to check.0

Ensure the columns required for the evaluation are present in the dataset.

Example:

```py
>>> from datasets import load_dataset
>>> from evaluate import evaluator
>>> data = load_dataset("rotten_tomatoes', split="train")
>>> evaluator.check_required_columns(data, {"input_column": "text", "label_column": "label"})
```

**Parameters:**

data (`str` or `Dataset`) : Specifies the dataset we will run evaluation on.

columns_names (`List[str]`) : List of column names to check in the dataset. The keys are the arguments to the [evaluate.EvaluationModule.compute()](/docs/evaluate/main/en/package_reference/main_classes#evaluate.EvaluationModule.compute) method, while the values are the column names to check.
#### compute_metric[[evaluate.Evaluator.compute_metric]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/base.py#L517)

Compute and return metrics.
#### get_dataset_split[[evaluate.Evaluator.get_dataset_split]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/base.py#L321)

Infers which split to use if `None` is given.

Example:

```py
>>> from evaluate import evaluator
>>> evaluator("text-classification").get_dataset_split(data="rotten_tomatoes")
WARNING:evaluate.evaluator.base:Dataset split not defined! Automatically evaluating with split: TEST
'test'
```

**Parameters:**

data (`str`) : Name of dataset.

subset (`str`) : Name of config for datasets with multiple configurations (e.g. 'glue/cola').

split (`str`, defaults to `None`) : Split to use.

**Returns:**

``split``

`str` containing which split to use
#### load_data[[evaluate.Evaluator.load_data]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/base.py#L350)

Load dataset with given subset and split.

Example:

```py
>>> from evaluate import evaluator
>>> evaluator("text-classification").load_data(data="rotten_tomatoes", split="train")
Dataset({
    features: ['text', 'label'],
    num_rows: 8530
})
```

**Parameters:**

data (`Dataset` or `str`, defaults to `None`) : Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset name, and load it. Otherwise we assume it represents a pre-loaded dataset.

subset (`str`, defaults to `None`) : Specifies dataset subset to be passed to `name` in `load_dataset`. To be used with datasets with several configurations (e.g. glue/sst2).

split (`str`, defaults to `None`) : User-defined dataset split by name (e.g. train, validation, test). Supports slice-split (`test[:n]`). If not defined and data is a `str` type, will automatically select the best one via `choose_split()`.

**Returns:**

`data (`Dataset`)`

Loaded dataset which will be used for evaluation.
#### predictions_processor[[evaluate.Evaluator.predictions_processor]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/base.py#L211)

A core method of the `Evaluator` class, which processes the pipeline outputs for compatibility with the metric.
#### prepare_data[[evaluate.Evaluator.prepare_data]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/base.py#L390)

Prepare data.

Example:

```py
>>> from evaluate import evaluator
>>> from datasets import load_dataset

>>> ds = load_dataset("rotten_tomatoes", split="train")
>>> evaluator("text-classification").prepare_data(ds, input_column="text", second_input_column=None, label_column="label")
```

**Parameters:**

data (`Dataset`) : Specifies the dataset we will run evaluation on.

input_column (`str`, defaults to `"text"`) : The name of the column containing the text feature in the dataset specified by `data`.

second_input_column(`str`, *optional*) : The name of the column containing the second text feature if there is one. Otherwise, set to `None`.

label_column (`str`, defaults to `"label"`) : The name of the column containing the labels in the dataset specified by `data`.

**Returns:**

``dict``

metric inputs.
`list`:  pipeline inputs.
#### prepare_metric[[evaluate.Evaluator.prepare_metric]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/base.py#L480)

Prepare metric.

Example:

```py
>>> from evaluate import evaluator
>>> evaluator("text-classification").prepare_metric("accuracy")
```

**Parameters:**

metric (`str` or [EvaluationModule](/docs/evaluate/main/en/package_reference/main_classes#evaluate.EvaluationModule), defaults to `None`) : Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and load it. Otherwise we assume it represents a pre-loaded metric.

**Returns:**

The loaded metric.
#### prepare_pipeline[[evaluate.Evaluator.prepare_pipeline]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/base.py#L422)

Prepare pipeline.

Example:

```py
>>> from evaluate import evaluator
>>> evaluator("text-classification").prepare_pipeline(model_or_pipeline="distilbert-base-uncased")
```

**Parameters:**

model_or_pipeline (`str` or [Pipeline](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#transformers.Pipeline) or `Callable` or [PreTrainedModel](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel) or `TFPreTrainedModel`, defaults to `None`) : If the argument in not specified, we initialize the default pipeline for the task. If the argument is of the type `str` or is a model instance, we use it to initialize a new [Pipeline](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#transformers.Pipeline) with the given model. Otherwise we assume the argument specifies a pre-initialized pipeline.

preprocessor ([PreTrainedTokenizerBase](https://huggingface.co/docs/transformers/main/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase) or [FeatureExtractionMixin](https://huggingface.co/docs/transformers/main/en/main_classes/feature_extractor#transformers.FeatureExtractionMixin), *optional*, defaults to `None`) : Argument can be used to overwrite a default preprocessor if `model_or_pipeline` represents a model for which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore this argument.

**Returns:**

The initialized pipeline.

## The task specific evaluators

### ImageClassificationEvaluator[[evaluate.ImageClassificationEvaluator]]

#### evaluate.ImageClassificationEvaluator[[evaluate.ImageClassificationEvaluator]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/image_classification.py#L49)

Image classification evaluator.
This image classification evaluator can currently be loaded from [evaluator()](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.evaluator) using the default task name
`image-classification`.
Methods in this class assume a data format compatible with the `ImageClassificationPipeline`.

computeevaluate.ImageClassificationEvaluator.computehttps://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/image_classification.py#L68[{"name": "model_or_pipeline", "val": ": typing.Union[str, ForwardRef('Pipeline'), typing.Callable, ForwardRef('PreTrainedModel'), ForwardRef('TFPreTrainedModel')] = None"}, {"name": "data", "val": ": typing.Union[str, datasets.arrow_dataset.Dataset] = None"}, {"name": "subset", "val": ": typing.Optional[str] = None"}, {"name": "split", "val": ": typing.Optional[str] = None"}, {"name": "metric", "val": ": typing.Union[str, evaluate.module.EvaluationModule] = None"}, {"name": "tokenizer", "val": ": typing.Union[str, ForwardRef('PreTrainedTokenizer'), NoneType] = None"}, {"name": "feature_extractor", "val": ": typing.Union[str, ForwardRef('FeatureExtractionMixin'), NoneType] = None"}, {"name": "strategy", "val": ": typing.Literal['simple', 'bootstrap'] = 'simple'"}, {"name": "confidence_level", "val": ": float = 0.95"}, {"name": "n_resamples", "val": ": int = 9999"}, {"name": "device", "val": ": int = None"}, {"name": "random_state", "val": ": typing.Optional[int] = None"}, {"name": "input_column", "val": ": str = 'image'"}, {"name": "label_column", "val": ": str = 'label'"}, {"name": "label_mapping", "val": ": typing.Optional[typing.Dict[str, numbers.Number]] = None"}]- **model_or_pipeline** (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) --
  If the argument in not specified, we initialize the default pipeline for the task (in this case
  `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or
  is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the
  argument specifies a pre-initialized pipeline.
- **data** (`str` or `Dataset`, defaults to `None`) --
  Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset
  name, and load it. Otherwise we assume it represents a pre-loaded dataset.
- **subset** (`str`, defaults to `None`) --
  Defines which dataset subset to load. If `None` is passed the default subset is loaded.
- **split** (`str`, defaults to `None`) --
  Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.
- **metric** (`str` or `EvaluationModule`, defaults to `None`) --
  Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and
  load it. Otherwise we assume it represents a pre-loaded metric.
- **tokenizer** (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) --
  Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for
  which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore
  this argument.
- **strategy** (`Literal["simple", "bootstrap"]`, defaults to "simple") --
  specifies the evaluation strategy. Possible values are:
  - `"simple"` - we evaluate the metric and return the scores.
  - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each
  of the returned metric keys, using `scipy`'s `bootstrap` method
  https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.
- **confidence_level** (`float`, defaults to `0.95`) --
  The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **n_resamples** (`int`, defaults to `9999`) --
  The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **device** (`int`, defaults to `None`) --
  Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive
  integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and
  CUDA:0 used if available, CPU otherwise.
- **random_state** (`int`, *optional*, defaults to `None`) --
  The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for
  debugging.

- **input_column** (`str`, defaults to `"image"`) --
  The name of the column containing the images as PIL ImageFile in the dataset specified by `data`.
- **label_column** (`str`, defaults to `"label"`) --
  The name of the column containing the labels in the dataset specified by `data`.
- **label_mapping** (`Dict[str, Number]`, *optional*, defaults to `None`) --
  We want to map class labels defined by the model in the pipeline to values consistent with those
  defined in the `label_column` of the `data` dataset.0A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

Compute the metric for a given pipeline and dataset combination.

Examples:
```python
>>> from evaluate import evaluator
>>> from datasets import load_dataset
>>> task_evaluator = evaluator("image-classification")
>>> data = load_dataset("beans", split="test[:40]")
>>> results = task_evaluator.compute(
>>>     model_or_pipeline="nateraw/vit-base-beans",
>>>     data=data,
>>>     label_column="labels",
>>>     metric="accuracy",
>>>     label_mapping={'angular_leaf_spot': 0, 'bean_rust': 1, 'healthy': 2},
>>>     strategy="bootstrap"
>>> )
```

**Parameters:**

model_or_pipeline (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) : If the argument in not specified, we initialize the default pipeline for the task (in this case `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the argument specifies a pre-initialized pipeline.

data (`str` or `Dataset`, defaults to `None`) : Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset name, and load it. Otherwise we assume it represents a pre-loaded dataset.

subset (`str`, defaults to `None`) : Defines which dataset subset to load. If `None` is passed the default subset is loaded.

split (`str`, defaults to `None`) : Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.

metric (`str` or `EvaluationModule`, defaults to `None`) : Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and load it. Otherwise we assume it represents a pre-loaded metric.

tokenizer (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) : Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore this argument.

strategy (`Literal["simple", "bootstrap"]`, defaults to "simple") : specifies the evaluation strategy. Possible values are: - `"simple"` - we evaluate the metric and return the scores. - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each of the returned metric keys, using `scipy`'s `bootstrap` method https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.

confidence_level (`float`, defaults to `0.95`) : The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

n_resamples (`int`, defaults to `9999`) : The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

device (`int`, defaults to `None`) : Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and CUDA:0 used if available, CPU otherwise.

random_state (`int`, *optional*, defaults to `None`) : The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for debugging.   

input_column (`str`, defaults to `"image"`) : The name of the column containing the images as PIL ImageFile in the dataset specified by `data`.

label_column (`str`, defaults to `"label"`) : The name of the column containing the labels in the dataset specified by `data`.

label_mapping (`Dict[str, Number]`, *optional*, defaults to `None`) : We want to map class labels defined by the model in the pipeline to values consistent with those defined in the `label_column` of the `data` dataset.

**Returns:**

A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

### QuestionAnsweringEvaluator[[evaluate.QuestionAnsweringEvaluator]]

#### evaluate.QuestionAnsweringEvaluator[[evaluate.QuestionAnsweringEvaluator]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/question_answering.py#L75)

Question answering evaluator. This evaluator handles
[**extractive** question answering](https://huggingface.co/docs/transformers/task_summary#extractive-question-answering),
where the answer to the question is extracted from a context.

This question answering evaluator can currently be loaded from [evaluator()](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.evaluator) using the default task name
`question-answering`.

Methods in this class assume a data format compatible with the
`QuestionAnsweringPipeline`.

computeevaluate.QuestionAnsweringEvaluator.computehttps://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/question_answering.py#L144[{"name": "model_or_pipeline", "val": ": typing.Union[str, ForwardRef('Pipeline'), typing.Callable, ForwardRef('PreTrainedModel'), ForwardRef('TFPreTrainedModel')] = None"}, {"name": "data", "val": ": typing.Union[str, datasets.arrow_dataset.Dataset] = None"}, {"name": "subset", "val": ": typing.Optional[str] = None"}, {"name": "split", "val": ": typing.Optional[str] = None"}, {"name": "metric", "val": ": typing.Union[str, evaluate.module.EvaluationModule] = None"}, {"name": "tokenizer", "val": ": typing.Union[str, ForwardRef('PreTrainedTokenizer'), NoneType] = None"}, {"name": "strategy", "val": ": typing.Literal['simple', 'bootstrap'] = 'simple'"}, {"name": "confidence_level", "val": ": float = 0.95"}, {"name": "n_resamples", "val": ": int = 9999"}, {"name": "device", "val": ": int = None"}, {"name": "random_state", "val": ": typing.Optional[int] = None"}, {"name": "question_column", "val": ": str = 'question'"}, {"name": "context_column", "val": ": str = 'context'"}, {"name": "id_column", "val": ": str = 'id'"}, {"name": "label_column", "val": ": str = 'answers'"}, {"name": "squad_v2_format", "val": ": typing.Optional[bool] = None"}]- **model_or_pipeline** (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) --
  If the argument in not specified, we initialize the default pipeline for the task (in this case
  `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or
  is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the
  argument specifies a pre-initialized pipeline.
- **data** (`str` or `Dataset`, defaults to `None`) --
  Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset
  name, and load it. Otherwise we assume it represents a pre-loaded dataset.
- **subset** (`str`, defaults to `None`) --
  Defines which dataset subset to load. If `None` is passed the default subset is loaded.
- **split** (`str`, defaults to `None`) --
  Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.
- **metric** (`str` or `EvaluationModule`, defaults to `None`) --
  Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and
  load it. Otherwise we assume it represents a pre-loaded metric.
- **tokenizer** (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) --
  Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for
  which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore
  this argument.
- **strategy** (`Literal["simple", "bootstrap"]`, defaults to "simple") --
  specifies the evaluation strategy. Possible values are:
  - `"simple"` - we evaluate the metric and return the scores.
  - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each
  of the returned metric keys, using `scipy`'s `bootstrap` method
  https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.
- **confidence_level** (`float`, defaults to `0.95`) --
  The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **n_resamples** (`int`, defaults to `9999`) --
  The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **device** (`int`, defaults to `None`) --
  Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive
  integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and
  CUDA:0 used if available, CPU otherwise.
- **random_state** (`int`, *optional*, defaults to `None`) --
  The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for
  debugging.

- **question_column** (`str`, defaults to `"question"`) --
  The name of the column containing the question in the dataset specified by `data`.
- **context_column** (`str`, defaults to `"context"`) --
  The name of the column containing the context in the dataset specified by `data`.
- **id_column** (`str`, defaults to `"id"`) --
  The name of the column containing the identification field of the question and answer pair in the
  dataset specified by `data`.
- **label_column** (`str`, defaults to `"answers"`) --
  The name of the column containing the answers in the dataset specified by `data`.
- **squad_v2_format** (`bool`, *optional*, defaults to `None`) --
  Whether the dataset follows the format of squad_v2 dataset. This is the case when the provided dataset
  has questions where the answer is not in the context, more specifically when are answers as
  `{"text": [], "answer_start": []}` in the answer column. If all questions have at least one answer, this parameter
  should be set to `False`. If this parameter is not provided, the format will be automatically inferred.0A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

Compute the metric for a given pipeline and dataset combination.

Examples:
```python
>>> from evaluate import evaluator
>>> from datasets import load_dataset
>>> task_evaluator = evaluator("question-answering")
>>> data = load_dataset("squad", split="validation[:2]")
>>> results = task_evaluator.compute(
>>>     model_or_pipeline="sshleifer/tiny-distilbert-base-cased-distilled-squad",
>>>     data=data,
>>>     metric="squad",
>>> )
```

> [!TIP]
> Datasets where the answer may be missing in the context are supported, for example SQuAD v2 dataset. In this case, it is safer to pass `squad_v2_format=True` to
> the compute() call.

```python
>>> from evaluate import evaluator
>>> from datasets import load_dataset
>>> task_evaluator = evaluator("question-answering")
>>> data = load_dataset("squad_v2", split="validation[:2]")
>>> results = task_evaluator.compute(
>>>     model_or_pipeline="mrm8488/bert-tiny-finetuned-squadv2",
>>>     data=data,
>>>     metric="squad_v2",
>>>     squad_v2_format=True,
>>> )
```

**Parameters:**

model_or_pipeline (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) : If the argument in not specified, we initialize the default pipeline for the task (in this case `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the argument specifies a pre-initialized pipeline.

data (`str` or `Dataset`, defaults to `None`) : Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset name, and load it. Otherwise we assume it represents a pre-loaded dataset.

subset (`str`, defaults to `None`) : Defines which dataset subset to load. If `None` is passed the default subset is loaded.

split (`str`, defaults to `None`) : Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.

metric (`str` or `EvaluationModule`, defaults to `None`) : Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and load it. Otherwise we assume it represents a pre-loaded metric.

tokenizer (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) : Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore this argument.

strategy (`Literal["simple", "bootstrap"]`, defaults to "simple") : specifies the evaluation strategy. Possible values are: - `"simple"` - we evaluate the metric and return the scores. - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each of the returned metric keys, using `scipy`'s `bootstrap` method https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.

confidence_level (`float`, defaults to `0.95`) : The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

n_resamples (`int`, defaults to `9999`) : The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

device (`int`, defaults to `None`) : Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and CUDA:0 used if available, CPU otherwise.

random_state (`int`, *optional*, defaults to `None`) : The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for debugging.   

question_column (`str`, defaults to `"question"`) : The name of the column containing the question in the dataset specified by `data`.

context_column (`str`, defaults to `"context"`) : The name of the column containing the context in the dataset specified by `data`.

id_column (`str`, defaults to `"id"`) : The name of the column containing the identification field of the question and answer pair in the dataset specified by `data`.

label_column (`str`, defaults to `"answers"`) : The name of the column containing the answers in the dataset specified by `data`.

squad_v2_format (`bool`, *optional*, defaults to `None`) : Whether the dataset follows the format of squad_v2 dataset. This is the case when the provided dataset has questions where the answer is not in the context, more specifically when are answers as `{"text": [], "answer_start": []}` in the answer column. If all questions have at least one answer, this parameter should be set to `False`. If this parameter is not provided, the format will be automatically inferred.

**Returns:**

A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

### TextClassificationEvaluator[[evaluate.TextClassificationEvaluator]]

#### evaluate.TextClassificationEvaluator[[evaluate.TextClassificationEvaluator]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/text_classification.py#L51)

Text classification evaluator.
This text classification evaluator can currently be loaded from [evaluator()](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.evaluator) using the default task name
`text-classification` or with a `"sentiment-analysis"` alias.
Methods in this class assume a data format compatible with the [TextClassificationPipeline](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#transformers.TextClassificationPipeline) - a single textual
feature as input and a categorical label as output.

computeevaluate.TextClassificationEvaluator.computehttps://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/text_classification.py#L89[{"name": "model_or_pipeline", "val": ": typing.Union[str, ForwardRef('Pipeline'), typing.Callable, ForwardRef('PreTrainedModel'), ForwardRef('TFPreTrainedModel')] = None"}, {"name": "data", "val": ": typing.Union[str, datasets.arrow_dataset.Dataset] = None"}, {"name": "subset", "val": ": typing.Optional[str] = None"}, {"name": "split", "val": ": typing.Optional[str] = None"}, {"name": "metric", "val": ": typing.Union[str, evaluate.module.EvaluationModule] = None"}, {"name": "tokenizer", "val": ": typing.Union[str, ForwardRef('PreTrainedTokenizer'), NoneType] = None"}, {"name": "feature_extractor", "val": ": typing.Union[str, ForwardRef('FeatureExtractionMixin'), NoneType] = None"}, {"name": "strategy", "val": ": typing.Literal['simple', 'bootstrap'] = 'simple'"}, {"name": "confidence_level", "val": ": float = 0.95"}, {"name": "n_resamples", "val": ": int = 9999"}, {"name": "device", "val": ": int = None"}, {"name": "random_state", "val": ": typing.Optional[int] = None"}, {"name": "input_column", "val": ": str = 'text'"}, {"name": "second_input_column", "val": ": typing.Optional[str] = None"}, {"name": "label_column", "val": ": str = 'label'"}, {"name": "label_mapping", "val": ": typing.Optional[typing.Dict[str, numbers.Number]] = None"}]- **model_or_pipeline** (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) --
  If the argument in not specified, we initialize the default pipeline for the task (in this case
  `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or
  is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the
  argument specifies a pre-initialized pipeline.
- **data** (`str` or `Dataset`, defaults to `None`) --
  Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset
  name, and load it. Otherwise we assume it represents a pre-loaded dataset.
- **subset** (`str`, defaults to `None`) --
  Defines which dataset subset to load. If `None` is passed the default subset is loaded.
- **split** (`str`, defaults to `None`) --
  Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.
- **metric** (`str` or `EvaluationModule`, defaults to `None`) --
  Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and
  load it. Otherwise we assume it represents a pre-loaded metric.
- **tokenizer** (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) --
  Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for
  which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore
  this argument.
- **strategy** (`Literal["simple", "bootstrap"]`, defaults to "simple") --
  specifies the evaluation strategy. Possible values are:
  - `"simple"` - we evaluate the metric and return the scores.
  - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each
  of the returned metric keys, using `scipy`'s `bootstrap` method
  https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.
- **confidence_level** (`float`, defaults to `0.95`) --
  The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **n_resamples** (`int`, defaults to `9999`) --
  The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **device** (`int`, defaults to `None`) --
  Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive
  integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and
  CUDA:0 used if available, CPU otherwise.
- **random_state** (`int`, *optional*, defaults to `None`) --
  The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for
  debugging.

- **input_column** (`str`, *optional*, defaults to `"text"`) --
  The name of the column containing the text feature in the dataset specified by `data`.
- **second_input_column** (`str`, *optional*, defaults to `None`) --
  The name of the second column containing the text features. This may be useful for classification tasks
  as MNLI, where two columns are used.
- **label_column** (`str`, defaults to `"label"`) --
  The name of the column containing the labels in the dataset specified by `data`.
- **label_mapping** (`Dict[str, Number]`, *optional*, defaults to `None`) --
  We want to map class labels defined by the model in the pipeline to values consistent with those
  defined in the `label_column` of the `data` dataset.0A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

Compute the metric for a given pipeline and dataset combination.

Examples:
```python
>>> from evaluate import evaluator
>>> from datasets import load_dataset
>>> task_evaluator = evaluator("text-classification")
>>> data = load_dataset("imdb", split="test[:2]")
>>> results = task_evaluator.compute(
>>>     model_or_pipeline="huggingface/prunebert-base-uncased-6-finepruned-w-distil-mnli",
>>>     data=data,
>>>     metric="accuracy",
>>>     label_mapping={"LABEL_0": 0.0, "LABEL_1": 1.0},
>>>     strategy="bootstrap",
>>>     n_resamples=10,
>>>     random_state=0
>>> )
```

**Parameters:**

model_or_pipeline (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) : If the argument in not specified, we initialize the default pipeline for the task (in this case `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the argument specifies a pre-initialized pipeline.

data (`str` or `Dataset`, defaults to `None`) : Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset name, and load it. Otherwise we assume it represents a pre-loaded dataset.

subset (`str`, defaults to `None`) : Defines which dataset subset to load. If `None` is passed the default subset is loaded.

split (`str`, defaults to `None`) : Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.

metric (`str` or `EvaluationModule`, defaults to `None`) : Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and load it. Otherwise we assume it represents a pre-loaded metric.

tokenizer (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) : Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore this argument.

strategy (`Literal["simple", "bootstrap"]`, defaults to "simple") : specifies the evaluation strategy. Possible values are: - `"simple"` - we evaluate the metric and return the scores. - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each of the returned metric keys, using `scipy`'s `bootstrap` method https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.

confidence_level (`float`, defaults to `0.95`) : The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

n_resamples (`int`, defaults to `9999`) : The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

device (`int`, defaults to `None`) : Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and CUDA:0 used if available, CPU otherwise.

random_state (`int`, *optional*, defaults to `None`) : The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for debugging.   

input_column (`str`, *optional*, defaults to `"text"`) : The name of the column containing the text feature in the dataset specified by `data`.

second_input_column (`str`, *optional*, defaults to `None`) : The name of the second column containing the text features. This may be useful for classification tasks as MNLI, where two columns are used.

label_column (`str`, defaults to `"label"`) : The name of the column containing the labels in the dataset specified by `data`.

label_mapping (`Dict[str, Number]`, *optional*, defaults to `None`) : We want to map class labels defined by the model in the pipeline to values consistent with those defined in the `label_column` of the `data` dataset.

**Returns:**

A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

### TokenClassificationEvaluator[[evaluate.TokenClassificationEvaluator]]

#### evaluate.TokenClassificationEvaluator[[evaluate.TokenClassificationEvaluator]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/token_classification.py#L84)

Token classification evaluator.

This token classification evaluator can currently be loaded from [evaluator()](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.evaluator) using the default task name
`token-classification`.

Methods in this class assume a data format compatible with the [TokenClassificationPipeline](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#transformers.TokenClassificationPipeline).

computeevaluate.TokenClassificationEvaluator.computehttps://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/token_classification.py#L212[{"name": "model_or_pipeline", "val": ": typing.Union[str, ForwardRef('Pipeline'), typing.Callable, ForwardRef('PreTrainedModel'), ForwardRef('TFPreTrainedModel')] = None"}, {"name": "data", "val": ": typing.Union[str, datasets.arrow_dataset.Dataset] = None"}, {"name": "subset", "val": ": typing.Optional[str] = None"}, {"name": "split", "val": ": str = None"}, {"name": "metric", "val": ": typing.Union[str, evaluate.module.EvaluationModule] = None"}, {"name": "tokenizer", "val": ": typing.Union[str, ForwardRef('PreTrainedTokenizer'), NoneType] = None"}, {"name": "strategy", "val": ": typing.Literal['simple', 'bootstrap'] = 'simple'"}, {"name": "confidence_level", "val": ": float = 0.95"}, {"name": "n_resamples", "val": ": int = 9999"}, {"name": "device", "val": ": typing.Optional[int] = None"}, {"name": "random_state", "val": ": typing.Optional[int] = None"}, {"name": "input_column", "val": ": str = 'tokens'"}, {"name": "label_column", "val": ": str = 'ner_tags'"}, {"name": "join_by", "val": ": typing.Optional[str] = ' '"}]- **model_or_pipeline** (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) --
  If the argument in not specified, we initialize the default pipeline for the task (in this case
  `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or
  is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the
  argument specifies a pre-initialized pipeline.
- **data** (`str` or `Dataset`, defaults to `None`) --
  Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset
  name, and load it. Otherwise we assume it represents a pre-loaded dataset.
- **subset** (`str`, defaults to `None`) --
  Defines which dataset subset to load. If `None` is passed the default subset is loaded.
- **split** (`str`, defaults to `None`) --
  Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.
- **metric** (`str` or `EvaluationModule`, defaults to `None`) --
  Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and
  load it. Otherwise we assume it represents a pre-loaded metric.
- **tokenizer** (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) --
  Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for
  which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore
  this argument.
- **strategy** (`Literal["simple", "bootstrap"]`, defaults to "simple") --
  specifies the evaluation strategy. Possible values are:
  - `"simple"` - we evaluate the metric and return the scores.
  - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each
  of the returned metric keys, using `scipy`'s `bootstrap` method
  https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.
- **confidence_level** (`float`, defaults to `0.95`) --
  The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **n_resamples** (`int`, defaults to `9999`) --
  The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **device** (`int`, defaults to `None`) --
  Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive
  integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and
  CUDA:0 used if available, CPU otherwise.
- **random_state** (`int`, *optional*, defaults to `None`) --
  The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for
  debugging.

- **input_column** (`str`, defaults to `"tokens"`) --
  The name of the column containing the tokens feature in the dataset specified by `data`.
- **label_column** (`str`, defaults to `"label"`) --
  The name of the column containing the labels in the dataset specified by `data`.
- **join_by** (`str`, *optional*, defaults to `" "`) --
  This evaluator supports dataset whose input column is a list of words. This parameter specifies how to join
  words to generate a string input. This is especially useful for languages that do not separate words by a space.0A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

Compute the metric for a given pipeline and dataset combination.

The dataset input and label columns are expected to be formatted as a list of words and a list of labels respectively, following [conll2003 dataset](https://huggingface.co/datasets/conll2003). Datasets whose inputs are single strings, and labels are a list of offset are not supported.

Examples:
```python
>>> from evaluate import evaluator
>>> from datasets import load_dataset
>>> task_evaluator = evaluator("token-classification")
>>> data = load_dataset("conll2003", split="validation[:2]")
>>> results = task_evaluator.compute(
>>>     model_or_pipeline="elastic/distilbert-base-uncased-finetuned-conll03-english",
>>>     data=data,
>>>     metric="seqeval",
>>> )
```

> [!TIP]
> For example, the following dataset format is accepted by the evaluator:
>
> ```python
> dataset = Dataset.from_dict(
>     mapping={
>         "tokens": [["New", "York", "is", "a", "city", "and", "Felix", "a", "person", "."]],
>         "ner_tags": [[1, 2, 0, 0, 0, 0, 3, 0, 0, 0]],
>     },
>     features=Features({
>         "tokens": Sequence(feature=Value(dtype="string")),
>         "ner_tags": Sequence(feature=ClassLabel(names=["O", "B-LOC", "I-LOC", "B-PER", "I-PER"])),
>         }),
> )
> ```

> [!WARNING]
> For example, the following dataset format is **not** accepted by the evaluator:
>
> ```python
> dataset = Dataset.from_dict(
>     mapping={
>         "tokens": [["New York is a city and Felix a person."]],
>         "starts": [[0, 23]],
>         "ends": [[7, 27]],
>         "ner_tags": [["LOC", "PER"]],
>     },
>     features=Features({
>         "tokens": Value(dtype="string"),
>         "starts": Sequence(feature=Value(dtype="int32")),
>         "ends": Sequence(feature=Value(dtype="int32")),
>         "ner_tags": Sequence(feature=Value(dtype="string")),
>     }),
> )
> ```

**Parameters:**

model_or_pipeline (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) : If the argument in not specified, we initialize the default pipeline for the task (in this case `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the argument specifies a pre-initialized pipeline.

data (`str` or `Dataset`, defaults to `None`) : Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset name, and load it. Otherwise we assume it represents a pre-loaded dataset.

subset (`str`, defaults to `None`) : Defines which dataset subset to load. If `None` is passed the default subset is loaded.

split (`str`, defaults to `None`) : Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.

metric (`str` or `EvaluationModule`, defaults to `None`) : Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and load it. Otherwise we assume it represents a pre-loaded metric.

tokenizer (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) : Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore this argument.

strategy (`Literal["simple", "bootstrap"]`, defaults to "simple") : specifies the evaluation strategy. Possible values are: - `"simple"` - we evaluate the metric and return the scores. - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each of the returned metric keys, using `scipy`'s `bootstrap` method https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.

confidence_level (`float`, defaults to `0.95`) : The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

n_resamples (`int`, defaults to `9999`) : The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

device (`int`, defaults to `None`) : Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and CUDA:0 used if available, CPU otherwise.

random_state (`int`, *optional*, defaults to `None`) : The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for debugging.   

input_column (`str`, defaults to `"tokens"`) : The name of the column containing the tokens feature in the dataset specified by `data`.

label_column (`str`, defaults to `"label"`) : The name of the column containing the labels in the dataset specified by `data`.

join_by (`str`, *optional*, defaults to `" "`) : This evaluator supports dataset whose input column is a list of words. This parameter specifies how to join words to generate a string input. This is especially useful for languages that do not separate words by a space.

**Returns:**

A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

### TextGenerationEvaluator[[evaluate.TextGenerationEvaluator]]

#### evaluate.TextGenerationEvaluator[[evaluate.TextGenerationEvaluator]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/text_generation.py#L31)

Text generation evaluator.
This Text generation evaluator can currently be loaded from [evaluator()](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.evaluator) using the default task name
`text-generation`.
Methods in this class assume a data format compatible with the [TextGenerationPipeline](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#transformers.TextGenerationPipeline).

computeevaluate.TextGenerationEvaluator.computehttps://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/base.py#L218[{"name": "model_or_pipeline", "val": ": typing.Union[str, ForwardRef('Pipeline'), typing.Callable, ForwardRef('PreTrainedModel'), ForwardRef('TFPreTrainedModel')] = None"}, {"name": "data", "val": ": typing.Union[str, datasets.arrow_dataset.Dataset] = None"}, {"name": "subset", "val": ": typing.Optional[str] = None"}, {"name": "split", "val": ": typing.Optional[str] = None"}, {"name": "metric", "val": ": typing.Union[str, evaluate.module.EvaluationModule] = None"}, {"name": "tokenizer", "val": ": typing.Union[str, ForwardRef('PreTrainedTokenizer'), NoneType] = None"}, {"name": "feature_extractor", "val": ": typing.Union[str, ForwardRef('FeatureExtractionMixin'), NoneType] = None"}, {"name": "strategy", "val": ": typing.Literal['simple', 'bootstrap'] = 'simple'"}, {"name": "confidence_level", "val": ": float = 0.95"}, {"name": "n_resamples", "val": ": int = 9999"}, {"name": "device", "val": ": int = None"}, {"name": "random_state", "val": ": typing.Optional[int] = None"}, {"name": "input_column", "val": ": str = 'text'"}, {"name": "label_column", "val": ": str = 'label'"}, {"name": "label_mapping", "val": ": typing.Optional[typing.Dict[str, numbers.Number]] = None"}]

### Text2TextGenerationEvaluator[[evaluate.Text2TextGenerationEvaluator]]

#### evaluate.Text2TextGenerationEvaluator[[evaluate.Text2TextGenerationEvaluator]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/text2text_generation.py#L88)

Text2Text generation evaluator.
This Text2Text generation evaluator can currently be loaded from [evaluator()](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.evaluator) using the default task name
`text2text-generation`.
Methods in this class assume a data format compatible with the `Text2TextGenerationPipeline`.

computeevaluate.Text2TextGenerationEvaluator.computehttps://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/text2text_generation.py#L105[{"name": "model_or_pipeline", "val": ": typing.Union[str, ForwardRef('Pipeline'), typing.Callable, ForwardRef('PreTrainedModel'), ForwardRef('TFPreTrainedModel')] = None"}, {"name": "data", "val": ": typing.Union[str, datasets.arrow_dataset.Dataset] = None"}, {"name": "subset", "val": ": typing.Optional[str] = None"}, {"name": "split", "val": ": typing.Optional[str] = None"}, {"name": "metric", "val": ": typing.Union[str, evaluate.module.EvaluationModule] = None"}, {"name": "tokenizer", "val": ": typing.Union[str, ForwardRef('PreTrainedTokenizer'), NoneType] = None"}, {"name": "strategy", "val": ": typing.Literal['simple', 'bootstrap'] = 'simple'"}, {"name": "confidence_level", "val": ": float = 0.95"}, {"name": "n_resamples", "val": ": int = 9999"}, {"name": "device", "val": ": int = None"}, {"name": "random_state", "val": ": typing.Optional[int] = None"}, {"name": "input_column", "val": ": str = 'text'"}, {"name": "label_column", "val": ": str = 'label'"}, {"name": "generation_kwargs", "val": ": dict = None"}]- **model_or_pipeline** (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) --
  If the argument in not specified, we initialize the default pipeline for the task (in this case
  `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or
  is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the
  argument specifies a pre-initialized pipeline.
- **data** (`str` or `Dataset`, defaults to `None`) --
  Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset
  name, and load it. Otherwise we assume it represents a pre-loaded dataset.
- **subset** (`str`, defaults to `None`) --
  Defines which dataset subset to load. If `None` is passed the default subset is loaded.
- **split** (`str`, defaults to `None`) --
  Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.
- **metric** (`str` or `EvaluationModule`, defaults to `None`) --
  Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and
  load it. Otherwise we assume it represents a pre-loaded metric.
- **tokenizer** (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) --
  Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for
  which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore
  this argument.
- **strategy** (`Literal["simple", "bootstrap"]`, defaults to "simple") --
  specifies the evaluation strategy. Possible values are:
  - `"simple"` - we evaluate the metric and return the scores.
  - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each
  of the returned metric keys, using `scipy`'s `bootstrap` method
  https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.
- **confidence_level** (`float`, defaults to `0.95`) --
  The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **n_resamples** (`int`, defaults to `9999`) --
  The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **device** (`int`, defaults to `None`) --
  Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive
  integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and
  CUDA:0 used if available, CPU otherwise.
- **random_state** (`int`, *optional*, defaults to `None`) --
  The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for
  debugging.

- **input_column** (`str`, defaults to `"text"`) --
  the name of the column containing the input text in the dataset specified by `data`.
- **label_column** (`str`, defaults to `"label"`) --
  the name of the column containing the labels in the dataset specified by `data`.
- **generation_kwargs** (`Dict`, *optional*, defaults to `None`) --
  The generation kwargs are passed to the pipeline and set the text generation strategy.0A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

Compute the metric for a given pipeline and dataset combination.

Examples:
```python
>>> from evaluate import evaluator
>>> from datasets import load_dataset
>>> task_evaluator = evaluator("text2text-generation")
>>> data = load_dataset("cnn_dailymail", "3.0.0", split="validation[:40]")
>>> results = task_evaluator.compute(
>>>     model_or_pipeline="facebook/bart-large-cnn",
>>>     data=data,
>>>     input_column="article",
>>>     label_column="highlights",
>>>     metric="rouge",
>>> )
```

**Parameters:**

model_or_pipeline (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) : If the argument in not specified, we initialize the default pipeline for the task (in this case `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the argument specifies a pre-initialized pipeline.

data (`str` or `Dataset`, defaults to `None`) : Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset name, and load it. Otherwise we assume it represents a pre-loaded dataset.

subset (`str`, defaults to `None`) : Defines which dataset subset to load. If `None` is passed the default subset is loaded.

split (`str`, defaults to `None`) : Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.

metric (`str` or `EvaluationModule`, defaults to `None`) : Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and load it. Otherwise we assume it represents a pre-loaded metric.

tokenizer (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) : Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore this argument.

strategy (`Literal["simple", "bootstrap"]`, defaults to "simple") : specifies the evaluation strategy. Possible values are: - `"simple"` - we evaluate the metric and return the scores. - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each of the returned metric keys, using `scipy`'s `bootstrap` method https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.

confidence_level (`float`, defaults to `0.95`) : The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

n_resamples (`int`, defaults to `9999`) : The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

device (`int`, defaults to `None`) : Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and CUDA:0 used if available, CPU otherwise.

random_state (`int`, *optional*, defaults to `None`) : The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for debugging. 

input_column (`str`, defaults to `"text"`) : the name of the column containing the input text in the dataset specified by `data`.

label_column (`str`, defaults to `"label"`) : the name of the column containing the labels in the dataset specified by `data`.

generation_kwargs (`Dict`, *optional*, defaults to `None`) : The generation kwargs are passed to the pipeline and set the text generation strategy.

**Returns:**

A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

### SummarizationEvaluator[[evaluate.SummarizationEvaluator]]

#### evaluate.SummarizationEvaluator[[evaluate.SummarizationEvaluator]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/text2text_generation.py#L152)

Text summarization evaluator.
This text summarization evaluator can currently be loaded from [evaluator()](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.evaluator) using the default task name
`summarization`.
Methods in this class assume a data format compatible with the [SummarizationEvaluator](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.SummarizationEvaluator).

computeevaluate.SummarizationEvaluator.computehttps://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/text2text_generation.py#L166[{"name": "model_or_pipeline", "val": ": typing.Union[str, ForwardRef('Pipeline'), typing.Callable, ForwardRef('PreTrainedModel'), ForwardRef('TFPreTrainedModel')] = None"}, {"name": "data", "val": ": typing.Union[str, datasets.arrow_dataset.Dataset] = None"}, {"name": "subset", "val": ": typing.Optional[str] = None"}, {"name": "split", "val": ": typing.Optional[str] = None"}, {"name": "metric", "val": ": typing.Union[str, evaluate.module.EvaluationModule] = None"}, {"name": "tokenizer", "val": ": typing.Union[str, ForwardRef('PreTrainedTokenizer'), NoneType] = None"}, {"name": "strategy", "val": ": typing.Literal['simple', 'bootstrap'] = 'simple'"}, {"name": "confidence_level", "val": ": float = 0.95"}, {"name": "n_resamples", "val": ": int = 9999"}, {"name": "device", "val": ": int = None"}, {"name": "random_state", "val": ": typing.Optional[int] = None"}, {"name": "input_column", "val": ": str = 'text'"}, {"name": "label_column", "val": ": str = 'label'"}, {"name": "generation_kwargs", "val": ": dict = None"}]- **model_or_pipeline** (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) --
  If the argument in not specified, we initialize the default pipeline for the task (in this case
  `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or
  is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the
  argument specifies a pre-initialized pipeline.
- **data** (`str` or `Dataset`, defaults to `None`) --
  Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset
  name, and load it. Otherwise we assume it represents a pre-loaded dataset.
- **subset** (`str`, defaults to `None`) --
  Defines which dataset subset to load. If `None` is passed the default subset is loaded.
- **split** (`str`, defaults to `None`) --
  Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.
- **metric** (`str` or `EvaluationModule`, defaults to `None`) --
  Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and
  load it. Otherwise we assume it represents a pre-loaded metric.
- **tokenizer** (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) --
  Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for
  which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore
  this argument.
- **strategy** (`Literal["simple", "bootstrap"]`, defaults to "simple") --
  specifies the evaluation strategy. Possible values are:
  - `"simple"` - we evaluate the metric and return the scores.
  - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each
  of the returned metric keys, using `scipy`'s `bootstrap` method
  https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.
- **confidence_level** (`float`, defaults to `0.95`) --
  The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **n_resamples** (`int`, defaults to `9999`) --
  The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **device** (`int`, defaults to `None`) --
  Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive
  integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and
  CUDA:0 used if available, CPU otherwise.
- **random_state** (`int`, *optional*, defaults to `None`) --
  The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for
  debugging.

- **input_column** (`str`, defaults to `"text"`) --
  the name of the column containing the input text in the dataset specified by `data`.
- **label_column** (`str`, defaults to `"label"`) --
  the name of the column containing the labels in the dataset specified by `data`.
- **generation_kwargs** (`Dict`, *optional*, defaults to `None`) --
  The generation kwargs are passed to the pipeline and set the text generation strategy.0A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

Compute the metric for a given pipeline and dataset combination.

Examples:
```python
>>> from evaluate import evaluator
>>> from datasets import load_dataset
>>> task_evaluator = evaluator("summarization")
>>> data = load_dataset("cnn_dailymail", "3.0.0", split="validation[:40]")
>>> results = task_evaluator.compute(
>>>     model_or_pipeline="facebook/bart-large-cnn",
>>>     data=data,
>>>     input_column="article",
>>>     label_column="highlights",
>>> )
```

**Parameters:**

model_or_pipeline (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) : If the argument in not specified, we initialize the default pipeline for the task (in this case `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the argument specifies a pre-initialized pipeline.

data (`str` or `Dataset`, defaults to `None`) : Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset name, and load it. Otherwise we assume it represents a pre-loaded dataset.

subset (`str`, defaults to `None`) : Defines which dataset subset to load. If `None` is passed the default subset is loaded.

split (`str`, defaults to `None`) : Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.

metric (`str` or `EvaluationModule`, defaults to `None`) : Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and load it. Otherwise we assume it represents a pre-loaded metric.

tokenizer (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) : Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore this argument.

strategy (`Literal["simple", "bootstrap"]`, defaults to "simple") : specifies the evaluation strategy. Possible values are: - `"simple"` - we evaluate the metric and return the scores. - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each of the returned metric keys, using `scipy`'s `bootstrap` method https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.

confidence_level (`float`, defaults to `0.95`) : The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

n_resamples (`int`, defaults to `9999`) : The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

device (`int`, defaults to `None`) : Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and CUDA:0 used if available, CPU otherwise.

random_state (`int`, *optional*, defaults to `None`) : The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for debugging. 

input_column (`str`, defaults to `"text"`) : the name of the column containing the input text in the dataset specified by `data`.

label_column (`str`, defaults to `"label"`) : the name of the column containing the labels in the dataset specified by `data`.

generation_kwargs (`Dict`, *optional*, defaults to `None`) : The generation kwargs are passed to the pipeline and set the text generation strategy.

**Returns:**

A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

### TranslationEvaluator[[evaluate.TranslationEvaluator]]

#### evaluate.TranslationEvaluator[[evaluate.TranslationEvaluator]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/text2text_generation.py#L211)

Translation evaluator.
This translation generation evaluator can currently be loaded from [evaluator()](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.evaluator) using the default task name
`translation`.
Methods in this class assume a data format compatible with the `TranslationPipeline`.

computeevaluate.TranslationEvaluator.computehttps://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/text2text_generation.py#L225[{"name": "model_or_pipeline", "val": ": typing.Union[str, ForwardRef('Pipeline'), typing.Callable, ForwardRef('PreTrainedModel'), ForwardRef('TFPreTrainedModel')] = None"}, {"name": "data", "val": ": typing.Union[str, datasets.arrow_dataset.Dataset] = None"}, {"name": "subset", "val": ": typing.Optional[str] = None"}, {"name": "split", "val": ": typing.Optional[str] = None"}, {"name": "metric", "val": ": typing.Union[str, evaluate.module.EvaluationModule] = None"}, {"name": "tokenizer", "val": ": typing.Union[str, ForwardRef('PreTrainedTokenizer'), NoneType] = None"}, {"name": "strategy", "val": ": typing.Literal['simple', 'bootstrap'] = 'simple'"}, {"name": "confidence_level", "val": ": float = 0.95"}, {"name": "n_resamples", "val": ": int = 9999"}, {"name": "device", "val": ": int = None"}, {"name": "random_state", "val": ": typing.Optional[int] = None"}, {"name": "input_column", "val": ": str = 'text'"}, {"name": "label_column", "val": ": str = 'label'"}, {"name": "generation_kwargs", "val": ": dict = None"}]- **model_or_pipeline** (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) --
  If the argument in not specified, we initialize the default pipeline for the task (in this case
  `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or
  is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the
  argument specifies a pre-initialized pipeline.
- **data** (`str` or `Dataset`, defaults to `None`) --
  Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset
  name, and load it. Otherwise we assume it represents a pre-loaded dataset.
- **subset** (`str`, defaults to `None`) --
  Defines which dataset subset to load. If `None` is passed the default subset is loaded.
- **split** (`str`, defaults to `None`) --
  Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.
- **metric** (`str` or `EvaluationModule`, defaults to `None`) --
  Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and
  load it. Otherwise we assume it represents a pre-loaded metric.
- **tokenizer** (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) --
  Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for
  which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore
  this argument.
- **strategy** (`Literal["simple", "bootstrap"]`, defaults to "simple") --
  specifies the evaluation strategy. Possible values are:
  - `"simple"` - we evaluate the metric and return the scores.
  - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each
  of the returned metric keys, using `scipy`'s `bootstrap` method
  https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.
- **confidence_level** (`float`, defaults to `0.95`) --
  The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **n_resamples** (`int`, defaults to `9999`) --
  The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **device** (`int`, defaults to `None`) --
  Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive
  integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and
  CUDA:0 used if available, CPU otherwise.
- **random_state** (`int`, *optional*, defaults to `None`) --
  The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for
  debugging.

- **input_column** (`str`, defaults to `"text"`) --
  the name of the column containing the input text in the dataset specified by `data`.
- **label_column** (`str`, defaults to `"label"`) --
  the name of the column containing the labels in the dataset specified by `data`.
- **generation_kwargs** (`Dict`, *optional*, defaults to `None`) --
  The generation kwargs are passed to the pipeline and set the text generation strategy.0A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

Compute the metric for a given pipeline and dataset combination.

Examples:
```python
>>> from evaluate import evaluator
>>> from datasets import load_dataset
>>> task_evaluator = evaluator("translation")
>>> data = load_dataset("wmt19", "fr-de", split="validation[:40]")
>>> data = data.map(lambda x: {"text": x["translation"]["de"], "label": x["translation"]["fr"]})
>>> results = task_evaluator.compute(
>>>     model_or_pipeline="Helsinki-NLP/opus-mt-de-fr",
>>>     data=data,
>>> )
```

**Parameters:**

model_or_pipeline (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) : If the argument in not specified, we initialize the default pipeline for the task (in this case `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the argument specifies a pre-initialized pipeline.

data (`str` or `Dataset`, defaults to `None`) : Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset name, and load it. Otherwise we assume it represents a pre-loaded dataset.

subset (`str`, defaults to `None`) : Defines which dataset subset to load. If `None` is passed the default subset is loaded.

split (`str`, defaults to `None`) : Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.

metric (`str` or `EvaluationModule`, defaults to `None`) : Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and load it. Otherwise we assume it represents a pre-loaded metric.

tokenizer (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) : Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore this argument.

strategy (`Literal["simple", "bootstrap"]`, defaults to "simple") : specifies the evaluation strategy. Possible values are: - `"simple"` - we evaluate the metric and return the scores. - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each of the returned metric keys, using `scipy`'s `bootstrap` method https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.

confidence_level (`float`, defaults to `0.95`) : The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

n_resamples (`int`, defaults to `9999`) : The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

device (`int`, defaults to `None`) : Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and CUDA:0 used if available, CPU otherwise.

random_state (`int`, *optional*, defaults to `None`) : The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for debugging. 

input_column (`str`, defaults to `"text"`) : the name of the column containing the input text in the dataset specified by `data`.

label_column (`str`, defaults to `"label"`) : the name of the column containing the labels in the dataset specified by `data`.

generation_kwargs (`Dict`, *optional*, defaults to `None`) : The generation kwargs are passed to the pipeline and set the text generation strategy.

**Returns:**

A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

### AutomaticSpeechRecognitionEvaluator[[evaluate.AutomaticSpeechRecognitionEvaluator]]

#### evaluate.AutomaticSpeechRecognitionEvaluator[[evaluate.AutomaticSpeechRecognitionEvaluator]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/automatic_speech_recognition.py#L47)

Automatic speech recognition evaluator.
This automatic speech recognition evaluator can currently be loaded from [evaluator()](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.evaluator) using the default task name
`automatic-speech-recognition`.
Methods in this class assume a data format compatible with the `AutomaticSpeechRecognitionPipeline`.

computeevaluate.AutomaticSpeechRecognitionEvaluator.computehttps://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/automatic_speech_recognition.py#L63[{"name": "model_or_pipeline", "val": ": typing.Union[str, ForwardRef('Pipeline'), typing.Callable, ForwardRef('PreTrainedModel'), ForwardRef('TFPreTrainedModel')] = None"}, {"name": "data", "val": ": typing.Union[str, datasets.arrow_dataset.Dataset] = None"}, {"name": "subset", "val": ": typing.Optional[str] = None"}, {"name": "split", "val": ": typing.Optional[str] = None"}, {"name": "metric", "val": ": typing.Union[str, evaluate.module.EvaluationModule] = None"}, {"name": "tokenizer", "val": ": typing.Union[str, ForwardRef('PreTrainedTokenizer'), NoneType] = None"}, {"name": "strategy", "val": ": typing.Literal['simple', 'bootstrap'] = 'simple'"}, {"name": "confidence_level", "val": ": float = 0.95"}, {"name": "n_resamples", "val": ": int = 9999"}, {"name": "device", "val": ": int = None"}, {"name": "random_state", "val": ": typing.Optional[int] = None"}, {"name": "input_column", "val": ": str = 'path'"}, {"name": "label_column", "val": ": str = 'sentence'"}, {"name": "generation_kwargs", "val": ": dict = None"}]- **model_or_pipeline** (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) --
  If the argument in not specified, we initialize the default pipeline for the task (in this case
  `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or
  is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the
  argument specifies a pre-initialized pipeline.
- **data** (`str` or `Dataset`, defaults to `None`) --
  Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset
  name, and load it. Otherwise we assume it represents a pre-loaded dataset.
- **subset** (`str`, defaults to `None`) --
  Defines which dataset subset to load. If `None` is passed the default subset is loaded.
- **split** (`str`, defaults to `None`) --
  Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.
- **metric** (`str` or `EvaluationModule`, defaults to `None`) --
  Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and
  load it. Otherwise we assume it represents a pre-loaded metric.
- **tokenizer** (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) --
  Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for
  which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore
  this argument.
- **strategy** (`Literal["simple", "bootstrap"]`, defaults to "simple") --
  specifies the evaluation strategy. Possible values are:
  - `"simple"` - we evaluate the metric and return the scores.
  - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each
  of the returned metric keys, using `scipy`'s `bootstrap` method
  https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.
- **confidence_level** (`float`, defaults to `0.95`) --
  The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **n_resamples** (`int`, defaults to `9999`) --
  The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **device** (`int`, defaults to `None`) --
  Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive
  integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and
  CUDA:0 used if available, CPU otherwise.
- **random_state** (`int`, *optional*, defaults to `None`) --
  The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for
  debugging.

- **input_column** (`str`, defaults to `"path"`) --
  the name of the column containing the input audio path in the dataset specified by `data`.
- **label_column** (`str`, defaults to `"sentence"`) --
  the name of the column containing the labels in the dataset specified by `data`.
- **generation_kwargs** (`Dict`, *optional*, defaults to `None`) --
  The generation kwargs are passed to the pipeline and set the text generation strategy.0A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

Compute the metric for a given pipeline and dataset combination.

Examples:
```python
>>> from evaluate import evaluator
>>> from datasets import load_dataset
>>> task_evaluator = evaluator("automatic-speech-recognition")
>>> data = load_dataset("mozilla-foundation/common_voice_11_0", "en", split="validation[:40]")
>>> results = task_evaluator.compute(
>>>     model_or_pipeline="https://huggingface.co/openai/whisper-tiny.en",
>>>     data=data,
>>>     input_column="path",
>>>     label_column="sentence",
>>>     metric="wer",
>>> )
```

**Parameters:**

model_or_pipeline (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) : If the argument in not specified, we initialize the default pipeline for the task (in this case `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the argument specifies a pre-initialized pipeline.

data (`str` or `Dataset`, defaults to `None`) : Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset name, and load it. Otherwise we assume it represents a pre-loaded dataset.

subset (`str`, defaults to `None`) : Defines which dataset subset to load. If `None` is passed the default subset is loaded.

split (`str`, defaults to `None`) : Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.

metric (`str` or `EvaluationModule`, defaults to `None`) : Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and load it. Otherwise we assume it represents a pre-loaded metric.

tokenizer (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) : Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore this argument.

strategy (`Literal["simple", "bootstrap"]`, defaults to "simple") : specifies the evaluation strategy. Possible values are: - `"simple"` - we evaluate the metric and return the scores. - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each of the returned metric keys, using `scipy`'s `bootstrap` method https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.

confidence_level (`float`, defaults to `0.95`) : The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

n_resamples (`int`, defaults to `9999`) : The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

device (`int`, defaults to `None`) : Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and CUDA:0 used if available, CPU otherwise.

random_state (`int`, *optional*, defaults to `None`) : The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for debugging.   

input_column (`str`, defaults to `"path"`) : the name of the column containing the input audio path in the dataset specified by `data`.

label_column (`str`, defaults to `"sentence"`) : the name of the column containing the labels in the dataset specified by `data`.

generation_kwargs (`Dict`, *optional*, defaults to `None`) : The generation kwargs are passed to the pipeline and set the text generation strategy.

**Returns:**

A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

### AudioClassificationEvaluator[[evaluate.AudioClassificationEvaluator]]

#### evaluate.AudioClassificationEvaluator[[evaluate.AudioClassificationEvaluator]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/audio_classification.py#L75)

Audio classification evaluator.
This audio classification evaluator can currently be loaded from [evaluator()](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.evaluator) using the default task name
`audio-classification`.
Methods in this class assume a data format compatible with the [transformers.AudioClassificationPipeline](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#transformers.AudioClassificationPipeline).

computeevaluate.AudioClassificationEvaluator.computehttps://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/audio_classification.py#L94[{"name": "model_or_pipeline", "val": ": typing.Union[str, ForwardRef('Pipeline'), typing.Callable, ForwardRef('PreTrainedModel'), ForwardRef('TFPreTrainedModel')] = None"}, {"name": "data", "val": ": typing.Union[str, datasets.arrow_dataset.Dataset] = None"}, {"name": "subset", "val": ": typing.Optional[str] = None"}, {"name": "split", "val": ": typing.Optional[str] = None"}, {"name": "metric", "val": ": typing.Union[str, evaluate.module.EvaluationModule] = None"}, {"name": "tokenizer", "val": ": typing.Union[str, ForwardRef('PreTrainedTokenizer'), NoneType] = None"}, {"name": "feature_extractor", "val": ": typing.Union[str, ForwardRef('FeatureExtractionMixin'), NoneType] = None"}, {"name": "strategy", "val": ": typing.Literal['simple', 'bootstrap'] = 'simple'"}, {"name": "confidence_level", "val": ": float = 0.95"}, {"name": "n_resamples", "val": ": int = 9999"}, {"name": "device", "val": ": int = None"}, {"name": "random_state", "val": ": typing.Optional[int] = None"}, {"name": "input_column", "val": ": str = 'file'"}, {"name": "label_column", "val": ": str = 'label'"}, {"name": "label_mapping", "val": ": typing.Optional[typing.Dict[str, numbers.Number]] = None"}]- **model_or_pipeline** (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) --
  If the argument in not specified, we initialize the default pipeline for the task (in this case
  `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or
  is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the
  argument specifies a pre-initialized pipeline.
- **data** (`str` or `Dataset`, defaults to `None`) --
  Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset
  name, and load it. Otherwise we assume it represents a pre-loaded dataset.
- **subset** (`str`, defaults to `None`) --
  Defines which dataset subset to load. If `None` is passed the default subset is loaded.
- **split** (`str`, defaults to `None`) --
  Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.
- **metric** (`str` or `EvaluationModule`, defaults to `None`) --
  Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and
  load it. Otherwise we assume it represents a pre-loaded metric.
- **tokenizer** (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) --
  Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for
  which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore
  this argument.
- **strategy** (`Literal["simple", "bootstrap"]`, defaults to "simple") --
  specifies the evaluation strategy. Possible values are:
  - `"simple"` - we evaluate the metric and return the scores.
  - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each
  of the returned metric keys, using `scipy`'s `bootstrap` method
  https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.
- **confidence_level** (`float`, defaults to `0.95`) --
  The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **n_resamples** (`int`, defaults to `9999`) --
  The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **device** (`int`, defaults to `None`) --
  Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive
  integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and
  CUDA:0 used if available, CPU otherwise.
- **random_state** (`int`, *optional*, defaults to `None`) --
  The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for
  debugging.

- **input_column** (`str`, defaults to `"file"`) --
  The name of the column containing either the audio files or a raw waveform, represented as a numpy array, in the dataset specified by `data`.
- **label_column** (`str`, defaults to `"label"`) --
  The name of the column containing the labels in the dataset specified by `data`.
- **label_mapping** (`Dict[str, Number]`, *optional*, defaults to `None`) --
  We want to map class labels defined by the model in the pipeline to values consistent with those
  defined in the `label_column` of the `data` dataset.0A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

Compute the metric for a given pipeline and dataset combination.

Examples:

> [!TIP]
> Remember that, in order to process audio files, you need ffmpeg installed (https://ffmpeg.org/download.html)

```python
>>> from evaluate import evaluator
>>> from datasets import load_dataset

>>> task_evaluator = evaluator("audio-classification")
>>> data = load_dataset("superb", 'ks', split="test[:40]")
>>> results = task_evaluator.compute(
>>>     model_or_pipeline=""superb/wav2vec2-base-superb-ks"",
>>>     data=data,
>>>     label_column="label",
>>>     input_column="file",
>>>     metric="accuracy",
>>>     label_mapping={0: "yes", 1: "no", 2: "up", 3: "down"}
>>> )
```

> [!TIP]
> The evaluator supports raw audio data as well, in the form of a numpy array. However, be aware that calling
> the audio column automatically decodes and resamples the audio files, which can be slow for large datasets.

```python
>>> from evaluate import evaluator
>>> from datasets import load_dataset

>>> task_evaluator = evaluator("audio-classification")
>>> data = load_dataset("superb", 'ks', split="test[:40]")
>>> data = data.map(lambda example: {"audio": example["audio"]["array"]})
>>> results = task_evaluator.compute(
>>>     model_or_pipeline=""superb/wav2vec2-base-superb-ks"",
>>>     data=data,
>>>     label_column="label",
>>>     input_column="audio",
>>>     metric="accuracy",
>>>     label_mapping={0: "yes", 1: "no", 2: "up", 3: "down"}
>>> )
```

**Parameters:**

model_or_pipeline (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) : If the argument in not specified, we initialize the default pipeline for the task (in this case `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the argument specifies a pre-initialized pipeline.

data (`str` or `Dataset`, defaults to `None`) : Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset name, and load it. Otherwise we assume it represents a pre-loaded dataset.

subset (`str`, defaults to `None`) : Defines which dataset subset to load. If `None` is passed the default subset is loaded.

split (`str`, defaults to `None`) : Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.

metric (`str` or `EvaluationModule`, defaults to `None`) : Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and load it. Otherwise we assume it represents a pre-loaded metric.

tokenizer (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) : Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore this argument.

strategy (`Literal["simple", "bootstrap"]`, defaults to "simple") : specifies the evaluation strategy. Possible values are: - `"simple"` - we evaluate the metric and return the scores. - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each of the returned metric keys, using `scipy`'s `bootstrap` method https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.

confidence_level (`float`, defaults to `0.95`) : The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

n_resamples (`int`, defaults to `9999`) : The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

device (`int`, defaults to `None`) : Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and CUDA:0 used if available, CPU otherwise.

random_state (`int`, *optional*, defaults to `None`) : The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for debugging.   

input_column (`str`, defaults to `"file"`) : The name of the column containing either the audio files or a raw waveform, represented as a numpy array, in the dataset specified by `data`.

label_column (`str`, defaults to `"label"`) : The name of the column containing the labels in the dataset specified by `data`.

label_mapping (`Dict[str, Number]`, *optional*, defaults to `None`) : We want to map class labels defined by the model in the pipeline to values consistent with those defined in the `label_column` of the `data` dataset.

**Returns:**

A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

