# Evaluate

## Docs

- [Installation](https://huggingface.co/docs/evaluate/main/installation.md)
- [Using the `evaluator` with custom pipelines](https://huggingface.co/docs/evaluate/main/custom_evaluator.md)
- [Evaluate on the Hub](https://huggingface.co/docs/evaluate/main/index.md)
- [A quick tour](https://huggingface.co/docs/evaluate/main/a_quick_tour.md)
- [🤗 Transformers](https://huggingface.co/docs/evaluate/main/transformers_integrations.md)
- [Creating and sharing a new evaluation](https://huggingface.co/docs/evaluate/main/creating_and_sharing.md)
- [Scikit-Learn](https://huggingface.co/docs/evaluate/main/sklearn_integrations.md)
- [Working with Keras and Tensorflow](https://huggingface.co/docs/evaluate/main/keras_integrations.md)
- [Considerations for model evaluation](https://huggingface.co/docs/evaluate/main/considerations.md)
- [Choosing a metric for your task](https://huggingface.co/docs/evaluate/main/choosing_a_metric.md)
- [Types of Evaluations in 🤗 Evaluate](https://huggingface.co/docs/evaluate/main/types_of_evaluations.md)
- [Using the `evaluator`](https://huggingface.co/docs/evaluate/main/base_evaluator.md)
- [Creating an EvaluationSuite](https://huggingface.co/docs/evaluate/main/evaluation_suite.md)
- [Evaluator](https://huggingface.co/docs/evaluate/main/package_reference/evaluator_classes.md)
- [Main classes](https://huggingface.co/docs/evaluate/main/package_reference/main_classes.md)
- [Saving methods](https://huggingface.co/docs/evaluate/main/package_reference/saving_methods.md)
- [Loading methods](https://huggingface.co/docs/evaluate/main/package_reference/loading_methods.md)
- [Hub methods](https://huggingface.co/docs/evaluate/main/package_reference/hub_methods.md)
- [Logging methods](https://huggingface.co/docs/evaluate/main/package_reference/logging_methods.md)
- [Visualization methods](https://huggingface.co/docs/evaluate/main/package_reference/visualization_methods.md)

### Installation
https://huggingface.co/docs/evaluate/main/installation.md

# Installation

Before you start, you will need to setup your environment and install the appropriate packages. 🤗 Evaluate is tested on **Python 3.7+**.

## Virtual environment

You should install 🤗 Evaluate in a [virtual environment](https://docs.python.org/3/library/venv.html) to keep everything neat and tidy.

1. Create and navigate to your project directory:

   ```bash
   mkdir ~/my-project
   cd ~/my-project
   ```

2. Start a virtual environment inside the directory:

   ```bash
   python -m venv .env
   ```

3. Activate and deactivate the virtual environment with the following commands:

   ```bash
   # Activate the virtual environment
   source .env/bin/activate
   
   # Deactivate the virtual environment
   source .env/bin/deactivate
   ```

Once you have created your virtual environment, you can install 🤗 Evaluate in it.

## pip

The most straightforward way to install 🤗 Evaluate is with pip:

```bash
pip install evaluate
```

Run the following command to check if 🤗 Evaluate has been properly installed:

```bash
python -c "import evaluate; print(evaluate.load('exact_match').compute(references=['hello'], predictions=['hello']))"
```

This should return:

```bash
{'exact_match': 1.0}
```

## source

Building 🤗 Evaluate from source lets you make changes to the code base. To install from source, clone the repository and install with the following commands:

```bash
git clone https://github.com/huggingface/evaluate.git
cd evaluate
pip install -e .
```

Again, you can check if 🤗 Evaluate has been properly installed with:

```bash
python -c "import evaluate; print(evaluate.load('exact_match').compute(references=['hello'], predictions=['hello']))"
```

### Using the `evaluator` with custom pipelines
https://huggingface.co/docs/evaluate/main/custom_evaluator.md

# Using the `evaluator` with custom pipelines

The evaluator is designed to work with `transformer` pipelines out-of-the-box. However, in many cases you might have a model or pipeline that's not part of the `transformer` ecosystem. You can still use `evaluator` to easily compute metrics for them. In this guide we show how to do this for a Scikit-Learn [pipeline](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html#sklearn.pipeline.Pipeline) and a Spacy [pipeline](https://spacy.io). Let's start with the Scikit-Learn case.

## Scikit-Learn

First we need to train a model. We'll train a simple text classifier on the [IMDb dataset](https://huggingface.co/datasets/imdb), so let's start by downloading the dataset:

```py
from datasets import load_dataset

ds = load_dataset("imdb")
```

Then we can build a simple TF-IDF preprocessor and Naive Bayes classifier wrapped in a `Pipeline`:

```py
from sklearn.pipeline import Pipeline
from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.feature_extraction.text import CountVectorizer

text_clf = Pipeline([
        ('vect', CountVectorizer()),
        ('tfidf', TfidfTransformer()),
        ('clf', MultinomialNB()),
])

text_clf.fit(ds["train"]["text"], ds["train"]["label"])
```

Following the convention in the `TextClassificationPipeline` of `transformers` our pipeline should be callable and return a list of dictionaries. In addition we use the `task` attribute to check if the pipeline is compatible with the `evaluator`. We can write a small wrapper class for that purpose:

```py
class ScikitEvalPipeline:
    def __init__(self, pipeline):
        self.pipeline = pipeline
        self.task = "text-classification"

    def __call__(self, input_texts, **kwargs):
        return [{"label": p} for p in self.pipeline.predict(input_texts)]

pipe = ScikitEvalPipeline(text_clf)
```

We can now pass this `pipeline` to the `evaluator`:

```py
from evaluate import evaluator

task_evaluator = evaluator("text-classification")
task_evaluator.compute(pipe, ds["test"], "accuracy")

>>> {'accuracy': 0.82956}
```

Implementing that simple wrapper is all that's needed to use any model from any framework with the `evaluator`. In the `__call__` you can implement all logic necessary for efficient forward passes through your model.

## Spacy

We'll use the `polarity` feature of the `spacytextblob` project to get a simple sentiment analyzer. First you'll need to install the project and download the resources:

```bash
pip install spacytextblob
python -m textblob.download_corpora
python -m spacy download en_core_web_sm
```

Then we can simply load the `nlp` pipeline and add the `spacytextblob` pipeline:
```py
import spacy

nlp = spacy.load('en_core_web_sm')
nlp.add_pipe('spacytextblob')
```

This snippet shows how we can use the `polarity` feature added with `spacytextblob` to get the sentiment of a text:

```py
texts = ["This movie is horrible", "This movie is awesome"]
results = nlp.pipe(texts)

for txt, res in zip(texts, results):
    print(f"{text} | Polarity: {res._.blob.polarity}")
```

Now we can wrap it in a simple wrapper class like in the Scikit-Learn example before. It just has to return a list of dictionaries with the predicted lables. If the polarity is larger than 0 we'll predict positive sentiment and negative otherwise:

```py
class SpacyEvalPipeline:
    def __init__(self, nlp):
        self.nlp = nlp
        self.task = "text-classification"

    def __call__(self, input_texts, **kwargs):
        results =[]
        for p in self.nlp.pipe(input_texts):
            if p._.blob.polarity>=0:
                results.append({"label": 1})
            else:
                results.append({"label": 0})
        return results

pipe = SpacyEvalPipeline(nlp)
```

That class is compatible with the `evaluator` and we can use the same instance from the previous examlpe along with the IMDb test set:

```py
eval.compute(pipe, ds["test"], "accuracy")
>>> {'accuracy': 0.6914}
```

This will take a little longer than the Scikit-Learn example but after roughly 10-15min you will have the evaluation results!

### Evaluate on the Hub
https://huggingface.co/docs/evaluate/main/index.md

# Evaluate on the Hub

    
    
    

You can evaluate AI models on the Hub in multiple ways and this page will guide you through the different options:

- **Community Leaderboards** bring together the best models for a given task or domain and make them accessible to everyone by ranking them.
- **Model Cards** provide a comprehensive overview of a model's capabilities from the author's perspective.
- **Libraries and Packages** give you the tools to evaluate your models on the Hub.

## Eval Results on the Hub

The Hub provides a decentralized system for tracking model evaluation results. Benchmark datasets can host leaderboards, and model repos store evaluation scores that automatically appear on both the model page and the benchmark's leaderboard.

![Eval Results on the Hub](https://huggingface.co/huggingface/documentation-images/resolve/main/evaluation-results/benchmark-preview.png)

You can add evaluation results to any model by submitting a YAML file to the `.eval_results/` folder in the model repo. These results display with badges indicating whether they are verified, community-provided, or linked to a benchmark leaderboard.

For full details on adding evaluation results to models and registering benchmark datasets, see the [Evaluation Results documentation](https://huggingface.co/docs/hub/eval-results).

## Community Managed Leaderboards

Community leaderboards show how a model performs on a given task or domain. For example, there are leaderboards for question answering, reasoning, classification, vision, and audio. If you're tackling a new task, you can use a leaderboard to see how a model performs on it.

Here are some examples of community leaderboards:

| Leaderboard | Model Type | Description |
| --- | --- | --- |
| [MTEB](https://huggingface.co/spaces/mteb/leaderboard)| Embedding | The Massive Text Embedding Benchmark leaderboard compares 100+ text and image embedding models across 1000+ languages. Refer to the publication of each selectable benchmark for details on metrics, languages, tasks, and task types. Anyone is welcome to add a model, add benchmarks, help improve zero-shot annotations, or propose other changes to the leaderboard. |
| [GAIA](https://huggingface.co/spaces/gaia-benchmark/leaderboard)| Agentic | GAIA is a benchmark which aims at evaluating next-generation LLMs (LLMs with augmented capabilities due to added tooling, efficient prompting, access to search, etc). (See [the paper](https://arxiv.org/abs/2311.12983) for more details.) |
| [OpenVLM Leaderboard](https://huggingface.co/spaces/opencompass/open_vlm_leaderboard)| Vision Language Models | The OpenVLM Leaderboard evaluates 272+ Vision-Language Models (including GPT-4v, Gemini, QwenVLPlus, LLaVA) across 31 different multi-modal benchmarks using the VLMEvalKit framework. It focuses on open-source VLMs and publicly available API models. |
| [Open ASR Leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard)| Audio | The Open ASR Leaderboard ranks and evaluates speech recognition models on the Hugging Face Hub. Models are ranked based on their Average WER, from lowest to highest. |
| [LLM-Perf Leaderboard](https://huggingface.co/spaces/llm-perf/leaderboard)| LLM Performance | The 🤗 LLM-Perf Leaderboard 🏋️ is a leaderboard at the intersection of quality and performance. Its aim is to benchmark the performance (latency, throughput, memory & energy) of Large Language Models (LLMs) with different hardware, backends and optimizations using Optimum-Benchmark. |

There are many more leaderboards on the Hub. Check out all the leaderboards via this [search](https://huggingface.co/spaces?category=model-benchmarking) or use this [dedicated Space](https://huggingface.co/spaces/OpenEvals/find-a-leaderboard) to find a leaderboard for your task.

## Model Cards

Model cards provide an overview of a model's capabilities evaluated by the community or the model's author. They are a great way to understand a model's capabilities and limitations.

![Qwen model card](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/evaluate-docs/qwen-model-card.png)

Unlike leaderboards, model card evaluation scores are often created by the author, rather than by the community.

> [!TIP]
> For information on reporting results, see details on [the Model Card Evaluation Results metadata](https://huggingface.co/docs/hub/en/model-cards#evaluation-results).

## Libraries and packages

There are a number of open-source libraries and packages that you can use to evaluate your models on the Hub. These are useful if you want to evaluate a custom model or performance on a custom evaluation task.

### LightEval

LightEval is a library for evaluating LLMs. It is designed to be comprehensive and customizable. Visit the LightEval [repository](https://github.com/huggingface/lighteval) for more information.

> [!TIP]
> For more recent evaluation approaches that are popular on the Hugging Face Hub that are currently more actively maintained, check out [LightEval](https://github.com/huggingface/lighteval).

### 🤗 Evaluate

A library for easily evaluating machine learning models and datasets.

With a single line of code, you get access to dozens of evaluation methods for different domains (NLP, Computer Vision, Reinforcement Learning, and more!). Be it on your local machine or in a distributed training setup, you can evaluate your models in a consistent and reproducible way!

Visit the 🤗 Evaluate [organization](https://huggingface.co/evaluate-metric) for a full list of available metrics. Each metric has a dedicated Space with an interactive demo for how to use the metric, and a documentation card detailing the metrics limitations and usage.

  
    Tutorials
      Learn the basics and become familiar with loading, computing, and saving with 🤗 Evaluate. Start here if you are using 🤗 Evaluate for the first time!
    
    How-to guides
      Practical guides to help you achieve a specific goal. Take a look at these guides to learn how to use 🤗 Evaluate to solve real-world problems.
    
    Conceptual guides
      High-level explanations for building a better understanding of important topics such as considerations going into evaluating a model or dataset and the difference between metrics, measurements, and comparisons.
   
    Reference
      Technical descriptions of how 🤗 Evaluate classes and methods work.

### A quick tour
https://huggingface.co/docs/evaluate/main/a_quick_tour.md

# A quick tour

🤗 Evaluate provides access to a wide range of evaluation tools. It covers a range of modalities such as text, computer vision, audio, etc. as well as tools to evaluate models or datasets. These tools are split into three categories.

## Types of evaluations

There are different aspects of a typical machine learning pipeline that can be evaluated and for each aspect 🤗 Evaluate provides a tool:

- **Metric**: A metric is used to evaluate a model's performance and usually involves the model's predictions as well as some ground truth labels. You can find all integrated metrics at [evaluate-metric](https://huggingface.co/evaluate-metric).
- **Comparison**: A comparison is used to compare two models. This can for example be done by comparing their predictions to ground truth labels and computing their agreement. You can find all integrated comparisons at [evaluate-comparison](https://huggingface.co/evaluate-comparison).
- **Measurement**: The dataset is as important as the model trained on it. With measurements one can investigate a dataset's properties. You can find all integrated measurements at [evaluate-measurement](https://huggingface.co/evaluate-measurement).

Each of these evaluation modules live on Hugging Face Hub as a Space. They come with an interactive widget and a documentation card documenting its use and limitations. For example [accuracy](https://huggingface.co/spaces/evaluate-metric/accuracy):

    

Each metric, comparison, and measurement is a separate Python module, but for using any of them, there is a single entry point: [evaluate.load()](/docs/evaluate/main/en/package_reference/loading_methods#evaluate.load)!

## Load

Any metric, comparison, or measurement is loaded with the `evaluate.load` function:

```py
>>> import evaluate
>>> accuracy = evaluate.load("accuracy")
```

If you want to make sure you are loading the right type of evaluation (especially if there are name clashes) you can explicitly pass the type:

```py
>>> word_length = evaluate.load("word_length", module_type="measurement")
```

### Community modules

Besides the modules implemented in 🤗 Evaluate you can also load any community module by specifying the repository ID of the metric implementation:

```py
>>> element_count = evaluate.load("lvwerra/element_count", module_type="measurement")
```

See the [Creating and Sharing Guide](/docs/evaluate/main/en/creating_and_sharing) for information about uploading custom metrics.

### List available modules

With [list_evaluation_modules()](/docs/evaluate/main/en/package_reference/loading_methods#evaluate.list_evaluation_modules) you can check what modules are available on the hub. You can also filter for a specific modules and skip community metrics if you want. You can also see additional information such as likes:

```python
>>> evaluate.list_evaluation_modules(
...   module_type="comparison",
...   include_community=False,
...   with_details=True)

[{'name': 'mcnemar', 'type': 'comparison', 'community': False, 'likes': 1},
 {'name': 'exact_match', 'type': 'comparison', 'community': False, 'likes': 0}]
```

## Module attributes

All evalution modules come with a range of useful attributes that help to use a module stored in a [EvaluationModuleInfo](/docs/evaluate/main/en/package_reference/main_classes#evaluate.EvaluationModuleInfo) object.

|Attribute|Description|
|---|---|
|`description`|A short description of the evaluation module.|
|`citation`|A BibTex string for citation when available.|
|`features`|A `Features` object defining the input format.|
|`inputs_description`|This is equivalent to the modules docstring.|
|`homepage`|The homepage of the module.|
|`license`|The license of the module.|
|`codebase_urls`|Link to the code behind the module.|
|`reference_urls`|Additional reference URLs.|

Let's have a look at a few examples. First, let's look at the `description` attribute of the accuracy metric:

```py
>>> accuracy = evaluate.load("accuracy")
>>> accuracy.description
Accuracy is the proportion of correct predictions among the total number of cases processed. It can be computed with:
Accuracy = (TP + TN) / (TP + TN + FP + FN)
 Where:
TP: True positive
TN: True negative
FP: False positive
FN: False negative
```

You can see that it describes how the metric works in theory. If you use this metric for your work, especially if it is an academic publication you want to reference it properly. For that you can look at the `citation` attribute:

```py
>>> accuracy.citation
@article{scikit-learn,
  title={Scikit-learn: Machine Learning in {P}ython},
  author={Pedregosa, F. and Varoquaux, G. and Gramfort, A. and Michel, V.
         and Thirion, B. and Grisel, O. and Blondel, M. and Prettenhofer, P.
         and Weiss, R. and Dubourg, V. and Vanderplas, J. and Passos, A. and
         Cournapeau, D. and Brucher, M. and Perrot, M. and Duchesnay, E.},
  journal={Journal of Machine Learning Research},
  volume={12},
  pages={2825--2830},
  year={2011}
}
```

Before we can apply a metric or other evaluation module to a use-case, we need to know what the input format of the metric is:

```py
>>> accuracy.features
{
    'predictions': Value(dtype='int32', id=None),
    'references': Value(dtype='int32', id=None)
}
```

> [!TIP]
> Note that features always describe the type of a single input element. In general we will add lists of elements so you can always think of a list around the types in `features`. Evaluate accepts various input formats (Python lists, NumPy arrays, PyTorch tensors, etc.) and converts them to an appropriate format for storage and computation.

## Compute

Now that we know how the evaluation module works and what should go in there we want to actually use it! When it comes to computing the actual score there are two main ways to do it:

1. All-in-one
2. Incremental

In the incremental approach the necessary inputs are added to the module with [EvaluationModule.add()](/docs/evaluate/main/en/package_reference/main_classes#evaluate.EvaluationModule.add) or [EvaluationModule.add_batch()](/docs/evaluate/main/en/package_reference/main_classes#evaluate.EvaluationModule.add_batch) and the score is calculated at the end with [EvaluationModule.compute()](/docs/evaluate/main/en/package_reference/main_classes#evaluate.EvaluationModule.compute). Alternatively, one can pass all the inputs at once to `compute()`. Let's have a look at the two approaches.

### How to compute

The simplest way to calculate the score of an evaluation module is by calling `compute()` directly with the necessary inputs. Simply pass the inputs as seen in `features` to the `compute()` method.

```py
>>> accuracy.compute(references=[0,1,0,1], predictions=[1,0,0,1])
{'accuracy': 0.5}
```
Evaluation modules return the results in a dictionary. However, in some instances you build up the predictions iteratively or in a distributed fashion in which case `add()` or `add_batch()` are useful.

### Calculate a single metric or a batch of metrics

In many evaluation pipelines you build the predictions iteratively such as in a for-loop. In that case you could store the predictions in a list and at the end pass them to `compute()`. With `add()` and `add_batch()` you can circumvent the step of storing the predictions separately. If you are only creating single predictions at a time you can use `add()`:

```py
>>> for ref, pred in zip([0,1,0,1], [1,0,0,1]):
>>>     accuracy.add(references=ref, predictions=pred)
>>> accuracy.compute()
{'accuracy': 0.5}
```

Once you have gathered all predictions you can call `compute()` to compute the score based on all stored values. When getting predictions and references in batches you can use `add_batch()` which adds a list elements for later processing. The rest works as with `add()`:

```py
>>> for refs, preds in zip([[0,1],[0,1]], [[1,0],[0,1]]):
>>>     accuracy.add_batch(references=refs, predictions=preds)
>>> accuracy.compute()
{'accuracy': 0.5}
```

This is especially useful when you need to get the predictions from your model in batches:

```py
>>> for model_inputs, gold_standards in evaluation_dataset:
>>>     predictions = model(model_inputs)
>>>     metric.add_batch(references=gold_standards, predictions=predictions)
>>> metric.compute()
```

### Distributed evaluation

Computing metrics in a distributed environment can be tricky. Metric evaluation is executed in separate Python processes, or nodes, on different subsets of a dataset. Typically, when a metric score is additive (`f(AuB) = f(A) + f(B)`), you can use distributed reduce operations to gather the scores for each subset of the dataset. But when a metric is non-additive (`f(AuB) ≠ f(A) + f(B)`), it's not that simple. For example, you can't take the sum of the [F1](https://huggingface.co/spaces/evaluate-metric/f1) scores of each data subset as your **final metric**.

A common way to overcome this issue is to fallback on single process evaluation. The metrics are evaluated on a single GPU, which becomes inefficient.

🤗 Evaluate solves this issue by only computing the final metric on the first node. The predictions and references are computed and provided to the metric separately for each node. These are temporarily stored in an Apache Arrow table, avoiding cluttering the GPU or CPU memory. When you are ready to `compute()` the final metric, the first node is able to access the predictions and references stored on all the other nodes. Once it has gathered all the predictions and references, `compute()` will perform the final metric evaluation.

This solution allows 🤗 Evaluate to perform distributed predictions, which is important for evaluation speed in distributed settings. At the same time, you can also use complex non-additive metrics without wasting valuable GPU or CPU memory.

## Combining several evaluations

Often one wants to not only evaluate a single metric but a range of different metrics capturing different aspects of a model. E.g. for classification it is usually a good idea to compute F1-score, recall, and precision in addition to accuracy to get a better picture of model performance. Naturally, you can load a bunch of metrics and call them sequentially. However, a more convenient way is to use the [combine()](/docs/evaluate/main/en/package_reference/main_classes#evaluate.combine) function to bundle them together:

```python
>>> clf_metrics = evaluate.combine(["accuracy", "f1", "precision", "recall"])
```

The `combine` function accepts both the list of names of the metrics as well as an instantiated modules. The `compute` call then computes each metric:

```python
>>> clf_metrics.compute(predictions=[0, 1, 0], references=[0, 1, 1])

{
  'accuracy': 0.667,
  'f1': 0.667,
  'precision': 1.0,
  'recall': 0.5
}
```

## Save and push to the Hub

Saving and sharing evaluation results is an important step. We provide the [evaluate.save()](/docs/evaluate/main/en/package_reference/saving_methods#evaluate.save) function to easily save metrics results. You can either pass a specific filename or a directory. In the latter case, the results are saved in a file with an automatically created file name. Besides the directory or file name, the function takes any key-value pairs as inputs and stores them in a JSON file.

```py
>>> result = accuracy.compute(references=[0,1,0,1], predictions=[1,0,0,1])

>>> hyperparams = {"model": "bert-base-uncased"}
>>> evaluate.save("./results/", experiment="run 42", **result, **hyperparams)
PosixPath('results/result-2022_05_30-22_09_11.json')
```

The content of the JSON file look like the following:

```json
{
    "experiment": "run 42",
    "accuracy": 0.5,
    "model": "bert-base-uncased",
    "_timestamp": "2022-05-30T22:09:11.959469",
    "_git_commit_hash": "123456789abcdefghijkl",
    "_evaluate_version": "0.1.0",
    "_python_version": "3.9.12 (main, Mar 26 2022, 15:51:15) \n[Clang 13.1.6 (clang-1316.0.21.2)]",
    "_interpreter_path": "/Users/leandro/git/evaluate/env/bin/python"
}
```

In addition to the specified fields, it also contains useful system information for reproducing the results.

Besides storing the results locally, you should report them on the model's repository on the Hub. With the [evaluate.push_to_hub()](/docs/evaluate/main/en/package_reference/hub_methods#evaluate.push_to_hub) function, you can easily report evaluation results to the model's repository:

```py
evaluate.push_to_hub(
  model_id="huggingface/gpt2-wikitext2",  # model repository on hub
  metric_value=0.5,                       # metric value
  metric_type="bleu",                     # metric name, e.g. accuracy.name
  metric_name="BLEU",                     # pretty name which is displayed
  dataset_type="wikitext",                # dataset name on the hub
  dataset_name="WikiText",                # pretty name
  dataset_split="test",                   # dataset split used
  task_type="text-generation",            # task id, see https://github.com/huggingface/evaluate/blob/main/src/evaluate/config.py#L154-L192
  task_name="Text Generation"             # pretty name for task
)
```

## Evaluator

The [evaluate.evaluator()](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.evaluator) provides automated evaluation and only requires a model, dataset, metric in contrast to the metrics in `EvaluationModule`s that require the model's predictions. As such it is easier to evaluate a model on a dataset with a given metric as the inference is handled internally. To make that possible it uses the [pipeline](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#transformers.pipeline) abstraction from `transformers`. However, you can use your own framework as long as it follows the `pipeline` interface.

To make an evaluation with the `evaluator` let's load a `transformers` pipeline (but you can pass your own custom inference class for any framework as long as it follows the pipeline call API) with an model trained on IMDb, the IMDb test split and the accuracy metric.

```python
from transformers import pipeline
from datasets import load_dataset
from evaluate import evaluator
import evaluate

pipe = pipeline("text-classification", model="lvwerra/distilbert-imdb", device=0)
data = load_dataset("imdb", split="test").shuffle().select(range(1000))
metric = evaluate.load("accuracy")
```

Then you can create an evaluator for text classification and pass the three objects to the `compute()` method. With the label mapping `evaluate` provides a method to align the pipeline outputs with the label column in the dataset:

```python
>>> task_evaluator = evaluator("text-classification")

>>> results = task_evaluator.compute(model_or_pipeline=pipe, data=data, metric=metric,
...                        label_mapping={"NEGATIVE": 0, "POSITIVE": 1},)

>>> print(results)
{'accuracy': 0.934}
```

Calculating the value of the metric alone is often not enough to know if a model performs significantly better than another one. With _bootstrapping_ `evaluate` computes confidence intervals and the standard error which helps estimate how stable a score is:

```python
>>> results = eval.compute(model_or_pipeline=pipe, data=data, metric=metric,
...                        label_mapping={"NEGATIVE": 0, "POSITIVE": 1},
...                        strategy="bootstrap", n_resamples=200)

>>> print(results)
{'accuracy':
    {
      'confidence_interval': (0.906, 0.9406749892841922),
      'standard_error': 0.00865213251082787,
      'score': 0.923
    }
}
```

The evaluator expects a `"text"` and `"label"` column for the data input. If your dataset differs you can provide the columns with the keywords `input_column="text"` and `label_column="label"`. Currently only `"text-classification"` is supported with more tasks being added in the future.

## Visualization

When comparing several models, sometimes it's hard to spot the differences in their performance simply by looking at their scores. Also often there is not a single best model but there are trade-offs between e.g. latency and accuracy as larger models might have better performance but are also slower. We are gradually adding different visualization approaches, such as plots, to make choosing the best model for a use-case easier.

For instance, if you have a list of results from multiple models (as dictionaries), you can feed them into the `radar_plot()` function:

```python
import evaluate
from evaluate.visualization import radar_plot

>>> data = [
   {"accuracy": 0.99, "precision": 0.8, "f1": 0.95, "latency_in_seconds": 33.6},
   {"accuracy": 0.98, "precision": 0.87, "f1": 0.91, "latency_in_seconds": 11.2},
   {"accuracy": 0.98, "precision": 0.78, "f1": 0.88, "latency_in_seconds": 87.6}, 
   {"accuracy": 0.88, "precision": 0.78, "f1": 0.81, "latency_in_seconds": 101.6}
   ]
>>> model_names = ["Model 1", "Model 2", "Model 3", "Model 4"]
>>> plot = radar_plot(data=data, model_names=model_names)
>>> plot.show()
```

Which lets you visually compare the 4 models and choose the optimal one for you, based on one or several metrics:

    

## Running evaluation on a suite of tasks

It can be useful to evaluate models on a variety of different tasks to understand their downstream performance. The [EvaluationSuite](evaluation_suite) enables evaluation of models on a collection of tasks. Tasks can be constructed as ([evaluator](base_evaluator), dataset, metric) tuples and passed to an [EvaluationSuite](evaluation_suite) stored on the Hugging Face Hub as a Space, or locally as a Python script. See the [evaluator documentation](base_evaluator) for a list of currently supported tasks.

`EvaluationSuite` scripts can be defined as follows, and supports Python code for data preprocessing.

```python
import evaluate
from evaluate.evaluation_suite import SubTask

class Suite(evaluate.EvaluationSuite):

    def __init__(self, name):
        super().__init__(name)

        self.suite = [
            SubTask(
                task_type="text-classification",
                data="imdb",
                split="test[:1]",
                args_for_task={
                    "metric": "accuracy",
                    "input_column": "text",
                    "label_column": "label",
                    "label_mapping": {
                        "LABEL_0": 0.0,
                        "LABEL_1": 1.0
                    }
                }
            ),
            SubTask(
                task_type="text-classification",
                data="sst2",
                split="test[:1]",
                args_for_task={
                    "metric": "accuracy",
                    "input_column": "sentence",
                    "label_column": "label",
                    "label_mapping": {
                        "LABEL_0": 0.0,
                        "LABEL_1": 1.0
                    }
                }
            )
        ]
```

Evaluation can be run by loading the `EvaluationSuite` and calling the `run()` method with a model or pipeline.

```
>>> from evaluate import EvaluationSuite
>>> suite = EvaluationSuite.load('mathemakitten/sentiment-evaluation-suite')
>>> results = suite.run("huggingface/prunebert-base-uncased-6-finepruned-w-distil-mnli")
```

|   accuracy |   total_time_in_seconds |   samples_per_second |   latency_in_seconds | task_name   |
|------------:|---------------------:|--------------------------:|:----------------|:-----------|
|        0.3 |                4.62804  |              2.16074 |            0.462804  | imdb        |
|        0   |                0.686388 |             14.569   |            0.0686388 | sst2        |

### 🤗 Transformers
https://huggingface.co/docs/evaluate/main/transformers_integrations.md

# 🤗 Transformers

To run the 🤗 Transformers examples make sure you have installed the following libraries:

```bash
pip install datasets transformers torch evaluate nltk rouge_score
```

## Trainer

The metrics in `evaluate` can be easily integrated with the [Trainer](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Trainer). The `Trainer` accepts a `compute_metrics` keyword argument that passes a function to compute metrics. One can specify the evaluation interval with `eval_strategy` in the `TrainerArguments`, and based on that, the model is evaluated accordingly, and the predictions and labels passed to `compute_metrics`.

```python
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForSequenceClassification, TrainingArguments, Trainer
import numpy as np
import evaluate

# Prepare and tokenize dataset
dataset = load_dataset("yelp_review_full")
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

def tokenize_function(examples):
    return tokenizer(examples["text"], padding="max_length", truncation=True)

tokenized_datasets = dataset.map(tokenize_function, batched=True)

small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(200))
small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(200))

# Setup evaluation 
metric = evaluate.load("accuracy")

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return metric.compute(predictions=predictions, references=labels)

# Load pretrained model and evaluate model after each epoch
model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=5)
training_args = TrainingArguments(output_dir="test_trainer", eval_strategy="epoch")

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=small_train_dataset,
    eval_dataset=small_eval_dataset,
    compute_metrics=compute_metrics,
)

trainer.train()
```

## Seq2SeqTrainer

We can use the [Seq2SeqTrainer](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Seq2SeqTrainer) for sequence-to-sequence tasks such as translation or summarization. For such generative tasks usually metrics such as ROUGE or BLEU are evaluated. However, these metrics require that we generate some text with the model rather than a single forward pass as with e.g. classification. The `Seq2SeqTrainer` allows for the use of the generate method when setting `predict_with_generate=True` which will generate text for each sample in the evaluation set. That means we evaluate generated text within the `compute_metric` function. We just need to decode the predictions and labels first.

```python
import nltk
from datasets import load_dataset
import evaluate
import numpy as np
from transformers import AutoTokenizer, DataCollatorForSeq2Seq
from transformers import AutoModelForSeq2SeqLM, Seq2SeqTrainingArguments, Seq2SeqTrainer

# Prepare and tokenize dataset
billsum = load_dataset("billsum", split="ca_test").shuffle(seed=42).select(range(200))
billsum = billsum.train_test_split(test_size=0.2)
tokenizer = AutoTokenizer.from_pretrained("t5-small")
prefix = "summarize: "

def preprocess_function(examples):
    inputs = [prefix + doc for doc in examples["text"]]
    model_inputs = tokenizer(inputs, max_length=1024, truncation=True)

    labels = tokenizer(text_target=examples["summary"], max_length=128, truncation=True)

    model_inputs["labels"] = labels["input_ids"]
    return model_inputs

tokenized_billsum = billsum.map(preprocess_function, batched=True)

# Setup evaluation
nltk.download("punkt_tab", quiet=True)
metric = evaluate.load("rouge")

def compute_metrics(eval_preds):
    preds, labels = eval_preds

    # decode preds and labels
    labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
    decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)
    decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

    # rougeLSum expects newline after each sentence
    decoded_preds = ["\n".join(nltk.sent_tokenize(pred.strip())) for pred in decoded_preds]
    decoded_labels = ["\n".join(nltk.sent_tokenize(label.strip())) for label in decoded_labels]

    result = metric.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)
    return result

# Load pretrained model and evaluate model after each epoch
model = AutoModelForSeq2SeqLM.from_pretrained("t5-small")
data_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=model)

training_args = Seq2SeqTrainingArguments(
    output_dir="./results",
    eval_strategy="epoch",
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=4,
    weight_decay=0.01,
    save_total_limit=3,
    num_train_epochs=2,
    fp16=True,
    predict_with_generate=True
)

trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_billsum["train"],
    eval_dataset=tokenized_billsum["test"],
    tokenizer=tokenizer,
    data_collator=data_collator,
    compute_metrics=compute_metrics
)

trainer.train()
```

You can use any `evaluate` metric with the `Trainer` and `Seq2SeqTrainer` as long as they are compatible with the task and predictions. In case you don't want to train a model but just evaluate an existing model you can replace `trainer.train()` with `trainer.evaluate()` in the above scripts.

### Creating and sharing a new evaluation
https://huggingface.co/docs/evaluate/main/creating_and_sharing.md

# Creating and sharing a new evaluation

## Setup

Before you can create a new metric make sure you have all the necessary dependencies installed:

```bash
pip install evaluate[template]
```

Also make sure your Hugging Face token is registered so you can connect to the Hugging Face Hub:

```bash
huggingface-cli login
```

## Create

All evaluation modules, be it metrics, comparisons, or measurements live on the 🤗 Hub in a [Space](https://huggingface.co/docs/hub/spaces) (see for example [Accuracy](https://huggingface.co/spaces/evaluate-metric/accuracy)). In principle, you could setup a new Space and add a new module following the same structure. However, we added a CLI that makes creating a new evaluation module much easier:

```bash
evaluate-cli create "My Metric" --module_type "metric"
```

This will create a new Space on the 🤗 Hub, clone it locally, and populate it with a template. Instructions on how to fill the template will be displayed in the terminal, but are also explained here in more detail.

For more information about Spaces, see the [Spaces documentation](https://huggingface.co/docs/hub/spaces).

## Module script

The evaluation module script (the file with suffix `*.py`) is the core of the new module and includes all the code for computing the evaluation.

### Attributes

Start by adding some information about your evalution module in `EvaluationModule._info()`. The most important attributes you should specify are:

1. `EvaluationModuleInfo.description` provides a brief description about your evalution module.

2. `EvaluationModuleInfo.citation` contains a BibTex citation for the evalution module.

3. `EvaluationModuleInfo.inputs_description` describes the expected inputs and outputs. It may also provide an example usage of the evalution module.

4. `EvaluationModuleInfo.features` defines the name and type of the predictions and references. This has to be either a single `datasets.Features` object or a list of `datasets.Features` objects if multiple input types are allowed.

Then, we can move on to prepare everything before the actual computation.

### Download

Some evaluation modules require some external data such as NLTK that requires resources or the BLEURT metric that requires checkpoints. You can implement these downloads in `EvaluationModule._download_and_prepare()`, which downloads and caches the resources via the `dlmanager`. A simplified example on how BLEURT downloads and loads a checkpoint:

```py
def _download_and_prepare(self, dl_manager):
    model_path = dl_manager.download_and_extract(CHECKPOINT_URLS[self.config_name])
    self.scorer = score.BleurtScorer(os.path.join(model_path, self.config_name))
```

Or if you need to download the NLTK `"punkt_tab"` resources:

```py
def _download_and_prepare(self, dl_manager):
    import nltk
    nltk.download("punkt_tab")
```

Next, we need to define how the computation of the evaluation module works.

### Compute

The computation is performed in the `EvaluationModule._compute()` method. It takes the same arguments as `EvaluationModuleInfo.features` and should then return the result as a dictionary. Here an example of an exact match metric:

```py
def _compute(self, references, predictions):
    em = sum([r==p for r, p in zip(references, predictions)])/len(references)
    return {"exact_match": em}
```

This method is used when you call `.compute()` later on.

## Readme

When you use the `evalute-cli` to setup the evaluation module the Readme structure and instructions are automatically created. It should include a general description of the metric, information about its input/output format, examples as well as information about its limiations or biases and references.

## Requirements

If your evaluation modules has additional dependencies (e.g. `sklearn` or `nltk`) the `requirements.txt` files is the place to put them. The file follows the `pip` format and you can list all dependencies there.

## App

The `app.py` is where the Spaces widget lives. In general it looks like the following and does not require any changes:

```py
import evaluate
from evaluate.utils import launch_gradio_widget

module = evaluate.load("lvwerra/element_count")
launch_gradio_widget(module)
```

If you want a custom widget you could add your gradio app here.

## Push to Hub

Finally, when you are done with all the above changes it is time to push your evaluation module to the hub. To do so navigate to the folder of your module and git add/commit/push the changes to the hub:

```
cd PATH_TO_MODULE
git add .
git commit -m "Add my new, shiny module."
git push
```
Tada 🎉! Your evaluation module is now on the 🤗 Hub and ready to be used by everybody!

### Scikit-Learn
https://huggingface.co/docs/evaluate/main/sklearn_integrations.md

# Scikit-Learn

To run the scikit-learn examples make sure you have installed the following library:

```bash
pip install -U scikit-learn
```

The metrics in `evaluate` can be easily integrated with an Scikit-Learn estimator or [pipeline](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html#sklearn.pipeline.Pipeline).  

However, these metrics require that we generate the predictions from the model. The predictions and labels from the estimators can be passed to `evaluate` mertics to compute the required values.

```python
import numpy as np
np.random.seed(0)
import evaluate
from sklearn.compose import ColumnTransformer
from sklearn.datasets import fetch_openml
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
```

Load data from https://www.openml.org/d/40945:

```python
X, y = fetch_openml("titanic", version=1, as_frame=True, return_X_y=True)
```

Alternatively X and y can be obtained directly from the frame attribute:

```python
X = titanic.frame.drop('survived', axis=1)
y = titanic.frame['survived']
```

We create the preprocessing pipelines for both numeric and categorical data. Note that pclass could either be treated as a categorical or numeric feature.

```python
numeric_features = ["age", "fare"]
numeric_transformer = Pipeline(
    steps=[("imputer", SimpleImputer(strategy="median")), ("scaler", StandardScaler())]
)

categorical_features = ["embarked", "sex", "pclass"]
categorical_transformer = OneHotEncoder(handle_unknown="ignore")

preprocessor = ColumnTransformer(
    transformers=[
        ("num", numeric_transformer, numeric_features),
        ("cat", categorical_transformer, categorical_features),
    ]
)
```

Append classifier to preprocessing pipeline. Now we have a full prediction pipeline.

```python
clf = Pipeline(
    steps=[("preprocessor", preprocessor), ("classifier", LogisticRegression())]
)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)
```

As `Evaluate` metrics use lists as inputs for references and predictions, we need to convert them to Python lists.

```python
# Evaluate metrics accept lists as inputs for values of references and predictions

y_test = y_test.tolist()
y_pred = y_pred.tolist()

# Accuracy

accuracy_metric = evaluate.load("accuracy")
accuracy = accuracy_metric.compute(references=y_test, predictions=y_pred)
print("Accuracy:", accuracy)
# Accuracy: 0.79
```

You can use any suitable `evaluate` metric with the estimators as long as they are compatible with the task and predictions.

### Working with Keras and Tensorflow
https://huggingface.co/docs/evaluate/main/keras_integrations.md

# Working with Keras and Tensorflow

Evaluate can be easily intergrated into your Keras and Tensorflow workflow. We'll demonstrate two ways of incorporating Evaluate into model training, using the Fashion MNIST example dataset. We'll train a standard classifier to predict two classes from this dataset, and show how to use a metric as a callback during training or afterwards for evaluation. 

```python
import numpy as np
from tensorflow import keras
from tensorflow.keras import layers
import evaluate

# We pull example code from Keras.io's guide on classifying with MNIST
# Located here: https://keras.io/examples/vision/mnist_convnet/

# Model / data parameters
input_shape = (28, 28, 1)

# Load the data and split it between train and test sets
(x_train, y_train), (x_test, y_test) = keras.datasets.fashion_mnist.load_data()

# Only select tshirts/tops and trousers, classes 0 and 1
def get_tshirts_tops_and_trouser(x_vals, y_vals):
    mask = np.where((y_vals == 0) | (y_vals == 1))
    return x_vals[mask], y_vals[mask]

x_train, y_train = get_tshirts_tops_and_trouser(x_train, y_train)
x_test, y_test = get_tshirts_tops_and_trouser(x_test, y_test)

# Scale images to the [0, 1] range
x_train = x_train.astype("float32") / 255
x_test = x_test.astype("float32") / 255

x_train = np.expand_dims(x_train, -1)
x_test = np.expand_dims(x_test, -1)

model = keras.Sequential(
    [
        keras.Input(shape=input_shape),
        layers.Conv2D(32, kernel_size=(3, 3), activation="relu"),
        layers.MaxPooling2D(pool_size=(2, 2)),
        layers.Conv2D(64, kernel_size=(3, 3), activation="relu"),
        layers.MaxPooling2D(pool_size=(2, 2)),
        layers.Flatten(),
        layers.Dropout(0.5),
        layers.Dense(1, activation="sigmoid"),
    ]
)
```

## Callbacks

Suppose we want to keep track of model metrics while a model is training. We can use a Callback in order to calculate this metric during training, after an epoch ends. 

We'll define a callback here that will take a metric name and our training data, and have it calculate a metric after the epoch ends. 

```python
class MetricsCallback(keras.callbacks.Callback):

    def __init__(self, metric_name, x_data, y_data) -> None:
        super(MetricsCallback, self).__init__()

        self.x_data = x_data
        self.y_data = y_data
        self.metric_name = metric_name
        self.metric = evaluate.load(metric_name)

    def on_epoch_end(self, epoch, logs=dict()):
        m = self.model 
        # Ensure we get labels of "1" or "0"
        training_preds = np.round(m.predict(self.x_data))
        training_labels = self.y_data

        # Compute score and save
        score = self.metric.compute(predictions = training_preds, references = training_labels)
        
        logs.update(score)
```

We can pass this class to the `callbacks` keyword-argument to use it during training:

```python
batch_size = 128
epochs = 2

model.compile(loss="binary_crossentropy", optimizer="adam")

model_history = model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, validation_split=0.1, 
callbacks = [MetricsCallback(x_data = x_train, y_data = y_train, metric_name = "accuracy")])
```

## Using an Evaluate Metric for... Evaluation!

We can also use the same metric after model training! Here, we show how to check accuracy of the model after training on the test set:

```python
acc = evaluate.load("accuracy")
# Round the predictions to turn them into "0" or "1" labels
test_preds = np.round(model.predict(x_test))
test_labels = y_test
```

```python
print("Test accuracy is : ", acc.compute(predictions = test_preds, references = test_labels))
# Test accuracy is : 0.9855
```

### Considerations for model evaluation
https://huggingface.co/docs/evaluate/main/considerations.md

# Considerations for model evaluation

Developing an ML model is rarely a one-shot deal: it often involves multiple stages of defining the model architecture and tuning hyper-parameters before converging on a final set. Responsible model evaluation is a key part of this process, and 🤗 Evaluate is here to help!

Here are some things to keep in mind when evaluating your model using the 🤗 Evaluate library:

## Properly splitting your data

Good evaluation generally requires three splits of your dataset:

- **train**: this is used for training your model.
- **validation**: this is used for validating the model hyperparameters.
- **test**: this is used for evaluating your model.

Many of the datasets on the 🤗 Hub are separated into 2 splits: `train` and `validation`; others are split into 3 splits (`train`, `validation` and `test`) -- make sure to use the right split for the right purpose!

Some datasets on the 🤗 Hub are already separated into these three splits. However, there are also many that only have a train/validation or only train split.

If the dataset you're using doesn't have a predefined train-test split, it is up to you to define which part of the dataset you want to use for training your model and  which you want to use for hyperparameter tuning or final evaluation.

> [!WARNING]
> Training and evaluating on the same split can misrepresent your results! If you overfit on your training data the evaluation results on that split will look great but the model will perform poorly on new data.

Depending on the size of the dataset, you can keep anywhere from 10-30% for evaluation and the rest for training, while aiming to set up the test set to reflect the production data as close as possible. Check out [this thread](https://discuss.huggingface.co/t/how-to-split-main-dataset-into-train-dev-test-as-datasetdict/1090) for a more in-depth discussion of dataset splitting!

## The impact of class imbalance

While many academic datasets, such as the [IMDb dataset](https://huggingface.co/datasets/imdb) of movie reviews, are perfectly balanced, most real-world datasets are not. In machine learning a *balanced dataset* corresponds to a datasets where all labels are represented equally. In the case of the IMDb dataset this means that there are as many positive as negative reviews in the dataset. In an imbalanced dataset this is not the case: in fraud detection for example there are usually many more non-fraud cases than fraud cases in the dataset.

Having an imbalanced dataset can skew the results of your metrics. Imagine a dataset with 99 "non-fraud" cases and 1 "fraud" case. A simple model that always predicts "non-fraud" cases would give yield a 99% accuracy which might sound good at first until you realize that you will never catch a fraud case.

Often, using more than one metric can help get a better idea of your model’s performance from different points of view. For instance, metrics like **[recall](https://huggingface.co/metrics/recall)** and **[precision](https://huggingface.co/metrics/precision)** can be used together, and the **[f1 score](https://huggingface.co/metrics/f1)** is actually the harmonic mean of the two.

In cases where a dataset is balanced, using [accuracy](https://huggingface.co/metrics/accuracy) can reflect the overall model performance:

![Balanced Labels](https://huggingface.co/datasets/evaluate/media/resolve/main/balanced-classes.png)

In cases where there is an imbalance, using [F1 score](https://huggingface.co/metrics/f1) can be a better representation of performance, given that it encompasses both precision and recall.

![Imbalanced Labels](https://huggingface.co/datasets/evaluate/media/resolve/main/imbalanced-classes.png)

Using accuracy in an imbalanced setting is less ideal, since it is not sensitive to minority classes and will not faithfully reflect model performance on them.

## Offline vs. online model evaluation

There are multiple ways to evaluate models, and an important distinction is offline versus online evaluation:

**Offline evaluation** is done before deploying a model or using insights generated from a model, using static datasets and metrics.

**Online evaluation** means evaluating how a model is performing after deployment and during its use in production.

These two types of evaluation can use different metrics and measure different aspects of model performance. For example, offline evaluation can compare a model to other models based on their performance on common benchmarks, whereas online evaluation will evaluate aspects such as latency and accuracy of the model based on production data (for example, the number of user queries that it was able to address).

## Trade-offs in model evaluation

When evaluating models in practice, there are often trade-offs that have to be made between different aspects of model performance: for instance, choosing a model that is slightly less accurate but that has a faster inference time, compared to a high-accuracy that has a higher memory footprint and requires access to more GPUs.

Here are other aspects of model performance to consider during evaluation:

### Interpretability

When evaluating models, **interpretability** (i.e. the ability to *interpret* results)  can be very important, especially when deploying models in production.

For instance, metrics such as [exact match](https://huggingface.co/spaces/evaluate-metric/exact_match) have a set range (between 0 and 1, or 0% and 100%) and are easily understandable to users: for a pair of strings, the exact match score is 1 if the two strings are the exact same, and 0 otherwise.

Other metrics, such as [BLEU](https://huggingface.co/spaces/evaluate-metric/exact_match) are harder to interpret: while they also range between 0 and 1, they can vary greatly depending on which parameters are used to generate the scores, especially when different tokenization and normalization techniques are used (see the [metric card](https://huggingface.co/spaces/evaluate-metric/bleu/blob/main/README.md) for more information about BLEU limitations). This means that it is difficult to interpret a BLEU score without having more information about the procedure used for obtaining it.

Interpretability can be more or less important depending on the evaluation use case, but it is a useful aspect of model evaluation to keep in mind, since communicating and comparing model evaluations is an important part of responsible machine learning.

### Inference speed and memory footprint

While recent years have seen increasingly large ML models achieve high performance on a large variety of tasks and benchmarks, deploying these multi-billion parameter models in practice can be a challenge in itself, and many organizations lack the resources for this. This is why considering the **inference speed** and **memory footprint** of models is important, especially when doing online model evaluation.

Inference speed refers to the time that it takes for a model to make a prediction -- this will vary depending on the hardware used and the way in which models are queried, e.g. in real time via an API or in batch jobs that run once a day.

Memory footprint refers to the size of the model weights and how much hardware memory they occupy. If a model is too large to fit on a single GPU or CPU, then it has to be split over multiple ones, which can be more or less difficult depending on the model architecture and the deployment method.

When doing online model evaluation, there is often a trade-off to be done between inference speed and accuracy or precision, whereas this is less the case for offline evaluation.

## Limitations and bias

All models and all metrics have their limitations and biases, which depend on the way in which they were trained, the data that was used, and their intended uses. It is important to measure and communicate these limitations clearly to prevent misuse and unintended impacts, for instance via [model cards](https://huggingface.co/course/chapter4/4?fw=pt) which document the training and evaluation process.

Measuring biases can be done by evaluating models on datasets such as [Wino Bias](https://huggingface.co/datasets/wino_bias) or [MD Gender Bias](https://huggingface.co/datasets/md_gender_bias), and by doing [Interactive Error Analyis](https://huggingface.co/spaces/nazneen/error-analysis) to try to identify which subsets of the evaluation dataset a model performs poorly on.

We are currently working on additional measurements that can be used to quantify different dimensions of bias in both models and datasets -- stay tuned for more documentation on this topic!

### Choosing a metric for your task
https://huggingface.co/docs/evaluate/main/choosing_a_metric.md

# Choosing a metric for your task

**So you've trained your model and want to see how well it’s doing on a dataset of your choice. Where do you start?**

There is no “one size fits all” approach to choosing an evaluation metric, but some good guidelines to keep in mind are:

## Categories of metrics

There are 3 high-level categories of metrics:

1. *Generic metrics*, which can be applied to a variety of situations and datasets, such as precision and accuracy.
2. *Task-specific metrics*, which are limited to a given task, such as Machine Translation (often evaluated using metrics [BLEU](https://huggingface.co/metrics/bleu) or [ROUGE](https://huggingface.co/metrics/rouge)) or Named Entity Recognition (often evaluated with [seqeval](https://huggingface.co/metrics/seqeval)).
3. *Dataset-specific metrics*, which aim to measure model performance on specific benchmarks: for instance, the [GLUE benchmark](https://huggingface.co/datasets/glue) has a dedicated [evaluation metric](https://huggingface.co/metrics/glue).

Let's look at each of these three cases:

### Generic metrics

Many of the metrics used in the Machine Learning community are quite generic and can be applied in a variety of tasks and datasets.

This is the case for metrics like [accuracy](https://huggingface.co/metrics/accuracy) and [precision](https://huggingface.co/metrics/precision), which can be used for evaluating labeled (supervised) datasets, as well as [perplexity](https://huggingface.co/metrics/perplexity), which can be used for evaluating different kinds of (unsupervised) generative tasks.

To see the input structure of a given metric, you can look at its metric card. For example, in the case of [precision](https://huggingface.co/metrics/precision), the format is:
```
>>> precision_metric = evaluate.load("precision")
>>> results = precision_metric.compute(references=[0, 1], predictions=[0, 1])
>>> print(results)
{'precision': 1.0}
```

### Task-specific metrics

Popular ML tasks like Machine Translation and Named Entity Recognition have specific metrics that can be used to compare models. For example, a series of different metrics have been proposed for text generation, ranging from [BLEU](https://huggingface.co/metrics/bleu) and its derivatives such as [GoogleBLEU](https://huggingface.co/metrics/google_bleu) and [GLEU](https://huggingface.co/metrics/gleu), but also [ROUGE](https://huggingface.co/metrics/rouge), [MAUVE](https://huggingface.co/metrics/mauve), etc.

You can find the right metric for your task by:

- **Looking at the [Task pages](https://huggingface.co/tasks)** to see what metrics can be used for evaluating models for a given task.
- **Checking out leaderboards** on sites like [Papers With Code](https://paperswithcode.com/) (you can search by task and by dataset).
-  **Reading the metric cards** for the relevant metrics and see which ones are a good fit for your use case. For example, see the [BLEU metric card](https://github.com/huggingface/evaluate/tree/main/metrics/bleu) or [SQuaD metric card](https://github.com/huggingface/evaluate/tree/main/metrics/squad).
- **Looking at papers and blog posts** published on the topic and see what metrics they report. This can change over time, so try to pick papers from the last couple of years!

### Dataset-specific metrics

Some datasets have specific metrics associated with them -- this is especially in the case of popular benchmarks like [GLUE](https://huggingface.co/metrics/glue) and [SQuAD](https://huggingface.co/metrics/squad).

> [!WARNING]
> 💡
> GLUE is actually a collection of different subsets on different tasks, so first you need to choose the one that corresponds to the NLI task, such as mnli, which is described as “crowdsourced collection of sentence pairs with textual entailment annotations”

If you are evaluating your model on a benchmark dataset like the ones mentioned above, you can use its dedicated evaluation metric. Make sure you respect the format that they require. For example, to evaluate your model on the [SQuAD](https://huggingface.co/datasets/squad) dataset, you need to feed the `question` and `context` into your model and return the `prediction_text`, which should be compared with the `references` (based on matching the `id` of the question) :

```
>>> from evaluate import load
>>> squad_metric = load("squad")
>>> predictions = [{'prediction_text': '1976', 'id': '56e10a3be3433e1400422b22'}]
>>> references = [{'answers': {'answer_start': [97], 'text': ['1976']}, 'id': '56e10a3be3433e1400422b22'}]
>>> results = squad_metric.compute(predictions=predictions, references=references)
>>> results
{'exact_match': 100.0, 'f1': 100.0}
```

You can find examples of dataset structures by consulting the "Dataset Preview" function or the dataset card for a given dataset, and you can see how to use its dedicated evaluation function based on the metric card.

### Types of Evaluations in 🤗 Evaluate
https://huggingface.co/docs/evaluate/main/types_of_evaluations.md

# Types of Evaluations in 🤗 Evaluate

The goal of the 🤗 Evaluate library is to support different types of evaluation, depending on different goals, datasets and models.

Here are the types of evaluations that are currently supported with a few examples for each:

## Metrics
A metric measures the performance of a model on a given dataset. This is often based on an existing ground truth (i.e. a set of references), but there are also *referenceless metrics* which allow evaluating generated text by leveraging a pretrained model such as [GPT-2](https://huggingface.co/gpt2).

Examples of metrics include:
- [Accuracy](https://huggingface.co/metrics/accuracy) : the proportion of correct predictions among the total number of cases processed.
- [Exact Match](https://huggingface.co/metrics/exact_match): the rate at which the input predicted strings exactly match their references.
- [Mean Intersection over union (IoUO)](https://huggingface.co/metrics/mean_iou): the area of overlap between the predicted segmentation of an image and the ground truth divided by the area of union between the predicted segmentation and the ground truth.

Metrics are often used to track model performance on benchmark datasets, and to report progress on tasks such as [machine translation](https://huggingface.co/tasks/translation) and [image classification](https://huggingface.co/tasks/image-classification).

## Comparisons

Comparisons can be useful to compare the performance of two or more models on a single test dataset.

For instance, the [McNemar Test](https://github.com/huggingface/evaluate/tree/main/comparisons/mcnemar) is a paired nonparametric statistical hypothesis test that takes the predictions of two models and compares them, aiming to measure whether the models's predictions diverge or not. The p value it outputs, which ranges from `0.0` to `1.0`, indicates the difference between the two models' predictions, with a lower p value indicating a more significant difference.

Comparisons have yet to be systematically used when comparing and reporting model performance, however they are useful tools to go beyond simply comparing leaderboard scores and for getting more information on the way model prediction differ.

## Measurements

In the 🤗 Evaluate library, measurements are tools for gaining more insights on datasets and model predictions.

For instance, in the case of datasets, it can be useful to calculate the [average word length](https://github.com/huggingface/evaluate/tree/main/measurements/word_length) of a dataset's entries, and how it is distributed -- this can help when choosing the maximum input length for [Tokenizer](https://huggingface.co/docs/transformers/main_classes/tokenizer).

In the case of model predictions, it can help to calculate the average [perplexity](https://huggingface.co/metrics/perplexity) of model predictions using different models such as [GPT-2](https://huggingface.co/gpt2) and [BERT](https://huggingface.co/bert-base-uncased), which can indicate the quality of generated text when no reference is available.

All three types of evaluation supported by the 🤗 Evaluate library are meant to be mutually complementary, and help our community carry out more mindful and responsible evaluation.

We will continue adding more types of metrics, measurements and comparisons in coming months, and are counting on community involvement (via [PRs](https://github.com/huggingface/evaluate/compare) and [issues](https://github.com/huggingface/evaluate/issues/new/choose)) to make the library as extensive and inclusive as possible!

### Using the `evaluator`
https://huggingface.co/docs/evaluate/main/base_evaluator.md

# Using the `evaluator`

The `Evaluator` classes allow to evaluate a  triplet of model, dataset, and metric. The models wrapped in a pipeline, responsible for handling all preprocessing and post-processing and out-of-the-box, `Evaluator`s support transformers pipelines for the supported tasks, but custom pipelines can be passed, as showcased in the section [Using the `evaluator` with custom pipelines](custom_evaluator).

Currently supported tasks are:
- `"text-classification"`: will use the [TextClassificationEvaluator](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.TextClassificationEvaluator).
- `"token-classification"`: will use the [TokenClassificationEvaluator](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.TokenClassificationEvaluator).
- `"question-answering"`: will use the [QuestionAnsweringEvaluator](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.QuestionAnsweringEvaluator).
- `"image-classification"`: will use the [ImageClassificationEvaluator](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.ImageClassificationEvaluator).
- `"text-generation"`: will use the [TextGenerationEvaluator](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.TextGenerationEvaluator).
- `"text2text-generation"`: will use the [Text2TextGenerationEvaluator](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.Text2TextGenerationEvaluator).
- `"summarization"`: will use the [SummarizationEvaluator](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.SummarizationEvaluator).
- `"translation"`: will use the [TranslationEvaluator](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.TranslationEvaluator).
- `"automatic-speech-recognition"`: will use the [AutomaticSpeechRecognitionEvaluator](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.AutomaticSpeechRecognitionEvaluator).
- `"audio-classification"`: will use the [AudioClassificationEvaluator](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.AudioClassificationEvaluator).

To run an `Evaluator` with several tasks in a single call, use the [EvaluationSuite](evaluation_suite), which runs evaluations on a collection of `SubTask`s.

Each task has its own set of requirements for the dataset format and pipeline output, make sure to check them out for your custom use case. Let's have a look at some of them and see how you can use the evaluator to evalute a single or multiple of models, datasets, and metrics at the same time.

## Text classification

The text classification evaluator can be used to evaluate text models on classification datasets such as IMDb. Beside the model, data, and metric inputs it takes the following optional inputs:

- `input_column="text"`: with this argument the column with the data for the pipeline can be specified.
- `label_column="label"`: with this argument the column with the labels for the evaluation can be specified.
- `label_mapping=None`: the label mapping aligns the labels in the pipeline output with the labels need for evaluation. E.g. the labels in `label_column` can be integers (`0`/`1`) whereas the pipeline can produce label names such as `"positive"`/`"negative"`. With that dictionary the pipeline outputs are mapped to the labels.

By default the `"accuracy"` metric is computed.

### Evaluate models on the Hub

There are several ways to pass a model to the evaluator: you can pass the name of a model on the Hub, you can load a `transformers` model and pass it to the evaluator or you can pass an initialized `transformers.Pipeline`. Alternatively you can pass any callable function that behaves like a `pipeline` call for the task in any framework.

So any of the following works:

```py
from datasets import load_dataset
from evaluate import evaluator
from transformers import AutoModelForSequenceClassification, pipeline

data = load_dataset("imdb", split="test").shuffle(seed=42).select(range(1000))
task_evaluator = evaluator("text-classification")

# 1. Pass a model name or path
eval_results = task_evaluator.compute(
    model_or_pipeline="lvwerra/distilbert-imdb",
    data=data,
    label_mapping={"NEGATIVE": 0, "POSITIVE": 1}
)

# 2. Pass an instantiated model
model = AutoModelForSequenceClassification.from_pretrained("lvwerra/distilbert-imdb")

eval_results = task_evaluator.compute(
    model_or_pipeline=model,
    data=data,
    label_mapping={"NEGATIVE": 0, "POSITIVE": 1}
)

# 3. Pass an instantiated pipeline
pipe = pipeline("text-classification", model="lvwerra/distilbert-imdb")

eval_results = task_evaluator.compute(
    model_or_pipeline=pipe,
    data=data,
    label_mapping={"NEGATIVE": 0, "POSITIVE": 1}
)
print(eval_results)
```
> [!TIP]
> Without specifying a device, the default for model inference will be the first GPU on the machine if one is available, and else CPU. If you want to use a specific device you can pass `device` to `compute` where -1 will use the GPU and a positive integer (starting with 0) will use the associated CUDA device.

The results will look as follows:
```python
{
    'accuracy': 0.918,
    'latency_in_seconds': 0.013,
    'samples_per_second': 78.887,
    'total_time_in_seconds': 12.676
}
```

Note that evaluation results include both the requested metric, and information about the time it took to obtain predictions through the pipeline.

> [!TIP]
> The time performances can give useful indication on model speed for inference but should be taken with a grain of salt: they include all the processing that goes on in the pipeline. This may include tokenizing, post-processing, that may be different depending on the model. Furthermore, it depends a lot on the hardware you are running the evaluation on and you may be able to improve the performance by optimizing things like the batch size.

### Evaluate multiple metrics

With the [combine()](/docs/evaluate/main/en/package_reference/main_classes#evaluate.combine) function one can bundle several metrics into an object that behaves like a single metric. We can use this to evaluate several metrics at once with the evaluator:

```python
import evaluate

eval_results = task_evaluator.compute(
    model_or_pipeline="lvwerra/distilbert-imdb",
    data=data,
    metric=evaluate.combine(["accuracy", "recall", "precision", "f1"]),
    label_mapping={"NEGATIVE": 0, "POSITIVE": 1}
)
print(eval_results)

```
The results will look as follows:
```python
{
    'accuracy': 0.918,
    'f1': 0.916,
    'precision': 0.9147,
    'recall': 0.9187,
    'latency_in_seconds': 0.013,
    'samples_per_second': 78.887,
    'total_time_in_seconds': 12.676
}
```

Next let's have a look at token classification.

## Token Classification

With the token classification evaluator one can evaluate models for tasks such as NER or POS tagging. It has the following specific arguments:

- `input_column="text"`: with this argument the column with the data for the pipeline can be specified.
- `label_column="label"`: with this argument the column with the labels for the evaluation can be specified.
- `label_mapping=None`: the label mapping aligns the labels in the pipeline output with the labels need for evaluation. E.g. the labels in `label_column` can be integers (`0`/`1`) whereas the pipeline can produce label names such as `"positive"`/`"negative"`. With that dictionary the pipeline outputs are mapped to the labels.
- `join_by=" "`: While most datasets are already tokenized the pipeline expects a string. Thus the tokens need to be joined before passing to the pipeline. By default they are joined with a whitespace.

Let's have a look how we can use the evaluator to benchmark several models.

### Benchmarking several models

Here is an example where several models can be compared thanks to the `evaluator` in only a few lines of code, abstracting away the preprocessing, inference, postprocessing, metric computation:

```python
import pandas as pd
from datasets import load_dataset
from evaluate import evaluator
from transformers import pipeline

models = [
    "xlm-roberta-large-finetuned-conll03-english",
    "dbmdz/bert-large-cased-finetuned-conll03-english",
    "elastic/distilbert-base-uncased-finetuned-conll03-english",
    "dbmdz/electra-large-discriminator-finetuned-conll03-english",
    "gunghio/distilbert-base-multilingual-cased-finetuned-conll2003-ner",
    "philschmid/distilroberta-base-ner-conll2003",
    "Jorgeutd/albert-base-v2-finetuned-ner",
]

data = load_dataset("conll2003", split="validation").shuffle().select(range(1000))
task_evaluator = evaluator("token-classification")

results = []
for model in models:
    results.append(
        task_evaluator.compute(
            model_or_pipeline=model, data=data, metric="seqeval"
            )
        )

df = pd.DataFrame(results, index=models)
df[["overall_f1", "overall_accuracy", "total_time_in_seconds", "samples_per_second", "latency_in_seconds"]]
```

The result is a table that looks like this:

|   model                                                            |   overall_f1 |   overall_accuracy |   total_time_in_seconds |   samples_per_second |   latency_in_seconds |
|:-------------------------------------------------------------------|-------------:|-------------------:|------------------------:|---------------------:|---------------------:|
| Jorgeutd/albert-base-v2-finetuned-ner                              |        0.941 |              0.989 |                   4.515 |              221.468 |                0.005 |
| dbmdz/bert-large-cased-finetuned-conll03-english                   |        0.962 |              0.881 |                  11.648 |               85.850 |                0.012 |
| dbmdz/electra-large-discriminator-finetuned-conll03-english        |        0.965 |              0.881 |                  11.456 |               87.292 |                0.011 |
| elastic/distilbert-base-uncased-finetuned-conll03-english          |        0.940 |              0.989 |                   2.318 |              431.378 |                0.002 |
| gunghio/distilbert-base-multilingual-cased-finetuned-conll2003-ner |        0.947 |              0.991 |                   2.376 |              420.873 |                0.002 |
| philschmid/distilroberta-base-ner-conll2003                        |        0.961 |              0.994 |                   2.436 |              410.579 |                0.002 |
| xlm-roberta-large-finetuned-conll03-english                        |        0.969 |              0.882 |                  11.996 |               83.359 |                0.012 |

### Visualizing results

You can feed in the `results` list above into the `plot_radar()` function to visualize different aspects of their performance and choose the model that is the best fit, depending on the metric(s) that are relevant to your use case:

```python
import evaluate
from evaluate.visualization import radar_plot

>>> plot = radar_plot(data=results, model_names=models, invert_range=["latency_in_seconds"])
>>> plot.show()
```

    

Don't forget to specify `invert_range` for metrics for which smaller is better (such as the case for latency in seconds).

If you want to save the plot locally, you can use the `plot.savefig()` function with the option `bbox_inches='tight'`, to make sure no part of the image gets cut off.

## Question Answering

With the question-answering evaluator one can evaluate models for QA without needing to worry about the complicated pre- and post-processing that's required for these models. It has the following specific arguments:

- `question_column="question"`: the name of the column containing the question in the dataset
- `context_column="context"`: the name of the column containing the context
- `id_column="id"`: the name of the column cointaing the identification field of the question and answer pair
- `label_column="answers"`: the name of the column containing the answers
- `squad_v2_format=None`: whether the dataset follows the format of squad_v2 dataset where a question may have no answer in the context. If this parameter is not provided, the format will be automatically inferred.

Let's have a look how we can evaluate QA models and compute confidence intervals at the same time.

### Confidence intervals

Every evaluator comes with the options to compute confidence intervals using [bootstrapping](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html). Simply pass `strategy="bootstrap"` and set the number of resanmples with `n_resamples`.

```python
from datasets import load_dataset
from evaluate import evaluator

task_evaluator = evaluator("question-answering")

data = load_dataset("squad", split="validation[:1000]")
eval_results = task_evaluator.compute(
    model_or_pipeline="distilbert-base-uncased-distilled-squad",
    data=data,
    metric="squad",
    strategy="bootstrap",
    n_resamples=30
)
```

Results include confidence intervals as well as error estimates as follows:

```python
{
    'exact_match':
    {
        'confidence_interval': (79.67, 84.54),
        'score': 82.30,
        'standard_error': 1.28
    },
    'f1':
    {
        'confidence_interval': (85.30, 88.88),
        'score': 87.23,
        'standard_error': 0.97
    },
    'latency_in_seconds': 0.0085,
    'samples_per_second': 117.31,
    'total_time_in_seconds': 8.52
 }
```

## Image classification

With the image classification evaluator we can evaluate any image classifier. It uses the same keyword arguments at the text classifier:

- `input_column="image"`: the name of the column containing the images as PIL ImageFile
- `label_column="label"`: the name of the column containing the labels
- `label_mapping=None`: We want to map class labels defined by the model in the pipeline to values consistent with those defined in the `label_column`

Let's have a look at how can evaluate image classification models on large datasets.

### Handling large datasets

The evaluator can be used on large datasets! Below, an example shows how to use it on ImageNet-1k for image classification. Beware that this example will require to download ~150 GB.

```python
data = load_dataset("imagenet-1k", split="validation", token=True)

pipe = pipeline(
    task="image-classification",
    model="facebook/deit-small-distilled-patch16-224"
)

task_evaluator = evaluator("image-classification")
eval_results = task_evaluator.compute(
    model_or_pipeline=pipe,
    data=data,
    metric="accuracy",
    label_mapping=pipe.model.config.label2id
)
```

Since we are using `datasets` to store data we make use of a technique called memory mappings. This means that the dataset is never fully loaded into memory which saves a lot of RAM. Running the above code only uses roughly 1.5 GB of RAM while the validation split is more than 30 GB big.

### Creating an EvaluationSuite
https://huggingface.co/docs/evaluate/main/evaluation_suite.md

# Creating an EvaluationSuite

It can be useful to evaluate models on a variety of different tasks to understand their downstream performance. Assessing the model on several types of tasks can reveal gaps in performance along some axis. For example, when training a language model, it is often useful to measure perplexity on an in-domain corpus, but also to concurrently evaluate on tasks which test for general language capabilities like natural language entailment or question-answering, or tasks designed to probe the model along fairness and bias dimensions.

The `EvaluationSuite` provides a way to compose any number of ([evaluator](base_evaluator), dataset, metric) tuples as a SubTask to evaluate a model on a collection of several evaluation tasks. See the [evaluator documentation](base_evaluator) for a list of currently supported tasks.

A new `EvaluationSuite` is made up of a list of `SubTask` classes, each defining an evaluation task. The Python file containing the definition can be uploaded to a Space on the Hugging Face Hub so it can be shared with the community or saved/loaded locally as a Python script.

Some datasets require additional preprocessing before passing them to an `Evaluator`. You can set a `data_preprocessor` for each `SubTask` which is applied via a `map` operation using the `datasets` library. Keyword arguments for the `Evaluator` can be passed down through the `args_for_task` attribute.

To create a new `EvaluationSuite`, create a [new Space](https://huggingface.co/new-space) with a .py file which matches the name of the Space, add the below template to a Python file, and fill in the attributes for a new task.

The mandatory attributes for a new `SubTask` are `task_type` and `data`.
1. `task_type` maps to the tasks currently supported by the Evaluator.
2. `data` can be an instantiated Hugging Face dataset object or the name of a dataset.
3. `subset` and `split` can be used to define which name and split of the dataset should be used for evaluation.
4. `args_for_task` should be a dictionary with kwargs to be passed to the Evaluator.

```python
import evaluate
from evaluate.evaluation_suite import SubTask

class Suite(evaluate.EvaluationSuite):

    def __init__(self, name):
        super().__init__(name)
        self.preprocessor = lambda x: {"text": x["text"].lower()}
        self.suite = [
            SubTask(
                task_type="text-classification",
                data="glue",
                subset="sst2",
                split="validation[:10]",
                args_for_task={
                    "metric": "accuracy",
                    "input_column": "sentence",
                    "label_column": "label",
                    "label_mapping": {
                        "LABEL_0": 0.0,
                        "LABEL_1": 1.0
                    }
                }
            ),
            SubTask(
                task_type="text-classification",
                data="glue",
                subset="rte",
                split="validation[:10]",
                args_for_task={
                    "metric": "accuracy",
                    "input_column": "sentence1",
                    "second_input_column": "sentence2",
                    "label_column": "label",
                    "label_mapping": {
                        "LABEL_0": 0,
                        "LABEL_1": 1
                    }
                }
            )
        ]
```

An `EvaluationSuite` can be loaded by name from the Hugging Face Hub, or locally by providing a path, and run with the `run(model_or_pipeline)` method. The evaluation results are returned along with their task names and information about the time it took to obtain predictions through the pipeline. These can be easily displayed with a `pandas.DataFrame`:

```
>>> from evaluate import EvaluationSuite
>>> suite = EvaluationSuite.load('mathemakitten/glue-evaluation-suite')
>>> results = suite.run("gpt2")
```

|   accuracy |   total_time_in_seconds |   samples_per_second |   latency_in_seconds | task_name   |
|-----------:|------------------------:|---------------------:|---------------------:|:------------|
|        0.5 |                0.740811 |             13.4987  |            0.0740811 | glue/sst2   |
|        0.4 |                1.67552  |              5.9683  |            0.167552  | glue/rte    |

### Evaluator
https://huggingface.co/docs/evaluate/main/package_reference/evaluator_classes.md

# Evaluator

The evaluator classes for automatic evaluation.

## Evaluator classes[[evaluate.evaluator]]

The main entry point for using the evaluator:

#### evaluate.evaluator[[evaluate.evaluator]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/__init__.py#L113)

Utility factory method to build an [Evaluator](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.Evaluator).
Evaluators encapsulate a task and a default metric name. They leverage `pipeline` functionality from `transformers`
to simplify the evaluation of multiple combinations of models, datasets and metrics for a given task.

Examples:
```python
>>> from evaluate import evaluator
>>> # Sentiment analysis evaluator
>>> evaluator("sentiment-analysis")
```

**Parameters:**

task (`str`) : The task defining which evaluator will be returned. Currently accepted tasks are: - `"image-classification"`: will return a [ImageClassificationEvaluator](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.ImageClassificationEvaluator). - `"question-answering"`: will return a [QuestionAnsweringEvaluator](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.QuestionAnsweringEvaluator). - `"text-classification"` (alias `"sentiment-analysis"` available): will return a [TextClassificationEvaluator](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.TextClassificationEvaluator). - `"token-classification"`: will return a [TokenClassificationEvaluator](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.TokenClassificationEvaluator).

**Returns:**

`[Evaluator](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.Evaluator)`

An evaluator suitable for the task.

The base class for all evaluator classes:

#### evaluate.Evaluator[[evaluate.Evaluator]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/base.py#L102)

The [Evaluator](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.Evaluator) class is the class from which all evaluators inherit. Refer to this class for methods shared across
different evaluators.
Base class implementing evaluator operations.

check_required_columnsevaluate.Evaluator.check_required_columnshttps://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/base.py#L295[{"name": "data", "val": ": typing.Union[str, datasets.arrow_dataset.Dataset]"}, {"name": "columns_names", "val": ": typing.Dict[str, str]"}]- **data** (`str` or `Dataset`) --
  Specifies the dataset we will run evaluation on.
- **columns_names** (`List[str]`) --
  List of column names to check in the dataset. The keys are the arguments to the [evaluate.EvaluationModule.compute()](/docs/evaluate/main/en/package_reference/main_classes#evaluate.EvaluationModule.compute) method,
  while the values are the column names to check.0

Ensure the columns required for the evaluation are present in the dataset.

Example:

```py
>>> from datasets import load_dataset
>>> from evaluate import evaluator
>>> data = load_dataset("rotten_tomatoes', split="train")
>>> evaluator.check_required_columns(data, {"input_column": "text", "label_column": "label"})
```

**Parameters:**

data (`str` or `Dataset`) : Specifies the dataset we will run evaluation on.

columns_names (`List[str]`) : List of column names to check in the dataset. The keys are the arguments to the [evaluate.EvaluationModule.compute()](/docs/evaluate/main/en/package_reference/main_classes#evaluate.EvaluationModule.compute) method, while the values are the column names to check.
#### compute_metric[[evaluate.Evaluator.compute_metric]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/base.py#L517)

Compute and return metrics.
#### get_dataset_split[[evaluate.Evaluator.get_dataset_split]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/base.py#L321)

Infers which split to use if `None` is given.

Example:

```py
>>> from evaluate import evaluator
>>> evaluator("text-classification").get_dataset_split(data="rotten_tomatoes")
WARNING:evaluate.evaluator.base:Dataset split not defined! Automatically evaluating with split: TEST
'test'
```

**Parameters:**

data (`str`) : Name of dataset.

subset (`str`) : Name of config for datasets with multiple configurations (e.g. 'glue/cola').

split (`str`, defaults to `None`) : Split to use.

**Returns:**

``split``

`str` containing which split to use
#### load_data[[evaluate.Evaluator.load_data]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/base.py#L350)

Load dataset with given subset and split.

Example:

```py
>>> from evaluate import evaluator
>>> evaluator("text-classification").load_data(data="rotten_tomatoes", split="train")
Dataset({
    features: ['text', 'label'],
    num_rows: 8530
})
```

**Parameters:**

data (`Dataset` or `str`, defaults to `None`) : Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset name, and load it. Otherwise we assume it represents a pre-loaded dataset.

subset (`str`, defaults to `None`) : Specifies dataset subset to be passed to `name` in `load_dataset`. To be used with datasets with several configurations (e.g. glue/sst2).

split (`str`, defaults to `None`) : User-defined dataset split by name (e.g. train, validation, test). Supports slice-split (`test[:n]`). If not defined and data is a `str` type, will automatically select the best one via `choose_split()`.

**Returns:**

`data (`Dataset`)`

Loaded dataset which will be used for evaluation.
#### predictions_processor[[evaluate.Evaluator.predictions_processor]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/base.py#L211)

A core method of the `Evaluator` class, which processes the pipeline outputs for compatibility with the metric.
#### prepare_data[[evaluate.Evaluator.prepare_data]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/base.py#L390)

Prepare data.

Example:

```py
>>> from evaluate import evaluator
>>> from datasets import load_dataset

>>> ds = load_dataset("rotten_tomatoes", split="train")
>>> evaluator("text-classification").prepare_data(ds, input_column="text", second_input_column=None, label_column="label")
```

**Parameters:**

data (`Dataset`) : Specifies the dataset we will run evaluation on.

input_column (`str`, defaults to `"text"`) : The name of the column containing the text feature in the dataset specified by `data`.

second_input_column(`str`, *optional*) : The name of the column containing the second text feature if there is one. Otherwise, set to `None`.

label_column (`str`, defaults to `"label"`) : The name of the column containing the labels in the dataset specified by `data`.

**Returns:**

``dict``

metric inputs.
`list`:  pipeline inputs.
#### prepare_metric[[evaluate.Evaluator.prepare_metric]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/base.py#L480)

Prepare metric.

Example:

```py
>>> from evaluate import evaluator
>>> evaluator("text-classification").prepare_metric("accuracy")
```

**Parameters:**

metric (`str` or [EvaluationModule](/docs/evaluate/main/en/package_reference/main_classes#evaluate.EvaluationModule), defaults to `None`) : Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and load it. Otherwise we assume it represents a pre-loaded metric.

**Returns:**

The loaded metric.
#### prepare_pipeline[[evaluate.Evaluator.prepare_pipeline]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/base.py#L422)

Prepare pipeline.

Example:

```py
>>> from evaluate import evaluator
>>> evaluator("text-classification").prepare_pipeline(model_or_pipeline="distilbert-base-uncased")
```

**Parameters:**

model_or_pipeline (`str` or [Pipeline](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#transformers.Pipeline) or `Callable` or [PreTrainedModel](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel) or `TFPreTrainedModel`, defaults to `None`) : If the argument in not specified, we initialize the default pipeline for the task. If the argument is of the type `str` or is a model instance, we use it to initialize a new [Pipeline](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#transformers.Pipeline) with the given model. Otherwise we assume the argument specifies a pre-initialized pipeline.

preprocessor ([PreTrainedTokenizerBase](https://huggingface.co/docs/transformers/main/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase) or [FeatureExtractionMixin](https://huggingface.co/docs/transformers/main/en/main_classes/feature_extractor#transformers.FeatureExtractionMixin), *optional*, defaults to `None`) : Argument can be used to overwrite a default preprocessor if `model_or_pipeline` represents a model for which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore this argument.

**Returns:**

The initialized pipeline.

## The task specific evaluators

### ImageClassificationEvaluator[[evaluate.ImageClassificationEvaluator]]

#### evaluate.ImageClassificationEvaluator[[evaluate.ImageClassificationEvaluator]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/image_classification.py#L49)

Image classification evaluator.
This image classification evaluator can currently be loaded from [evaluator()](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.evaluator) using the default task name
`image-classification`.
Methods in this class assume a data format compatible with the `ImageClassificationPipeline`.

computeevaluate.ImageClassificationEvaluator.computehttps://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/image_classification.py#L68[{"name": "model_or_pipeline", "val": ": typing.Union[str, ForwardRef('Pipeline'), typing.Callable, ForwardRef('PreTrainedModel'), ForwardRef('TFPreTrainedModel')] = None"}, {"name": "data", "val": ": typing.Union[str, datasets.arrow_dataset.Dataset] = None"}, {"name": "subset", "val": ": typing.Optional[str] = None"}, {"name": "split", "val": ": typing.Optional[str] = None"}, {"name": "metric", "val": ": typing.Union[str, evaluate.module.EvaluationModule] = None"}, {"name": "tokenizer", "val": ": typing.Union[str, ForwardRef('PreTrainedTokenizer'), NoneType] = None"}, {"name": "feature_extractor", "val": ": typing.Union[str, ForwardRef('FeatureExtractionMixin'), NoneType] = None"}, {"name": "strategy", "val": ": typing.Literal['simple', 'bootstrap'] = 'simple'"}, {"name": "confidence_level", "val": ": float = 0.95"}, {"name": "n_resamples", "val": ": int = 9999"}, {"name": "device", "val": ": int = None"}, {"name": "random_state", "val": ": typing.Optional[int] = None"}, {"name": "input_column", "val": ": str = 'image'"}, {"name": "label_column", "val": ": str = 'label'"}, {"name": "label_mapping", "val": ": typing.Optional[typing.Dict[str, numbers.Number]] = None"}]- **model_or_pipeline** (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) --
  If the argument in not specified, we initialize the default pipeline for the task (in this case
  `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or
  is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the
  argument specifies a pre-initialized pipeline.
- **data** (`str` or `Dataset`, defaults to `None`) --
  Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset
  name, and load it. Otherwise we assume it represents a pre-loaded dataset.
- **subset** (`str`, defaults to `None`) --
  Defines which dataset subset to load. If `None` is passed the default subset is loaded.
- **split** (`str`, defaults to `None`) --
  Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.
- **metric** (`str` or `EvaluationModule`, defaults to `None`) --
  Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and
  load it. Otherwise we assume it represents a pre-loaded metric.
- **tokenizer** (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) --
  Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for
  which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore
  this argument.
- **strategy** (`Literal["simple", "bootstrap"]`, defaults to "simple") --
  specifies the evaluation strategy. Possible values are:
  - `"simple"` - we evaluate the metric and return the scores.
  - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each
  of the returned metric keys, using `scipy`'s `bootstrap` method
  https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.
- **confidence_level** (`float`, defaults to `0.95`) --
  The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **n_resamples** (`int`, defaults to `9999`) --
  The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **device** (`int`, defaults to `None`) --
  Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive
  integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and
  CUDA:0 used if available, CPU otherwise.
- **random_state** (`int`, *optional*, defaults to `None`) --
  The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for
  debugging.

- **input_column** (`str`, defaults to `"image"`) --
  The name of the column containing the images as PIL ImageFile in the dataset specified by `data`.
- **label_column** (`str`, defaults to `"label"`) --
  The name of the column containing the labels in the dataset specified by `data`.
- **label_mapping** (`Dict[str, Number]`, *optional*, defaults to `None`) --
  We want to map class labels defined by the model in the pipeline to values consistent with those
  defined in the `label_column` of the `data` dataset.0A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

Compute the metric for a given pipeline and dataset combination.

Examples:
```python
>>> from evaluate import evaluator
>>> from datasets import load_dataset
>>> task_evaluator = evaluator("image-classification")
>>> data = load_dataset("beans", split="test[:40]")
>>> results = task_evaluator.compute(
>>>     model_or_pipeline="nateraw/vit-base-beans",
>>>     data=data,
>>>     label_column="labels",
>>>     metric="accuracy",
>>>     label_mapping={'angular_leaf_spot': 0, 'bean_rust': 1, 'healthy': 2},
>>>     strategy="bootstrap"
>>> )
```

**Parameters:**

model_or_pipeline (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) : If the argument in not specified, we initialize the default pipeline for the task (in this case `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the argument specifies a pre-initialized pipeline.

data (`str` or `Dataset`, defaults to `None`) : Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset name, and load it. Otherwise we assume it represents a pre-loaded dataset.

subset (`str`, defaults to `None`) : Defines which dataset subset to load. If `None` is passed the default subset is loaded.

split (`str`, defaults to `None`) : Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.

metric (`str` or `EvaluationModule`, defaults to `None`) : Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and load it. Otherwise we assume it represents a pre-loaded metric.

tokenizer (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) : Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore this argument.

strategy (`Literal["simple", "bootstrap"]`, defaults to "simple") : specifies the evaluation strategy. Possible values are: - `"simple"` - we evaluate the metric and return the scores. - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each of the returned metric keys, using `scipy`'s `bootstrap` method https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.

confidence_level (`float`, defaults to `0.95`) : The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

n_resamples (`int`, defaults to `9999`) : The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

device (`int`, defaults to `None`) : Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and CUDA:0 used if available, CPU otherwise.

random_state (`int`, *optional*, defaults to `None`) : The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for debugging.   

input_column (`str`, defaults to `"image"`) : The name of the column containing the images as PIL ImageFile in the dataset specified by `data`.

label_column (`str`, defaults to `"label"`) : The name of the column containing the labels in the dataset specified by `data`.

label_mapping (`Dict[str, Number]`, *optional*, defaults to `None`) : We want to map class labels defined by the model in the pipeline to values consistent with those defined in the `label_column` of the `data` dataset.

**Returns:**

A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

### QuestionAnsweringEvaluator[[evaluate.QuestionAnsweringEvaluator]]

#### evaluate.QuestionAnsweringEvaluator[[evaluate.QuestionAnsweringEvaluator]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/question_answering.py#L75)

Question answering evaluator. This evaluator handles
[**extractive** question answering](https://huggingface.co/docs/transformers/task_summary#extractive-question-answering),
where the answer to the question is extracted from a context.

This question answering evaluator can currently be loaded from [evaluator()](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.evaluator) using the default task name
`question-answering`.

Methods in this class assume a data format compatible with the
`QuestionAnsweringPipeline`.

computeevaluate.QuestionAnsweringEvaluator.computehttps://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/question_answering.py#L144[{"name": "model_or_pipeline", "val": ": typing.Union[str, ForwardRef('Pipeline'), typing.Callable, ForwardRef('PreTrainedModel'), ForwardRef('TFPreTrainedModel')] = None"}, {"name": "data", "val": ": typing.Union[str, datasets.arrow_dataset.Dataset] = None"}, {"name": "subset", "val": ": typing.Optional[str] = None"}, {"name": "split", "val": ": typing.Optional[str] = None"}, {"name": "metric", "val": ": typing.Union[str, evaluate.module.EvaluationModule] = None"}, {"name": "tokenizer", "val": ": typing.Union[str, ForwardRef('PreTrainedTokenizer'), NoneType] = None"}, {"name": "strategy", "val": ": typing.Literal['simple', 'bootstrap'] = 'simple'"}, {"name": "confidence_level", "val": ": float = 0.95"}, {"name": "n_resamples", "val": ": int = 9999"}, {"name": "device", "val": ": int = None"}, {"name": "random_state", "val": ": typing.Optional[int] = None"}, {"name": "question_column", "val": ": str = 'question'"}, {"name": "context_column", "val": ": str = 'context'"}, {"name": "id_column", "val": ": str = 'id'"}, {"name": "label_column", "val": ": str = 'answers'"}, {"name": "squad_v2_format", "val": ": typing.Optional[bool] = None"}]- **model_or_pipeline** (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) --
  If the argument in not specified, we initialize the default pipeline for the task (in this case
  `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or
  is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the
  argument specifies a pre-initialized pipeline.
- **data** (`str` or `Dataset`, defaults to `None`) --
  Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset
  name, and load it. Otherwise we assume it represents a pre-loaded dataset.
- **subset** (`str`, defaults to `None`) --
  Defines which dataset subset to load. If `None` is passed the default subset is loaded.
- **split** (`str`, defaults to `None`) --
  Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.
- **metric** (`str` or `EvaluationModule`, defaults to `None`) --
  Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and
  load it. Otherwise we assume it represents a pre-loaded metric.
- **tokenizer** (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) --
  Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for
  which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore
  this argument.
- **strategy** (`Literal["simple", "bootstrap"]`, defaults to "simple") --
  specifies the evaluation strategy. Possible values are:
  - `"simple"` - we evaluate the metric and return the scores.
  - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each
  of the returned metric keys, using `scipy`'s `bootstrap` method
  https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.
- **confidence_level** (`float`, defaults to `0.95`) --
  The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **n_resamples** (`int`, defaults to `9999`) --
  The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **device** (`int`, defaults to `None`) --
  Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive
  integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and
  CUDA:0 used if available, CPU otherwise.
- **random_state** (`int`, *optional*, defaults to `None`) --
  The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for
  debugging.

- **question_column** (`str`, defaults to `"question"`) --
  The name of the column containing the question in the dataset specified by `data`.
- **context_column** (`str`, defaults to `"context"`) --
  The name of the column containing the context in the dataset specified by `data`.
- **id_column** (`str`, defaults to `"id"`) --
  The name of the column containing the identification field of the question and answer pair in the
  dataset specified by `data`.
- **label_column** (`str`, defaults to `"answers"`) --
  The name of the column containing the answers in the dataset specified by `data`.
- **squad_v2_format** (`bool`, *optional*, defaults to `None`) --
  Whether the dataset follows the format of squad_v2 dataset. This is the case when the provided dataset
  has questions where the answer is not in the context, more specifically when are answers as
  `{"text": [], "answer_start": []}` in the answer column. If all questions have at least one answer, this parameter
  should be set to `False`. If this parameter is not provided, the format will be automatically inferred.0A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

Compute the metric for a given pipeline and dataset combination.

Examples:
```python
>>> from evaluate import evaluator
>>> from datasets import load_dataset
>>> task_evaluator = evaluator("question-answering")
>>> data = load_dataset("squad", split="validation[:2]")
>>> results = task_evaluator.compute(
>>>     model_or_pipeline="sshleifer/tiny-distilbert-base-cased-distilled-squad",
>>>     data=data,
>>>     metric="squad",
>>> )
```

> [!TIP]
> Datasets where the answer may be missing in the context are supported, for example SQuAD v2 dataset. In this case, it is safer to pass `squad_v2_format=True` to
> the compute() call.

```python
>>> from evaluate import evaluator
>>> from datasets import load_dataset
>>> task_evaluator = evaluator("question-answering")
>>> data = load_dataset("squad_v2", split="validation[:2]")
>>> results = task_evaluator.compute(
>>>     model_or_pipeline="mrm8488/bert-tiny-finetuned-squadv2",
>>>     data=data,
>>>     metric="squad_v2",
>>>     squad_v2_format=True,
>>> )
```

**Parameters:**

model_or_pipeline (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) : If the argument in not specified, we initialize the default pipeline for the task (in this case `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the argument specifies a pre-initialized pipeline.

data (`str` or `Dataset`, defaults to `None`) : Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset name, and load it. Otherwise we assume it represents a pre-loaded dataset.

subset (`str`, defaults to `None`) : Defines which dataset subset to load. If `None` is passed the default subset is loaded.

split (`str`, defaults to `None`) : Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.

metric (`str` or `EvaluationModule`, defaults to `None`) : Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and load it. Otherwise we assume it represents a pre-loaded metric.

tokenizer (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) : Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore this argument.

strategy (`Literal["simple", "bootstrap"]`, defaults to "simple") : specifies the evaluation strategy. Possible values are: - `"simple"` - we evaluate the metric and return the scores. - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each of the returned metric keys, using `scipy`'s `bootstrap` method https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.

confidence_level (`float`, defaults to `0.95`) : The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

n_resamples (`int`, defaults to `9999`) : The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

device (`int`, defaults to `None`) : Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and CUDA:0 used if available, CPU otherwise.

random_state (`int`, *optional*, defaults to `None`) : The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for debugging.   

question_column (`str`, defaults to `"question"`) : The name of the column containing the question in the dataset specified by `data`.

context_column (`str`, defaults to `"context"`) : The name of the column containing the context in the dataset specified by `data`.

id_column (`str`, defaults to `"id"`) : The name of the column containing the identification field of the question and answer pair in the dataset specified by `data`.

label_column (`str`, defaults to `"answers"`) : The name of the column containing the answers in the dataset specified by `data`.

squad_v2_format (`bool`, *optional*, defaults to `None`) : Whether the dataset follows the format of squad_v2 dataset. This is the case when the provided dataset has questions where the answer is not in the context, more specifically when are answers as `{"text": [], "answer_start": []}` in the answer column. If all questions have at least one answer, this parameter should be set to `False`. If this parameter is not provided, the format will be automatically inferred.

**Returns:**

A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

### TextClassificationEvaluator[[evaluate.TextClassificationEvaluator]]

#### evaluate.TextClassificationEvaluator[[evaluate.TextClassificationEvaluator]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/text_classification.py#L51)

Text classification evaluator.
This text classification evaluator can currently be loaded from [evaluator()](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.evaluator) using the default task name
`text-classification` or with a `"sentiment-analysis"` alias.
Methods in this class assume a data format compatible with the [TextClassificationPipeline](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#transformers.TextClassificationPipeline) - a single textual
feature as input and a categorical label as output.

computeevaluate.TextClassificationEvaluator.computehttps://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/text_classification.py#L89[{"name": "model_or_pipeline", "val": ": typing.Union[str, ForwardRef('Pipeline'), typing.Callable, ForwardRef('PreTrainedModel'), ForwardRef('TFPreTrainedModel')] = None"}, {"name": "data", "val": ": typing.Union[str, datasets.arrow_dataset.Dataset] = None"}, {"name": "subset", "val": ": typing.Optional[str] = None"}, {"name": "split", "val": ": typing.Optional[str] = None"}, {"name": "metric", "val": ": typing.Union[str, evaluate.module.EvaluationModule] = None"}, {"name": "tokenizer", "val": ": typing.Union[str, ForwardRef('PreTrainedTokenizer'), NoneType] = None"}, {"name": "feature_extractor", "val": ": typing.Union[str, ForwardRef('FeatureExtractionMixin'), NoneType] = None"}, {"name": "strategy", "val": ": typing.Literal['simple', 'bootstrap'] = 'simple'"}, {"name": "confidence_level", "val": ": float = 0.95"}, {"name": "n_resamples", "val": ": int = 9999"}, {"name": "device", "val": ": int = None"}, {"name": "random_state", "val": ": typing.Optional[int] = None"}, {"name": "input_column", "val": ": str = 'text'"}, {"name": "second_input_column", "val": ": typing.Optional[str] = None"}, {"name": "label_column", "val": ": str = 'label'"}, {"name": "label_mapping", "val": ": typing.Optional[typing.Dict[str, numbers.Number]] = None"}]- **model_or_pipeline** (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) --
  If the argument in not specified, we initialize the default pipeline for the task (in this case
  `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or
  is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the
  argument specifies a pre-initialized pipeline.
- **data** (`str` or `Dataset`, defaults to `None`) --
  Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset
  name, and load it. Otherwise we assume it represents a pre-loaded dataset.
- **subset** (`str`, defaults to `None`) --
  Defines which dataset subset to load. If `None` is passed the default subset is loaded.
- **split** (`str`, defaults to `None`) --
  Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.
- **metric** (`str` or `EvaluationModule`, defaults to `None`) --
  Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and
  load it. Otherwise we assume it represents a pre-loaded metric.
- **tokenizer** (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) --
  Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for
  which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore
  this argument.
- **strategy** (`Literal["simple", "bootstrap"]`, defaults to "simple") --
  specifies the evaluation strategy. Possible values are:
  - `"simple"` - we evaluate the metric and return the scores.
  - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each
  of the returned metric keys, using `scipy`'s `bootstrap` method
  https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.
- **confidence_level** (`float`, defaults to `0.95`) --
  The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **n_resamples** (`int`, defaults to `9999`) --
  The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **device** (`int`, defaults to `None`) --
  Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive
  integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and
  CUDA:0 used if available, CPU otherwise.
- **random_state** (`int`, *optional*, defaults to `None`) --
  The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for
  debugging.

- **input_column** (`str`, *optional*, defaults to `"text"`) --
  The name of the column containing the text feature in the dataset specified by `data`.
- **second_input_column** (`str`, *optional*, defaults to `None`) --
  The name of the second column containing the text features. This may be useful for classification tasks
  as MNLI, where two columns are used.
- **label_column** (`str`, defaults to `"label"`) --
  The name of the column containing the labels in the dataset specified by `data`.
- **label_mapping** (`Dict[str, Number]`, *optional*, defaults to `None`) --
  We want to map class labels defined by the model in the pipeline to values consistent with those
  defined in the `label_column` of the `data` dataset.0A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

Compute the metric for a given pipeline and dataset combination.

Examples:
```python
>>> from evaluate import evaluator
>>> from datasets import load_dataset
>>> task_evaluator = evaluator("text-classification")
>>> data = load_dataset("imdb", split="test[:2]")
>>> results = task_evaluator.compute(
>>>     model_or_pipeline="huggingface/prunebert-base-uncased-6-finepruned-w-distil-mnli",
>>>     data=data,
>>>     metric="accuracy",
>>>     label_mapping={"LABEL_0": 0.0, "LABEL_1": 1.0},
>>>     strategy="bootstrap",
>>>     n_resamples=10,
>>>     random_state=0
>>> )
```

**Parameters:**

model_or_pipeline (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) : If the argument in not specified, we initialize the default pipeline for the task (in this case `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the argument specifies a pre-initialized pipeline.

data (`str` or `Dataset`, defaults to `None`) : Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset name, and load it. Otherwise we assume it represents a pre-loaded dataset.

subset (`str`, defaults to `None`) : Defines which dataset subset to load. If `None` is passed the default subset is loaded.

split (`str`, defaults to `None`) : Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.

metric (`str` or `EvaluationModule`, defaults to `None`) : Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and load it. Otherwise we assume it represents a pre-loaded metric.

tokenizer (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) : Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore this argument.

strategy (`Literal["simple", "bootstrap"]`, defaults to "simple") : specifies the evaluation strategy. Possible values are: - `"simple"` - we evaluate the metric and return the scores. - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each of the returned metric keys, using `scipy`'s `bootstrap` method https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.

confidence_level (`float`, defaults to `0.95`) : The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

n_resamples (`int`, defaults to `9999`) : The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

device (`int`, defaults to `None`) : Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and CUDA:0 used if available, CPU otherwise.

random_state (`int`, *optional*, defaults to `None`) : The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for debugging.   

input_column (`str`, *optional*, defaults to `"text"`) : The name of the column containing the text feature in the dataset specified by `data`.

second_input_column (`str`, *optional*, defaults to `None`) : The name of the second column containing the text features. This may be useful for classification tasks as MNLI, where two columns are used.

label_column (`str`, defaults to `"label"`) : The name of the column containing the labels in the dataset specified by `data`.

label_mapping (`Dict[str, Number]`, *optional*, defaults to `None`) : We want to map class labels defined by the model in the pipeline to values consistent with those defined in the `label_column` of the `data` dataset.

**Returns:**

A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

### TokenClassificationEvaluator[[evaluate.TokenClassificationEvaluator]]

#### evaluate.TokenClassificationEvaluator[[evaluate.TokenClassificationEvaluator]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/token_classification.py#L84)

Token classification evaluator.

This token classification evaluator can currently be loaded from [evaluator()](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.evaluator) using the default task name
`token-classification`.

Methods in this class assume a data format compatible with the [TokenClassificationPipeline](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#transformers.TokenClassificationPipeline).

computeevaluate.TokenClassificationEvaluator.computehttps://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/token_classification.py#L212[{"name": "model_or_pipeline", "val": ": typing.Union[str, ForwardRef('Pipeline'), typing.Callable, ForwardRef('PreTrainedModel'), ForwardRef('TFPreTrainedModel')] = None"}, {"name": "data", "val": ": typing.Union[str, datasets.arrow_dataset.Dataset] = None"}, {"name": "subset", "val": ": typing.Optional[str] = None"}, {"name": "split", "val": ": str = None"}, {"name": "metric", "val": ": typing.Union[str, evaluate.module.EvaluationModule] = None"}, {"name": "tokenizer", "val": ": typing.Union[str, ForwardRef('PreTrainedTokenizer'), NoneType] = None"}, {"name": "strategy", "val": ": typing.Literal['simple', 'bootstrap'] = 'simple'"}, {"name": "confidence_level", "val": ": float = 0.95"}, {"name": "n_resamples", "val": ": int = 9999"}, {"name": "device", "val": ": typing.Optional[int] = None"}, {"name": "random_state", "val": ": typing.Optional[int] = None"}, {"name": "input_column", "val": ": str = 'tokens'"}, {"name": "label_column", "val": ": str = 'ner_tags'"}, {"name": "join_by", "val": ": typing.Optional[str] = ' '"}]- **model_or_pipeline** (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) --
  If the argument in not specified, we initialize the default pipeline for the task (in this case
  `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or
  is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the
  argument specifies a pre-initialized pipeline.
- **data** (`str` or `Dataset`, defaults to `None`) --
  Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset
  name, and load it. Otherwise we assume it represents a pre-loaded dataset.
- **subset** (`str`, defaults to `None`) --
  Defines which dataset subset to load. If `None` is passed the default subset is loaded.
- **split** (`str`, defaults to `None`) --
  Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.
- **metric** (`str` or `EvaluationModule`, defaults to `None`) --
  Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and
  load it. Otherwise we assume it represents a pre-loaded metric.
- **tokenizer** (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) --
  Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for
  which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore
  this argument.
- **strategy** (`Literal["simple", "bootstrap"]`, defaults to "simple") --
  specifies the evaluation strategy. Possible values are:
  - `"simple"` - we evaluate the metric and return the scores.
  - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each
  of the returned metric keys, using `scipy`'s `bootstrap` method
  https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.
- **confidence_level** (`float`, defaults to `0.95`) --
  The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **n_resamples** (`int`, defaults to `9999`) --
  The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **device** (`int`, defaults to `None`) --
  Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive
  integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and
  CUDA:0 used if available, CPU otherwise.
- **random_state** (`int`, *optional*, defaults to `None`) --
  The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for
  debugging.

- **input_column** (`str`, defaults to `"tokens"`) --
  The name of the column containing the tokens feature in the dataset specified by `data`.
- **label_column** (`str`, defaults to `"label"`) --
  The name of the column containing the labels in the dataset specified by `data`.
- **join_by** (`str`, *optional*, defaults to `" "`) --
  This evaluator supports dataset whose input column is a list of words. This parameter specifies how to join
  words to generate a string input. This is especially useful for languages that do not separate words by a space.0A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

Compute the metric for a given pipeline and dataset combination.

The dataset input and label columns are expected to be formatted as a list of words and a list of labels respectively, following [conll2003 dataset](https://huggingface.co/datasets/conll2003). Datasets whose inputs are single strings, and labels are a list of offset are not supported.

Examples:
```python
>>> from evaluate import evaluator
>>> from datasets import load_dataset
>>> task_evaluator = evaluator("token-classification")
>>> data = load_dataset("conll2003", split="validation[:2]")
>>> results = task_evaluator.compute(
>>>     model_or_pipeline="elastic/distilbert-base-uncased-finetuned-conll03-english",
>>>     data=data,
>>>     metric="seqeval",
>>> )
```

> [!TIP]
> For example, the following dataset format is accepted by the evaluator:
>
> ```python
> dataset = Dataset.from_dict(
>     mapping={
>         "tokens": [["New", "York", "is", "a", "city", "and", "Felix", "a", "person", "."]],
>         "ner_tags": [[1, 2, 0, 0, 0, 0, 3, 0, 0, 0]],
>     },
>     features=Features({
>         "tokens": Sequence(feature=Value(dtype="string")),
>         "ner_tags": Sequence(feature=ClassLabel(names=["O", "B-LOC", "I-LOC", "B-PER", "I-PER"])),
>         }),
> )
> ```

> [!WARNING]
> For example, the following dataset format is **not** accepted by the evaluator:
>
> ```python
> dataset = Dataset.from_dict(
>     mapping={
>         "tokens": [["New York is a city and Felix a person."]],
>         "starts": [[0, 23]],
>         "ends": [[7, 27]],
>         "ner_tags": [["LOC", "PER"]],
>     },
>     features=Features({
>         "tokens": Value(dtype="string"),
>         "starts": Sequence(feature=Value(dtype="int32")),
>         "ends": Sequence(feature=Value(dtype="int32")),
>         "ner_tags": Sequence(feature=Value(dtype="string")),
>     }),
> )
> ```

**Parameters:**

model_or_pipeline (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) : If the argument in not specified, we initialize the default pipeline for the task (in this case `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the argument specifies a pre-initialized pipeline.

data (`str` or `Dataset`, defaults to `None`) : Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset name, and load it. Otherwise we assume it represents a pre-loaded dataset.

subset (`str`, defaults to `None`) : Defines which dataset subset to load. If `None` is passed the default subset is loaded.

split (`str`, defaults to `None`) : Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.

metric (`str` or `EvaluationModule`, defaults to `None`) : Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and load it. Otherwise we assume it represents a pre-loaded metric.

tokenizer (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) : Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore this argument.

strategy (`Literal["simple", "bootstrap"]`, defaults to "simple") : specifies the evaluation strategy. Possible values are: - `"simple"` - we evaluate the metric and return the scores. - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each of the returned metric keys, using `scipy`'s `bootstrap` method https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.

confidence_level (`float`, defaults to `0.95`) : The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

n_resamples (`int`, defaults to `9999`) : The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

device (`int`, defaults to `None`) : Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and CUDA:0 used if available, CPU otherwise.

random_state (`int`, *optional*, defaults to `None`) : The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for debugging.   

input_column (`str`, defaults to `"tokens"`) : The name of the column containing the tokens feature in the dataset specified by `data`.

label_column (`str`, defaults to `"label"`) : The name of the column containing the labels in the dataset specified by `data`.

join_by (`str`, *optional*, defaults to `" "`) : This evaluator supports dataset whose input column is a list of words. This parameter specifies how to join words to generate a string input. This is especially useful for languages that do not separate words by a space.

**Returns:**

A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

### TextGenerationEvaluator[[evaluate.TextGenerationEvaluator]]

#### evaluate.TextGenerationEvaluator[[evaluate.TextGenerationEvaluator]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/text_generation.py#L31)

Text generation evaluator.
This Text generation evaluator can currently be loaded from [evaluator()](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.evaluator) using the default task name
`text-generation`.
Methods in this class assume a data format compatible with the [TextGenerationPipeline](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#transformers.TextGenerationPipeline).

computeevaluate.TextGenerationEvaluator.computehttps://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/base.py#L218[{"name": "model_or_pipeline", "val": ": typing.Union[str, ForwardRef('Pipeline'), typing.Callable, ForwardRef('PreTrainedModel'), ForwardRef('TFPreTrainedModel')] = None"}, {"name": "data", "val": ": typing.Union[str, datasets.arrow_dataset.Dataset] = None"}, {"name": "subset", "val": ": typing.Optional[str] = None"}, {"name": "split", "val": ": typing.Optional[str] = None"}, {"name": "metric", "val": ": typing.Union[str, evaluate.module.EvaluationModule] = None"}, {"name": "tokenizer", "val": ": typing.Union[str, ForwardRef('PreTrainedTokenizer'), NoneType] = None"}, {"name": "feature_extractor", "val": ": typing.Union[str, ForwardRef('FeatureExtractionMixin'), NoneType] = None"}, {"name": "strategy", "val": ": typing.Literal['simple', 'bootstrap'] = 'simple'"}, {"name": "confidence_level", "val": ": float = 0.95"}, {"name": "n_resamples", "val": ": int = 9999"}, {"name": "device", "val": ": int = None"}, {"name": "random_state", "val": ": typing.Optional[int] = None"}, {"name": "input_column", "val": ": str = 'text'"}, {"name": "label_column", "val": ": str = 'label'"}, {"name": "label_mapping", "val": ": typing.Optional[typing.Dict[str, numbers.Number]] = None"}]

### Text2TextGenerationEvaluator[[evaluate.Text2TextGenerationEvaluator]]

#### evaluate.Text2TextGenerationEvaluator[[evaluate.Text2TextGenerationEvaluator]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/text2text_generation.py#L88)

Text2Text generation evaluator.
This Text2Text generation evaluator can currently be loaded from [evaluator()](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.evaluator) using the default task name
`text2text-generation`.
Methods in this class assume a data format compatible with the `Text2TextGenerationPipeline`.

computeevaluate.Text2TextGenerationEvaluator.computehttps://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/text2text_generation.py#L105[{"name": "model_or_pipeline", "val": ": typing.Union[str, ForwardRef('Pipeline'), typing.Callable, ForwardRef('PreTrainedModel'), ForwardRef('TFPreTrainedModel')] = None"}, {"name": "data", "val": ": typing.Union[str, datasets.arrow_dataset.Dataset] = None"}, {"name": "subset", "val": ": typing.Optional[str] = None"}, {"name": "split", "val": ": typing.Optional[str] = None"}, {"name": "metric", "val": ": typing.Union[str, evaluate.module.EvaluationModule] = None"}, {"name": "tokenizer", "val": ": typing.Union[str, ForwardRef('PreTrainedTokenizer'), NoneType] = None"}, {"name": "strategy", "val": ": typing.Literal['simple', 'bootstrap'] = 'simple'"}, {"name": "confidence_level", "val": ": float = 0.95"}, {"name": "n_resamples", "val": ": int = 9999"}, {"name": "device", "val": ": int = None"}, {"name": "random_state", "val": ": typing.Optional[int] = None"}, {"name": "input_column", "val": ": str = 'text'"}, {"name": "label_column", "val": ": str = 'label'"}, {"name": "generation_kwargs", "val": ": dict = None"}]- **model_or_pipeline** (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) --
  If the argument in not specified, we initialize the default pipeline for the task (in this case
  `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or
  is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the
  argument specifies a pre-initialized pipeline.
- **data** (`str` or `Dataset`, defaults to `None`) --
  Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset
  name, and load it. Otherwise we assume it represents a pre-loaded dataset.
- **subset** (`str`, defaults to `None`) --
  Defines which dataset subset to load. If `None` is passed the default subset is loaded.
- **split** (`str`, defaults to `None`) --
  Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.
- **metric** (`str` or `EvaluationModule`, defaults to `None`) --
  Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and
  load it. Otherwise we assume it represents a pre-loaded metric.
- **tokenizer** (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) --
  Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for
  which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore
  this argument.
- **strategy** (`Literal["simple", "bootstrap"]`, defaults to "simple") --
  specifies the evaluation strategy. Possible values are:
  - `"simple"` - we evaluate the metric and return the scores.
  - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each
  of the returned metric keys, using `scipy`'s `bootstrap` method
  https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.
- **confidence_level** (`float`, defaults to `0.95`) --
  The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **n_resamples** (`int`, defaults to `9999`) --
  The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **device** (`int`, defaults to `None`) --
  Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive
  integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and
  CUDA:0 used if available, CPU otherwise.
- **random_state** (`int`, *optional*, defaults to `None`) --
  The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for
  debugging.

- **input_column** (`str`, defaults to `"text"`) --
  the name of the column containing the input text in the dataset specified by `data`.
- **label_column** (`str`, defaults to `"label"`) --
  the name of the column containing the labels in the dataset specified by `data`.
- **generation_kwargs** (`Dict`, *optional*, defaults to `None`) --
  The generation kwargs are passed to the pipeline and set the text generation strategy.0A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

Compute the metric for a given pipeline and dataset combination.

Examples:
```python
>>> from evaluate import evaluator
>>> from datasets import load_dataset
>>> task_evaluator = evaluator("text2text-generation")
>>> data = load_dataset("cnn_dailymail", "3.0.0", split="validation[:40]")
>>> results = task_evaluator.compute(
>>>     model_or_pipeline="facebook/bart-large-cnn",
>>>     data=data,
>>>     input_column="article",
>>>     label_column="highlights",
>>>     metric="rouge",
>>> )
```

**Parameters:**

model_or_pipeline (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) : If the argument in not specified, we initialize the default pipeline for the task (in this case `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the argument specifies a pre-initialized pipeline.

data (`str` or `Dataset`, defaults to `None`) : Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset name, and load it. Otherwise we assume it represents a pre-loaded dataset.

subset (`str`, defaults to `None`) : Defines which dataset subset to load. If `None` is passed the default subset is loaded.

split (`str`, defaults to `None`) : Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.

metric (`str` or `EvaluationModule`, defaults to `None`) : Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and load it. Otherwise we assume it represents a pre-loaded metric.

tokenizer (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) : Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore this argument.

strategy (`Literal["simple", "bootstrap"]`, defaults to "simple") : specifies the evaluation strategy. Possible values are: - `"simple"` - we evaluate the metric and return the scores. - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each of the returned metric keys, using `scipy`'s `bootstrap` method https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.

confidence_level (`float`, defaults to `0.95`) : The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

n_resamples (`int`, defaults to `9999`) : The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

device (`int`, defaults to `None`) : Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and CUDA:0 used if available, CPU otherwise.

random_state (`int`, *optional*, defaults to `None`) : The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for debugging. 

input_column (`str`, defaults to `"text"`) : the name of the column containing the input text in the dataset specified by `data`.

label_column (`str`, defaults to `"label"`) : the name of the column containing the labels in the dataset specified by `data`.

generation_kwargs (`Dict`, *optional*, defaults to `None`) : The generation kwargs are passed to the pipeline and set the text generation strategy.

**Returns:**

A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

### SummarizationEvaluator[[evaluate.SummarizationEvaluator]]

#### evaluate.SummarizationEvaluator[[evaluate.SummarizationEvaluator]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/text2text_generation.py#L152)

Text summarization evaluator.
This text summarization evaluator can currently be loaded from [evaluator()](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.evaluator) using the default task name
`summarization`.
Methods in this class assume a data format compatible with the [SummarizationEvaluator](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.SummarizationEvaluator).

computeevaluate.SummarizationEvaluator.computehttps://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/text2text_generation.py#L166[{"name": "model_or_pipeline", "val": ": typing.Union[str, ForwardRef('Pipeline'), typing.Callable, ForwardRef('PreTrainedModel'), ForwardRef('TFPreTrainedModel')] = None"}, {"name": "data", "val": ": typing.Union[str, datasets.arrow_dataset.Dataset] = None"}, {"name": "subset", "val": ": typing.Optional[str] = None"}, {"name": "split", "val": ": typing.Optional[str] = None"}, {"name": "metric", "val": ": typing.Union[str, evaluate.module.EvaluationModule] = None"}, {"name": "tokenizer", "val": ": typing.Union[str, ForwardRef('PreTrainedTokenizer'), NoneType] = None"}, {"name": "strategy", "val": ": typing.Literal['simple', 'bootstrap'] = 'simple'"}, {"name": "confidence_level", "val": ": float = 0.95"}, {"name": "n_resamples", "val": ": int = 9999"}, {"name": "device", "val": ": int = None"}, {"name": "random_state", "val": ": typing.Optional[int] = None"}, {"name": "input_column", "val": ": str = 'text'"}, {"name": "label_column", "val": ": str = 'label'"}, {"name": "generation_kwargs", "val": ": dict = None"}]- **model_or_pipeline** (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) --
  If the argument in not specified, we initialize the default pipeline for the task (in this case
  `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or
  is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the
  argument specifies a pre-initialized pipeline.
- **data** (`str` or `Dataset`, defaults to `None`) --
  Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset
  name, and load it. Otherwise we assume it represents a pre-loaded dataset.
- **subset** (`str`, defaults to `None`) --
  Defines which dataset subset to load. If `None` is passed the default subset is loaded.
- **split** (`str`, defaults to `None`) --
  Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.
- **metric** (`str` or `EvaluationModule`, defaults to `None`) --
  Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and
  load it. Otherwise we assume it represents a pre-loaded metric.
- **tokenizer** (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) --
  Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for
  which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore
  this argument.
- **strategy** (`Literal["simple", "bootstrap"]`, defaults to "simple") --
  specifies the evaluation strategy. Possible values are:
  - `"simple"` - we evaluate the metric and return the scores.
  - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each
  of the returned metric keys, using `scipy`'s `bootstrap` method
  https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.
- **confidence_level** (`float`, defaults to `0.95`) --
  The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **n_resamples** (`int`, defaults to `9999`) --
  The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **device** (`int`, defaults to `None`) --
  Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive
  integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and
  CUDA:0 used if available, CPU otherwise.
- **random_state** (`int`, *optional*, defaults to `None`) --
  The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for
  debugging.

- **input_column** (`str`, defaults to `"text"`) --
  the name of the column containing the input text in the dataset specified by `data`.
- **label_column** (`str`, defaults to `"label"`) --
  the name of the column containing the labels in the dataset specified by `data`.
- **generation_kwargs** (`Dict`, *optional*, defaults to `None`) --
  The generation kwargs are passed to the pipeline and set the text generation strategy.0A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

Compute the metric for a given pipeline and dataset combination.

Examples:
```python
>>> from evaluate import evaluator
>>> from datasets import load_dataset
>>> task_evaluator = evaluator("summarization")
>>> data = load_dataset("cnn_dailymail", "3.0.0", split="validation[:40]")
>>> results = task_evaluator.compute(
>>>     model_or_pipeline="facebook/bart-large-cnn",
>>>     data=data,
>>>     input_column="article",
>>>     label_column="highlights",
>>> )
```

**Parameters:**

model_or_pipeline (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) : If the argument in not specified, we initialize the default pipeline for the task (in this case `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the argument specifies a pre-initialized pipeline.

data (`str` or `Dataset`, defaults to `None`) : Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset name, and load it. Otherwise we assume it represents a pre-loaded dataset.

subset (`str`, defaults to `None`) : Defines which dataset subset to load. If `None` is passed the default subset is loaded.

split (`str`, defaults to `None`) : Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.

metric (`str` or `EvaluationModule`, defaults to `None`) : Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and load it. Otherwise we assume it represents a pre-loaded metric.

tokenizer (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) : Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore this argument.

strategy (`Literal["simple", "bootstrap"]`, defaults to "simple") : specifies the evaluation strategy. Possible values are: - `"simple"` - we evaluate the metric and return the scores. - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each of the returned metric keys, using `scipy`'s `bootstrap` method https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.

confidence_level (`float`, defaults to `0.95`) : The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

n_resamples (`int`, defaults to `9999`) : The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

device (`int`, defaults to `None`) : Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and CUDA:0 used if available, CPU otherwise.

random_state (`int`, *optional*, defaults to `None`) : The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for debugging. 

input_column (`str`, defaults to `"text"`) : the name of the column containing the input text in the dataset specified by `data`.

label_column (`str`, defaults to `"label"`) : the name of the column containing the labels in the dataset specified by `data`.

generation_kwargs (`Dict`, *optional*, defaults to `None`) : The generation kwargs are passed to the pipeline and set the text generation strategy.

**Returns:**

A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

### TranslationEvaluator[[evaluate.TranslationEvaluator]]

#### evaluate.TranslationEvaluator[[evaluate.TranslationEvaluator]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/text2text_generation.py#L211)

Translation evaluator.
This translation generation evaluator can currently be loaded from [evaluator()](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.evaluator) using the default task name
`translation`.
Methods in this class assume a data format compatible with the `TranslationPipeline`.

computeevaluate.TranslationEvaluator.computehttps://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/text2text_generation.py#L225[{"name": "model_or_pipeline", "val": ": typing.Union[str, ForwardRef('Pipeline'), typing.Callable, ForwardRef('PreTrainedModel'), ForwardRef('TFPreTrainedModel')] = None"}, {"name": "data", "val": ": typing.Union[str, datasets.arrow_dataset.Dataset] = None"}, {"name": "subset", "val": ": typing.Optional[str] = None"}, {"name": "split", "val": ": typing.Optional[str] = None"}, {"name": "metric", "val": ": typing.Union[str, evaluate.module.EvaluationModule] = None"}, {"name": "tokenizer", "val": ": typing.Union[str, ForwardRef('PreTrainedTokenizer'), NoneType] = None"}, {"name": "strategy", "val": ": typing.Literal['simple', 'bootstrap'] = 'simple'"}, {"name": "confidence_level", "val": ": float = 0.95"}, {"name": "n_resamples", "val": ": int = 9999"}, {"name": "device", "val": ": int = None"}, {"name": "random_state", "val": ": typing.Optional[int] = None"}, {"name": "input_column", "val": ": str = 'text'"}, {"name": "label_column", "val": ": str = 'label'"}, {"name": "generation_kwargs", "val": ": dict = None"}]- **model_or_pipeline** (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) --
  If the argument in not specified, we initialize the default pipeline for the task (in this case
  `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or
  is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the
  argument specifies a pre-initialized pipeline.
- **data** (`str` or `Dataset`, defaults to `None`) --
  Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset
  name, and load it. Otherwise we assume it represents a pre-loaded dataset.
- **subset** (`str`, defaults to `None`) --
  Defines which dataset subset to load. If `None` is passed the default subset is loaded.
- **split** (`str`, defaults to `None`) --
  Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.
- **metric** (`str` or `EvaluationModule`, defaults to `None`) --
  Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and
  load it. Otherwise we assume it represents a pre-loaded metric.
- **tokenizer** (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) --
  Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for
  which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore
  this argument.
- **strategy** (`Literal["simple", "bootstrap"]`, defaults to "simple") --
  specifies the evaluation strategy. Possible values are:
  - `"simple"` - we evaluate the metric and return the scores.
  - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each
  of the returned metric keys, using `scipy`'s `bootstrap` method
  https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.
- **confidence_level** (`float`, defaults to `0.95`) --
  The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **n_resamples** (`int`, defaults to `9999`) --
  The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **device** (`int`, defaults to `None`) --
  Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive
  integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and
  CUDA:0 used if available, CPU otherwise.
- **random_state** (`int`, *optional*, defaults to `None`) --
  The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for
  debugging.

- **input_column** (`str`, defaults to `"text"`) --
  the name of the column containing the input text in the dataset specified by `data`.
- **label_column** (`str`, defaults to `"label"`) --
  the name of the column containing the labels in the dataset specified by `data`.
- **generation_kwargs** (`Dict`, *optional*, defaults to `None`) --
  The generation kwargs are passed to the pipeline and set the text generation strategy.0A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

Compute the metric for a given pipeline and dataset combination.

Examples:
```python
>>> from evaluate import evaluator
>>> from datasets import load_dataset
>>> task_evaluator = evaluator("translation")
>>> data = load_dataset("wmt19", "fr-de", split="validation[:40]")
>>> data = data.map(lambda x: {"text": x["translation"]["de"], "label": x["translation"]["fr"]})
>>> results = task_evaluator.compute(
>>>     model_or_pipeline="Helsinki-NLP/opus-mt-de-fr",
>>>     data=data,
>>> )
```

**Parameters:**

model_or_pipeline (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) : If the argument in not specified, we initialize the default pipeline for the task (in this case `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the argument specifies a pre-initialized pipeline.

data (`str` or `Dataset`, defaults to `None`) : Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset name, and load it. Otherwise we assume it represents a pre-loaded dataset.

subset (`str`, defaults to `None`) : Defines which dataset subset to load. If `None` is passed the default subset is loaded.

split (`str`, defaults to `None`) : Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.

metric (`str` or `EvaluationModule`, defaults to `None`) : Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and load it. Otherwise we assume it represents a pre-loaded metric.

tokenizer (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) : Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore this argument.

strategy (`Literal["simple", "bootstrap"]`, defaults to "simple") : specifies the evaluation strategy. Possible values are: - `"simple"` - we evaluate the metric and return the scores. - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each of the returned metric keys, using `scipy`'s `bootstrap` method https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.

confidence_level (`float`, defaults to `0.95`) : The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

n_resamples (`int`, defaults to `9999`) : The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

device (`int`, defaults to `None`) : Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and CUDA:0 used if available, CPU otherwise.

random_state (`int`, *optional*, defaults to `None`) : The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for debugging. 

input_column (`str`, defaults to `"text"`) : the name of the column containing the input text in the dataset specified by `data`.

label_column (`str`, defaults to `"label"`) : the name of the column containing the labels in the dataset specified by `data`.

generation_kwargs (`Dict`, *optional*, defaults to `None`) : The generation kwargs are passed to the pipeline and set the text generation strategy.

**Returns:**

A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

### AutomaticSpeechRecognitionEvaluator[[evaluate.AutomaticSpeechRecognitionEvaluator]]

#### evaluate.AutomaticSpeechRecognitionEvaluator[[evaluate.AutomaticSpeechRecognitionEvaluator]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/automatic_speech_recognition.py#L47)

Automatic speech recognition evaluator.
This automatic speech recognition evaluator can currently be loaded from [evaluator()](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.evaluator) using the default task name
`automatic-speech-recognition`.
Methods in this class assume a data format compatible with the `AutomaticSpeechRecognitionPipeline`.

computeevaluate.AutomaticSpeechRecognitionEvaluator.computehttps://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/automatic_speech_recognition.py#L63[{"name": "model_or_pipeline", "val": ": typing.Union[str, ForwardRef('Pipeline'), typing.Callable, ForwardRef('PreTrainedModel'), ForwardRef('TFPreTrainedModel')] = None"}, {"name": "data", "val": ": typing.Union[str, datasets.arrow_dataset.Dataset] = None"}, {"name": "subset", "val": ": typing.Optional[str] = None"}, {"name": "split", "val": ": typing.Optional[str] = None"}, {"name": "metric", "val": ": typing.Union[str, evaluate.module.EvaluationModule] = None"}, {"name": "tokenizer", "val": ": typing.Union[str, ForwardRef('PreTrainedTokenizer'), NoneType] = None"}, {"name": "strategy", "val": ": typing.Literal['simple', 'bootstrap'] = 'simple'"}, {"name": "confidence_level", "val": ": float = 0.95"}, {"name": "n_resamples", "val": ": int = 9999"}, {"name": "device", "val": ": int = None"}, {"name": "random_state", "val": ": typing.Optional[int] = None"}, {"name": "input_column", "val": ": str = 'path'"}, {"name": "label_column", "val": ": str = 'sentence'"}, {"name": "generation_kwargs", "val": ": dict = None"}]- **model_or_pipeline** (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) --
  If the argument in not specified, we initialize the default pipeline for the task (in this case
  `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or
  is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the
  argument specifies a pre-initialized pipeline.
- **data** (`str` or `Dataset`, defaults to `None`) --
  Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset
  name, and load it. Otherwise we assume it represents a pre-loaded dataset.
- **subset** (`str`, defaults to `None`) --
  Defines which dataset subset to load. If `None` is passed the default subset is loaded.
- **split** (`str`, defaults to `None`) --
  Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.
- **metric** (`str` or `EvaluationModule`, defaults to `None`) --
  Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and
  load it. Otherwise we assume it represents a pre-loaded metric.
- **tokenizer** (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) --
  Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for
  which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore
  this argument.
- **strategy** (`Literal["simple", "bootstrap"]`, defaults to "simple") --
  specifies the evaluation strategy. Possible values are:
  - `"simple"` - we evaluate the metric and return the scores.
  - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each
  of the returned metric keys, using `scipy`'s `bootstrap` method
  https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.
- **confidence_level** (`float`, defaults to `0.95`) --
  The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **n_resamples** (`int`, defaults to `9999`) --
  The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **device** (`int`, defaults to `None`) --
  Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive
  integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and
  CUDA:0 used if available, CPU otherwise.
- **random_state** (`int`, *optional*, defaults to `None`) --
  The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for
  debugging.

- **input_column** (`str`, defaults to `"path"`) --
  the name of the column containing the input audio path in the dataset specified by `data`.
- **label_column** (`str`, defaults to `"sentence"`) --
  the name of the column containing the labels in the dataset specified by `data`.
- **generation_kwargs** (`Dict`, *optional*, defaults to `None`) --
  The generation kwargs are passed to the pipeline and set the text generation strategy.0A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

Compute the metric for a given pipeline and dataset combination.

Examples:
```python
>>> from evaluate import evaluator
>>> from datasets import load_dataset
>>> task_evaluator = evaluator("automatic-speech-recognition")
>>> data = load_dataset("mozilla-foundation/common_voice_11_0", "en", split="validation[:40]")
>>> results = task_evaluator.compute(
>>>     model_or_pipeline="https://huggingface.co/openai/whisper-tiny.en",
>>>     data=data,
>>>     input_column="path",
>>>     label_column="sentence",
>>>     metric="wer",
>>> )
```

**Parameters:**

model_or_pipeline (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) : If the argument in not specified, we initialize the default pipeline for the task (in this case `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the argument specifies a pre-initialized pipeline.

data (`str` or `Dataset`, defaults to `None`) : Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset name, and load it. Otherwise we assume it represents a pre-loaded dataset.

subset (`str`, defaults to `None`) : Defines which dataset subset to load. If `None` is passed the default subset is loaded.

split (`str`, defaults to `None`) : Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.

metric (`str` or `EvaluationModule`, defaults to `None`) : Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and load it. Otherwise we assume it represents a pre-loaded metric.

tokenizer (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) : Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore this argument.

strategy (`Literal["simple", "bootstrap"]`, defaults to "simple") : specifies the evaluation strategy. Possible values are: - `"simple"` - we evaluate the metric and return the scores. - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each of the returned metric keys, using `scipy`'s `bootstrap` method https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.

confidence_level (`float`, defaults to `0.95`) : The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

n_resamples (`int`, defaults to `9999`) : The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

device (`int`, defaults to `None`) : Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and CUDA:0 used if available, CPU otherwise.

random_state (`int`, *optional*, defaults to `None`) : The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for debugging.   

input_column (`str`, defaults to `"path"`) : the name of the column containing the input audio path in the dataset specified by `data`.

label_column (`str`, defaults to `"sentence"`) : the name of the column containing the labels in the dataset specified by `data`.

generation_kwargs (`Dict`, *optional*, defaults to `None`) : The generation kwargs are passed to the pipeline and set the text generation strategy.

**Returns:**

A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

### AudioClassificationEvaluator[[evaluate.AudioClassificationEvaluator]]

#### evaluate.AudioClassificationEvaluator[[evaluate.AudioClassificationEvaluator]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/audio_classification.py#L75)

Audio classification evaluator.
This audio classification evaluator can currently be loaded from [evaluator()](/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.evaluator) using the default task name
`audio-classification`.
Methods in this class assume a data format compatible with the [transformers.AudioClassificationPipeline](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#transformers.AudioClassificationPipeline).

computeevaluate.AudioClassificationEvaluator.computehttps://github.com/huggingface/evaluate/blob/main/src/evaluate/evaluator/audio_classification.py#L94[{"name": "model_or_pipeline", "val": ": typing.Union[str, ForwardRef('Pipeline'), typing.Callable, ForwardRef('PreTrainedModel'), ForwardRef('TFPreTrainedModel')] = None"}, {"name": "data", "val": ": typing.Union[str, datasets.arrow_dataset.Dataset] = None"}, {"name": "subset", "val": ": typing.Optional[str] = None"}, {"name": "split", "val": ": typing.Optional[str] = None"}, {"name": "metric", "val": ": typing.Union[str, evaluate.module.EvaluationModule] = None"}, {"name": "tokenizer", "val": ": typing.Union[str, ForwardRef('PreTrainedTokenizer'), NoneType] = None"}, {"name": "feature_extractor", "val": ": typing.Union[str, ForwardRef('FeatureExtractionMixin'), NoneType] = None"}, {"name": "strategy", "val": ": typing.Literal['simple', 'bootstrap'] = 'simple'"}, {"name": "confidence_level", "val": ": float = 0.95"}, {"name": "n_resamples", "val": ": int = 9999"}, {"name": "device", "val": ": int = None"}, {"name": "random_state", "val": ": typing.Optional[int] = None"}, {"name": "input_column", "val": ": str = 'file'"}, {"name": "label_column", "val": ": str = 'label'"}, {"name": "label_mapping", "val": ": typing.Optional[typing.Dict[str, numbers.Number]] = None"}]- **model_or_pipeline** (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) --
  If the argument in not specified, we initialize the default pipeline for the task (in this case
  `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or
  is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the
  argument specifies a pre-initialized pipeline.
- **data** (`str` or `Dataset`, defaults to `None`) --
  Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset
  name, and load it. Otherwise we assume it represents a pre-loaded dataset.
- **subset** (`str`, defaults to `None`) --
  Defines which dataset subset to load. If `None` is passed the default subset is loaded.
- **split** (`str`, defaults to `None`) --
  Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.
- **metric** (`str` or `EvaluationModule`, defaults to `None`) --
  Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and
  load it. Otherwise we assume it represents a pre-loaded metric.
- **tokenizer** (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) --
  Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for
  which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore
  this argument.
- **strategy** (`Literal["simple", "bootstrap"]`, defaults to "simple") --
  specifies the evaluation strategy. Possible values are:
  - `"simple"` - we evaluate the metric and return the scores.
  - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each
  of the returned metric keys, using `scipy`'s `bootstrap` method
  https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.
- **confidence_level** (`float`, defaults to `0.95`) --
  The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **n_resamples** (`int`, defaults to `9999`) --
  The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.
- **device** (`int`, defaults to `None`) --
  Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive
  integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and
  CUDA:0 used if available, CPU otherwise.
- **random_state** (`int`, *optional*, defaults to `None`) --
  The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for
  debugging.

- **input_column** (`str`, defaults to `"file"`) --
  The name of the column containing either the audio files or a raw waveform, represented as a numpy array, in the dataset specified by `data`.
- **label_column** (`str`, defaults to `"label"`) --
  The name of the column containing the labels in the dataset specified by `data`.
- **label_mapping** (`Dict[str, Number]`, *optional*, defaults to `None`) --
  We want to map class labels defined by the model in the pipeline to values consistent with those
  defined in the `label_column` of the `data` dataset.0A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

Compute the metric for a given pipeline and dataset combination.

Examples:

> [!TIP]
> Remember that, in order to process audio files, you need ffmpeg installed (https://ffmpeg.org/download.html)

```python
>>> from evaluate import evaluator
>>> from datasets import load_dataset

>>> task_evaluator = evaluator("audio-classification")
>>> data = load_dataset("superb", 'ks', split="test[:40]")
>>> results = task_evaluator.compute(
>>>     model_or_pipeline=""superb/wav2vec2-base-superb-ks"",
>>>     data=data,
>>>     label_column="label",
>>>     input_column="file",
>>>     metric="accuracy",
>>>     label_mapping={0: "yes", 1: "no", 2: "up", 3: "down"}
>>> )
```

> [!TIP]
> The evaluator supports raw audio data as well, in the form of a numpy array. However, be aware that calling
> the audio column automatically decodes and resamples the audio files, which can be slow for large datasets.

```python
>>> from evaluate import evaluator
>>> from datasets import load_dataset

>>> task_evaluator = evaluator("audio-classification")
>>> data = load_dataset("superb", 'ks', split="test[:40]")
>>> data = data.map(lambda example: {"audio": example["audio"]["array"]})
>>> results = task_evaluator.compute(
>>>     model_or_pipeline=""superb/wav2vec2-base-superb-ks"",
>>>     data=data,
>>>     label_column="label",
>>>     input_column="audio",
>>>     metric="accuracy",
>>>     label_mapping={0: "yes", 1: "no", 2: "up", 3: "down"}
>>> )
```

**Parameters:**

model_or_pipeline (`str` or `Pipeline` or `Callable` or `PreTrainedModel` or `TFPreTrainedModel`, defaults to `None`) : If the argument in not specified, we initialize the default pipeline for the task (in this case `text-classification` or its alias - `sentiment-analysis`). If the argument is of the type `str` or is a model instance, we use it to initialize a new `Pipeline` with the given model. Otherwise we assume the argument specifies a pre-initialized pipeline.

data (`str` or `Dataset`, defaults to `None`) : Specifies the dataset we will run evaluation on. If it is of type `str`, we treat it as the dataset name, and load it. Otherwise we assume it represents a pre-loaded dataset.

subset (`str`, defaults to `None`) : Defines which dataset subset to load. If `None` is passed the default subset is loaded.

split (`str`, defaults to `None`) : Defines which dataset split to load. If `None` is passed, infers based on the `choose_split` function.

metric (`str` or `EvaluationModule`, defaults to `None`) : Specifies the metric we use in evaluator. If it is of type `str`, we treat it as the metric name, and load it. Otherwise we assume it represents a pre-loaded metric.

tokenizer (`str` or `PreTrainedTokenizer`, *optional*, defaults to `None`) : Argument can be used to overwrite a default tokenizer if `model_or_pipeline` represents a model for which we build a pipeline. If `model_or_pipeline` is `None` or a pre-initialized pipeline, we ignore this argument.

strategy (`Literal["simple", "bootstrap"]`, defaults to "simple") : specifies the evaluation strategy. Possible values are: - `"simple"` - we evaluate the metric and return the scores. - `"bootstrap"` - on top of computing the metric scores, we calculate the confidence interval for each of the returned metric keys, using `scipy`'s `bootstrap` method https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html.

confidence_level (`float`, defaults to `0.95`) : The `confidence_level` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

n_resamples (`int`, defaults to `9999`) : The `n_resamples` value passed to `bootstrap` if `"bootstrap"` strategy is chosen.

device (`int`, defaults to `None`) : Device ordinal for CPU/GPU support of the pipeline. Setting this to -1 will leverage CPU, a positive integer will run the model on the associated CUDA device ID. If `None` is provided it will be inferred and CUDA:0 used if available, CPU otherwise.

random_state (`int`, *optional*, defaults to `None`) : The `random_state` value passed to `bootstrap` if `"bootstrap"` strategy is chosen. Useful for debugging.   

input_column (`str`, defaults to `"file"`) : The name of the column containing either the audio files or a raw waveform, represented as a numpy array, in the dataset specified by `data`.

label_column (`str`, defaults to `"label"`) : The name of the column containing the labels in the dataset specified by `data`.

label_mapping (`Dict[str, Number]`, *optional*, defaults to `None`) : We want to map class labels defined by the model in the pipeline to values consistent with those defined in the `label_column` of the `data` dataset.

**Returns:**

A `Dict`. The keys represent metric keys calculated for the `metric` spefied in function arguments. For the
`"simple"` strategy, the value is the metric score. For the `"bootstrap"` strategy, the value is a `Dict`
containing the score, the confidence interval and the standard error calculated for each metric key.

### Main classes
https://huggingface.co/docs/evaluate/main/package_reference/main_classes.md

# Main classes

## EvaluationModuleInfo[[evaluate.EvaluationModuleInfo]]

The base class `EvaluationModuleInfo` implements a the logic for the subclasses `MetricInfo`, `ComparisonInfo`, and `MeasurementInfo`.

#### evaluate.EvaluationModuleInfo[[evaluate.EvaluationModuleInfo]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/info.py#L35)

Base class to store information about an evaluation used for `MetricInfo`, `ComparisonInfo`,
and `MeasurementInfo`.

`EvaluationModuleInfo` documents an evaluation, including its name, version, and features.
See the constructor arguments and properties for a full list.

Note: Not all fields are known on construction and may be updated later.

from_directoryevaluate.EvaluationModuleInfo.from_directoryhttps://github.com/huggingface/evaluate/blob/main/src/evaluate/info.py#L92[{"name": "metric_info_dir", "val": ""}]- **metric_info_dir** (`str`) --
  The directory containing the `metric_info` JSON file. This
  should be the root directory of a specific metric version.0
Create `EvaluationModuleInfo` from the JSON file in `metric_info_dir`.

Example:

```py
>>> my_metric = EvaluationModuleInfo.from_directory("/path/to/directory/")
```

**Parameters:**

metric_info_dir (`str`) : The directory containing the `metric_info` JSON file. This should be the root directory of a specific metric version.
#### write_to_directory[[evaluate.EvaluationModuleInfo.write_to_directory]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/info.py#L72)

Write `EvaluationModuleInfo` as JSON to `metric_info_dir`.
Also save the license separately in LICENSE.

Example:

```py
>>> my_metric.info.write_to_directory("/path/to/directory/")
```

**Parameters:**

metric_info_dir (`str`) : The directory to save `metric_info_dir` to.

#### evaluate.MetricInfo[[evaluate.MetricInfo]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/info.py#L122)

Information about a metric.

`EvaluationModuleInfo` documents a metric, including its name, version, and features.
See the constructor arguments and properties for a full list.

Note: Not all fields are known on construction and may be updated later.

#### evaluate.ComparisonInfo[[evaluate.ComparisonInfo]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/info.py#L135)

Information about a comparison.

`EvaluationModuleInfo` documents a comparison, including its name, version, and features.
See the constructor arguments and properties for a full list.

Note: Not all fields are known on construction and may be updated later.

#### evaluate.MeasurementInfo[[evaluate.MeasurementInfo]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/info.py#L148)

Information about a measurement.

`EvaluationModuleInfo` documents a measurement, including its name, version, and features.
See the constructor arguments and properties for a full list.

Note: Not all fields are known on construction and may be updated later.

## EvaluationModule[[evaluate.EvaluationModule]]

The base class `EvaluationModule` implements a the logic for the subclasses `Metric`, `Comparison`, and `Measurement`.

#### evaluate.EvaluationModule[[evaluate.EvaluationModule]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/module.py#L149)

A `EvaluationModule` is the base class and common API for metrics, comparisons, and measurements.

addevaluate.EvaluationModule.addhttps://github.com/huggingface/evaluate/blob/main/src/evaluate/module.py#L548[{"name": "prediction", "val": " = None"}, {"name": "reference", "val": " = None"}, {"name": "**kwargs", "val": ""}]- **prediction** (`list/array/tensor`, *optional*) --
  Predictions.
- **reference** (`list/array/tensor`, *optional*) --
  References.0
Add one prediction and reference for the evaluation module's stack.

Example:

```py
>>> import evaluate
>>> accuracy = evaluate.load("accuracy")
>>> accuracy.add(references=[0,1], predictions=[1,0])
```

**Parameters:**

config_name (`str`) : This is used to define a hash specific to a module computation script and prevents the module's data to be overridden when the module loading script is modified.

keep_in_memory (`bool`) : Keep all predictions and references in memory. Not possible in distributed settings.

cache_dir (`str`) : Path to a directory in which temporary prediction/references data will be stored. The data directory should be located on a shared file-system in distributed setups.

num_process (`int`) : Specify the total number of nodes in a distributed settings. This is useful to compute module in distributed setups (in particular non-additive modules like F1).

process_id (`int`) : Specify the id of the current process in a distributed setup (between 0 and num_process-1) This is useful to compute module in distributed setups (in particular non-additive metrics like F1).

seed (`int`, optional) : If specified, this will temporarily set numpy's random seed when [compute()](/docs/evaluate/main/en/package_reference/main_classes#evaluate.EvaluationModule.compute) is run.

experiment_id (`str`) : A specific experiment id. This is used if several distributed evaluations share the same file system. This is useful to compute module in distributed setups (in particular non-additive metrics like F1).

hash (`str`) : Used to identify the evaluation module according to the hashed file contents.

max_concurrent_cache_files (`int`) : Max number of concurrent module cache files (default `10000`).

timeout (`Union[int, float]`) : Timeout in second for distributed setting synchronization.
#### add_batch[[evaluate.EvaluationModule.add_batch]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/module.py#L488)

Add a batch of predictions and references for the evaluation module's stack.

Example:

```py
>>> import evaluate
>>> accuracy = evaluate.load("accuracy")
>>> for refs, preds in zip([[0,1],[0,1]], [[1,0],[0,1]]):
...     accuracy.add_batch(references=refs, predictions=preds)
```

**Parameters:**

predictions (`list/array/tensor`, *optional*) : Predictions.

references (`list/array/tensor`, *optional*) : References.
#### compute[[evaluate.EvaluationModule.compute]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/module.py#L415)

Compute the evaluation module.

Usage of positional arguments is not allowed to prevent mistakes.

```py
>>> import evaluate
>>> accuracy =  evaluate.load("accuracy")
>>> accuracy.compute(predictions=[0, 1, 1, 0], references=[0, 1, 0, 1])
```

**Parameters:**

predictions (`list/array/tensor`, *optional*) : Predictions.

references (`list/array/tensor`, *optional*) : References.

- ****kwargs** (optional) : Keyword arguments that will be forwarded to the evaluation module [compute()](/docs/evaluate/main/en/package_reference/main_classes#evaluate.EvaluationModule.compute) method (see details in the docstring).

**Returns:**

``dict` or `None``

- Dictionary with the results if this evaluation module is run on the main process (`process_id == 0`).
- `None` if the evaluation module is not run on the main process (`process_id != 0`).
#### download_and_prepare[[evaluate.EvaluationModule.download_and_prepare]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/module.py#L677)

Downloads and prepares evaluation module for reading.

Example:

```py
>>> import evaluate
```

**Parameters:**

download_config (`DownloadConfig`, *optional*) : Specific download configuration parameters.

dl_manager (`DownloadManager`, *optional*) : Specific download manager to use.

#### evaluate.Metric[[evaluate.Metric]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/module.py#L782)

A Metric is the base class and common API for all metrics.

**Parameters:**

config_name (`str`) : This is used to define a hash specific to a metric computation script and prevents the metric's data to be overridden when the metric loading script is modified.

keep_in_memory (`bool`) : Keep all predictions and references in memory. Not possible in distributed settings.

cache_dir (`str`) : Path to a directory in which temporary prediction/references data will be stored. The data directory should be located on a shared file-system in distributed setups.

num_process (`int`) : Specify the total number of nodes in a distributed settings. This is useful to compute metrics in distributed setups (in particular non-additive metrics like F1).

process_id (`int`) : Specify the id of the current process in a distributed setup (between 0 and num_process-1) This is useful to compute metrics in distributed setups (in particular non-additive metrics like F1).

seed (`int`, *optional*) : If specified, this will temporarily set numpy's random seed when [compute()](/docs/evaluate/main/en/package_reference/main_classes#evaluate.EvaluationModule.compute) is run.

experiment_id (`str`) : A specific experiment id. This is used if several distributed evaluations share the same file system. This is useful to compute metrics in distributed setups (in particular non-additive metrics like F1).

max_concurrent_cache_files (`int`) : Max number of concurrent metric cache files (default `10000`).

timeout (`Union[int, float]`) : Timeout in second for distributed setting synchronization.

#### evaluate.Comparison[[evaluate.Comparison]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/module.py#L812)

A Comparison is the base class and common API for all comparisons.

**Parameters:**

config_name (`str`) : This is used to define a hash specific to a comparison computation script and prevents the comparison's data to be overridden when the comparison loading script is modified.

keep_in_memory (`bool`) : Keep all predictions and references in memory. Not possible in distributed settings.

cache_dir (`str`) : Path to a directory in which temporary prediction/references data will be stored. The data directory should be located on a shared file-system in distributed setups.

num_process (`int`) : Specify the total number of nodes in a distributed settings. This is useful to compute  comparisons in distributed setups (in particular non-additive comparisons).

process_id (`int`) : Specify the id of the current process in a distributed setup (between 0 and num_process-1) This is useful to compute  comparisons in distributed setups (in particular non-additive comparisons).

seed (`int`, *optional*) : If specified, this will temporarily set numpy's random seed when [compute()](/docs/evaluate/main/en/package_reference/main_classes#evaluate.EvaluationModule.compute) is run.

experiment_id (`str`) : A specific experiment id. This is used if several distributed evaluations share the same file system. This is useful to compute  comparisons in distributed setups (in particular non-additive comparisons).

max_concurrent_cache_files (`int`) : Max number of concurrent comparison cache files (default `10000`).

timeout (`Union[int, float]`) : Timeout in second for distributed setting synchronization.

#### evaluate.Measurement[[evaluate.Measurement]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/module.py#L842)

A Measurement is the base class and common API for all measurements.

**Parameters:**

config_name (`str`) : This is used to define a hash specific to a measurement computation script and prevents the measurement's data to be overridden when the measurement loading script is modified.

keep_in_memory (`bool`) : Keep all predictions and references in memory. Not possible in distributed settings.

cache_dir (`str`) : Path to a directory in which temporary prediction/references data will be stored. The data directory should be located on a shared file-system in distributed setups.

num_process (`int`) : Specify the total number of nodes in a distributed settings. This is useful to compute measurements in distributed setups (in particular non-additive measurements).

process_id (`int`) : Specify the id of the current process in a distributed setup (between 0 and num_process-1) This is useful to compute measurements in distributed setups (in particular non-additive measurements).

seed (`int`, *optional*) : If specified, this will temporarily set numpy's random seed when [compute()](/docs/evaluate/main/en/package_reference/main_classes#evaluate.EvaluationModule.compute) is run.

experiment_id (`str`) : A specific experiment id. This is used if several distributed evaluations share the same file system. This is useful to compute measurements in distributed setups (in particular non-additive measurements).

max_concurrent_cache_files (`int`) : Max number of concurrent measurement cache files (default `10000`).

timeout (`Union[int, float]`) : Timeout in second for distributed setting synchronization.

## CombinedEvaluations[[evaluate.combine]]

The `combine` function allows to combine multiple `EvaluationModule`s into a single `CombinedEvaluations`.

#### evaluate.combine[[evaluate.combine]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/module.py#L1008)

Combines several metrics, comparisons, or measurements into a single `CombinedEvaluations` object that
can be used like a single evaluation module.

If two scores have the same name, then they are prefixed with their module names.
And if two modules have the same name, please use a dictionary to give them different names, otherwise an integer id is appended to the prefix.

Examples:

```py
>>> import evaluate
>>> accuracy = evaluate.load("accuracy")
>>> f1 = evaluate.load("f1")
>>> clf_metrics = combine(["accuracy", "f1"])
```

**Parameters:**

evaluations (`Union[list, dict]`) : A list or dictionary of evaluation modules. The modules can either be passed as strings or loaded `EvaluationModule`s. If a dictionary is passed its keys are the names used and the values the modules. The names are used as prefix in case there are name overlaps in the returned results of each module or if `force_prefix=True`.

force_prefix (`bool`, *optional*, defaults to `False`) : If `True` all scores from the modules are prefixed with their name. If a dictionary is passed the keys are used as name otherwise the module's name.

#### evaluate.CombinedEvaluations[[evaluate.CombinedEvaluations]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/module.py#L872)

addevaluate.CombinedEvaluations.addhttps://github.com/huggingface/evaluate/blob/main/src/evaluate/module.py#L895[{"name": "prediction", "val": " = None"}, {"name": "reference", "val": " = None"}, {"name": "**kwargs", "val": ""}]- **predictions** (`list/array/tensor`, *optional*) --
  Predictions.
- **references** (`list/array/tensor`, *optional*) --
  References.0
Add one prediction and reference for each evaluation module's stack.

Example:

```py
>>> import evaluate
>>> accuracy = evaluate.load("accuracy")
>>> f1 = evaluate.load("f1")
>>> clf_metrics = combine(["accuracy", "f1"])
>>> for ref, pred in zip([0,1,0,1], [1,0,0,1]):
...     clf_metrics.add(references=ref, predictions=pred)
```

**Parameters:**

predictions (`list/array/tensor`, *optional*) : Predictions.

references (`list/array/tensor`, *optional*) : References.
#### add_batch[[evaluate.CombinedEvaluations.add_batch]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/module.py#L920)

Add a batch of predictions and references for each evaluation module's stack.

Example:
```py
>>> import evaluate
>>> accuracy = evaluate.load("accuracy")
>>> f1 = evaluate.load("f1")
>>> clf_metrics = combine(["accuracy", "f1"])
>>> for refs, preds in zip([[0,1],[0,1]], [[1,0],[0,1]]):
...     clf_metrics.add(references=refs, predictions=preds)
```

**Parameters:**

predictions (`list/array/tensor`, *optional*) : Predictions.

references (`list/array/tensor`, *optional*) : References.
#### compute[[evaluate.CombinedEvaluations.compute]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/module.py#L944)

Compute each evaluation module.

Usage of positional arguments is not allowed to prevent mistakes.

Example:

```py
>>> import evaluate
>>> accuracy = evaluate.load("accuracy")
>>> f1 = evaluate.load("f1")
>>> clf_metrics = combine(["accuracy", "f1"])
>>> clf_metrics.compute(predictions=[0,1], references=[1,1])
{'accuracy': 0.5, 'f1': 0.6666666666666666}
```

**Parameters:**

predictions (`list/array/tensor`, *optional*) : Predictions.

references (`list/array/tensor`, *optional*) : References.

- ****kwargs** (*optional*) : Keyword arguments that will be forwarded to the evaluation module [compute()](/docs/evaluate/main/en/package_reference/main_classes#evaluate.EvaluationModule.compute) method (see details in the docstring).

**Returns:**

``dict` or `None``

- Dictionary with the results if this evaluation module is run on the main process (`process_id == 0`).
- `None` if the evaluation module is not run on the main process (`process_id != 0`).

### Saving methods
https://huggingface.co/docs/evaluate/main/package_reference/saving_methods.md

# Saving methods

Methods for saving evaluations results:

## Save[[evaluate.save]]

#### evaluate.save[[evaluate.save]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/saving.py#L13)

Saves results to a JSON file. Also saves system information such as current time, current commit
hash if inside a repository, and Python system information.

Example:
```py
>>> import evaluate
>>> result = {"bleu": 0.7}
>>> params = {"model": "gpt-2"}
>>> evaluate.save("./results/", **result, **params)
```

**Parameters:**

path_or_file (`str`) : Path or file to store the file. If only a folder is provided the results file will be saved in the format `"result-%Y_%m_%d-%H_%M_%S.json"`.

### Loading methods
https://huggingface.co/docs/evaluate/main/package_reference/loading_methods.md

# Loading methods

Methods for listing and loading evaluation modules:

## List[[evaluate.list_evaluation_modules]]

#### evaluate.list_evaluation_modules[[evaluate.list_evaluation_modules]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/inspect.py#L35)

List all evaluation modules available on the Hugging Face Hub.

Example:

```py
>>> from evaluate import list_evaluation_modules
>>> list_evaluation_modules(module_type="metric")
```

**Parameters:**

module_type (`str`, *optional*, defaults to `None`) : Type of evaluation modules to list. Has to be one of `'metric'`, `'comparison'`, or `'measurement'`. If `None`, all types are listed.

include_community (`bool`, *optional*, defaults to `True`) : Include community modules in the list.

with_details (`bool`, *optional*, defaults to `False`) : Return the full details on the metrics instead of only the ID.

**Returns:**

`List[Union[str, dict]]`

## Load[[evaluate.load]]

#### evaluate.load[[evaluate.load]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/loading.py#L692)

Load a [EvaluationModule](/docs/evaluate/main/en/package_reference/main_classes#evaluate.EvaluationModule).

Example:

```py
>>> from evaluate import load
>>> accuracy = load("accuracy")
```

**Parameters:**

path (`str`) : Path to the evaluation processing script with the evaluation builder. Can be either: - a local path to processing script or the directory containing the script (if the script has the same name as the directory), e.g. `'./metrics/rouge'` or `'./metrics/rouge/rouge.py'` - a evaluation module identifier on the HuggingFace evaluate repo e.g. `'rouge'` or `'bleu'` that are in either `'metrics/'`, `'comparisons/'`, or `'measurements/'` depending on the provided `module_type`

config_name (`str`, *optional*) : Selecting a configuration for the metric (e.g. the GLUE metric has a configuration for each subset).

module_type (`str`, default `'metric'`) : Type of evaluation module, can be one of `'metric'`, `'comparison'`, or `'measurement'`.

process_id (`int`, *optional*) : For distributed evaluation: id of the process.

num_process (`int`, *optional*) : For distributed evaluation: total number of processes.

cache_dir (`str`, *optional*) : Path to store the temporary predictions and references (default to `~/.cache/huggingface/evaluate/`).

experiment_id (`str`) : A specific experiment id. This is used if several distributed evaluations share the same file system. This is useful to compute metrics in distributed setups (in particular non-additive metrics like F1).

keep_in_memory (`bool`) : Whether to store the temporary results in memory (defaults to `False`).

download_config (`~evaluate.DownloadConfig`, *optional*) : Specific download configuration parameters.

download_mode (`DownloadMode`, defaults to `REUSE_DATASET_IF_EXISTS`) : Download/generate mode.

revision (`Union[str, evaluate.Version]`, *optional*) : If specified, the module will be loaded from the datasets repository at this version. By default it is set to the local version of the lib. Specifying a version that is different from your local version of the lib might cause compatibility issues.

**Returns:**

[evaluate.EvaluationModule](/docs/evaluate/main/en/package_reference/main_classes#evaluate.EvaluationModule)

### Hub methods
https://huggingface.co/docs/evaluate/main/package_reference/hub_methods.md

# Hub methods

Methods for using the Hugging Face Hub:

## Push to hub [[evaluate.push_to_hub]]

#### evaluate.push_to_hub[[evaluate.push_to_hub]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/hub.py#L14)

Pushes the result of a metric to the metadata of a model repository in the Hub.

Example:

```python
>>> push_to_hub(
...     model_id="huggingface/gpt2-wikitext2",
...     metric_value=0.5
...     metric_type="bleu",
...     metric_name="BLEU",
...     dataset_name="WikiText",
...     dataset_type="wikitext",
...     dataset_split="test",
...     task_type="text-generation",
...     task_name="Text Generation"
... )
```

**Parameters:**

model_id (`str`) : Model id from https://hf.co/models.

task_type (`str`) : Task id, refer to the [Hub allowed tasks](https://github.com/huggingface/evaluate/blob/main/src/evaluate/config.py#L154) for allowed values.

dataset_type (`str`) : Dataset id from https://hf.co/datasets.

dataset_name (`str`) : Pretty name for the dataset.

metric_type (`str`) : Metric id from https://hf.co/metrics.

metric_name (`str`) : Pretty name for the metric.

metric_value (`float`) : Computed metric value.

task_name (`str`, *optional*) : Pretty name for the task.

dataset_config (`str`, *optional*) : Dataset configuration used in [load_dataset](https://huggingface.co/docs/datasets/main/en/package_reference/loading_methods#datasets.load_dataset). See [load_dataset](https://huggingface.co/docs/datasets/main/en/package_reference/loading_methods#datasets.load_dataset) for more info.

dataset_split (`str`, *optional*) : Name of split used for metric computation.

dataset_revision (`str`, *optional*) : Git hash for the specific version of the dataset.

dataset_args (`dict[str, int]`, *optional*) : Additional arguments passed to [load_dataset](https://huggingface.co/docs/datasets/main/en/package_reference/loading_methods#datasets.load_dataset).

metric_config (`str`, *optional*) : Configuration for the metric (e.g. the GLUE metric has a configuration for each subset).

metric_args (`dict[str, int]`, *optional*) : Arguments passed during [compute()](/docs/evaluate/main/en/package_reference/main_classes#evaluate.EvaluationModule.compute).

overwrite (`bool`, *optional*, defaults to `False`) : If set to `True` an existing metric field can be overwritten, otherwise attempting to overwrite any existing fields will cause an error.

### Logging methods
https://huggingface.co/docs/evaluate/main/package_reference/logging_methods.md

# Logging methods

🤗 Evaluate strives to be transparent and explicit about how it works, but this can be quite verbose at times. We have included a series of logging methods which allow you to easily adjust the level of verbosity of the entire library. Currently the default verbosity of the library is set to `WARNING`.

To change the level of verbosity, use one of the direct setters. For instance, here is how to change the verbosity to the `INFO` level:

```py
import evaluate
evaluate.logging.set_verbosity_info()
```

You can also use the environment variable `EVALUATE_VERBOSITY` to override the default verbosity, and set it to one of the following: `debug`, `info`, `warning`, `error`, `critical`:

```bash
EVALUATE_VERBOSITY=error ./myprogram.py
```

All the methods of this logging module are documented below. The main ones are:

- [logging.get_verbosity()](/docs/evaluate/main/en/package_reference/logging_methods#evaluate.utils.logging.get_verbosity) to get the current level of verbosity in the logger
- [logging.set_verbosity()](/docs/evaluate/main/en/package_reference/logging_methods#evaluate.utils.logging.set_verbosity) to set the verbosity to the level of your choice

In order from the least to the most verbose (with their corresponding `int` values):

1. `logging.CRITICAL` or `logging.FATAL` (int value, 50): only report the most critical errors.
2. `logging.ERROR` (int value, 40): only report errors.
3. `logging.WARNING` or `logging.WARN` (int value, 30): only reports error and warnings. This the default level used by the library.
4. `logging.INFO` (int value, 20): reports error, warnings and basic information.
5. `logging.DEBUG` (int value, 10): report all information.

By default, `tqdm` progress bars will be displayed during evaluate download and processing. [logging.disable_progress_bar()](/docs/evaluate/main/en/package_reference/logging_methods#evaluate.disable_progress_bar) and [logging.enable_progress_bar()](/docs/evaluate/main/en/package_reference/logging_methods#evaluate.enable_progress_bar) can be used to suppress or unsuppress this behavior. 

## Functions[[evaluate.utils.logging.get_verbosity]]

#### evaluate.utils.logging.get_verbosity[[evaluate.utils.logging.get_verbosity]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/utils/logging.py#L85)

Return the current level for the Hugging Face Evaluate library's root logger.

> [!TIP]
> Hugging Face Evaluate library has following logging levels:
>     - `evaluate.logging.CRITICAL`, `evaluate.logging.FATAL`
>     - `evaluate.logging.ERROR`
>     - `evaluate.logging.WARNING`, `evaluate.logging.WARN`
>     - `evaluate.logging.INFO`
>     - `evaluate.logging.DEBUG`

**Returns:**

Logging level, e.g., `evaluate.logging.DEBUG` and `evaluate.logging.INFO`.

#### evaluate.utils.logging.set_verbosity[[evaluate.utils.logging.set_verbosity]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/utils/logging.py#L101)

Set the level for the Hugging Face Evaluate library's root logger.

**Parameters:**

verbosity : Logging level, e.g., `evaluate.logging.DEBUG` and `evaluate.logging.INFO`.

#### evaluate.utils.logging.set_verbosity_info[[evaluate.utils.logging.set_verbosity_info]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/utils/logging.py#L110)

Set the level for the Hugging Face Evaluate library's root logger to `INFO`.

This will display most of the logging information and tqdm bars.

Shortcut to `evaluate.logging.set_verbosity(evaluate.logging.INFO)`.

#### evaluate.utils.logging.set_verbosity_warning[[evaluate.utils.logging.set_verbosity_warning]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/utils/logging.py#L120)

Set the level for the Hugging Face Evaluate library's root logger to `WARNING`.

This will display only the warning and errors logging information and tqdm bars.

Shortcut to `evaluate.logging.set_verbosity(evaluate.logging.WARNING)`.

#### evaluate.utils.logging.set_verbosity_debug[[evaluate.utils.logging.set_verbosity_debug]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/utils/logging.py#L130)

Set the level for the Hugging Face Evaluate library's root logger to `DEBUG`.

This will display all the logging information and tqdm bars.

Shortcut to `evaluate.logging.set_verbosity(evaluate.logging.DEBUG)`.

#### evaluate.utils.logging.set_verbosity_error[[evaluate.utils.logging.set_verbosity_error]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/utils/logging.py#L140)

Set the level for the Hugging Face Evaluate library's root logger to `ERROR`.

This will display only the errors logging information and tqdm bars.

Shortcut to `evaluate.logging.set_verbosity(evaluate.logging.ERROR)`.

#### evaluate.utils.logging.disable_propagation[[evaluate.utils.logging.disable_propagation]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/utils/logging.py#L150)

Disable propagation of the library log outputs.
Note that log propagation is disabled by default.

#### evaluate.utils.logging.enable_propagation[[evaluate.utils.logging.enable_propagation]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/utils/logging.py#L157)

Enable propagation of the library log outputs.
Please disable the Hugging Face Evaluate library's default handler to prevent double logging if the root logger has
been configured.

#### evaluate.utils.logging.get_logger[[evaluate.utils.logging.get_logger]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/utils/logging.py#L78)

Return a logger with the specified name.

#### evaluate.enable_progress_bar[[evaluate.enable_progress_bar]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/utils/logging.py#L221)

Enable tqdm progress bar.

#### evaluate.disable_progress_bar[[evaluate.disable_progress_bar]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/utils/logging.py#L227)

Enable tqdm progress bar.

## Levels

### evaluate.logging.CRITICAL

evaluate.logging.CRITICAL = 50

### evaluate.logging.DEBUG

evaluate.logging.DEBUG = 10

### evaluate.logging.ERROR

evaluate.logging.ERROR = 40

### evaluate.logging.FATAL

evaluate.logging.FATAL = 50

### evaluate.logging.INFO

evaluate.logging.INFO = 20

### evaluate.logging.NOTSET

evaluate.logging.NOTSET = 0

### evaluate.logging.WARN

evaluate.logging.WARN = 30

### evaluate.logging.WARNING

evaluate.logging.WARNING = 30

### Visualization methods
https://huggingface.co/docs/evaluate/main/package_reference/visualization_methods.md

# Visualization methods

Methods for visualizing evaluations results:

## Radar Plot[[evaluate.visualization.radar_plot]]

#### evaluate.visualization.radar_plot[[evaluate.visualization.radar_plot]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/visualization.py#L138)

Create a complex radar chart with different scales for each variable
Source: https://towardsdatascience.com/how-to-create-and-visualize-complex-radar-charts-f7764d0f3652

**Parameters:**

data (`List[dict]`) : the results (list of metric + value pairs). E.g. data = [{"accuracy": 0.9, "precision":0.8},{"accuracy": 0.7, "precision":0.6}]

names (`List[dict]`) : model names. E.g. names = ["model1", "model 2", ...]

invert_range (`List[dict]`, optional) : the metrics to invert (in cases when smaller is better, e.g. speed) E.g. invert_range=["latency_in_seconds"]

config (`dict`, optional) : a specification of the formatting configurations, namely:  - rad_ln_args (`dict`, default `{"visible": True}`): The visibility of the radial (circle) lines.  - outer_ring (`dict`, default `{"visible": True}`): The visibility of the outer ring.  - angle_ln_args (`dict`, default `{"visible": True}`): The visibility of the angle lines.  - rgrid_tick_lbls_args (`dict`, default `{"fontsize": 12}`): The font size of the tick labels on the scales.  - theta_tick_lbls (`dict`, default `{"fontsize": 12}`): The font size of the variable labels on the plot.  - theta_tick_lbls_pad (`int`, default `3`): The padding of the variable labels on the plot.  - theta_tick_lbls_brk_lng_wrds (`bool`, default `True` ): Whether long words in the label are broken up or not.  - theta_tick_lbls_txt_wrap (`int`, default `15`): Text wrap for tick labels  - incl_endpoint (`bool`, default `False`): Include value endpoints on calse  - marker (`str`, default `"o"`): the shape of the marker used in the radar plot.  - markersize (`int`, default `3`): the shape of the marker used in the radar plot.  - legend_loc (`str`, default `"upper right"`): the location of the legend in the radar plot. Must be one of: 'upper left', 'upper right', 'lower left', 'lower right'.  - bbox_to_anchor (`tuple`, default `(2, 1)`: anchor for the legend.

fig (`matplotlib.figure.Figure`, optional) : figure used to plot the radar plot.

**Returns:**

`matplotlib.figure.Figure`
