# Main classes

## EvaluationModuleInfo[[evaluate.EvaluationModuleInfo]]

The base class `EvaluationModuleInfo` implements a the logic for the subclasses `MetricInfo`, `ComparisonInfo`, and `MeasurementInfo`.

#### evaluate.EvaluationModuleInfo[[evaluate.EvaluationModuleInfo]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/info.py#L35)

Base class to store information about an evaluation used for `MetricInfo`, `ComparisonInfo`,
and `MeasurementInfo`.

`EvaluationModuleInfo` documents an evaluation, including its name, version, and features.
See the constructor arguments and properties for a full list.

Note: Not all fields are known on construction and may be updated later.

from_directoryevaluate.EvaluationModuleInfo.from_directoryhttps://github.com/huggingface/evaluate/blob/main/src/evaluate/info.py#L92[{"name": "metric_info_dir", "val": ""}]- **metric_info_dir** (`str`) --
  The directory containing the `metric_info` JSON file. This
  should be the root directory of a specific metric version.0
Create `EvaluationModuleInfo` from the JSON file in `metric_info_dir`.

Example:

```py
>>> my_metric = EvaluationModuleInfo.from_directory("/path/to/directory/")
```

**Parameters:**

metric_info_dir (`str`) : The directory containing the `metric_info` JSON file. This should be the root directory of a specific metric version.
#### write_to_directory[[evaluate.EvaluationModuleInfo.write_to_directory]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/info.py#L72)

Write `EvaluationModuleInfo` as JSON to `metric_info_dir`.
Also save the license separately in LICENSE.

Example:

```py
>>> my_metric.info.write_to_directory("/path/to/directory/")
```

**Parameters:**

metric_info_dir (`str`) : The directory to save `metric_info_dir` to.

#### evaluate.MetricInfo[[evaluate.MetricInfo]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/info.py#L122)

Information about a metric.

`EvaluationModuleInfo` documents a metric, including its name, version, and features.
See the constructor arguments and properties for a full list.

Note: Not all fields are known on construction and may be updated later.

#### evaluate.ComparisonInfo[[evaluate.ComparisonInfo]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/info.py#L135)

Information about a comparison.

`EvaluationModuleInfo` documents a comparison, including its name, version, and features.
See the constructor arguments and properties for a full list.

Note: Not all fields are known on construction and may be updated later.

#### evaluate.MeasurementInfo[[evaluate.MeasurementInfo]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/info.py#L148)

Information about a measurement.

`EvaluationModuleInfo` documents a measurement, including its name, version, and features.
See the constructor arguments and properties for a full list.

Note: Not all fields are known on construction and may be updated later.

## EvaluationModule[[evaluate.EvaluationModule]]

The base class `EvaluationModule` implements a the logic for the subclasses `Metric`, `Comparison`, and `Measurement`.

#### evaluate.EvaluationModule[[evaluate.EvaluationModule]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/module.py#L149)

A `EvaluationModule` is the base class and common API for metrics, comparisons, and measurements.

addevaluate.EvaluationModule.addhttps://github.com/huggingface/evaluate/blob/main/src/evaluate/module.py#L548[{"name": "prediction", "val": " = None"}, {"name": "reference", "val": " = None"}, {"name": "**kwargs", "val": ""}]- **prediction** (`list/array/tensor`, *optional*) --
  Predictions.
- **reference** (`list/array/tensor`, *optional*) --
  References.0
Add one prediction and reference for the evaluation module's stack.

Example:

```py
>>> import evaluate
>>> accuracy = evaluate.load("accuracy")
>>> accuracy.add(references=[0,1], predictions=[1,0])
```

**Parameters:**

config_name (`str`) : This is used to define a hash specific to a module computation script and prevents the module's data to be overridden when the module loading script is modified.

keep_in_memory (`bool`) : Keep all predictions and references in memory. Not possible in distributed settings.

cache_dir (`str`) : Path to a directory in which temporary prediction/references data will be stored. The data directory should be located on a shared file-system in distributed setups.

num_process (`int`) : Specify the total number of nodes in a distributed settings. This is useful to compute module in distributed setups (in particular non-additive modules like F1).

process_id (`int`) : Specify the id of the current process in a distributed setup (between 0 and num_process-1) This is useful to compute module in distributed setups (in particular non-additive metrics like F1).

seed (`int`, optional) : If specified, this will temporarily set numpy's random seed when [compute()](/docs/evaluate/main/en/package_reference/main_classes#evaluate.EvaluationModule.compute) is run.

experiment_id (`str`) : A specific experiment id. This is used if several distributed evaluations share the same file system. This is useful to compute module in distributed setups (in particular non-additive metrics like F1).

hash (`str`) : Used to identify the evaluation module according to the hashed file contents.

max_concurrent_cache_files (`int`) : Max number of concurrent module cache files (default `10000`).

timeout (`Union[int, float]`) : Timeout in second for distributed setting synchronization.
#### add_batch[[evaluate.EvaluationModule.add_batch]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/module.py#L488)

Add a batch of predictions and references for the evaluation module's stack.

Example:

```py
>>> import evaluate
>>> accuracy = evaluate.load("accuracy")
>>> for refs, preds in zip([[0,1],[0,1]], [[1,0],[0,1]]):
...     accuracy.add_batch(references=refs, predictions=preds)
```

**Parameters:**

predictions (`list/array/tensor`, *optional*) : Predictions.

references (`list/array/tensor`, *optional*) : References.
#### compute[[evaluate.EvaluationModule.compute]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/module.py#L415)

Compute the evaluation module.

Usage of positional arguments is not allowed to prevent mistakes.

```py
>>> import evaluate
>>> accuracy =  evaluate.load("accuracy")
>>> accuracy.compute(predictions=[0, 1, 1, 0], references=[0, 1, 0, 1])
```

**Parameters:**

predictions (`list/array/tensor`, *optional*) : Predictions.

references (`list/array/tensor`, *optional*) : References.

- ****kwargs** (optional) : Keyword arguments that will be forwarded to the evaluation module [compute()](/docs/evaluate/main/en/package_reference/main_classes#evaluate.EvaluationModule.compute) method (see details in the docstring).

**Returns:**

``dict` or `None``

- Dictionary with the results if this evaluation module is run on the main process (`process_id == 0`).
- `None` if the evaluation module is not run on the main process (`process_id != 0`).
#### download_and_prepare[[evaluate.EvaluationModule.download_and_prepare]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/module.py#L677)

Downloads and prepares evaluation module for reading.

Example:

```py
>>> import evaluate
```

**Parameters:**

download_config (`DownloadConfig`, *optional*) : Specific download configuration parameters.

dl_manager (`DownloadManager`, *optional*) : Specific download manager to use.

#### evaluate.Metric[[evaluate.Metric]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/module.py#L782)

A Metric is the base class and common API for all metrics.

**Parameters:**

config_name (`str`) : This is used to define a hash specific to a metric computation script and prevents the metric's data to be overridden when the metric loading script is modified.

keep_in_memory (`bool`) : Keep all predictions and references in memory. Not possible in distributed settings.

cache_dir (`str`) : Path to a directory in which temporary prediction/references data will be stored. The data directory should be located on a shared file-system in distributed setups.

num_process (`int`) : Specify the total number of nodes in a distributed settings. This is useful to compute metrics in distributed setups (in particular non-additive metrics like F1).

process_id (`int`) : Specify the id of the current process in a distributed setup (between 0 and num_process-1) This is useful to compute metrics in distributed setups (in particular non-additive metrics like F1).

seed (`int`, *optional*) : If specified, this will temporarily set numpy's random seed when [compute()](/docs/evaluate/main/en/package_reference/main_classes#evaluate.EvaluationModule.compute) is run.

experiment_id (`str`) : A specific experiment id. This is used if several distributed evaluations share the same file system. This is useful to compute metrics in distributed setups (in particular non-additive metrics like F1).

max_concurrent_cache_files (`int`) : Max number of concurrent metric cache files (default `10000`).

timeout (`Union[int, float]`) : Timeout in second for distributed setting synchronization.

#### evaluate.Comparison[[evaluate.Comparison]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/module.py#L812)

A Comparison is the base class and common API for all comparisons.

**Parameters:**

config_name (`str`) : This is used to define a hash specific to a comparison computation script and prevents the comparison's data to be overridden when the comparison loading script is modified.

keep_in_memory (`bool`) : Keep all predictions and references in memory. Not possible in distributed settings.

cache_dir (`str`) : Path to a directory in which temporary prediction/references data will be stored. The data directory should be located on a shared file-system in distributed setups.

num_process (`int`) : Specify the total number of nodes in a distributed settings. This is useful to compute  comparisons in distributed setups (in particular non-additive comparisons).

process_id (`int`) : Specify the id of the current process in a distributed setup (between 0 and num_process-1) This is useful to compute  comparisons in distributed setups (in particular non-additive comparisons).

seed (`int`, *optional*) : If specified, this will temporarily set numpy's random seed when [compute()](/docs/evaluate/main/en/package_reference/main_classes#evaluate.EvaluationModule.compute) is run.

experiment_id (`str`) : A specific experiment id. This is used if several distributed evaluations share the same file system. This is useful to compute  comparisons in distributed setups (in particular non-additive comparisons).

max_concurrent_cache_files (`int`) : Max number of concurrent comparison cache files (default `10000`).

timeout (`Union[int, float]`) : Timeout in second for distributed setting synchronization.

#### evaluate.Measurement[[evaluate.Measurement]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/module.py#L842)

A Measurement is the base class and common API for all measurements.

**Parameters:**

config_name (`str`) : This is used to define a hash specific to a measurement computation script and prevents the measurement's data to be overridden when the measurement loading script is modified.

keep_in_memory (`bool`) : Keep all predictions and references in memory. Not possible in distributed settings.

cache_dir (`str`) : Path to a directory in which temporary prediction/references data will be stored. The data directory should be located on a shared file-system in distributed setups.

num_process (`int`) : Specify the total number of nodes in a distributed settings. This is useful to compute measurements in distributed setups (in particular non-additive measurements).

process_id (`int`) : Specify the id of the current process in a distributed setup (between 0 and num_process-1) This is useful to compute measurements in distributed setups (in particular non-additive measurements).

seed (`int`, *optional*) : If specified, this will temporarily set numpy's random seed when [compute()](/docs/evaluate/main/en/package_reference/main_classes#evaluate.EvaluationModule.compute) is run.

experiment_id (`str`) : A specific experiment id. This is used if several distributed evaluations share the same file system. This is useful to compute measurements in distributed setups (in particular non-additive measurements).

max_concurrent_cache_files (`int`) : Max number of concurrent measurement cache files (default `10000`).

timeout (`Union[int, float]`) : Timeout in second for distributed setting synchronization.

## CombinedEvaluations[[evaluate.combine]]

The `combine` function allows to combine multiple `EvaluationModule`s into a single `CombinedEvaluations`.

#### evaluate.combine[[evaluate.combine]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/module.py#L1008)

Combines several metrics, comparisons, or measurements into a single `CombinedEvaluations` object that
can be used like a single evaluation module.

If two scores have the same name, then they are prefixed with their module names.
And if two modules have the same name, please use a dictionary to give them different names, otherwise an integer id is appended to the prefix.

Examples:

```py
>>> import evaluate
>>> accuracy = evaluate.load("accuracy")
>>> f1 = evaluate.load("f1")
>>> clf_metrics = combine(["accuracy", "f1"])
```

**Parameters:**

evaluations (`Union[list, dict]`) : A list or dictionary of evaluation modules. The modules can either be passed as strings or loaded `EvaluationModule`s. If a dictionary is passed its keys are the names used and the values the modules. The names are used as prefix in case there are name overlaps in the returned results of each module or if `force_prefix=True`.

force_prefix (`bool`, *optional*, defaults to `False`) : If `True` all scores from the modules are prefixed with their name. If a dictionary is passed the keys are used as name otherwise the module's name.

#### evaluate.CombinedEvaluations[[evaluate.CombinedEvaluations]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/module.py#L872)

addevaluate.CombinedEvaluations.addhttps://github.com/huggingface/evaluate/blob/main/src/evaluate/module.py#L895[{"name": "prediction", "val": " = None"}, {"name": "reference", "val": " = None"}, {"name": "**kwargs", "val": ""}]- **predictions** (`list/array/tensor`, *optional*) --
  Predictions.
- **references** (`list/array/tensor`, *optional*) --
  References.0
Add one prediction and reference for each evaluation module's stack.

Example:

```py
>>> import evaluate
>>> accuracy = evaluate.load("accuracy")
>>> f1 = evaluate.load("f1")
>>> clf_metrics = combine(["accuracy", "f1"])
>>> for ref, pred in zip([0,1,0,1], [1,0,0,1]):
...     clf_metrics.add(references=ref, predictions=pred)
```

**Parameters:**

predictions (`list/array/tensor`, *optional*) : Predictions.

references (`list/array/tensor`, *optional*) : References.
#### add_batch[[evaluate.CombinedEvaluations.add_batch]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/module.py#L920)

Add a batch of predictions and references for each evaluation module's stack.

Example:
```py
>>> import evaluate
>>> accuracy = evaluate.load("accuracy")
>>> f1 = evaluate.load("f1")
>>> clf_metrics = combine(["accuracy", "f1"])
>>> for refs, preds in zip([[0,1],[0,1]], [[1,0],[0,1]]):
...     clf_metrics.add(references=refs, predictions=preds)
```

**Parameters:**

predictions (`list/array/tensor`, *optional*) : Predictions.

references (`list/array/tensor`, *optional*) : References.
#### compute[[evaluate.CombinedEvaluations.compute]]

[Source](https://github.com/huggingface/evaluate/blob/main/src/evaluate/module.py#L944)

Compute each evaluation module.

Usage of positional arguments is not allowed to prevent mistakes.

Example:

```py
>>> import evaluate
>>> accuracy = evaluate.load("accuracy")
>>> f1 = evaluate.load("f1")
>>> clf_metrics = combine(["accuracy", "f1"])
>>> clf_metrics.compute(predictions=[0,1], references=[1,1])
{'accuracy': 0.5, 'f1': 0.6666666666666666}
```

**Parameters:**

predictions (`list/array/tensor`, *optional*) : Predictions.

references (`list/array/tensor`, *optional*) : References.

- ****kwargs** (*optional*) : Keyword arguments that will be forwarded to the evaluation module [compute()](/docs/evaluate/main/en/package_reference/main_classes#evaluate.EvaluationModule.compute) method (see details in the docstring).

**Returns:**

``dict` or `None``

- Dictionary with the results if this evaluation module is run on the main process (`process_id == 0`).
- `None` if the evaluation module is not run on the main process (`process_id != 0`).

