---
language:
  - en
tags:
  - genomics
  - dna
  - masked-language-modeling
  - caduceus
library_name: transformers
pipeline_tag: fill-mask
---

# Caduceus-PS fish20 adapted

This checkpoint is Caduceus-PS adapted to the FishNALM 20-species fish-genome corpus for masked language modeling.

## Domain-adaptive pretraining

| Setting | Value |
| --- | --- |
| Base model | Caduceus-PS (~7.7M parameters) |
| Tokenization | Single nucleotide |
| Hardware | 4 × NVIDIA A100 80GB |
| Per-device batch size | 64 |
| Gradient accumulation steps | 1 |
| Effective global batch size | 256 |
| Peak learning rate | 1 × 10⁻⁴ |
| Corpus exposure | 1 epoch |

## Usage

This repository contains custom model and tokenizer code. Load it with `trust_remote_code=True`.

```python
from transformers import AutoModelForMaskedLM, AutoTokenizer

model_id = "<your-namespace>/Caduceus-PS-fish20-adapted"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(model_id, trust_remote_code=True)
```

The checkpoint is intended for DNA-sequence representation and masked-token prediction. Evaluate and adapt it for each downstream task before use.