manga-ocr-base-CPT-flex

English | 中文

Code, technical report and findings: GitHub muscgab/manga-ocr-CPT · Weights: Hugging Face · ModelScope

OCR for Japanese manga text boxes. This is mangaOCR after continued pre-training on our training data, with the architecture unchanged (ViT-B/16 encoder, 2-layer BERT decoder, autoregressive, 111M parameters) and a new vocabulary of 6,817 characters. Input: aspect ratio kept, resized into at most 196 patches of 16 px; the 14 × 14 position table is interpolated to the patch grid and padded patches are masked.

Recommended of the two models: clearly better than manga-ocr-base-CPT on elongated boxes (aspect ratio ≥ 8: EM 85.40% vs 76.64%) and horizontal boxes (87.02% vs 81.78%), and tied elsewhere.

This model comes from an experimental study; see the technical report.

Accuracy

JMangaBench_Read v1.3, 7,789 boxes, RIN v1 normalisation, batch 1, RTX 2080 Ti.

Model Score ↑ CER % ↓ EM % ↑ EM printed EM non-printed
manga-ocr-base-CPT-flex 97.45 1.53 91.42 94.16 83.00
manga-ocr-base-CPT 97.43 1.77 91.00 93.82 82.32
mangaOCR (original, package defaults) 90.23 4.84 78.25 88.04 48.17

Speed

RTX 2080 Ti, Read v1.3 speed protocol, preprocessing included; a page (8.7 boxes on average) is one batch.

Inference path Per crop (batch 1) Page p50 Peak throughput
CUDA path (CUDA graphs + GPU preprocessing) 11.0 ms 43.0 ms 260 crops/s (batch 32)
Plain PyTorch path 27.6 ms 71.4 ms 242 crops/s (batch 80)
Original manga-ocr package (fp16) 84.9 ms 242.4 ms 42 crops/s (batch 56)

Both paths give identical outputs on all 7,789 boxes.

Usage

The inference code is on GitHub; the weights are downloaded from this repository on first use.

pip install git+https://github.com/muscgab/manga-ocr-CPT
from manga_ocr_cpt import MangaOcrCPT
ocr = MangaOcrCPT("flex")              # or a local directory holding this repository
ocr("crop.png")                          # one crop -> str
ocr(["crop1.png", "crop2.png"])          # the crops of one page -> list[str]

Inputs are cropped text boxes (file paths, PIL images or uint8 RGB arrays); there is no text detection. With an NVIDIA GPU the CUDA path is used, otherwise plain PyTorch (CPU, Apple MPS, ...).

Training

  • Initialised from kha-white/manga-ocr-base; the 5,133 characters already in the original vocabulary keep their embeddings.
  • Data (same as manga-ocr-nar-small): 4,471,985 crops: synthetic printed text 1,795,869, synthetic handwriting 1,670,000, AnimeText-OCR real crops 1,006,116. Labels of 1–64 characters.
  • 139,749 steps, global batch 256 (8 epochs), 4 × RTX 2080 Ti; encoder fp16 mixed precision, decoder fp32. Released weights are from the last step.
  • Stopped by a false divergence alarm at step 126,200 and resumed from step 120,000 with a new data seed, so it ran 6,200 steps (about 1.59M samples) more than manga-ocr-base-CPT.

Files

File Contents
model.safetensors Weights, fp32 (444 MB)
config.json Architecture and input settings (input.fit selects the input)
nar_vocabulary_6817.json Vocabulary (6,817 characters)
LICENSE_MODEL CC BY-NC-SA 4.0
NOTICE.md Provenance and acknowledgements
SHA256SUMS Checksums

Limitations

  • Japanese only. At most 64 characters per box; longer boxes must be split first. Boxes of more than 32 characters are clearly weaker.
  • Handwriting and stylised lettering produce clearly more errors than printed text.
  • Aimed at dialogue, narration and small annotations; onomatopoeia is not a target.
  • The benchmark was built by the same author; parts of it were used during development. Not evaluated on scanned manga such as Manga109-s.

License

  • Weights: CC BY-NC-SA 4.0 (LICENSE_MODEL); commercial use requires separate permission. The training data AnimeText-OCR is CC BY-NC-SA 4.0.
  • Initialised from kha-white/manga-ocr-base (Apache License 2.0). See NOTICE.md.
  • The inference code on GitHub is Apache License 2.0.
Downloads last month
226
Safetensors
Model size
0.1B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for muscgab/manga-ocr-base-CPT-flex

Finetuned
(7)
this model
Quantizations
1 model

Dataset used to train muscgab/manga-ocr-base-CPT-flex