--- license: cc-by-nc-sa-4.0 language: - ja pipeline_tag: image-to-text base_model: kha-white/manga-ocr-base tags: - ocr - manga - japanese datasets: - JustANormalTinkerer/animetext-ocr tasks: - ocr-recognition frameworks: - pytorch --- # manga-ocr-base-CPT-flex [English](README.md) | [中文](README_zh.md) Code, technical report and findings: [GitHub muscgab/manga-ocr-CPT](https://github.com/muscgab/manga-ocr-CPT) · Weights: [Hugging Face](https://huggingface.co/muscgab/manga-ocr-base-CPT-flex) · [ModelScope](https://modelscope.cn/models/muscgab/manga-ocr-base-CPT-flex) OCR for Japanese manga text boxes. This is [mangaOCR](https://huggingface.co/kha-white/manga-ocr-base) after continued pre-training on our training data, with the architecture unchanged (ViT-B/16 encoder, 2-layer BERT decoder, autoregressive, 111M parameters) and a new vocabulary of 6,817 characters. Input: aspect ratio kept, resized into at most 196 patches of 16 px; the 14 × 14 position table is interpolated to the patch grid and padded patches are masked. Recommended of the two models: clearly better than manga-ocr-base-CPT on elongated boxes (aspect ratio ≥ 8: EM 85.40% vs 76.64%) and horizontal boxes (87.02% vs 81.78%), and tied elsewhere. This model comes from an experimental study; see the [technical report](https://github.com/muscgab/manga-ocr-CPT/blob/main/TECHNICAL_REPORT.md). ## Accuracy [JMangaBench_Read v1.3](https://github.com/muscgab/JMangaBench_Read), 7,789 boxes, RIN v1 normalisation, batch 1, RTX 2080 Ti. | Model | Score ↑ | CER % ↓ | EM % ↑ | EM printed | EM non-printed | |---|---|---|---|---|---| | **manga-ocr-base-CPT-flex** | **97.45** | **1.53** | **91.42** | 94.16 | 83.00 | | manga-ocr-base-CPT | 97.43 | 1.77 | 91.00 | 93.82 | 82.32 | | mangaOCR (original, package defaults) | 90.23 | 4.84 | 78.25 | 88.04 | 48.17 | ## Speed RTX 2080 Ti, Read v1.3 speed protocol, preprocessing included; a page (8.7 boxes on average) is one batch. | Inference path | Per crop (batch 1) | Page p50 | Peak throughput | |---|---|---|---| | CUDA path (CUDA graphs + GPU preprocessing) | 11.0 ms | 43.0 ms | 260 crops/s (batch 32) | | Plain PyTorch path | 27.6 ms | 71.4 ms | 242 crops/s (batch 80) | | Original manga-ocr package (fp16) | 84.9 ms | 242.4 ms | 42 crops/s (batch 56) | Both paths give identical outputs on all 7,789 boxes. ## Usage The inference code is on GitHub; the weights are downloaded from this repository on first use. ```bash pip install git+https://github.com/muscgab/manga-ocr-CPT ``` ```python from manga_ocr_cpt import MangaOcrCPT ocr = MangaOcrCPT("flex") # or a local directory holding this repository ocr("crop.png") # one crop -> str ocr(["crop1.png", "crop2.png"]) # the crops of one page -> list[str] ``` Inputs are cropped text boxes (file paths, PIL images or uint8 RGB arrays); there is no text detection. With an NVIDIA GPU the CUDA path is used, otherwise plain PyTorch (CPU, Apple MPS, ...). ## Training - Initialised from `kha-white/manga-ocr-base`; the 5,133 characters already in the original vocabulary keep their embeddings. - Data (same as manga-ocr-nar-small): 4,471,985 crops: synthetic printed text 1,795,869, synthetic handwriting 1,670,000, [AnimeText-OCR](https://huggingface.co/datasets/JustANormalTinkerer/animetext-ocr) real crops 1,006,116. Labels of 1–64 characters. - 139,749 steps, global batch 256 (8 epochs), 4 × RTX 2080 Ti; encoder fp16 mixed precision, decoder fp32. Released weights are from the last step. - Stopped by a false divergence alarm at step 126,200 and resumed from step 120,000 with a new data seed, so it ran 6,200 steps (about 1.59M samples) more than manga-ocr-base-CPT. ## Files | File | Contents | |---|---| | `model.safetensors` | Weights, fp32 (444 MB) | | `config.json` | Architecture and input settings (`input.fit` selects the input) | | `nar_vocabulary_6817.json` | Vocabulary (6,817 characters) | | `LICENSE_MODEL` | CC BY-NC-SA 4.0 | | `NOTICE.md` | Provenance and acknowledgements | | `SHA256SUMS` | Checksums | ## Limitations - Japanese only. At most 64 characters per box; longer boxes must be split first. Boxes of more than 32 characters are clearly weaker. - Handwriting and stylised lettering produce clearly more errors than printed text. - Aimed at dialogue, narration and small annotations; onomatopoeia is not a target. - The benchmark was built by the same author; parts of it were used during development. Not evaluated on scanned manga such as Manga109-s. ## License - Weights: CC BY-NC-SA 4.0 (`LICENSE_MODEL`); commercial use requires separate permission. The training data AnimeText-OCR is CC BY-NC-SA 4.0. - Initialised from kha-white/manga-ocr-base (Apache License 2.0). See `NOTICE.md`. - The inference code on GitHub is Apache License 2.0.