reference/parsebench/pipeline/README.md

ParseBench pipelines

Run Nemotron 3.5 Parse with single-, double-, or triple-pass inference, apply the bundled postprocessing, and evaluate with the official ParseBench scorer. Model weights, benchmark data, reference answers, and saved predictions are downloaded or generated separately.

Pipeline	Inference stages	Reproduced ParseBench score
Single	Full-page parse	72.72
Double	Full-page parse → Table/Chart crops from predicted boxes	74.88
Triple	Full-page parse → separate detection pass → Table/Chart crops	75.14
Triple, region union	Same stages, with additional full-page regions after overlap filtering	75.20

Scores are percentages, averaged over the five official category headlines, including Visual Grounding Element Pass Rate (EPR). These are measurements of the same checkpoint, not guarantees for every stochastic run. See expected_results.json and validation_metrics.json for exact scores, identities, and validation coverage.

The default triple-pass pipeline uses the separate detector's regions. To add Table/Chart regions found by the full-page parser, use --triple-regions union with run.py. This retains detector regions first, removes near-duplicates at IoU 0.85, suppresses added regions overlapping an original same-class box at IoU 0.5, and caps the region list at 80. It uses more crop requests. The matched full-dataset comparison improved by approximately 0.05 percentage points.

Postprocessing also removes literal repeated elements at the same coordinates, normalizes existing list bullets, removes discretionary soft hyphens between letters in textual roles, and repairs empty incomplete serialization prefixes. Code/math content is protected. Plain above-chart captions and original chart subtitles are preserved in their associated chart captions; notes below charts stay standalone. No new data values or axis/legend labels are invented.

Download and install

Use Linux, Python 3.12, and one NVIDIA Hopper H100 with 80 GB memory for the reference serving profile. Start in this pipeline directory. Authenticate with Hugging Face if needed.

python3.12 -m venv .serve-venv
source .serve-venv/bin/activate
python -m pip install 'huggingface-hub==1.24.0'

# Fetch the evaluated model/runtime. Use the helpers in this directory.
export PIPELINE="$(pwd)"
hf download nvidia/Nemotron-3.5-Parse \
  --revision e5e0406c85ec241b164907bf41ebd648a54e4ea2 \
  --exclude 'reference/**' '*.whl' --local-dir model

export MODEL="$(realpath model)"
export BENCH_WORK="$(pwd)/parsebench-work"
mkdir -p "$BENCH_WORK"
# Install the source from that exact historical revision. No wheel download is needed.
python -m pip install "${MODEL}[vllm,client]"
python -m pip install -r "$PIPELINE/requirements.txt"
python -m pip check

hf download llamaindex/ParseBench --repo-type dataset \
  --revision 2805a1d940f95a203e0ae4b88be9934f7765b3fc \
  --local-dir "$BENCH_WORK/public-data"

The packaged vLLM plugin supplies one-token MTP; no legacy sitecustomize or training checkout is needed. serve.py verifies the evaluated weight SHA-256 and key runtime versions before starting. PACKAGE_MANIFEST.json lists the bundled helper file hashes.

Prepare public data

Keep the scorer/data environment separate from serving:

python3.12 -m venv "$BENCH_WORK/score-venv"
source "$BENCH_WORK/score-venv/bin/activate"
python -m pip install -r "$PIPELINE/requirements-scorer-lock.txt"
python "$PIPELINE/prepare_data.py" \
  --snapshot "$BENCH_WORK/public-data" \
  --output "$BENCH_WORK/data" --workers 8

This prepares 2,078 unique inputs from the public PDFs and JSONL annotations. It sorts source PDF paths, assigns stable parsebench_00000 keys, and renders the first page of each source PDF at 150 DPI with PyMuPDF 1.26.3, RGB, and JPEG quality 92. Sources containing additional PDF pages are recorded in preparation.json; those additional pages are outside this measured input contract. The resulting images and annotations were checked against all 2,078 evaluated inputs with zero differences.

The output contains five compact train/*.tar shards, a per-image SHA-256 manifest, and a preparation receipt. Annotation data is used for dataset preparation and scoring; the inference client reads only page images and stable keys.

Start serving and run inference

Reactivate the serving environment from the original working directory:

source .serve-venv/bin/activate
python "$PIPELINE/prepare_inputs.py" --dataset-root "$BENCH_WORK/data" \
  --images-out "$BENCH_WORK/images" --manifest-out "$BENCH_WORK/inputs.jsonl"
python "$PIPELINE/serve.py" --model "$MODEL" --port 8000

In another terminal, restore the same MODEL, PIPELINE, and BENCH_WORK paths and activate the serving environment. Run all three pipelines:

python "$PIPELINE/run.py" \
  --images-dir "$BENCH_WORK/images" --manifest "$BENCH_WORK/inputs.jsonl" \
  --endpoint http://127.0.0.1:8000/v1 --mode all \
  --output "$BENCH_WORK/results"

To reproduce the 75.20 headline score, add --triple-regions union to the command above, or use --mode triple --triple-regions union to run only that profile.

Choose --mode single, --mode double, or --mode triple to run one pipeline. all shares the full-page pass across modes. Use a new output directory for each run. For a small smoke test, append:

--keys parsebench_00000,parsebench_00010,parsebench_00568,parsebench_01087,parsebench_01405,parsebench_01571,parsebench_02077

Default concurrency is 128. Full-page parsing and detection use temperature 0.2 and repetition penalty 1.05; crop parsing uses temperature 0.1 and repetition penalty 1.0. All requests use top-k 50, top-p 1.0, and seed 20261012. Output limits are 8,744 tokens for parsing and 4,096 for detection. Table/Chart crops use predicted boxes, a 4% plus 12-pixel margin, and JPEG quality 95. These settings are recorded in contract.json.

Crops are merged into the full-page output using ordered structural additions and guarded table replacement. All three modes automatically apply the same postprocessing:

• Convert predicted tables, charts, and semantic formatting into Markdown/HTML.
• Infer table headers from predicted spans and existing bold markup. No English header vocabulary or forced header bolding is applied.
• Preserve predicted panel and chart-caption context.
• For chart columns consistently containing numeric pairs such as 32 (16.6%), separate the existing values into Value and Percent subcolumns. This requires at least two matching nonempty cells and does not calculate or invent values.

These rules operate on predictions without benchmark categories, annotations, page-specific routing, or reference answers. They remain heuristics and can make errors on other document distributions.

Permanent outputs are compact JSONL/gzip files and receipts:

results/
  contract.json
  inputs.jsonl
  direct.jsonl.gz
  single/predictions.jsonl.gz
  double/predictions.jsonl.gz
  double/crop_predictions.jsonl
  triple/predictions.jsonl.gz
  triple/crop_predictions.jsonl
  triple/detection_predictions.jsonl.gz
  complete.json

Each final prediction includes key, raw_out, markdown, and coordinate_frame. Temporary crop images are deleted after each mode. Put --scratch on local storage when available. Timing receipts from all share an initial pass and should not be treated as independent throughput measurements.

Score with official ParseBench

The latest official scorer validated for this bundle is pinned to commit 9af4102e26b51f234ab5f037e19fec05c0696ec6 (October 16, 2026). Upstream source is downloaded separately and is not modified.

source "$BENCH_WORK/score-venv/bin/activate"
git clone https://github.com/run-llama/ParseBench.git "$BENCH_WORK/ParseBench"
git -C "$BENCH_WORK/ParseBench" checkout 9af4102e26b51f234ab5f037e19fec05c0696ec6

for mode in single double triple; do
  python "$PIPELINE/score.py" \
    --source "$BENCH_WORK/ParseBench/src" --dataset-root "$BENCH_WORK/data" \
    --predictions "$BENCH_WORK/results/$mode/predictions.jsonl.gz" \
    --output "$BENCH_WORK/results/$mode/score" --workers 16
done

The adapter passes the final Markdown and predicted geometry to the official evaluators. score.py verifies all 241 upstream Python source hashes, scores every page freshly with the default rule timeouts, and writes per-page results and independently checked aggregate means. Add --allow-partial only for smoke-test scoring. Full runs require 2,078 unique keys. Later upstream revisions require a separate validation and updated source hashes.

Expected finite metric counts are Tables 503, Charts 568, Content Faithfulness 506, Semantic Formatting 456, and Visual Grounding EPR 500. Formatting has 476 cases, of which 456 have applicable finite metrics. Errors are recorded and cause scoring to fail.

Replay postprocessing and validation

Reapply conversion to saved inference without a GPU or benchmark annotations:

source .serve-venv/bin/activate
python "$PIPELINE/postprocess.py" \
  --direct "$BENCH_WORK/results/direct.jsonl.gz" \
  --crops "$BENCH_WORK/results/triple/crop_predictions.jsonl" \
  --output "$BENCH_WORK/replay.jsonl.gz" --workers 8

Omit --crops for single-pass rendering; use the double-pass crops for double-pass replay.

The portable client generated all stages on all 2,078 public inputs using one H100 and the pinned release weights. Detector-only and union triple outputs share the same fresh full-page, detection and original-region crop responses. Every mode was scored freshly with the pinned official scorer. Public-data preparation reproduces the reference inputs byte-for-byte.

The corrected renderer was replayed on all 8,312 saved outputs without new inference. A conservative header rule changes four pages per mode while preserving raw predictions, geometry, cell text, order, and spans. Empty-title guards cover both chart-rendering paths. All four modes were then freshly scored with the pinned official scorer. validation_metrics.json preserves the original HTTP hashes separately from the final renderer and corrected prediction hashes.

Original licenses and notices are preserved in the accompanying files. SOURCE_MANIFEST.json records the extracted NVIDIA implementation sources and subsequent simplifications; PACKAGE_MANIFEST.json identifies this bundle.

License and environment notices

See this directory's license and third-party notices. The separately installed scoring environment is inventoried in benchmark_environment_components.json. The source, model runtime, scoring dependencies and benchmark data keep their respective licenses. The notice update does not change inference or scorer code.