Deploy initial accession explorer with a bounded GenBank annotation sample

.gitignore

@@ -0,0 +1,5 @@
+__pycache__/
+.venv/
+.env
+.cache/
+*.log

FEATURE_BACKLOG.md

@@ -0,0 +1,122 @@
+# Feature backlog
+
+Track desired features that are not yet implemented. Add new ideas here and update each item's status as work progresses. These entries describe future work; implementation details and priorities remain open unless explicitly agreed.
+
+Last updated: 2026-09-29.
+
+## Overview
+
+| ID | Feature | Status |
+| --- | --- | --- |
+| F001 | GenBank inference coverage map | Planned; not implemented |
+| F002 | Taxonomy map and navigation | Planned; not implemented |
+| F003 | Comparison with RefSeq annotations | Planned; not implemented |
+| F004 | Download annotations by accession, individually or in bulk | Planned; only individual sampled segment downloads exist |
+| F005 | Fast accession lookup and retrieval across the full dataset | Planned; only a local sample index exists |
+
+## F001 — GenBank inference coverage map
+
+**Goal:** Show how much of GenBank we currently have annotations for, and how that coverage grows as more inference completes.
+
+Desired behavior:
+
+- Provide a visual map or overview of coverage, with a way to explore covered and uncovered groups.
+- Show counts and proportions for annotated assemblies and bases, with the underlying totals visible.
+- Distinguish partial coverage from complete coverage, and distinguish the full annotation collection from the small sample loaded by the prototype.
+- Refresh the map as new inference outputs are published. Display when coverage was last updated.
+- Let users navigate from the overview to available assemblies and their annotations.
+
+Decisions and dependencies for implementation:
+
+- Define the GenBank inventory or snapshot used as the denominator, including any eligibility filters. Label these explicitly so the map's meaning stays clear as GenBank changes.
+- Define what counts as complete inference, including how to handle missing contigs, segments, or bases.
+- Build a coverage inventory from published outputs and inference metadata, deduplicating assembly versions and repeated runs.
+- Choose the visualization and refresh mechanism. Historical progress views are an optional extension.
+
+## F002 — Taxonomy map and navigation
+
+**Goal:** Provide a taxonomy-based view of the available genomes and annotations.
+
+Desired behavior:
+
+- Explore organisms through an interactive taxonomy map, with navigation from broad groups to individual taxa and assemblies.
+- Show how many assemblies or annotations are available within a selected group.
+- Let users open accession-level results from the map.
+- Consider sharing coverage information with F001 so users can see which taxonomic groups have been processed.
+
+Decisions and dependencies for implementation:
+
+- Choose a visualization, such as an expandable tree, treemap, or sunburst; no format is selected yet.
+- Choose and version a taxonomy source, and resolve assembly taxon IDs to lineages.
+- Define how to represent missing, unresolved, or changed taxon IDs.
+
+## F003 — Comparison with RefSeq annotations
+
+**Goal:** For sequences with RefSeq annotations, show where our predictions agree with or differ from those annotations.
+
+Desired behavior:
+
+- Display RefSeq annotation tracks alongside our strand-specific CDS probabilities in the same genomic region.
+- Highlight differences, with candidate categories including predicted coding regions outside RefSeq CDS features, RefSeq CDS regions with low predicted probability, and boundary or strand disagreements.
+- Let users inspect the underlying RefSeq features and prediction values for each difference.
+- Show the RefSeq annotation version and inference provenance used in the comparison.
+- Make unavailable or incompatible comparisons explicit.
+
+Decisions and dependencies for implementation:
+
+- Resolve GenBank records to the corresponding RefSeq assemblies and contigs, checking sequence versions and coordinate compatibility before overlaying tracks.
+- Decide how to handle sequences that require coordinate mapping rather than a direct overlay.
+- Define how probability tracks become comparable predicted intervals, including thresholds, strand handling, and boundary tolerance.
+- Select comparison metrics and feature scope, including how multiple transcripts and overlapping CDS features are treated.
+- Describe differences as disagreements with the reference annotations; a disagreement alone does not establish which annotation is correct.
+
+## F004 — Download annotations by accession
+
+**Goal:** Make it easy for users to download annotations for the assemblies or contigs they are interested in.
+
+Current baseline: the prototype can download one selected sampled segment as Parquet. Accession-level and bulk downloads are future work.
+
+Desired behavior:
+
+- Offer a download action directly from accession search results and assembly or contig views.
+- Let users select multiple accessions or paste/upload an accession list, then download the matching annotations together.
+- Include all available segments for a selected contig, and all available contigs for a selected assembly. Clearly flag incomplete coverage.
+- Show which requested accessions matched, were unavailable, or need a version choice. Record the exact versions included.
+- Include metadata, coordinate conventions, inference provenance, and a manifest of the exported contents.
+- Show estimated download size and progress; support preparing large exports without blocking browsing.
+
+Decisions and dependencies for implementation:
+
+- Choose export formats and packaging. Preserve original per-base probabilities in Parquet; consider additional formats for derived intervals once their interpretation is defined.
+- Define handling of duplicates, multiple inference runs, accession versions, and partial failures in bulk requests.
+- Decide whether sequence is included when available, and label its availability explicitly.
+- Define export size limits, temporary storage, and download expiration.
+- Use F005 to locate and retrieve requested records efficiently across the full collection.
+
+## F005 — Fast lookup and retrieval across the full dataset
+
+**Goal:** Keep accession searches and annotation retrieval responsive as the app expands beyond its bundled sample.
+
+Current baseline: the prototype indexes local sample metadata and reads selected Parquet row groups. It does not search the full bucket.
+
+Desired behavior:
+
+- Resolve assembly and contig accessions through a persistent metadata index, without scanning every annotation shard for each query.
+- Map each match to its exact accession version, inference run, bucket object, row group, and segment coordinates.
+- Retrieve only the data needed for the selected records or region, where the storage layout supports it.
+- Support bulk resolution for F004, and paginate large assembly results.
+- Update the index as inference outputs are published, showing the indexed snapshot or refresh time.
+- Distinguish an accession absent from the indexed inventory from an incomplete or stale index and from a retrieval failure.
+
+Decisions and dependencies for implementation:
+
+- Benchmark lookup latency, data transfer, and memory use on representative workloads before selecting an index backend or changing the storage layout.
+- Evaluate an accession-to-object/row-group index and metadata queries separately from fetching the large probability arrays.
+- Check remote range-read support and actual row-group sizes. A coordinate index alone does not guarantee efficient region reads; consider smaller genomic chunks or another layout if needed.
+- Define incremental indexing and consistent publication so index entries refer to available, identifiable object versions. Handle replacements and repeated inference runs explicitly.
+- Consider bounded caching of frequently accessed metadata and annotation chunks, with invalidation tied to data versions.
+- Share versioned inventory metadata with F001 and F002 where useful, keeping coverage summaries and lookup results consistent.
+
+## Adding and updating entries
+
+Give each new feature a stable ID, a goal, a status, desired behavior, and open decisions or dependencies. Link implementation work when it starts. Once shipped, mark the entry implemented and record any remaining scope rather than deleting its history.

README.md

@@ -1,13 +1,71 @@
 ---
-title: Genbank Annotation Explorer
-emoji: 👀
-colorFrom: blue
+title: GenBank Annotation Explorer
+emoji: 🧬
+colorFrom: green
 colorTo: blue
 sdk: gradio
-sdk_version: 6.29.0
-python_version: '3.13'
+sdk_version: 6.6.0
+python_version: '3.10'
 app_file: app.py
 pinned: false
+short_description: Explore sampled genome annotations by accession ID
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+Search a small, real subset of [HuggingFaceBio/genbank-annotations](https://huggingface.co/buckets/HuggingFaceBio/genbank-annotations) by assembly accession, contig accession, or the exact `source_key`. View strand-specific CDS probabilities, inspect provenance, and download original segment rows as Parquet.
+
+Future features and open design decisions are tracked in the [feature backlog](FEATURE_BACKLOG.md).
+
+## Run locally
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+pip install -r requirements.txt
+python app.py
+```
+
+Open http://localhost:7860. The bundled sample works offline and requires no token at runtime.
+
+## Deploy to a Space
+
+The preview Space is [HuggingFaceBio/genbank-annotation-explorer](https://huggingface.co/spaces/HuggingFaceBio/genbank-annotation-explorer), with private visibility.
+
+To deploy or update it, upload this directory, including `data/sample.parquet` and `data/manifest.json`. The README metadata configures the runtime, following the [Gradio Spaces documentation](https://huggingface.co/docs/hub/spaces-sdks-gradio). Keep the Space private for the initial preview: the bundled annotations will be accessible to its viewers.
+
+## Lookup behavior
+
+- Exact versioned accessions match only that version; unknown versions do not fall back.
+- Unversioned accessions return all matching versions in the sample. Input ignores case and surrounding whitespace.
+- Assembly lookups return all sampled contigs/segments for that assembly. This sample is not a complete assembly.
+- A miss means “not in this sample,” not “absent from GenBank/the bucket.”
+- Split contigs stay as separate segments, preserving their source coordinates and segment counts.
+
+## Data and coordinates
+
+The bucket contains model predictions: positive- and negative-strand per-base P(CDS), metadata, and sometimes sequence. It does not provide curated gene names or gene models. Coordinates are 0-based, end-exclusive. The initial viewport shows up to 10,000 bases; update the start/end fields to explore other regions. Large windows are averaged into at most 1,200 bins per strand. Downloads retain original probabilities and source columns without downsampling.
+
+`data/manifest.json` records the bucket object, content hash, source row indices, selection limits, and sample checksum. The default sample is limited to 24 complete segments and 2 million bases from one fungi chunk. It is for interface development, not a representative biological sample. Full per-base columns are loaded only for the selected segment; the accession index holds metadata only.
+
+## Rebuild the sample
+
+Use a Hugging Face login or `HF_TOKEN` with bucket read access. Never commit tokens.
+
+```bash
+pip install -r requirements-data.txt
+hf auth login
+python prepare_sample.py
+```
+
+The default source object is about 31 MB; automatic downloads are capped at 100 MB. Override `--source`, `--max-rows`, or `--max-bp` to choose another bounded sample. If the source already exists locally:
+
+```bash
+python prepare_sample.py --local-file /path/to/source.parquet
+```
+
+Source rows are selected intact, without truncating their probabilities. The app never scans or downloads the full bucket at startup. To support arbitrary accessions across the full bucket later, build a separate metadata index mapping accession → bucket object → row group, then retrieve matching probability data on demand.
+
+## Checks
+
+```bash
+python -m unittest discover -s tests -v
+```

app.py

@@ -0,0 +1,101 @@
+"""A small Hugging Face Space for exploring genome annotations by accession."""
+from pathlib import Path
+import os
+import tempfile
+
+# Shared compute hosts can have /tmp/gradio owned by a different user.
+os.environ.setdefault("GRADIO_TEMP_DIR", str(Path(tempfile.gettempdir()) / f"genbank-explorer-{os.getuid()}"))
+
+import gradio as gr
+import pyarrow.parquet as pq
+
+from catalog import Catalog
+
+
+def build_app(catalog=None):
+    catalog = catalog or Catalog()
+    all_ids = list(range(len(catalog.records)))
+    first = catalog.records[0]
+
+    def search(accession):
+        ids = catalog.lookup(accession)
+        choices = [(f"{catalog.records[i]['record_name']} · {catalog.records[i]['assembly_accession']} · "
+                    f"[{catalog.records[i]['segment_start_bp']:,}, {catalog.records[i]['segment_end_bp']:,})", str(i)) for i in ids]
+        if not str(accession or "").strip():
+            message = "Enter an assembly or contig accession. Try an example below."
+        elif not ids:
+            message = "No match in this sample. This does not mean the accession is absent from the full bucket."
+        else:
+            message = f"Found **{len(ids)} sampled segment(s)**. Assembly results include only contigs loaded in this prototype."
+        return message, catalog.table(ids), gr.Dropdown(choices=choices, value=str(ids[0]) if ids else None), None
+
+    def select_segment(index):
+        if index is None:
+            return {}, None, None, None, "Choose a matching segment.", None
+        record = catalog.records[int(index)]
+        start = record["segment_start_bp"]
+        end = min(record["segment_end_bp"], start + 10_000)
+        plot, step = catalog.window(index, start, end)
+        return record, start, end, plot, plot_note(step), None
+
+    def plot_note(step):
+        return ("Coordinates are **0-based, end-exclusive**. "
+                + ("Each point is one base." if step == 1 else f"Each point is the mean of up to **{step:,} bases**; short peaks can be smoothed.")
+                + " Download the segment for the original per-base probabilities.")
+
+    def update_window(index, start, end):
+        if index is None:
+            raise gr.Error("Look up an accession and choose a segment first.")
+        try:
+            plot, step = catalog.window(index, start, end)
+        except (ValueError, TypeError, OverflowError) as exc:
+            raise gr.Error(str(exc)) from exc
+        return plot, plot_note(step)
+
+    def export(index):
+        if index is None:
+            raise gr.Error("Choose a segment first.")
+        target = Path(tempfile.mkdtemp(prefix="genbank-export-")) / "annotation.parquet"
+        pq.write_table(catalog.segment_table(index), target, compression="zstd")
+        return str(target)
+
+    with gr.Blocks(title="GenBank Annotation Explorer", delete_cache=(3600, 3600)) as demo:
+        gr.Markdown("# GenBank Annotation Explorer\nExplore predicted coding regions by **assembly or contig accession**.")
+        gr.Markdown(f"**Prototype sample · {len(catalog.records):,} segments · "
+                    f"{len(catalog.manifest['assemblies']):,} assemblies · {catalog.manifest['bases']:,} bases**\n\n"
+                    "Source: [HuggingFaceBio/genbank-annotations](https://huggingface.co/buckets/HuggingFaceBio/genbank-annotations). "
+                    "These are model-predicted CDS probabilities, not curated gene features. Search covers the loaded sample only.")
+        with gr.Row():
+            accession = gr.Textbox(label="Accession ID", placeholder="Assembly (GCA_…) or contig accession", scale=5)
+            search_button = gr.Button("Find annotations", variant="primary", scale=1)
+        gr.Examples(examples=[[first["assembly_accession"]], [first["record_name"]],
+                              [first["record_name"].rsplit(".", 1)[0]]], inputs=accession)
+        status = gr.Markdown("Enter an accession or select an example. IDs are case-insensitive; version suffixes are optional.")
+        results = gr.Dataframe(value=catalog.table([]), interactive=False, label="Matching segments")
+        segment = gr.Dropdown(choices=[], label="Segment to explore", interactive=True)
+        with gr.Row():
+            start = gr.Number(label="Start (0-based, inclusive)", precision=0)
+            end = gr.Number(label="End (exclusive)", precision=0)
+            view = gr.Button("Update region")
+        plot = gr.LinePlot(x="Position (bp)", y="P(CDS)", color="Strand", y_lim=[0, 1],
+                           title="CDS probability by strand", height=320)
+        note = gr.Markdown("Choose a segment to see its probability tracks.")
+        with gr.Accordion("Segment metadata and provenance", open=False):
+            metadata = gr.JSON(label="Source metadata")
+        with gr.Row():
+            download = gr.Button("Prepare segment download")
+            file = gr.File(label="Original segment annotations (Parquet)", interactive=False)
+        with gr.Accordion("Accessions available in this sample", open=False):
+            gr.Dataframe(value=catalog.table(all_ids), interactive=False)
+            gr.JSON(value=catalog.manifest, label="Sample provenance")
+        outputs = [status, results, segment, file]
+        for event in (search_button.click, accession.submit):
+            event(search, accession, outputs).then(select_segment, segment, [metadata, start, end, plot, note, file])
+        segment.input(select_segment, segment, [metadata, start, end, plot, note, file])
+        view.click(update_window, [segment, start, end], [plot, note])
+        download.click(export, segment, file)
+    return demo
+
+
+if __name__ == "__main__":
+    build_app().queue(default_concurrency_limit=2).launch(server_name="0.0.0.0")

catalog.py

@@ -0,0 +1,84 @@
+"""Accession lookup and lazy, segment-level access to the local sample."""
+from collections import defaultdict
+import json
+from pathlib import Path
+import re
+
+import numpy as np
+import pandas as pd
+import pyarrow.parquet as pq
+
+ROOT = Path(__file__).resolve().parent
+PROBS = ["pred_prob_positive_strand_cds", "pred_prob_negative_strand_cds"]
+DISPLAY_COLUMNS = ["assembly_accession", "record_name", "organism_name", "division",
+                   "segment_start_bp", "segment_end_bp", "segment_index", "segment_count"]
+
+
+def normalize(value):
+    return str(value or "").strip().upper()
+
+
+def unversioned(value):
+    return re.sub(r"\.\d+$", "", value)
+
+
+class Catalog:
+    def __init__(self, directory=ROOT / "data"):
+        directory = Path(directory)
+        self.path = directory / "sample.parquet"
+        self.manifest = json.loads((directory / "manifest.json").read_text())
+        pf = pq.ParquetFile(self.path)
+        columns = [c for c in pf.schema_arrow.names if c not in PROBS and c != "sequence"]
+        self.records = pf.read(columns=columns).to_pylist()
+        self.locations = [(g, r) for g in range(pf.num_row_groups)
+                          for r in range(pf.metadata.row_group(g).num_rows)]
+        self.exact, self.base = defaultdict(set), defaultdict(set)
+        for i, record in enumerate(self.records):
+            for field in ("assembly_accession", "record_name", "source_key"):
+                key = normalize(record[field])
+                if key:
+                    self.exact[key].add(i)
+                    if field != "source_key":
+                        self.base[unversioned(key)].add(i)
+
+    def lookup(self, accession):
+        key = normalize(accession)
+        if not key:
+            return []
+        # A versioned query never silently falls back to another version.
+        matches = self.exact.get(key, set()) if re.search(r"\.\d+$", key) or "|" in key else self.base.get(key, set())
+        return sorted(matches, key=lambda i: (self.records[i]["assembly_accession"],
+                                             self.records[i]["record_name"],
+                                             self.records[i]["segment_index"]))
+
+    def table(self, ids):
+        return pd.DataFrame([self.records[i] for i in ids], columns=DISPLAY_COLUMNS)
+
+    def segment_table(self, index):
+        index = int(index)
+        if not 0 <= index < len(self.records):
+            raise ValueError("Choose a loaded segment.")
+        group, row = self.locations[index]
+        return pq.ParquetFile(self.path).read_row_group(group).slice(row, 1)
+
+    def window(self, index, start=None, end=None, max_points=1200):
+        record = self.records[int(index)]
+        lo, hi = record["segment_start_bp"], record["segment_end_bp"]
+        start = lo if start is None else int(start)
+        end = min(hi, start + 10_000) if end is None else int(end)
+        if not lo <= start < end <= hi:
+            raise ValueError(f"Enter a range within [{lo:,}, {hi:,}) with start < end.")
+        table = self.segment_table(index)
+        width = end - start
+        step = max(1, (width + max_points - 1) // max_points)
+        positions = np.arange(start, end, step)
+        frames = []
+        for column, strand in zip(PROBS, ["+ strand", "− strand"]):
+            values = table.column(column)[0].values.to_numpy(zero_copy_only=False).astype(np.float32)
+            if len(values) != hi - lo:
+                raise ValueError("Probability length does not match segment coordinates.")
+            values = values[start - lo:end - lo]
+            counts = np.minimum(step, width - np.arange(0, width, step))
+            means = np.add.reduceat(values, np.arange(0, width, step)) / counts
+            frames.append(pd.DataFrame({"Position (bp)": positions, "P(CDS)": means, "Strand": strand}))
+        return pd.concat(frames, ignore_index=True), step

data/manifest.json

@@ -0,0 +1,30 @@
+{
+  "bucket_id": "HuggingFaceBio/genbank-annotations",
+  "source_path": "annotations/fungi/s0000/s0000.chunk-00510.parquet",
+  "source_xet_hash": "f7454034faf6e7e0c9f80f51355269eafbf5f8447b3aa41166fb7558797dafb9",
+  "created_at": "2026-09-29T17:05:37.431500+00:00",
+  "selection": "First complete segments fitting the row and base budgets, in source order",
+  "max_rows": 24,
+  "max_bp": 2000000,
+  "rows": 12,
+  "bases": 1969534,
+  "source_rows": 48,
+  "source_row_indices": [
+    0,
+    1,
+    2,
+    3,
+    4,
+    5,
+    6,
+    7,
+    8,
+    9,
+    11,
+    46
+  ],
+  "assemblies": [
+    "GCA_055472895.1"
+  ],
+  "sha256": "f1128a4ee3215ceb51fe9ddbd6ae2228ce20e56a60579ee9b22011b945460b52"
+}

data/sample.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f1128a4ee3215ceb51fe9ddbd6ae2228ce20e56a60579ee9b22011b945460b52
+size 6999680

prepare_sample.py

@@ -0,0 +1,86 @@
+"""Extract a bounded, reproducible sample from one published bucket object."""
+import argparse
+from datetime import datetime, timezone
+import hashlib
+import json
+from pathlib import Path
+import tempfile
+
+import pyarrow as pa
+import pyarrow.parquet as pq
+
+BUCKET = "HuggingFaceBio/genbank-annotations"
+DEFAULT_SOURCE = "annotations/fungi/s0000/s0000.chunk-00510.parquet"
+ROOT = Path(__file__).resolve().parent
+
+
+def extract(source, output, source_path, max_rows=24, max_bp=2_000_000, xet_hash=None):
+    if max_rows < 1 or max_bp < 1:
+        raise ValueError("Sample limits must be positive.")
+    pf = pq.ParquetFile(source)
+    selected, bases = [], 0
+    # Read lengths before loading the much larger per-base columns.
+    lengths = pf.read(columns=["segment_bp_length"]).column(0).to_pylist()
+    for i, length in enumerate(lengths):
+        if 0 < length <= max_bp - bases:
+            selected.append(i)
+            bases += length
+        if len(selected) >= max_rows:
+            break
+    if not selected:
+        raise ValueError("No complete segments fit the sample budget.")
+    pieces, offset = [], 0
+    for group in range(pf.num_row_groups):
+        count = pf.metadata.row_group(group).num_rows
+        indices = [i - offset for i in selected if offset <= i < offset + count]
+        if indices:
+            pieces.append(pf.read_row_group(group).take(pa.array(indices)))
+        offset += count
+    sample = pa.concat_tables(pieces)
+    output = Path(output)
+    output.mkdir(parents=True, exist_ok=True)
+    target = output / "sample.parquet"
+    # One group per segment lets the app fetch just the selected segment.
+    pq.write_table(sample, target, compression="zstd", row_group_size=1)
+    metadata = sample.select([c for c in sample.column_names if not c.startswith("pred_prob_") and c != "sequence"]).to_pylist()
+    manifest = {
+        "bucket_id": BUCKET, "source_path": source_path, "source_xet_hash": xet_hash,
+        "created_at": datetime.now(timezone.utc).isoformat(),
+        "selection": "First complete segments fitting the row and base budgets, in source order",
+        "max_rows": max_rows, "max_bp": max_bp, "rows": sample.num_rows,
+        "bases": bases, "source_rows": pf.metadata.num_rows, "source_row_indices": selected,
+        "assemblies": sorted({r["assembly_accession"] for r in metadata}),
+        "sha256": hashlib.sha256(target.read_bytes()).hexdigest(),
+    }
+    (output / "manifest.json").write_text(json.dumps(manifest, indent=2) + "\n")
+    print(json.dumps(manifest, indent=2))
+    return manifest
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--source", default=DEFAULT_SOURCE)
+    parser.add_argument("--local-file", type=Path)
+    parser.add_argument("--output", type=Path, default=ROOT / "data")
+    parser.add_argument("--max-rows", type=int, default=24)
+    parser.add_argument("--max-bp", type=int, default=2_000_000)
+    args = parser.parse_args()
+    if args.local_file:
+        extract(args.local_file, args.output, args.source, args.max_rows, args.max_bp)
+    else:
+        from huggingface_hub import HfApi
+        api = HfApi()
+        entries = api.list_bucket_tree(BUCKET, args.source, recursive=False)
+        entry = next((e for e in entries if e.path == args.source), None)
+        if entry is None:
+            raise ValueError(f"Bucket object not found: {args.source}")
+        if entry.size > 100_000_000:
+            raise ValueError("Source exceeds the 100 MB download budget. Choose a smaller object or use --local-file.")
+        with tempfile.TemporaryDirectory() as temp:
+            local = Path(temp) / "source.parquet"
+            api.download_bucket_files(BUCKET, [(entry, local)], raise_on_missing_files=True)
+            extract(local, args.output, args.source, args.max_rows, args.max_bp, entry.xet_hash)
+
+
+if __name__ == "__main__":
+    main()

requirements-data.txt

@@ -0,0 +1,2 @@
+-r requirements.txt
+huggingface_hub==1.16.0

requirements.txt

@@ -0,0 +1,4 @@
+gradio==6.6.0
+numpy>=1.26,<3
+pandas>=2.2,<3
+pyarrow>=18,<24

tests/test_catalog.py

@@ -0,0 +1,89 @@
+import hashlib
+import json
+from pathlib import Path
+import tempfile
+import unittest
+
+import numpy as np
+import pyarrow as pa
+import pyarrow.parquet as pq
+
+from catalog import Catalog, ROOT
+
+
+class SampleTests(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls):
+        cls.catalog = Catalog()
+
+    def test_sample_integrity_and_budget(self):
+        manifest = self.catalog.manifest
+        self.assertEqual(hashlib.sha256(self.catalog.path.read_bytes()).hexdigest(), manifest['sha256'])
+        self.assertEqual(len(self.catalog.records), manifest['rows'])
+        self.assertLessEqual(manifest['rows'], manifest['max_rows'])
+        self.assertLessEqual(manifest['bases'], manifest['max_bp'])
+        for i, record in enumerate(self.catalog.records):
+            row = self.catalog.segment_table(i).to_pylist()[0]
+            length = record['segment_end_bp'] - record['segment_start_bp']
+            for key in ('pred_prob_positive_strand_cds', 'pred_prob_negative_strand_cds'):
+                values = np.asarray(row[key], dtype=float)
+                self.assertEqual(len(values), length)
+                self.assertTrue(np.all(np.isfinite(values)))
+                self.assertTrue(np.all((values >= 0) & (values <= 1)))
+
+    def test_real_accession_lookup(self):
+        first = self.catalog.records[0]
+        self.assertIn(0, self.catalog.lookup(first['source_key']))
+        self.assertIn(0, self.catalog.lookup('  ' + first['record_name'].lower() + ' '))
+        self.assertIn(0, self.catalog.lookup(first['record_name'].rsplit('.', 1)[0]))
+        expected = [i for i, r in enumerate(self.catalog.records) if r['assembly_accession'] == first['assembly_accession']]
+        self.assertCountEqual(self.catalog.lookup(first['assembly_accession']), expected)
+        self.assertEqual(self.catalog.lookup(first['record_name'].rsplit('.', 1)[0] + '.999999'), [])
+        self.assertEqual(self.catalog.lookup(''), [])
+        self.assertEqual(self.catalog.lookup('not-an-accession'), [])
+
+
+class CoordinateTests(unittest.TestCase):
+    def setUp(self):
+        self.temp = tempfile.TemporaryDirectory()
+        directory = Path(self.temp.name)
+        rows = []
+        for version in (1, 2):
+            rows.append({'assembly_accession': f'GCA_000001.{version}',
+                         'record_name': f'TEST000001.{version}',
+                         'source_key': f'GCA_000001.{version}|TEST000001.{version}',
+                         'segment_index': 1, 'segment_start_bp': 100,
+                         'segment_end_bp': 105,
+                         'pred_prob_positive_strand_cds': [0., 0.2, 0.4, 0.6, 0.8],
+                         'pred_prob_negative_strand_cds': [1., 0.8, 0.6, 0.4, 0.2]})
+        pq.write_table(pa.Table.from_pylist(rows), directory / 'sample.parquet')
+        (directory / 'manifest.json').write_text('{}')
+        self.catalog = Catalog(directory)
+
+    def tearDown(self):
+        self.temp.cleanup()
+
+    def test_version_resolution(self):
+        self.assertEqual(self.catalog.lookup('GCA_000001'), [0, 1])
+        self.assertEqual(self.catalog.lookup('GCA_000001.2'), [1])
+        self.assertEqual(self.catalog.lookup('TEST000001'), [0, 1])
+        self.assertEqual(self.catalog.lookup('GCA_000001.3'), [])
+
+    def test_absolute_coordinates_and_partial_last_bin(self):
+        frame, step = self.catalog.window(0, 100, 105, max_points=2)
+        self.assertEqual(step, 3)
+        positive = frame[frame.Strand == '+ strand']
+        self.assertEqual(positive['Position (bp)'].tolist(), [100, 103])
+        np.testing.assert_allclose(positive['P(CDS)'], [0.2, 0.7], atol=1e-6)
+        frame, step = self.catalog.window(0, 103, 105)
+        self.assertEqual(step, 1)
+        np.testing.assert_allclose(frame[frame.Strand == '+ strand']['P(CDS)'], [0.6, 0.8])
+
+    def test_invalid_ranges(self):
+        for start, end in [(99, 103), (100, 106), (103, 103), (104, 102)]:
+            with self.assertRaises(ValueError):
+                self.catalog.window(0, start, end)
+
+
+if __name__ == '__main__':
+    unittest.main()