Publish reviewed Realtime-Venus configurations, HEARTBEAT index and documentation

README.md

@@ -155,10 +155,11 @@ python download_models.py --model all --local-dir .
 ```
 
 Use `--model omni` or `--model audio` to download only the model you need.
-The downloader reads this repository's root `config.yaml` and downloads each
-selected model's complete directory, including weights, custom code, and
-assets. It saves a download manifest and uses the Hugging Face Hub's standard
-progress display and cache. Each download uses one repository revision.
+The downloader first fetches root `config.yaml`, which contains both models'
+complete configurations and directory mapping, then downloads the selected
+model directories in full, including weights, custom code, and assets.
+Each download uses one repository revision and the Hugging Face Hub's
+standard progress display and cache.
 
 Install the inference dependencies after downloading. For Omni or `all`:
 
@@ -185,8 +186,49 @@ paths = download_models(model="omni", local_dir=".")  # "omni", "audio", or "all
 model_dir = paths["omni"]  # pathlib.Path; use paths["audio"] for Audio
 ```
 
-The inference examples below load the downloaded local model directories. Run
-them from the same directory; all asset and output paths are relative to it.
+Each model directory's `config.json` references the complete configuration in
+root `config.yaml` at a fixed revision. During loading, the configuration class
+first checks for a matching YAML beside the model directories, then checks the
+Hub cache, and downloads the pinned file if needed and network access is allowed.
+The examples below load `./Realtime-Venus-Omni` or `./Realtime-Venus-Audio`; run
+them from the download directory, with asset and output paths relative to it.
+
+### First load with automatic root configuration download
+
+If you use `snapshot_download` to fetch only one model subdirectory, keep your
+existing download code and `allow_patterns` unchanged. For the first
+`AutoModel.from_pretrained` call, omit `local_files_only` or set it to `False`
+when the matching root YAML is not yet available locally or in the cache.
+The model configuration class will download and validate the referenced YAML;
+there is no need to download the other model directory.
+
+Alternatively, prepare only the configuration before using the offline model
+initialization examples below. This step does not load model weights:
+
+```python
+from transformers import AutoConfig
+
+model_dir = "./Realtime-Venus-Audio"  # or "./Realtime-Venus-Omni"
+config = AutoConfig.from_pretrained(model_dir, trust_remote_code=True)
+```
+
+### Offline loading after preparation
+
+The initialization examples below use `local_files_only=True`. They require the
+matching root YAML to be present beside the model directories or in the Hub
+cache, as well as the downloaded model files. If it is missing, loading fails
+with an explicit error; offline mode does not silently download it or fall back
+to another revision. `HF_HUB_OFFLINE=1` and `TRANSFORMERS_OFFLINE=1` also disable
+network access; set these variables before importing the libraries.
+
+If you pass `cache_dir`, use the same directory for preparation and later
+loading. Otherwise the loader follows Transformers' default cache configuration,
+including the legacy `TRANSFORMERS_CACHE` setting. Root downloads also preserve
+an explicitly supplied `proxies` argument. The root YAML's Hub cache is separate
+from the `.cache/huggingface` download metadata under a snapshot's `local_dir`.
+When moving a model to an offline machine, also provide the matching root YAML
+beside its directory or transfer the corresponding Hub cache. Copying only the
+model subdirectory does not transfer that cached dependency.
 
 As an independent alternative, the ModelScope CLI (installed separately) can
 download the entire mirror repository:
@@ -681,7 +723,11 @@ print("Saved generated speech to output/audio_full_duplex.wav")
 
 </details>
 
-## 9. 📝 Citation
+## 9. Data
+
+HEARTBEAT Challenge evaluates when to respond and what to say in continuous audio or audio-visual interaction, covering reciprocal audio turn-taking and streaming proactive understanding. The challenge grew out of Realtime-Venus; see the [data README](./data/HEARTBEAT_Challenge/README.md) for task details, data organization, and loading instructions.
+
+## 10. 📝 Citation
 
 If you find Realtime-Venus useful, please cite the technical report:
 
@@ -694,7 +740,7 @@ If you find Realtime-Venus useful, please cite the technical report:
 }
 ```
 
-## 10. 📄 License
+## 11. 📄 License
 
 This repository includes an [Apache License 2.0](./LICENSE). Please also review
 the licenses and acceptable-use terms of the upstream model, third-party

README_zh.md

@@ -112,7 +112,7 @@ hf download inclusionAI/Realtime-Venus download_models.py --local-dir .
 python download_models.py --model all --local-dir .
 ```
 
-如只需一款模型，将参数改为 `--model omni` 或 `--model audio`。下载器读取本仓库根目录的 `config.yaml`，下载所选模型的完整子目录，包括权重、自定义代码和资源文件；同时保存下载清单，并使用 Hugging Face Hub 的标准进度显示与缓存。单次下载中的所有文件均来自同一个仓库版本。
+如只需一款模型，将参数改为 `--model omni` 或 `--model audio`。下载器先获取根 `config.yaml`（包含两款模型的完整配置和目录映射），再下载所选模型的完整子目录，包括权重、自定义代码和资源文件。单次下载中的所有文件均来自同一个仓库版本，并使用 Hugging Face Hub 的标准进度显示与缓存。
 
 下载后安装推理依赖。选择 Omni 或 `all` 时运行：
 
@@ -136,7 +136,26 @@ paths = download_models(model="omni", local_dir=".")  # 可选 "omni"、"audio"
 model_dir = paths["omni"]  # pathlib.Path；Audio 使用 paths["audio"]
 ```
 
-下文推理示例直接加载下载后的本地模型目录。请在同一目录中运行，资源文件和输出路径均相对于当前目录。
+模型子目录中的 `config.json` 引用固定版本根 `config.yaml` 中的完整配置。加载时，配置类先检查模型目录父级是否有匹配的 YAML，再查找 Hub 缓存；仍缺失且允许联网时，自动下载引用指定的文件。下文示例分别从 `./Realtime-Venus-Omni` 或 `./Realtime-Venus-Audio` 加载模型；请在下载目录中运行，资源文件和输出路径均相对于该目录。
+
+### 首次加载：允许自动获取根配置
+
+如果通过 `snapshot_download` 仅下载一款模型的子目录，原下载代码和 `allow_patterns` 均无需修改。匹配的根 YAML 尚未在本地或缓存中准备好时，首次调用 `AutoModel.from_pretrained` 应省略 `local_files_only`，或将其设为 `False`。配置类会自动下载并校验引用的根 YAML，无需下载另一款模型目录。
+
+也可以先只解析并缓存配置，再执行下文的离线模型初始化示例。这一步不加载模型权重：
+
+```python
+from transformers import AutoConfig
+
+model_dir = "./Realtime-Venus-Audio"  # 或 "./Realtime-Venus-Omni"
+config = AutoConfig.from_pretrained(model_dir, trust_remote_code=True)
+```
+
+### 配置准备好后：离线加载
+
+下文初始化示例使用 `local_files_only=True`，前提是模型文件已下载完整，且模型目录父级或 Hub 缓存中已有匹配的根 YAML。缺少时会明确报错；离线模式不会偷偷联网，也不会回退到其他版本。`HF_HUB_OFFLINE=1` 和 `TRANSFORMERS_OFFLINE=1` 同样禁止联网，请在导入库之前设置。
+
+若显式传入 `cache_dir`，首次准备和后续加载应使用同一目录；未传入时，解析器遵循 Transformers 的默认缓存设置，包括旧的 `TRANSFORMERS_CACHE`。根文件下载也会保留调用方显式传入的 `proxies`。根 YAML 所用的 Hub 缓存，与 snapshot 的 `local_dir` 下 `.cache/huggingface` 下载元数据目录是两套机制。搬移到离线机器时，需要把匹配的根 YAML 一并放在模型目录父级，或迁移对应 Hub 缓存；仅复制模型子目录不会同时带走该缓存依赖。
 
 也可以独立使用 ModelScope CLI（需另行安装）下载整个镜像仓库：
 
@@ -594,7 +613,11 @@ print("Saved generated speech to output/audio_full_duplex.wav")
 
 </details>
 
-## 9. 📝 引用
+## 9. Data
+
+HEARTBEAT Challenge 评测系统在连续音频或音视频输入中判断何时回应、说什么的能力，涵盖双向音频轮次交互与流式主动理解两个赛道。该挑战赛由 Realtime-Venus 发展而来，具体任务、数据组织和读取方法见[数据 README](./data/HEARTBEAT_Challenge/README_zh.md)。
+
+## 10. 📝 引用
 
 如果 Realtime-Venus 对你的工作有帮助，请引用技术报告：
 
@@ -607,6 +630,6 @@ print("Saved generated speech to output/audio_full_duplex.wav")
 }
 ```
 
-## 10. 📄 许可证
+## 11. 📄 许可证
 
 本仓库包含 [Apache License 2.0](./LICENSE)。另请查阅上游模型、第三方库，以及与本模型配合使用的数据各自的许可证和可接受使用条款。

Realtime-Venus-Audio/config.json

@@ -1,323 +1,19 @@
 {
+  "model_type": "minicpmo",
   "architectures": [
     "MiniCPMO"
   ],
-  "attention_bias": false,
-  "attention_dropout": 0.0,
-  "audio_chunk_length": 1.0,
-  "audio_config": {
-    "_attn_implementation_autoset": true,
-    "_name_or_path": "openai/whisper-medium",
-    "activation_dropout": 0.0,
-    "activation_function": "gelu",
-    "apply_spec_augment": false,
-    "architectures": [
-      "MiniCPMWhisperEncoder"
-    ],
-    "attention_dropout": 0.0,
-    "begin_suppress_tokens": [
-      220,
-      50257
-    ],
-    "bos_token_id": 50257,
-    "classifier_proj_size": 256,
-    "d_model": 1024,
-    "decoder_attention_heads": 16,
-    "decoder_ffn_dim": 4096,
-    "decoder_layerdrop": 0.0,
-    "decoder_layers": 24,
-    "decoder_start_token_id": 50258,
-    "dropout": 0.0,
-    "dtype": "float32",
-    "encoder_attention_heads": 16,
-    "encoder_ffn_dim": 4096,
-    "encoder_layerdrop": 0.0,
-    "encoder_layers": 24,
-    "eos_token_id": 50257,
-    "forced_decoder_ids": [
-      [
-        1,
-        50259
-      ],
-      [
-        2,
-        50359
-      ],
-      [
-        3,
-        50363
-      ]
-    ],
-    "init_std": 0.02,
-    "mask_feature_length": 10,
-    "mask_feature_min_masks": 0,
-    "mask_feature_prob": 0.0,
-    "mask_time_length": 10,
-    "mask_time_min_masks": 2,
-    "mask_time_prob": 0.05,
-    "max_length": 448,
-    "max_source_positions": 1500,
-    "max_target_positions": 448,
-    "median_filter_width": 7,
-    "model_type": "whisper",
-    "num_hidden_layers": 24,
-    "num_mel_bins": 80,
-    "pad_token_id": 50257,
-    "scale_embedding": false,
-    "suppress_tokens": [
-      1,
-      2,
-      7,
-      8,
-      9,
-      10,
-      14,
-      25,
-      26,
-      27,
-      28,
-      29,
-      31,
-      58,
-      59,
-      60,
-      61,
-      62,
-      63,
-      90,
-      91,
-      92,
-      93,
-      359,
-      503,
-      522,
-      542,
-      873,
-      893,
-      902,
-      918,
-      922,
-      931,
-      1350,
-      1853,
-      1982,
-      2460,
-      2627,
-      3246,
-      3253,
-      3268,
-      3536,
-      3846,
-      3961,
-      4183,
-      4667,
-      6585,
-      6647,
-      7273,
-      9061,
-      9383,
-      10428,
-      10929,
-      11938,
-      12033,
-      12331,
-      12562,
-      13793,
-      14157,
-      14635,
-      15265,
-      15618,
-      16553,
-      16604,
-      18362,
-      18956,
-      20075,
-      21675,
-      22520,
-      26130,
-      26161,
-      26435,
-      28279,
-      29464,
-      31650,
-      32302,
-      32470,
-      36865,
-      42863,
-      47425,
-      49870,
-      50254,
-      50258,
-      50358,
-      50359,
-      50360,
-      50361,
-      50362
-    ],
-    "use_cache": true,
-    "use_weighted_layer_sum": false,
-    "vocab_size": 51865
-  },
-  "audio_pool_step": 5,
   "auto_map": {
     "AutoConfig": "configuration_minicpmo.MiniCPMOConfig",
     "AutoModel": "modeling_minicpmo.MiniCPMO",
     "AutoModelForCausalLM": "modeling_minicpmo.MiniCPMO"
   },
-  "batch_vision_input": true,
-  "bos_token_id": 151643,
-  "drop_vision_last_layer": false,
-  "dtype": "bfloat16",
-  "eos_token_id": 151645,
-  "head_dim": 128,
-  "hidden_act": "silu",
-  "hidden_size": 4096,
-  "image_size": 448,
-  "init_audio": true,
-  "init_tts": true,
-  "init_vision": true,
-  "initializer_range": 0.02,
-  "intermediate_size": 12288,
-  "layer_types": [
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention"
-  ],
-  "listen_speak_type": "asr",
-  "max_position_embeddings": 40960,
-  "max_window_layers": 36,
-  "model_type": "minicpmo",
-  "num_attention_heads": 32,
-  "num_hidden_layers": 36,
-  "num_key_value_heads": 8,
-  "patch_size": 14,
-  "query_num": 64,
-  "rms_norm_eps": 1e-06,
-  "rope_scaling": null,
-  "rope_theta": 1000000,
-  "slice_config": {
-    "max_slice_nums": 1,
-    "model_type": "minicpmv",
-    "patch_size": 14,
-    "scale_resolution": 448
-  },
-  "slice_mode": true,
-  "sliding_window": null,
-  "stream_input": true,
-  "tie_word_embeddings": false,
-  "transformers_version": "4.57.6",
-  "tts_config": {
-    "_attn_implementation_autoset": true,
-    "attention_type": "full_attention",
-    "attn_implementation": "eager",
-    "audio_bos_token_id": 151687,
-    "audio_tokenizer_sample_rate": 16000,
-    "audio_tokenizer_type": "s3tokenizer",
-    "aug_layer_loss_weight": false,
-    "aug_loss_weight": false,
-    "backbone_model": "llama",
-    "condition_type": "hidden_text_merge",
-    "cosyvoice_config_path": null,
-    "cosyvoice_model_dir": null,
-    "filter_tts_loss": false,
-    "hidden_act": "silu",
-    "hidden_size": 768,
-    "interleaved": false,
-    "intermediate_size": 3072,
-    "llm_dim": 4096,
-    "llm_dim_model_base": 256,
-    "llm_down_scale": false,
-    "llm_hidden_size": 4096,
-    "llm_intermediate_size": 768,
-    "long_weight": 0.1,
-    "max_position_embeddings": 4096,
-    "model_type": "minicpmtts",
-    "normalize_projected_hidden": true,
-    "num_attention_heads": 12,
-    "num_audio_tokens": 6562,
-    "num_hidden_layers": 20,
-    "num_key_value_heads": 12,
-    "num_mel_bins": 100,
-    "num_text_tokens": 152064,
-    "num_vq": 1,
-    "projector_type": "mlp",
-    "recomputed_chunks": 1,
-    "s3_stream_chunk_size": 25,
-    "s3_stream_generate": false,
-    "s3_stream_n_timesteps": 10,
-    "s3_stream_prelook_size": 3,
-    "short_weight": 0.1,
-    "streaming": false,
-    "streaming_audio_chunk_size": 50,
-    "streaming_sliding_window": false,
-    "streaming_sliding_window_audio_frame_rate": 50,
-    "streaming_sliding_window_audio_init_text_length": 10,
-    "streaming_sliding_window_audio_window_size": 300,
-    "streaming_sliding_window_average_speed": 5,
-    "streaming_sliding_window_fast_speed": 7,
-    "streaming_sliding_window_max_text_len": 500,
-    "streaming_sliding_window_slow_speed": 3,
-    "streaming_sliding_window_text_window_size": 50,
-    "streaming_text_chunk_max": 7,
-    "streaming_text_chunk_min": 3,
-    "streaming_text_reserved_len": 300,
-    "text_eos_token_id": 151692,
-    "tts_filter_loss_fix": false,
-    "use_llm_hidden_state": false,
-    "use_text": true,
-    "window_size": 2
-  },
-  "use_cache": true,
-  "use_image_id": true,
-  "use_sliding_window": false,
-  "version": "4.5",
-  "vision_batch_size": 16,
-  "vision_config": {
-    "_attn_implementation_autoset": true,
-    "attention_dropout": 0.0,
-    "hidden_act": "gelu_pytorch_tanh",
-    "hidden_size": 1152,
-    "image_size": 980,
-    "intermediate_size": 4304,
-    "layer_norm_eps": 1e-06,
-    "model_type": "siglip_vision_model",
-    "num_attention_heads": 16,
-    "num_channels": 3,
-    "num_hidden_layers": 27,
-    "patch_size": 14
-  },
-  "vocab_size": 151752
+  "venus_config_ref": {
+    "repo_id": "inclusionAI/Realtime-Venus",
+    "repo_type": "model",
+    "filename": "config.yaml",
+    "revision": "15d5b420cd8557e9582d41a57a80af3e4de18e47",
+    "sha256": "f08995a8bc63d59b01835321128ee1bd05a8a74d91fc8d6ef54b5780fefe84dd",
+    "model": "audio"
+  }
 }

Realtime-Venus-Audio/configuration_minicpmo.py

@@ -259,3 +259,105 @@ class MiniCPMOConfig(Qwen3Config):
         self.patch_size = self.vision_config.patch_size
 
         super().__init__(**kwargs)
+
+    @classmethod
+    def get_config_dict(cls, pretrained_model_name_or_path, **kwargs):
+        options = dict(kwargs)
+        cls._set_token_in_kwargs(options)
+        config, remaining = super().get_config_dict(pretrained_model_name_or_path, **kwargs)
+        return _expand_venus_config(config, pretrained_model_name_or_path, options, 'audio'), remaining
+
+
+"""This helper is embedded into each existing model configuration module."""
+
+
+def _expand_venus_config(config, model_path, options, expected_model):
+    import copy
+    import hashlib
+    import re
+    from pathlib import Path
+
+    ref = config.get('venus_config_ref')
+    if ref is None:
+        return config
+    expected_types = {'audio': 'minicpmo', 'omni': 'realtime_venus_omni'}
+    if not isinstance(ref, dict) or set(ref) != {'repo_id','repo_type','filename','revision','sha256','model'}:
+        raise ValueError('Invalid venus_config_ref fields')
+    if (ref['repo_id'] != 'inclusionAI/Realtime-Venus' or ref['repo_type'] != 'model'
+            or ref['filename'] != 'config.yaml' or ref['model'] != expected_model):
+        raise ValueError('venus_config_ref repository/file/model identity mismatch')
+    if not isinstance(ref['revision'],str) or not re.fullmatch(r'[0-9a-f]{40}', ref['revision']):
+        raise ValueError('venus_config_ref requires a fixed full commit SHA')
+    if not isinstance(ref['sha256'],str) or not re.fullmatch(r'[0-9a-f]{64}', ref['sha256']):
+        raise ValueError('venus_config_ref requires a SHA-256 digest')
+    if config.get('model_type') != expected_types[expected_model]:
+        raise ValueError('Child configuration model_type mismatch')
+
+    raw = None
+    path = Path(model_path)
+    subfolder = options.get('subfolder', '')
+    candidates = []
+    if path.is_dir():
+        model_dir = path / subfolder if subfolder else path
+        candidates.append(model_dir.parent / 'config.yaml')
+    elif path.is_file():
+        candidates.append(path.parent.parent / 'config.yaml')
+    for candidate in candidates:
+        if candidate.is_file():
+            value = candidate.read_bytes()
+            if hashlib.sha256(value).hexdigest() == ref['sha256']:
+                raw = value
+                break
+
+    if raw is None:
+        from huggingface_hub import hf_hub_download, try_to_load_from_cache
+        from transformers.utils.hub import TRANSFORMERS_CACHE
+        cache_dir = options.get('cache_dir') or TRANSFORMERS_CACHE
+        cached = try_to_load_from_cache(ref['repo_id'],ref['filename'],
+            revision=ref['revision'],repo_type='model',cache_dir=cache_dir)
+        if isinstance(cached,str):
+            raw = Path(cached).read_bytes()
+        else:
+            download_options = {k: options[k] for k in
+                ('token','local_files_only','force_download','user_agent','proxies') if k in options}
+            download_options['cache_dir'] = cache_dir
+            try:
+                resolved = hf_hub_download(repo_id=ref['repo_id'],repo_type='model',
+                    filename=ref['filename'],revision=ref['revision'],**download_options)
+            except OSError as error:
+                raise OSError('Required pinned Realtime-Venus config.yaml is unavailable locally/in cache; '
+                    'download that revision while online or place its verified YAML beside the model directory.') from error
+            raw = Path(resolved).read_bytes()
+    if hashlib.sha256(raw).hexdigest() != ref['sha256']:
+        raise ValueError('Pinned Realtime-Venus config.yaml SHA-256 mismatch')
+
+    import yaml
+    document = yaml.safe_load(raw)
+    if not isinstance(document,dict) or document.get('schema_version') != 3 or document.get('name') != 'Realtime-Venus':
+        raise ValueError('Invalid Realtime-Venus root YAML schema/name')
+    models = document.get('models')
+    entry = models.get(expected_model) if isinstance(models,dict) else None
+    expected_path = 'Realtime-Venus-Audio' if expected_model == 'audio' else 'Realtime-Venus-Omni'
+    if not isinstance(entry,dict) or entry.get('path') != expected_path:
+        raise ValueError('Root YAML model path mismatch')
+    full = entry.get('hf_config')
+    if not isinstance(full,dict) or 'venus_config_ref' in full or full.get('model_type') != expected_types[expected_model]:
+        raise ValueError('Root YAML hf_config model identity mismatch')
+    if full.get('architectures') != config.get('architectures'):
+        raise ValueError('Root YAML architecture mismatch')
+    for field in ('audio_config','vision_config','tts_config'):
+        if not isinstance(full.get(field),dict):
+            raise ValueError('Incomplete root hf_config: ' + field)
+    def auto_map_identity(mapping):
+        if not isinstance(mapping,dict):
+            raise ValueError('Invalid auto_map')
+        return {k:v.split('--')[-1] for k,v in mapping.items()}
+    if auto_map_identity(full.get('auto_map')) != auto_map_identity(config.get('auto_map')):
+        raise ValueError('Root YAML AutoClass identity mismatch')
+    expanded = copy.deepcopy(full)
+    # Keep the current model snapshot's code/weights revision, not the YAML's.
+    if '_commit_hash' in config:
+        expanded['_commit_hash'] = config['_commit_hash']
+    expanded['auto_map'] = copy.deepcopy(config['auto_map'])
+    expanded['_name_or_path'] = str(model_path)
+    return expanded

Realtime-Venus-Omni/config.json

@@ -1,285 +1,19 @@
 {
+  "model_type": "realtime_venus_omni",
   "architectures": [
     "RealtimeVenusOmni"
   ],
-  "version": "4.5",
-  "attention_bias": false,
-  "attention_dropout": 0.0,
-  "audio_chunk_length": 1.0,
-  "audio_config": {
-    "_attn_implementation_autoset": true,
-    "_name_or_path": "openai/whisper-medium",
-    "activation_dropout": 0.0,
-    "activation_function": "gelu",
-    "apply_spec_augment": false,
-    "architectures": [
-      "RealtimeVenusOmniWhisperEncoder"
-    ],
-    "attention_dropout": 0.0,
-    "begin_suppress_tokens": [
-      220,
-      50257
-    ],
-    "bos_token_id": 50257,
-    "classifier_proj_size": 256,
-    "d_model": 1024,
-    "decoder_attention_heads": 16,
-    "decoder_ffn_dim": 4096,
-    "decoder_layerdrop": 0.0,
-    "decoder_layers": 24,
-    "decoder_start_token_id": 50258,
-    "dropout": 0.0,
-    "encoder_attention_heads": 16,
-    "encoder_ffn_dim": 4096,
-    "encoder_layerdrop": 0.0,
-    "encoder_layers": 24,
-    "eos_token_id": 50257,
-    "forced_decoder_ids": [
-      [
-        1,
-        50259
-      ],
-      [
-        2,
-        50359
-      ],
-      [
-        3,
-        50363
-      ]
-    ],
-    "init_std": 0.02,
-    "mask_feature_length": 10,
-    "mask_feature_min_masks": 0,
-    "mask_feature_prob": 0.0,
-    "mask_time_length": 10,
-    "mask_time_min_masks": 2,
-    "mask_time_prob": 0.05,
-    "max_length": 448,
-    "max_source_positions": 1500,
-    "max_target_positions": 448,
-    "median_filter_width": 7,
-    "model_type": "whisper",
-    "num_hidden_layers": 24,
-    "num_mel_bins": 80,
-    "pad_token_id": 50257,
-    "scale_embedding": false,
-    "suppress_tokens": [
-      1,
-      2,
-      7,
-      8,
-      9,
-      10,
-      14,
-      25,
-      26,
-      27,
-      28,
-      29,
-      31,
-      58,
-      59,
-      60,
-      61,
-      62,
-      63,
-      90,
-      91,
-      92,
-      93,
-      359,
-      503,
-      522,
-      542,
-      873,
-      893,
-      902,
-      918,
-      922,
-      931,
-      1350,
-      1853,
-      1982,
-      2460,
-      2627,
-      3246,
-      3253,
-      3268,
-      3536,
-      3846,
-      3961,
-      4183,
-      4667,
-      6585,
-      6647,
-      7273,
-      9061,
-      9383,
-      10428,
-      10929,
-      11938,
-      12033,
-      12331,
-      12562,
-      13793,
-      14157,
-      14635,
-      15265,
-      15618,
-      16553,
-      16604,
-      18362,
-      18956,
-      20075,
-      21675,
-      22520,
-      26130,
-      26161,
-      26435,
-      28279,
-      29464,
-      31650,
-      32302,
-      32470,
-      36865,
-      42863,
-      47425,
-      49870,
-      50254,
-      50258,
-      50358,
-      50359,
-      50360,
-      50361,
-      50362
-    ],
-    "torch_dtype": "float32",
-    "use_cache": true,
-    "use_weighted_layer_sum": false,
-    "vocab_size": 51865
-  },
-  "audio_pool_step": 5,
   "auto_map": {
     "AutoConfig": "configuration_realtime_venus_omni.RealtimeVenusOmniConfig",
     "AutoModel": "modeling_realtime_venus_omni.RealtimeVenusOmni",
     "AutoModelForCausalLM": "modeling_realtime_venus_omni.RealtimeVenusOmni"
   },
-  "batch_vision_input": true,
-  "bos_token_id": 151643,
-  "drop_vision_last_layer": false,
-  "eos_token_id": 151645,
-  "head_dim": 128,
-  "hidden_act": "silu",
-  "hidden_size": 4096,
-  "image_size": 448,
-  "init_audio": true,
-  "init_tts": true,
-  "init_vision": true,
-  "initializer_range": 0.02,
-  "intermediate_size": 12288,
-  "listen_speak_type": "asr",
-  "max_position_embeddings": 40960,
-  "max_window_layers": 36,
-  "model_type": "realtime_venus_omni",
-  "num_attention_heads": 32,
-  "num_hidden_layers": 36,
-  "num_key_value_heads": 8,
-  "patch_size": 14,
-  "query_num": 64,
-  "rms_norm_eps": 1e-06,
-  "rope_scaling": null,
-  "rope_theta": 1000000,
-  "slice_config": {
-    "max_slice_nums": 1,
-    "model_type": "realtime_venus_omni_vision",
-    "patch_size": 14,
-    "scale_resolution": 448
-  },
-  "slice_mode": true,
-  "sliding_window": null,
-  "stream_input": true,
-  "tie_word_embeddings": false,
-  "torch_dtype": "bfloat16",
-  "transformers_version": "4.51.0",
-  "tts_config": {
-    "_attn_implementation_autoset": true,
-    "attention_type": "full_attention",
-    "attn_implementation": "sdpa",
-    "audio_bos_token_id": 151687,
-    "audio_tokenizer_sample_rate": 16000,
-    "audio_tokenizer_type": "s3tokenizer",
-    "aug_layer_loss_weight": false,
-    "aug_loss_weight": false,
-    "backbone_model": "llama",
-    "condition_type": "hidden_text_merge",
-    "cosyvoice_config_path": null,
-    "cosyvoice_model_dir": null,
-    "filter_tts_loss": false,
-    "hidden_act": "silu",
-    "hidden_size": 768,
-    "interleaved": false,
-    "intermediate_size": 3072,
-    "llm_dim": 4096,
-    "llm_dim_model_base": 256,
-    "llm_down_scale": false,
-    "llm_hidden_size": 4096,
-    "llm_intermediate_size": 768,
-    "long_weight": 0.1,
-    "max_position_embeddings": 4096,
-    "model_type": "realtime_venus_omni_tts",
-    "normalize_projected_hidden": true,
-    "num_attention_heads": 12,
-    "num_audio_tokens": 6562,
-    "num_hidden_layers": 20,
-    "num_key_value_heads": 12,
-    "num_mel_bins": 100,
-    "num_text_tokens": 152064,
-    "num_vq": 1,
-    "projector_type": "mlp",
-    "recomputed_chunks": 1,
-    "s3_stream_chunk_size": 25,
-    "s3_stream_generate": false,
-    "s3_stream_n_timesteps": 10,
-    "s3_stream_prelook_size": 3,
-    "short_weight": 0.1,
-    "streaming": false,
-    "streaming_audio_chunk_size": 50,
-    "streaming_sliding_window": false,
-    "streaming_sliding_window_audio_frame_rate": 50,
-    "streaming_sliding_window_audio_init_text_length": 10,
-    "streaming_sliding_window_audio_window_size": 300,
-    "streaming_sliding_window_average_speed": 5,
-    "streaming_sliding_window_fast_speed": 7,
-    "streaming_sliding_window_max_text_len": 500,
-    "streaming_sliding_window_slow_speed": 3,
-    "streaming_sliding_window_text_window_size": 50,
-    "streaming_text_chunk_max": 7,
-    "streaming_text_chunk_min": 3,
-    "streaming_text_reserved_len": 300,
-    "text_eos_token_id": 151692,
-    "tts_filter_loss_fix": false,
-    "use_llm_hidden_state": false,
-    "use_text": true,
-    "window_size": 2
-  },
-  "use_cache": true,
-  "use_image_id": true,
-  "use_sliding_window": false,
-  "vision_batch_size": 16,
-  "vision_config": {
-    "_attn_implementation_autoset": true,
-    "attention_dropout": 0.0,
-    "hidden_act": "gelu_pytorch_tanh",
-    "hidden_size": 1152,
-    "image_size": 980,
-    "intermediate_size": 4304,
-    "layer_norm_eps": 1e-06,
-    "model_type": "siglip_vision_model",
-    "num_attention_heads": 16,
-    "num_channels": 3,
-    "num_hidden_layers": 27,
-    "patch_size": 14
-  },
-  "vocab_size": 151752
+  "venus_config_ref": {
+    "repo_id": "inclusionAI/Realtime-Venus",
+    "repo_type": "model",
+    "filename": "config.yaml",
+    "revision": "15d5b420cd8557e9582d41a57a80af3e4de18e47",
+    "sha256": "f08995a8bc63d59b01835321128ee1bd05a8a74d91fc8d6ef54b5780fefe84dd",
+    "model": "omni"
+  }
 }

Realtime-Venus-Omni/configuration_realtime_venus_omni.py

@@ -259,3 +259,105 @@ class RealtimeVenusOmniConfig(Qwen3Config):
         self.patch_size = self.vision_config.patch_size
 
         super().__init__(**kwargs)
+
+    @classmethod
+    def get_config_dict(cls, pretrained_model_name_or_path, **kwargs):
+        options = dict(kwargs)
+        cls._set_token_in_kwargs(options)
+        config, remaining = super().get_config_dict(pretrained_model_name_or_path, **kwargs)
+        return _expand_venus_config(config, pretrained_model_name_or_path, options, 'omni'), remaining
+
+
+"""This helper is embedded into each existing model configuration module."""
+
+
+def _expand_venus_config(config, model_path, options, expected_model):
+    import copy
+    import hashlib
+    import re
+    from pathlib import Path
+
+    ref = config.get('venus_config_ref')
+    if ref is None:
+        return config
+    expected_types = {'audio': 'minicpmo', 'omni': 'realtime_venus_omni'}
+    if not isinstance(ref, dict) or set(ref) != {'repo_id','repo_type','filename','revision','sha256','model'}:
+        raise ValueError('Invalid venus_config_ref fields')
+    if (ref['repo_id'] != 'inclusionAI/Realtime-Venus' or ref['repo_type'] != 'model'
+            or ref['filename'] != 'config.yaml' or ref['model'] != expected_model):
+        raise ValueError('venus_config_ref repository/file/model identity mismatch')
+    if not isinstance(ref['revision'],str) or not re.fullmatch(r'[0-9a-f]{40}', ref['revision']):
+        raise ValueError('venus_config_ref requires a fixed full commit SHA')
+    if not isinstance(ref['sha256'],str) or not re.fullmatch(r'[0-9a-f]{64}', ref['sha256']):
+        raise ValueError('venus_config_ref requires a SHA-256 digest')
+    if config.get('model_type') != expected_types[expected_model]:
+        raise ValueError('Child configuration model_type mismatch')
+
+    raw = None
+    path = Path(model_path)
+    subfolder = options.get('subfolder', '')
+    candidates = []
+    if path.is_dir():
+        model_dir = path / subfolder if subfolder else path
+        candidates.append(model_dir.parent / 'config.yaml')
+    elif path.is_file():
+        candidates.append(path.parent.parent / 'config.yaml')
+    for candidate in candidates:
+        if candidate.is_file():
+            value = candidate.read_bytes()
+            if hashlib.sha256(value).hexdigest() == ref['sha256']:
+                raw = value
+                break
+
+    if raw is None:
+        from huggingface_hub import hf_hub_download, try_to_load_from_cache
+        from transformers.utils.hub import TRANSFORMERS_CACHE
+        cache_dir = options.get('cache_dir') or TRANSFORMERS_CACHE
+        cached = try_to_load_from_cache(ref['repo_id'],ref['filename'],
+            revision=ref['revision'],repo_type='model',cache_dir=cache_dir)
+        if isinstance(cached,str):
+            raw = Path(cached).read_bytes()
+        else:
+            download_options = {k: options[k] for k in
+                ('token','local_files_only','force_download','user_agent','proxies') if k in options}
+            download_options['cache_dir'] = cache_dir
+            try:
+                resolved = hf_hub_download(repo_id=ref['repo_id'],repo_type='model',
+                    filename=ref['filename'],revision=ref['revision'],**download_options)
+            except OSError as error:
+                raise OSError('Required pinned Realtime-Venus config.yaml is unavailable locally/in cache; '
+                    'download that revision while online or place its verified YAML beside the model directory.') from error
+            raw = Path(resolved).read_bytes()
+    if hashlib.sha256(raw).hexdigest() != ref['sha256']:
+        raise ValueError('Pinned Realtime-Venus config.yaml SHA-256 mismatch')
+
+    import yaml
+    document = yaml.safe_load(raw)
+    if not isinstance(document,dict) or document.get('schema_version') != 3 or document.get('name') != 'Realtime-Venus':
+        raise ValueError('Invalid Realtime-Venus root YAML schema/name')
+    models = document.get('models')
+    entry = models.get(expected_model) if isinstance(models,dict) else None
+    expected_path = 'Realtime-Venus-Audio' if expected_model == 'audio' else 'Realtime-Venus-Omni'
+    if not isinstance(entry,dict) or entry.get('path') != expected_path:
+        raise ValueError('Root YAML model path mismatch')
+    full = entry.get('hf_config')
+    if not isinstance(full,dict) or 'venus_config_ref' in full or full.get('model_type') != expected_types[expected_model]:
+        raise ValueError('Root YAML hf_config model identity mismatch')
+    if full.get('architectures') != config.get('architectures'):
+        raise ValueError('Root YAML architecture mismatch')
+    for field in ('audio_config','vision_config','tts_config'):
+        if not isinstance(full.get(field),dict):
+            raise ValueError('Incomplete root hf_config: ' + field)
+    def auto_map_identity(mapping):
+        if not isinstance(mapping,dict):
+            raise ValueError('Invalid auto_map')
+        return {k:v.split('--')[-1] for k,v in mapping.items()}
+    if auto_map_identity(full.get('auto_map')) != auto_map_identity(config.get('auto_map')):
+        raise ValueError('Root YAML AutoClass identity mismatch')
+    expanded = copy.deepcopy(full)
+    # Keep the current model snapshot's code/weights revision, not the YAML's.
+    if '_commit_hash' in config:
+        expanded['_commit_hash'] = config['_commit_hash']
+    expanded['auto_map'] = copy.deepcopy(config['auto_map'])
+    expanded['_name_or_path'] = str(model_path)
+    return expanded

Realtime-Venus-Omni/requirements.txt

@@ -47,3 +47,6 @@ huggingface_hub
 # aiofiles==23.2.1
 # onnxruntime==1.20.1
 # pydantic==2.10.6
+
+# Pinned root config.yaml resolution
+PyYAML>=6.0

config.yaml

@@ -1,6 +1,580 @@
+schema_version: 3
 name: Realtime-Venus
 models:
-  omni:
-    path: Realtime-Venus-Omni
   audio:
     path: Realtime-Venus-Audio
+    hf_config:
+      architectures:
+      - MiniCPMO
+      attention_bias: false
+      attention_dropout: 0.0
+      audio_chunk_length: 1.0
+      audio_config:
+        _attn_implementation_autoset: true
+        _name_or_path: openai/whisper-medium
+        activation_dropout: 0.0
+        activation_function: gelu
+        apply_spec_augment: false
+        architectures:
+        - MiniCPMWhisperEncoder
+        attention_dropout: 0.0
+        begin_suppress_tokens:
+        - 220
+        - 50257
+        bos_token_id: 50257
+        classifier_proj_size: 256
+        d_model: 1024
+        decoder_attention_heads: 16
+        decoder_ffn_dim: 4096
+        decoder_layerdrop: 0.0
+        decoder_layers: 24
+        decoder_start_token_id: 50258
+        dropout: 0.0
+        dtype: float32
+        encoder_attention_heads: 16
+        encoder_ffn_dim: 4096
+        encoder_layerdrop: 0.0
+        encoder_layers: 24
+        eos_token_id: 50257
+        forced_decoder_ids:
+        - - 1
+          - 50259
+        - - 2
+          - 50359
+        - - 3
+          - 50363
+        init_std: 0.02
+        mask_feature_length: 10
+        mask_feature_min_masks: 0
+        mask_feature_prob: 0.0
+        mask_time_length: 10
+        mask_time_min_masks: 2
+        mask_time_prob: 0.05
+        max_length: 448
+        max_source_positions: 1500
+        max_target_positions: 448
+        median_filter_width: 7
+        model_type: whisper
+        num_hidden_layers: 24
+        num_mel_bins: 80
+        pad_token_id: 50257
+        scale_embedding: false
+        suppress_tokens:
+        - 1
+        - 2
+        - 7
+        - 8
+        - 9
+        - 10
+        - 14
+        - 25
+        - 26
+        - 27
+        - 28
+        - 29
+        - 31
+        - 58
+        - 59
+        - 60
+        - 61
+        - 62
+        - 63
+        - 90
+        - 91
+        - 92
+        - 93
+        - 359
+        - 503
+        - 522
+        - 542
+        - 873
+        - 893
+        - 902
+        - 918
+        - 922
+        - 931
+        - 1350
+        - 1853
+        - 1982
+        - 2460
+        - 2627
+        - 3246
+        - 3253
+        - 3268
+        - 3536
+        - 3846
+        - 3961
+        - 4183
+        - 4667
+        - 6585
+        - 6647
+        - 7273
+        - 9061
+        - 9383
+        - 10428
+        - 10929
+        - 11938
+        - 12033
+        - 12331
+        - 12562
+        - 13793
+        - 14157
+        - 14635
+        - 15265
+        - 15618
+        - 16553
+        - 16604
+        - 18362
+        - 18956
+        - 20075
+        - 21675
+        - 22520
+        - 26130
+        - 26161
+        - 26435
+        - 28279
+        - 29464
+        - 31650
+        - 32302
+        - 32470
+        - 36865
+        - 42863
+        - 47425
+        - 49870
+        - 50254
+        - 50258
+        - 50358
+        - 50359
+        - 50360
+        - 50361
+        - 50362
+        use_cache: true
+        use_weighted_layer_sum: false
+        vocab_size: 51865
+      audio_pool_step: 5
+      auto_map:
+        AutoConfig: configuration_minicpmo.MiniCPMOConfig
+        AutoModel: modeling_minicpmo.MiniCPMO
+        AutoModelForCausalLM: modeling_minicpmo.MiniCPMO
+      batch_vision_input: true
+      bos_token_id: 151643
+      drop_vision_last_layer: false
+      dtype: bfloat16
+      eos_token_id: 151645
+      head_dim: 128
+      hidden_act: silu
+      hidden_size: 4096
+      image_size: 448
+      init_audio: true
+      init_tts: true
+      init_vision: true
+      initializer_range: 0.02
+      intermediate_size: 12288
+      layer_types:
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      - full_attention
+      listen_speak_type: asr
+      max_position_embeddings: 40960
+      max_window_layers: 36
+      model_type: minicpmo
+      num_attention_heads: 32
+      num_hidden_layers: 36
+      num_key_value_heads: 8
+      patch_size: 14
+      query_num: 64
+      rms_norm_eps: 1.0e-06
+      rope_scaling: null
+      rope_theta: 1000000
+      slice_config:
+        max_slice_nums: 1
+        model_type: minicpmv
+        patch_size: 14
+        scale_resolution: 448
+      slice_mode: true
+      sliding_window: null
+      stream_input: true
+      tie_word_embeddings: false
+      transformers_version: 4.57.6
+      tts_config:
+        _attn_implementation_autoset: true
+        attention_type: full_attention
+        attn_implementation: eager
+        audio_bos_token_id: 151687
+        audio_tokenizer_sample_rate: 16000
+        audio_tokenizer_type: s3tokenizer
+        aug_layer_loss_weight: false
+        aug_loss_weight: false
+        backbone_model: llama
+        condition_type: hidden_text_merge
+        cosyvoice_config_path: null
+        cosyvoice_model_dir: null
+        filter_tts_loss: false
+        hidden_act: silu
+        hidden_size: 768
+        interleaved: false
+        intermediate_size: 3072
+        llm_dim: 4096
+        llm_dim_model_base: 256
+        llm_down_scale: false
+        llm_hidden_size: 4096
+        llm_intermediate_size: 768
+        long_weight: 0.1
+        max_position_embeddings: 4096
+        model_type: minicpmtts
+        normalize_projected_hidden: true
+        num_attention_heads: 12
+        num_audio_tokens: 6562
+        num_hidden_layers: 20
+        num_key_value_heads: 12
+        num_mel_bins: 100
+        num_text_tokens: 152064
+        num_vq: 1
+        projector_type: mlp
+        recomputed_chunks: 1
+        s3_stream_chunk_size: 25
+        s3_stream_generate: false
+        s3_stream_n_timesteps: 10
+        s3_stream_prelook_size: 3
+        short_weight: 0.1
+        streaming: false
+        streaming_audio_chunk_size: 50
+        streaming_sliding_window: false
+        streaming_sliding_window_audio_frame_rate: 50
+        streaming_sliding_window_audio_init_text_length: 10
+        streaming_sliding_window_audio_window_size: 300
+        streaming_sliding_window_average_speed: 5
+        streaming_sliding_window_fast_speed: 7
+        streaming_sliding_window_max_text_len: 500
+        streaming_sliding_window_slow_speed: 3
+        streaming_sliding_window_text_window_size: 50
+        streaming_text_chunk_max: 7
+        streaming_text_chunk_min: 3
+        streaming_text_reserved_len: 300
+        text_eos_token_id: 151692
+        tts_filter_loss_fix: false
+        use_llm_hidden_state: false
+        use_text: true
+        window_size: 2
+      use_cache: true
+      use_image_id: true
+      use_sliding_window: false
+      version: '4.5'
+      vision_batch_size: 16
+      vision_config:
+        _attn_implementation_autoset: true
+        attention_dropout: 0.0
+        hidden_act: gelu_pytorch_tanh
+        hidden_size: 1152
+        image_size: 980
+        intermediate_size: 4304
+        layer_norm_eps: 1.0e-06
+        model_type: siglip_vision_model
+        num_attention_heads: 16
+        num_channels: 3
+        num_hidden_layers: 27
+        patch_size: 14
+      vocab_size: 151752
+  omni:
+    path: Realtime-Venus-Omni
+    hf_config:
+      architectures:
+      - RealtimeVenusOmni
+      version: '4.5'
+      attention_bias: false
+      attention_dropout: 0.0
+      audio_chunk_length: 1.0
+      audio_config:
+        _attn_implementation_autoset: true
+        _name_or_path: openai/whisper-medium
+        activation_dropout: 0.0
+        activation_function: gelu
+        apply_spec_augment: false
+        architectures:
+        - RealtimeVenusOmniWhisperEncoder
+        attention_dropout: 0.0
+        begin_suppress_tokens:
+        - 220
+        - 50257
+        bos_token_id: 50257
+        classifier_proj_size: 256
+        d_model: 1024
+        decoder_attention_heads: 16
+        decoder_ffn_dim: 4096
+        decoder_layerdrop: 0.0
+        decoder_layers: 24
+        decoder_start_token_id: 50258
+        dropout: 0.0
+        encoder_attention_heads: 16
+        encoder_ffn_dim: 4096
+        encoder_layerdrop: 0.0
+        encoder_layers: 24
+        eos_token_id: 50257
+        forced_decoder_ids:
+        - - 1
+          - 50259
+        - - 2
+          - 50359
+        - - 3
+          - 50363
+        init_std: 0.02
+        mask_feature_length: 10
+        mask_feature_min_masks: 0
+        mask_feature_prob: 0.0
+        mask_time_length: 10
+        mask_time_min_masks: 2
+        mask_time_prob: 0.05
+        max_length: 448
+        max_source_positions: 1500
+        max_target_positions: 448
+        median_filter_width: 7
+        model_type: whisper
+        num_hidden_layers: 24
+        num_mel_bins: 80
+        pad_token_id: 50257
+        scale_embedding: false
+        suppress_tokens:
+        - 1
+        - 2
+        - 7
+        - 8
+        - 9
+        - 10
+        - 14
+        - 25
+        - 26
+        - 27
+        - 28
+        - 29
+        - 31
+        - 58
+        - 59
+        - 60
+        - 61
+        - 62
+        - 63
+        - 90
+        - 91
+        - 92
+        - 93
+        - 359
+        - 503
+        - 522
+        - 542
+        - 873
+        - 893
+        - 902
+        - 918
+        - 922
+        - 931
+        - 1350
+        - 1853
+        - 1982
+        - 2460
+        - 2627
+        - 3246
+        - 3253
+        - 3268
+        - 3536
+        - 3846
+        - 3961
+        - 4183
+        - 4667
+        - 6585
+        - 6647
+        - 7273
+        - 9061
+        - 9383
+        - 10428
+        - 10929
+        - 11938
+        - 12033
+        - 12331
+        - 12562
+        - 13793
+        - 14157
+        - 14635
+        - 15265
+        - 15618
+        - 16553
+        - 16604
+        - 18362
+        - 18956
+        - 20075
+        - 21675
+        - 22520
+        - 26130
+        - 26161
+        - 26435
+        - 28279
+        - 29464
+        - 31650
+        - 32302
+        - 32470
+        - 36865
+        - 42863
+        - 47425
+        - 49870
+        - 50254
+        - 50258
+        - 50358
+        - 50359
+        - 50360
+        - 50361
+        - 50362
+        torch_dtype: float32
+        use_cache: true
+        use_weighted_layer_sum: false
+        vocab_size: 51865
+      audio_pool_step: 5
+      auto_map:
+        AutoConfig: configuration_realtime_venus_omni.RealtimeVenusOmniConfig
+        AutoModel: modeling_realtime_venus_omni.RealtimeVenusOmni
+        AutoModelForCausalLM: modeling_realtime_venus_omni.RealtimeVenusOmni
+      batch_vision_input: true
+      bos_token_id: 151643
+      drop_vision_last_layer: false
+      eos_token_id: 151645
+      head_dim: 128
+      hidden_act: silu
+      hidden_size: 4096
+      image_size: 448
+      init_audio: true
+      init_tts: true
+      init_vision: true
+      initializer_range: 0.02
+      intermediate_size: 12288
+      listen_speak_type: asr
+      max_position_embeddings: 40960
+      max_window_layers: 36
+      model_type: realtime_venus_omni
+      num_attention_heads: 32
+      num_hidden_layers: 36
+      num_key_value_heads: 8
+      patch_size: 14
+      query_num: 64
+      rms_norm_eps: 1.0e-06
+      rope_scaling: null
+      rope_theta: 1000000
+      slice_config:
+        max_slice_nums: 1
+        model_type: realtime_venus_omni_vision
+        patch_size: 14
+        scale_resolution: 448
+      slice_mode: true
+      sliding_window: null
+      stream_input: true
+      tie_word_embeddings: false
+      torch_dtype: bfloat16
+      transformers_version: 4.51.0
+      tts_config:
+        _attn_implementation_autoset: true
+        attention_type: full_attention
+        attn_implementation: sdpa
+        audio_bos_token_id: 151687
+        audio_tokenizer_sample_rate: 16000
+        audio_tokenizer_type: s3tokenizer
+        aug_layer_loss_weight: false
+        aug_loss_weight: false
+        backbone_model: llama
+        condition_type: hidden_text_merge
+        cosyvoice_config_path: null
+        cosyvoice_model_dir: null
+        filter_tts_loss: false
+        hidden_act: silu
+        hidden_size: 768
+        interleaved: false
+        intermediate_size: 3072
+        llm_dim: 4096
+        llm_dim_model_base: 256
+        llm_down_scale: false
+        llm_hidden_size: 4096
+        llm_intermediate_size: 768
+        long_weight: 0.1
+        max_position_embeddings: 4096
+        model_type: realtime_venus_omni_tts
+        normalize_projected_hidden: true
+        num_attention_heads: 12
+        num_audio_tokens: 6562
+        num_hidden_layers: 20
+        num_key_value_heads: 12
+        num_mel_bins: 100
+        num_text_tokens: 152064
+        num_vq: 1
+        projector_type: mlp
+        recomputed_chunks: 1
+        s3_stream_chunk_size: 25
+        s3_stream_generate: false
+        s3_stream_n_timesteps: 10
+        s3_stream_prelook_size: 3
+        short_weight: 0.1
+        streaming: false
+        streaming_audio_chunk_size: 50
+        streaming_sliding_window: false
+        streaming_sliding_window_audio_frame_rate: 50
+        streaming_sliding_window_audio_init_text_length: 10
+        streaming_sliding_window_audio_window_size: 300
+        streaming_sliding_window_average_speed: 5
+        streaming_sliding_window_fast_speed: 7
+        streaming_sliding_window_max_text_len: 500
+        streaming_sliding_window_slow_speed: 3
+        streaming_sliding_window_text_window_size: 50
+        streaming_text_chunk_max: 7
+        streaming_text_chunk_min: 3
+        streaming_text_reserved_len: 300
+        text_eos_token_id: 151692
+        tts_filter_loss_fix: false
+        use_llm_hidden_state: false
+        use_text: true
+        window_size: 2
+      use_cache: true
+      use_image_id: true
+      use_sliding_window: false
+      vision_batch_size: 16
+      vision_config:
+        _attn_implementation_autoset: true
+        attention_dropout: 0.0
+        hidden_act: gelu_pytorch_tanh
+        hidden_size: 1152
+        image_size: 980
+        intermediate_size: 4304
+        layer_norm_eps: 1.0e-06
+        model_type: siglip_vision_model
+        num_attention_heads: 16
+        num_channels: 3
+        num_hidden_layers: 27
+        patch_size: 14
+      vocab_size: 151752

data/HEARTBEAT_Challenge/README.md

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1cb37a587a0d22ef84ecec21e3b6759abfa84f7a28b1b71b8d350409fc7e5f4f
+size 6132

data/HEARTBEAT_Challenge/README_zh.md

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3d43b8b715a6d627949175cedcf4b906496487192777ee669c94c61a48cf83c6
+size 5726

data/HEARTBEAT_Challenge/fields/README.md

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f807a4ae8bc5a47f291ce7fd42ecfd648a98bbe9fef12d417f3c937e8073a4f4
+size 14541

data/HEARTBEAT_Challenge/fields/README_zh.md

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9d578d1d7763b74ab84cdcfb1836797fc309a16c4dfae8becb5ee67f9a0fac50
+size 15101

examples/heartbeat_access.py

@@ -0,0 +1,102 @@
+"""Lossless HEARTBEAT records in Zarr v2, addressed by a ReferenceFS index.
+
+Media stays in its original encoding. Row JSON uses schema-directed binary
+references, so arbitrary user strings/objects cannot collide with the marker.
+"""
+import base64
+import hashlib
+import json
+
+import fsspec
+import numpy as np
+import pyarrow as pa
+import zarr
+
+
+def open_root(config_location, *, chunk_cache_bytes=128*1024*1024):
+    fs = fsspec.filesystem('reference', fo=config_location, skip_instance_cache=True)
+    if '.zgroup' not in fs.references:
+        raise ValueError('Missing Zarr root metadata')
+    for key in fs.references:
+        if key.endswith('/.zarray'):
+            md = json.loads(fs.cat_file(key))
+            if len(md['shape']) != 1 or len(md['chunks']) != 1:
+                raise ValueError('Expected one-dimensional dense arrays')
+            prefix = key[:-len('.zarray')]
+            for i in range((md['shape'][0] + md['chunks'][0] - 1)//md['chunks'][0]):
+                if prefix + str(i) not in fs.references:
+                    raise ValueError('Missing dense chunk reference: ' + prefix + str(i))
+    store=fs.get_mapper('')
+    if chunk_cache_bytes:
+        # Adjacent samples frequently share a media block. Reuse compressed
+        # blocks instead of retransferring the same 16 MiB for each small clip.
+        store=zarr.LRUStoreCache(store,max_size=chunk_cache_bytes)
+    return zarr.open_group(store, mode='r', zarr_version=2), fs
+
+
+def restore(value, dtype, group):
+    if value is None:
+        return None
+    if pa.types.is_binary(dtype) or pa.types.is_large_binary(dtype):
+        i = value['venus_blob_index']
+        start, end = map(int, group['media_offsets'][i:i+2])
+        return group['media_data'][start:end].tobytes()
+    if pa.types.is_struct(dtype):
+        return {f.name: restore(value[f.name], f.type, group) for f in dtype}
+    if pa.types.is_list(dtype) or pa.types.is_large_list(dtype):
+        return [restore(v, dtype.value_type, group) for v in value]
+    return value
+
+
+def rows_from_shard(group, start=0, stop=None):
+    schema = pa.ipc.read_schema(pa.BufferReader(base64.b64decode(group.attrs['arrow_schema_base64'])))
+    n = group.attrs['num_rows']
+    for i in range(start, min(n, n if stop is None else stop)):
+        a, b = map(int, group['records_offsets'][i:i+2])
+        value = json.loads(group['records_data'][a:b].tobytes())
+        yield {f.name: restore(value[f.name], f.type, group) for f in schema}
+
+
+def generate_rows(config_location, track, task, split, subset=None, max_rows=None, _index_identity=None):
+    if max_rows is not None and max_rows <= 0:
+        return
+    root, fs = open_root(config_location)
+    path = f'data/HEARTBEAT_Challenge/{track}/{task}/{split}'
+    if subset:
+        path += '/' + subset
+    group = root[path]
+    count = 0
+    for key in sorted(group.group_keys()):
+        for row in rows_from_shard(group[key]):
+            yield row
+            count += 1
+            if max_rows is not None and count >= max_rows:
+                return
+
+
+def as_pandas(config_location, track, task, split, subset=None, max_rows=None):
+    import pandas as pd
+    return pd.DataFrame(generate_rows(config_location, track, task, split, subset, max_rows))
+
+
+def as_datasets(config_location, track, task, split, subset=None, *, streaming=False,
+                cache_dir=None, max_rows=None):
+    from datasets import Dataset, IterableDataset, Features
+    root, fs = open_root(config_location)
+    group = root[f'data/HEARTBEAT_Challenge/{track}/{task}/{split}' + ('/'+subset if subset else '')]
+    first = group[sorted(group.group_keys())[0]]
+    schema = pa.ipc.read_schema(pa.BufferReader(base64.b64decode(first.attrs['arrow_schema_base64'])))
+    features = Features.from_arrow_schema(schema)
+    kwargs = dict(config_location=str(config_location), track=track, task=task,
+                  split=split, subset=subset, max_rows=max_rows)
+    # In addition to immutable Hub revision paths, distinguish updated local
+    # reference indexes at the same filename without invalidating normal reuse.
+    kwargs['_index_identity'] = hashlib.sha256(json.dumps(
+        fs.references, sort_keys=True, separators=(',', ':')).encode()).hexdigest()
+    # Fixed commit URLs/local immutable indexes are part of generator identity.
+    if streaming:
+        return IterableDataset.from_generator(generate_rows, gen_kwargs=kwargs, features=features)
+    # Large original video blobs can overflow Arrow's 2 GiB binary-offset limit
+    # when accumulated into the default 1,000-record writer batch.
+    return Dataset.from_generator(generate_rows, gen_kwargs=kwargs, cache_dir=cache_dir,
+                                  features=features, writer_batch_size=1)

requirements-data.txt

@@ -0,0 +1,44 @@
+aiohappyeyeballs==2.7.1
+aiohttp==3.14.5
+aiosignal==1.4.0
+anyio==4.15.1
+asciitree==0.3.3
+attrs==26.1.0
+certifi==2026.10.23
+cffi==2.1.1
+charset-normalizer==3.5.2
+click==8.5.0
+datasets==5.0.1
+dill==0.4.1
+fasteners==0.20
+filelock==4.2.4
+frozenlist==1.8.0
+fsspec==2026.6.0
+h11==0.16.0
+hf-xet==1.7.0
+httpcore==1.0.9
+httpx==0.28.1
+huggingface_hub==1.33.0
+idna==3.20
+multidict==7.1.0
+multiprocess==0.70.19
+numcodecs==0.13.1
+numpy==1.26.4
+packaging==26.3
+pandas==2.3.3
+propcache==0.5.4
+pyarrow==25.0.1
+pycparser==3.12
+python-dateutil==3.0.0
+pytz==2026.5
+PyYAML==6.0.3
+requests==2.34.2
+six==1.17.0
+soundfile==0.14.0
+tqdm==4.70.1
+typing_extensions==4.16.0
+tzdata==2026.5
+urllib3==2.8.0
+xxhash==4.0.2
+yarl==1.25.1
+zarr==2.18.3