Sync from GitHub via hub-sync

.env.example

@@ -1,12 +1,24 @@
-# Realtime backend to use: "openai" or "gemini"
-BACKEND_PROVIDER="openai"
-# For Gemini set "gemini-3.1-flash-live-preview"
-MODEL_NAME="gpt-realtime"
+# Realtime backend to use: "huggingface", "openai", or "gemini"
+# Defaults to "huggingface" when unset.
+# BACKEND_PROVIDER="huggingface"
+# Optional model override for OpenAI Realtime or Gemini Live.
+# Hugging Face uses the server's model selection and ignores MODEL_NAME.
+# MODEL_NAME="gpt-realtime"
 
 # Set up your API key according to your backend
 OPENAI_API_KEY=
 # GEMINI_API_KEY=
 
+# Hugging Face connection mode: "deployed" or "local".
+# Deployed mode uses the built-in Hugging Face server.
+# Defaults to "deployed" when unset.
+# HF_REALTIME_CONNECTION_MODE="deployed"
+
+# Direct Hugging Face realtime endpoint for local/LAN backends.
+# Accepts either the base URL (`ws://127.0.0.1:8765/v1`) or the full realtime URL
+# (`ws://127.0.0.1:8765/v1/realtime`). Used when HF_REALTIME_CONNECTION_MODE=local.
+# HF_REALTIME_WS_URL="ws://127.0.0.1:8765/v1/realtime"
+
 # Local vision model (only used with --local-vision CLI flag)
 # By default, vision is handled by the selected realtime backend when the camera tool is used
 LOCAL_VISION_MODEL=HuggingFaceTB/SmolVLM2-2.2B-Instruct
@@ -17,8 +29,9 @@ HF_HOME=./cache
 # Hugging Face token for accessing datasets/models
 HF_TOKEN=
 
-# Profile selection (defaults to "default" when unset)
-REACHY_MINI_CUSTOM_PROFILE="example"
+# Optional built-in profile selector. Use a folder name from profiles/.
+# If unset, the app uses the default profile unless startup settings were saved from the UI.
+# REACHY_MINI_CUSTOM_PROFILE=mars_rover
 
 # Optional external profile/tool directories
 # REACHY_MINI_EXTERNAL_PROFILES_DIRECTORY=external_content/external_profiles

README.md

@@ -14,7 +14,7 @@ tags:
 
 # Reachy Mini conversation app
 
-Conversational app for the Reachy Mini robot combining real-time voice APIs (OpenAI Realtime or Gemini Live), vision pipelines, and choreographed motion libraries.
+Conversational app for the Reachy Mini robot combining realtime voice backends, vision pipelines, and choreographed motion libraries.
 
 ![Reachy Mini Dance](docs/assets/reachy_mini_dance.gif)
 
@@ -30,10 +30,11 @@ Conversational app for the Reachy Mini robot combining real-time voice APIs (Ope
 - [License](#license)
 
 ## Overview
-- Real-time audio conversation loop with `fastrtc` for low-latency streaming. Supports two backends:
-  - **OpenAI Realtime** (`gpt-realtime`) — default
-  - **Gemini Live** (`gemini-3.1-flash-live-preview`) — alternative, using the Google GenAI SDK
-- Vision processing uses the selected realtime backend by default (when camera tool is used), with optional on-device local vision using SmolVLM2 (CPU/GPU/MPS) via `--local-vision`.
+- Real-time audio conversation loop with `fastrtc` for low-latency streaming. Supported backends:
+  - **Hugging Face** - default, using the built-in Hugging Face server or your own local endpoint.
+  - **OpenAI Realtime** (`gpt-realtime`) - requires `OPENAI_API_KEY`.
+  - **Gemini Live** (`gemini-3.1-flash-live-preview`) - requires `GEMINI_API_KEY`.
+- Vision processing uses the selected realtime backend by default (when the camera tool is used), with optional on-device local vision using SmolVLM2 (CPU/GPU/MPS) via `--local-vision`.
 - Layered motion system queues primary moves (dances, emotions, goto poses, breathing) while blending speech-reactive wobble and head-tracking.
 - Async tool dispatch integrates robot motion, camera capture, and optional head-tracking capabilities through a Gradio web UI with live transcripts.
 
@@ -120,33 +121,64 @@ Some wheels (like PyTorch) are large and require compatible CUDA or CPU builds
 
 ## Configuration
 
-1. Copy `.env.example` to `.env`
-2. Fill in your API key and backend choice
+The default setup uses the Hugging Face backend and does not require an API key.
+
+Copy `.env.example` to `.env` when you want to switch backends, provide API keys, or point Hugging Face at your own local endpoint.
 
 | Variable | Description |
 |----------|-------------|
-| `OPENAI_API_KEY` | Optional override or fallback for OpenAI mode. In the headless settings flow, the app can use bundled OpenAI access when available; set your own key to override it or provide a fallback. |
+| `OPENAI_API_KEY` | Required for OpenAI Realtime mode. |
 | `GEMINI_API_KEY` | Required for Gemini mode. Also accepts `GOOGLE_API_KEY`. Get one at [aistudio.google.com](https://aistudio.google.com/apikey). |
-| `BACKEND_PROVIDER` | Realtime backend to use: `openai` (default) or `gemini`. |
-| `MODEL_NAME` | Optional model override for the selected backend. Defaults to `gpt-realtime` for OpenAI and `gemini-3.1-flash-live-preview` for Gemini Live. |
+| `BACKEND_PROVIDER` | Realtime backend to use: `huggingface` (default), `openai`, or `gemini`. |
+| `MODEL_NAME` | Optional model override for OpenAI Realtime or Gemini Live. Defaults to `gpt-realtime` for OpenAI and `gemini-3.1-flash-live-preview` for Gemini. Hugging Face uses the server's model selection. |
+| `HF_REALTIME_CONNECTION_MODE` | Hugging Face connection selector: `deployed` uses the built-in Hugging Face server; `local` uses `HF_REALTIME_WS_URL`. Defaults to `deployed`. |
+| `HF_REALTIME_WS_URL` | Direct websocket endpoint for your own Hugging Face backend. Accepts either a base URL like `ws://127.0.0.1:8765/v1` or the full websocket URL `ws://127.0.0.1:8765/v1/realtime`. Used when `HF_REALTIME_CONNECTION_MODE=local`. |
 | `HF_HOME` | Cache directory for local Hugging Face downloads (only used with `--local-vision` flag, defaults to `./cache`). |
 | `HF_TOKEN` | Optional token for Hugging Face access (for gated/private assets). |
 | `LOCAL_VISION_MODEL` | Hugging Face model path for local vision processing (only used with `--local-vision` flag, defaults to `HuggingFaceTB/SmolVLM2-2.2B-Instruct`). |
 
-### Switching to Gemini Live
+### Hugging Face Connection Modes
+
+Use the built-in Hugging Face server through the app-managed Space proxy. This is the default for a new install; set it explicitly only when you want to switch back from a saved local endpoint:
+
+```env
+BACKEND_PROVIDER=huggingface
+HF_REALTIME_CONNECTION_MODE=deployed
+```
+
+Run your own realtime voice backend using [speech-to-speech](https://github.com/huggingface/speech-to-speech) on the same machine as the conversation app:
+
+```env
+BACKEND_PROVIDER=huggingface
+HF_REALTIME_CONNECTION_MODE=local
+HF_REALTIME_WS_URL=ws://127.0.0.1:8765/v1/realtime
+```
 
-To use Gemini Live instead of OpenAI Realtime, update your `.env`:
+Run your own Hugging Face backend on your laptop and connect to it from Reachy Mini Wireless over the same Wi-Fi network:
 
 ```env
-BACKEND_PROVIDER="gemini"
-MODEL_NAME="gemini-3.1-flash-live-preview"
-GEMINI_API_KEY=your-gemini-api-key
+BACKEND_PROVIDER=huggingface
+HF_REALTIME_CONNECTION_MODE=local
+HF_REALTIME_WS_URL=ws://<your-laptop-lan-ip>:8765/v1/realtime
 ```
 
-`BACKEND_PROVIDER` is the primary switch. The app still falls back to `MODEL_NAME` for compatibility with older configs, and all features (tools, profiles, head tracking) work with both backends.
+For that LAN setup, make sure the backend listens on an address reachable from the robot, not only on `127.0.0.1`.
 
-> [!NOTE]
-> Gemini Live uses a different set of voices: Aoede, Charon, Fenrir, Kore (default), Leda, Orus, Puck, Zephyr. If your profile's `voice.txt` specifies an OpenAI voice, it will fall back to Kore.
+If the backend stays bound to loopback on your laptop, you can forward it into the robot over SSH instead:
+
+```bash
+ssh -N -R 8765:127.0.0.1:8765 <robot-user>@<robot-host>
+```
+
+Then set this on the robot:
+
+```env
+BACKEND_PROVIDER=huggingface
+HF_REALTIME_CONNECTION_MODE=local
+HF_REALTIME_WS_URL=ws://127.0.0.1:8765/v1/realtime
+```
+
+When using the headless settings UI, selecting `Hugging Face` lets you choose either the built-in server or a local `host:port` target. The UI writes `HF_REALTIME_CONNECTION_MODE` for you, and the local path writes `HF_REALTIME_WS_URL` with a default of `localhost:8765`.
 
 ## Running the app
 
@@ -167,7 +199,7 @@ The app runs in console mode by default. Add `--gradio` to launch a web UI at ht
 |--------|---------|-------------|
 | `--head-tracker {yolo,mediapipe}` | `None` | Select a head-tracking backend when a camera is available. `yolo` uses a local YOLO face detector, `mediapipe` comes from the `reachy_mini_toolbox` package. Requires the matching optional extra. |
 | `--no-camera` | `False` | Run without camera capture or head tracking. |
-| `--local-vision` | `False` | Use the local vision model (SmolVLM2) for camera-tool requests instead of the selected realtime backend vision. Requires `local_vision` extra to be installed. |
+| `--local-vision` | `False` | Use the local vision model (SmolVLM2) for camera-tool requests instead of the selected realtime backend. Requires `local_vision` extra to be installed. |
 | `--gradio` | `False` | Launch the Gradio web UI. Without this flag, runs in console mode. Required when running in simulation mode. |
 | `--robot-name` | `None` | Optional. Connect to a specific robot by name when running multiple daemons on the same subnet. See [Multiple robots on the same subnet](#advanced-features). |
 | `--debug` | `False` | Enable verbose logging for troubleshooting. |
@@ -205,7 +237,7 @@ reachy-mini-conversation-app --gradio
 | `stop_dance` | Clear queued dances. | Core install only. |
 | `play_emotion` | Play a recorded emotion clip via Hugging Face datasets. | Core install only. Uses the default open emotions dataset: [`pollen-robotics/reachy-mini-emotions-library`](https://huggingface.co/datasets/pollen-robotics/reachy-mini-emotions-library). |
 | `stop_emotion` | Clear queued emotions. | Core install only. |
-| `do_nothing` | Explicitly remain idle. | Core install only. |
+| `idle_do_nothing` | Explicitly remain idle during an idle turn. Not intended for normal conversation turns. | Core install only. |
 
 ## Advanced features
 
@@ -218,7 +250,9 @@ Built-in motion content is published as open Hugging Face datasets:
 
 Create custom profiles with dedicated instructions and enabled tools.
 
-Set `REACHY_MINI_CUSTOM_PROFILE=<name>` to load `profiles/<name>/` (see `.env.example`). If unset, the `default` profile is used.
+For normal usage, select a profile from the UI and save it for startup. That selection is persisted in `startup_settings.json`.
+
+If no startup settings have been saved yet, you can still seed startup from the environment with `REACHY_MINI_CUSTOM_PROFILE=<name>` to load `profiles/<name>/`. If neither is set, the `default` profile is used.
 
 Each profile should include `instructions.txt` (prompt text). `tools.txt` (list of allowed tools) is recommended. If missing for a non-default profile, the app falls back to `profiles/default/tools.txt`. Profiles can optionally contain custom tool implementations.
 
@@ -266,7 +300,7 @@ To create a locked variant of the app that cannot switch profiles, edit `src/rea
 ```python
 LOCKED_PROFILE: str | None = "mars_rover"  # Lock to this profile
 ```
-When `LOCKED_PROFILE` is set, the app always uses that profile, ignoring `REACHY_MINI_CUSTOM_PROFILE` env var & the Gradio UI shows "(locked)" and disables all profile editing controls.
+When `LOCKED_PROFILE` is set, the app always uses that profile, ignoring saved startup settings, `REACHY_MINI_CUSTOM_PROFILE`, and the Gradio UI. The UI shows "(locked)" and disables all profile editing controls.
 This is useful for creating dedicated clones of the app with a fixed personality. Clone scripts can simply edit this constant to lock the variant.
 
 </details>
@@ -294,9 +328,10 @@ external_content/
 
 **Environment variables:**
 
-Set these values in your `.env` (copy from `.env.example`):
+Set these values in your `.env` when you want env-driven external profile/tool selection:
 
 ```env
+# Optional fallback/manual profile selector:
 REACHY_MINI_CUSTOM_PROFILE=my_profile
 REACHY_MINI_EXTERNAL_PROFILES_DIRECTORY=./external_content/external_profiles
 REACHY_MINI_EXTERNAL_TOOLS_DIRECTORY=./external_content/external_tools

profiles/bored_teenager/tools.txt

@@ -3,6 +3,6 @@ stop_dance
 play_emotion
 stop_emotion
 camera
-do_nothing
+idle_do_nothing
 head_tracking
 move_head

profiles/captain_circuit/tools.txt

@@ -3,6 +3,6 @@ stop_dance
 play_emotion
 stop_emotion
 camera
-do_nothing
+idle_do_nothing
 head_tracking
 move_head

profiles/chess_coach/tools.txt

@@ -3,6 +3,6 @@ stop_dance
 play_emotion
 stop_emotion
 camera
-do_nothing
+idle_do_nothing
 head_tracking
 move_head

profiles/cosmic_kitchen/tools.txt

@@ -3,6 +3,6 @@ stop_dance
 play_emotion
 stop_emotion
 camera
-do_nothing
+idle_do_nothing
 head_tracking
 move_head

profiles/default/tools.txt

@@ -3,6 +3,6 @@ stop_dance
 play_emotion
 stop_emotion
 camera
-do_nothing
+idle_do_nothing
 head_tracking
 move_head

profiles/example/tools.txt

@@ -5,7 +5,7 @@ stop_dance
 play_emotion
 stop_emotion
 # camera
-# do_nothing
+# idle_do_nothing
 # head_tracking
 # move_head
 

profiles/hype_bot/tools.txt

@@ -3,6 +3,6 @@ stop_dance
 play_emotion
 stop_emotion
 camera
-do_nothing
+idle_do_nothing
 head_tracking
 move_head

profiles/mad_scientist_assistant/tools.txt

@@ -3,6 +3,6 @@ stop_dance
 play_emotion
 stop_emotion
 camera
-do_nothing
+idle_do_nothing
 head_tracking
 move_head

profiles/mars_rover/tools.txt

@@ -3,6 +3,6 @@ stop_dance
 play_emotion
 stop_emotion
 camera
-do_nothing
+idle_do_nothing
 head_tracking
 move_head

profiles/nature_documentarian/tools.txt

@@ -3,6 +3,6 @@ stop_dance
 play_emotion
 stop_emotion
 camera
-do_nothing
+idle_do_nothing
 head_tracking
 move_head

profiles/noir_detective/tools.txt

@@ -3,6 +3,6 @@ stop_dance
 play_emotion
 stop_emotion
 camera
-do_nothing
+idle_do_nothing
 head_tracking
 move_head

profiles/sorry_bro/tools.txt

@@ -3,6 +3,6 @@ stop_dance
 play_emotion
 stop_emotion
 camera
-do_nothing
+idle_do_nothing
 head_tracking
 move_head

profiles/tedai/tools.txt

@@ -3,5 +3,5 @@ stop_dance
 play_emotion
 stop_emotion
 camera
-do_nothing
+idle_do_nothing
 move_head

profiles/time_traveler/tools.txt

@@ -3,6 +3,6 @@ stop_dance
 play_emotion
 stop_emotion
 camera
-do_nothing
+idle_do_nothing
 head_tracking
 move_head

profiles/victorian_butler/tools.txt

@@ -3,6 +3,6 @@ stop_dance
 play_emotion
 stop_emotion
 camera
-do_nothing
+idle_do_nothing
 head_tracking
 move_head

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "reachy_mini_conversation_app"
-version = "0.5.0"
+version = "0.6.0"
 authors = [{ name = "Pollen Robotics", email = "contact@pollen-robotics.com" }]
 description = ""
 readme = "README.md"
@@ -16,6 +16,7 @@ dependencies = [
     "fastrtc>=0.0.34",
     "gradio==5.50.1.dev1",
     "huggingface-hub==1.3.0",
+    "httpx",
 
     #Environment variables
     "python-dotenv",
@@ -29,7 +30,7 @@ dependencies = [
     #Reachy mini
     "reachy_mini_dances_library",
     "reachy_mini_toolbox",
-    "reachy-mini>=1.6.4",
+    "reachy-mini>=1.7.1",
     "gradio_client>=1.13.3",
 ]
 

src/reachy_mini_conversation_app/audio/startup_config.py

@@ -0,0 +1,64 @@
+"""Startup configuration for the Reachy Mini audio processor."""
+
+from __future__ import annotations
+import logging
+from collections.abc import Sequence
+
+
+AudioControlValue = float | int
+AudioStartupParameter = tuple[str, tuple[AudioControlValue, ...]]
+WRITE_SETTLE_SECONDS = 0.1
+
+AUDIO_STARTUP_CONFIG: tuple[AudioStartupParameter, ...] = (
+    ("PP_AGCMAXGAIN", (10.0,)),
+    ("PP_MIN_NS", (0.8,)),
+    ("PP_MIN_NN", (0.8,)),
+    ("PP_GAMMA_E", (0.5,)),
+    ("PP_GAMMA_ETAIL", (0.5,)),
+    ("PP_NLATTENONOFF", (0,)),
+    ("PP_MGSCALE", (4.0, 1.0, 1.0)),
+)
+
+
+def apply_audio_startup_config(
+    robot: object,
+    *,
+    logger: logging.Logger | None = None,
+    verify: bool = True,
+    write_settle_seconds: float = WRITE_SETTLE_SECONDS,
+) -> bool:
+    """Apply the tuned XVF3800 audio configuration for the conversation app."""
+    log = logger or logging.getLogger(__name__)
+    audio = getattr(getattr(robot, "media", None), "audio", None)
+
+    if audio is None:
+        log.warning("Skipping Reachy audio startup config: robot media audio is unavailable.")
+        return False
+
+    apply_audio_config = getattr(audio, "apply_audio_config", None)
+    if not callable(apply_audio_config):
+        log.warning("Skipping Reachy audio startup config: SDK audio config API is unavailable.")
+        return False
+
+    try:
+        applied = bool(
+            apply_audio_config(
+                AUDIO_STARTUP_CONFIG,
+                verify=verify,
+                write_settle_seconds=write_settle_seconds,
+            )
+        )
+    except Exception as exc:
+        log.warning("Skipping Reachy audio startup config: SDK audio config failed: %s", exc)
+        return False
+
+    if applied:
+        log.info("Applied Reachy audio startup config: %s", _format_config(AUDIO_STARTUP_CONFIG))
+    else:
+        log.warning("Reachy audio startup config was not applied.")
+
+    return applied
+
+
+def _format_config(config: Sequence[AudioStartupParameter]) -> str:
+    return ", ".join(f"{name}={' '.join(str(value) for value in values)}" for name, values in config)

src/reachy_mini_conversation_app/base_realtime.py

@@ -0,0 +1,1017 @@
+import json
+import time
+import uuid
+import base64
+import random
+import asyncio
+import logging
+from abc import ABC, abstractmethod
+from typing import Any, Final, Tuple, ClassVar, Optional
+from datetime import datetime
+
+import numpy as np
+import gradio as gr
+from openai import AsyncOpenAI
+from fastrtc import AdditionalOutputs, wait_for_item, audio_to_int16
+from pydantic import Field, BaseModel
+from numpy.typing import NDArray
+from scipy.signal import resample
+from openai.types.realtime import (
+    RealtimeAudioConfigParam,
+    RealtimeToolsConfigParam,
+    RealtimeFunctionToolParam,
+    RealtimeAudioConfigOutputParam,
+    RealtimeResponseCreateParamsParam,
+    RealtimeSessionCreateRequestParam,
+)
+from websockets.exceptions import ConnectionClosedError
+from openai.resources.realtime.realtime import AsyncRealtimeConnection
+
+from reachy_mini_conversation_app.config import (
+    config,
+    get_default_voice_for_backend,
+    get_available_voices_for_backend,
+)
+from reachy_mini_conversation_app.tools.core_tools import ToolDependencies
+from reachy_mini_conversation_app.conversation_handler import ConversationHandler
+from reachy_mini_conversation_app.tools.background_tool_manager import (
+    ToolCallRoutine,
+    ToolNotification,
+    BackgroundToolManager,
+)
+
+
+logger = logging.getLogger(__name__)
+
+_RESPONSE_DONE_TIMEOUT: Final[float] = 30.0
+_RESPONSE_REJECTION_RETRY_DELAY: Final[float] = 0.5
+
+
+class InputTranscriptChunksByItem(BaseModel):
+    """Current item_id and its accumulated deltas. Only one item at a time."""
+
+    item_id: str | None = None
+    deltas: list[str] = Field(default_factory=list)
+
+
+def to_realtime_tools_config(tool_specs: list[dict[str, Any]]) -> RealtimeToolsConfigParam:
+    """Convert app tool specs to the OpenAI-compatible realtime session shape."""
+    realtime_tools: RealtimeToolsConfigParam = []
+    for spec in tool_specs:
+        tool_type = spec.get("type")
+        name = spec.get("name")
+        description = spec.get("description")
+        parameters = spec.get("parameters", {})
+
+        if tool_type != "function" or not isinstance(name, str):
+            raise ValueError(f"Unsupported realtime tool spec: {spec!r}")
+
+        realtime_tool = RealtimeFunctionToolParam(
+            type="function",
+            name=name,
+            parameters=parameters,
+        )
+        if isinstance(description, str):
+            realtime_tool["description"] = description
+        realtime_tools.append(realtime_tool)
+    return realtime_tools
+
+
+class BaseRealtimeHandler(ConversationHandler, ABC):
+    """Shared realtime stream handler for OpenAI-compatible client APIs."""
+
+    BACKEND_PROVIDER: ClassVar[str]
+    SAMPLE_RATE: ClassVar[int]
+    REFRESH_CLIENT_ON_RECONNECT: ClassVar[bool]
+    AUDIO_INPUT_COST_PER_1M: ClassVar[float]
+    AUDIO_OUTPUT_COST_PER_1M: ClassVar[float]
+    TEXT_INPUT_COST_PER_1M: ClassVar[float]
+    TEXT_OUTPUT_COST_PER_1M: ClassVar[float]
+    IMAGE_INPUT_COST_PER_1M: ClassVar[float]
+
+    _REQUIRED_PROVIDER_CONFIG: ClassVar[tuple[str, ...]] = (
+        "BACKEND_PROVIDER",
+        "SAMPLE_RATE",
+        "REFRESH_CLIENT_ON_RECONNECT",
+        "AUDIO_INPUT_COST_PER_1M",
+        "AUDIO_OUTPUT_COST_PER_1M",
+        "TEXT_INPUT_COST_PER_1M",
+        "TEXT_OUTPUT_COST_PER_1M",
+        "IMAGE_INPUT_COST_PER_1M",
+    )
+
+    def __init_subclass__(cls, **kwargs: Any) -> None:
+        """Require concrete providers to declare their provider configuration."""
+        super().__init_subclass__(**kwargs)
+        missing = [name for name in cls._REQUIRED_PROVIDER_CONFIG if name not in cls.__dict__]
+        if missing:
+            raise TypeError(f"{cls.__name__} must define provider config class variable(s): {', '.join(missing)}")
+
+    def __init__(
+        self,
+        deps: ToolDependencies,
+        gradio_mode: bool = False,
+        instance_path: Optional[str] = None,
+        startup_voice: Optional[str] = None,
+    ):
+        """Initialize the handler."""
+        sample_rate = self.SAMPLE_RATE
+        super().__init__(
+            expected_layout="mono",
+            output_sample_rate=sample_rate,
+            input_sample_rate=sample_rate,
+        )
+
+        self.deps = deps
+
+        self.output_sample_rate = sample_rate
+        self.input_sample_rate = sample_rate
+
+        self.client: AsyncOpenAI
+        self.connection: AsyncRealtimeConnection | None = None
+        self.output_queue: "asyncio.Queue[Tuple[int, NDArray[np.int16]] | AdditionalOutputs]" = asyncio.Queue()
+
+        self.last_activity_time = asyncio.get_event_loop().time()
+        self.start_time = asyncio.get_event_loop().time()
+        self.is_idle_tool_call = False
+        self.gradio_mode = gradio_mode
+        self.instance_path = instance_path
+        self._voice_override: str | None = self._normalize_startup_voice(startup_voice)
+        self._realtime_connect_query: dict[str, str] = {}
+
+        # Debouncing for partial transcripts
+        self.partial_transcript_task: asyncio.Task[None] | None = None
+        self.partial_debounce_delay = 0.5  # seconds
+        self.input_transcript_chunks_by_item = InputTranscriptChunksByItem()
+
+        # Internal lifecycle flags
+        self._connected_event: asyncio.Event = asyncio.Event()
+
+        # Background tool manager
+        self.tool_manager = BackgroundToolManager()
+
+        # Cost tracking
+        self.cumulative_cost: float = 0.0
+
+        # Response-in-progress guard: the Realtime API only allows one active
+        # response per conversation at a time.  A dedicated worker task
+        # (_response_sender_loop) dequeues and sends one request at a time
+        self._pending_responses: asyncio.Queue[dict[str, Any]] = asyncio.Queue()
+        self._response_done_event: asyncio.Event = asyncio.Event()
+        self._response_done_event.set()
+        self._response_started_or_rejected_event: asyncio.Event = asyncio.Event()
+        self._last_response_rejected: bool = False
+        self._turn_user_done_at: float | None = None
+        self._turn_response_created_at: float | None = None
+        self._turn_first_audio_at: float | None = None
+
+    @staticmethod
+    def _sanitize_tool_result_for_model(tool_name: str, tool_result: dict[str, Any]) -> dict[str, Any]:
+        """Remove bulky transport-only fields before echoing tool output back to the model."""
+        if tool_name == "camera" and "b64_im" in tool_result:
+            sanitized = dict(tool_result)
+            sanitized.pop("b64_im", None)
+            sanitized["image_attached"] = True
+            return sanitized
+        return tool_result
+
+    def _normalize_startup_voice(self, voice: str | None) -> str | None:
+        """Return a valid persisted startup voice for this backend, or None."""
+        available_voices = get_available_voices_for_backend(self.BACKEND_PROVIDER)
+        if voice in available_voices:
+            return voice
+        if voice:
+            logger.warning(
+                "Ignoring persisted startup voice %r for backend=%r; expected one of %s",
+                voice,
+                self.BACKEND_PROVIDER,
+                available_voices,
+            )
+        return None
+
+    def _response_done_timeout(self) -> float:
+        """Return the response completion timeout."""
+        return _RESPONSE_DONE_TIMEOUT
+
+    def _connection_closed_errors(self) -> tuple[type[BaseException], ...]:
+        """Return websocket closure exceptions handled as reconnectable/ignorable."""
+        return (ConnectionClosedError,)
+
+    @abstractmethod
+    def _get_session_instructions(self) -> str:
+        """Return session instructions for this backend."""
+
+    @abstractmethod
+    def _get_session_voice(self, default: str | None = None) -> str:
+        """Return the configured session voice for this backend."""
+
+    @abstractmethod
+    def _get_active_tool_specs(self) -> list[dict[str, Any]]:
+        """Return active tool specs for the current session dependencies."""
+
+    @abstractmethod
+    def _get_session_config(self, tool_specs: list[dict[str, Any]]) -> RealtimeSessionCreateRequestParam:
+        """Return the backend-specific realtime session config."""
+
+    async def _wait_for_output_item(self) -> Tuple[int, NDArray[np.int16]] | AdditionalOutputs | None:
+        """Wait for the next output item."""
+        return await wait_for_item(self.output_queue)  # type: ignore[no-any-return]
+
+    def _mark_activity(self, reason: str) -> None:
+        """Record non-idle conversation activity for the idle timer."""
+        self.last_activity_time = asyncio.get_event_loop().time()
+        logger.debug("last activity time updated to %s (%s)", self.last_activity_time, reason)
+
+    def copy(self) -> "BaseRealtimeHandler":
+        """Create a copy of the handler."""
+        return type(self)(
+            self.deps,
+            self.gradio_mode,
+            self.instance_path,
+            startup_voice=self._voice_override,
+        )
+
+    async def change_voice(self, voice: str) -> str:
+        """Change only the voice and restart the session."""
+        self._voice_override = voice
+        if getattr(self, "client", None) is not None:
+            try:
+                await self._restart_session()
+                return f"Voice changed to {voice}."
+            except Exception as e:
+                logger.warning("Failed to restart session for voice change: %s", e)
+                return "Voice change failed. Will take effect on next connection."
+        return "Voice changed. Will take effect on next connection."
+
+    def get_current_voice(self) -> str:
+        """Return the voice currently selected for this handler."""
+        default_voice = get_default_voice_for_backend(self.BACKEND_PROVIDER)
+        return self._voice_override or self._get_session_voice(default=default_voice)
+
+    async def apply_personality(self, profile: str | None) -> str:
+        """Apply a new personality (profile) at runtime if possible.
+
+        - Updates the global config's selected profile for subsequent calls.
+        - If a realtime connection is active, sends a session.update with the
+          freshly resolved instructions so the change takes effect immediately.
+
+        Returns a short status message for UI feedback.
+        """
+        try:
+            # Update the in-process config value and env
+            from reachy_mini_conversation_app.config import config as _config
+            from reachy_mini_conversation_app.config import set_custom_profile
+
+            set_custom_profile(profile)
+            logger.info(
+                "Set custom profile to %r (config=%r)", profile, getattr(_config, "REACHY_MINI_CUSTOM_PROFILE", None)
+            )
+
+            try:
+                instructions = self._get_session_instructions()
+                voice = self.get_current_voice()
+            except BaseException as e:  # catch SystemExit from prompt loader without crashing
+                logger.error("Failed to resolve personality content: %s", e)
+                return f"Failed to apply personality: {e}"
+
+            # Attempt a live update first, then force a full restart to ensure it sticks
+            if self.connection is not None:
+                try:
+                    await self.connection.session.update(
+                        session=RealtimeSessionCreateRequestParam(
+                            type="realtime",
+                            instructions=instructions,
+                            audio=RealtimeAudioConfigParam(
+                                output=RealtimeAudioConfigOutputParam(
+                                    voice=voice,
+                                ),
+                            ),
+                        ),
+                    )
+                    logger.info("Applied personality via live update: %s", profile or "built-in default")
+                except Exception as e:
+                    logger.warning("Live update failed; will restart session: %s", e)
+
+                # Force a real restart to guarantee the new instructions/voice
+                try:
+                    await self._restart_session()
+                    return "Applied personality and restarted realtime session."
+                except Exception as e:
+                    logger.warning("Failed to restart session after apply: %s", e)
+                    return "Applied personality. Will take effect on next connection."
+            else:
+                logger.info(
+                    "Applied personality recorded: %s (no live connection; will apply on next session)",
+                    profile or "built-in default",
+                )
+                return "Applied personality. Will take effect on next connection."
+        except Exception as e:
+            logger.error("Error applying personality '%s': %s", profile, e)
+            return f"Failed to apply personality: {e}"
+
+    async def _emit_debounced_partial(self, transcript: str, item_id: str, sequence_counter: int) -> None:
+        """Emit partial transcript after debounce delay."""
+        try:
+            await asyncio.sleep(self.partial_debounce_delay)
+
+            input_transcript = self.input_transcript_chunks_by_item
+            if input_transcript.item_id == item_id and len(input_transcript.deltas) - 1 == sequence_counter:
+                await self.output_queue.put(AdditionalOutputs({"role": "user_partial", "content": transcript}))
+                logger.debug(f"Debounced partial emitted: {transcript}")
+        except asyncio.CancelledError:
+            logger.debug("Debounced partial cancelled")
+            raise
+
+    def _record_partial_transcript_delta(
+        self,
+        input_transcript: InputTranscriptChunksByItem,
+        item_id: str,
+        delta: str,
+    ) -> None:
+        """Record a suffix delta for a partial transcript."""
+        if input_transcript.item_id != item_id:
+            input_transcript.item_id = item_id
+            input_transcript.deltas = [delta]
+        else:
+            input_transcript.deltas.append(delta)
+
+    def _compute_response_cost(self, usage: Any) -> float:
+        """Compute response cost using this backend's pricing."""
+        inp = getattr(usage, "input_token_details", None)
+        out = getattr(usage, "output_token_details", None)
+        cost = 0.0
+        if inp:
+            cost += (getattr(inp, "audio_tokens", 0) or 0) * self.AUDIO_INPUT_COST_PER_1M / 1e6
+            cost += (getattr(inp, "text_tokens", 0) or 0) * self.TEXT_INPUT_COST_PER_1M / 1e6
+            cost += (getattr(inp, "image_tokens", 0) or 0) * self.IMAGE_INPUT_COST_PER_1M / 1e6
+        if out:
+            cost += (getattr(out, "audio_tokens", 0) or 0) * self.AUDIO_OUTPUT_COST_PER_1M / 1e6
+            cost += (getattr(out, "text_tokens", 0) or 0) * self.TEXT_OUTPUT_COST_PER_1M / 1e6
+        return cost
+
+    async def _prepare_startup_credentials(self) -> None:
+        """Let providers collect any startup credentials they need."""
+
+    def _persist_credentials_if_needed(self) -> None:
+        """Let providers persist credentials after a successful session update."""
+
+    async def start_up(self) -> None:
+        """Start the handler with minimal retries on unexpected websocket closure."""
+        await self._prepare_startup_credentials()
+        self.client = await self._build_realtime_client()
+
+        max_attempts = 3
+        for attempt in range(1, max_attempts + 1):
+            try:
+                await self._run_realtime_session()
+                # Normal exit from the session, stop retrying
+                return
+            except self._connection_closed_errors() as e:
+                # Abrupt close (e.g., "no close frame received or sent") → retry
+                logger.warning("Realtime websocket closed unexpectedly (attempt %d/%d): %s", attempt, max_attempts, e)
+                if attempt < max_attempts:
+                    if self.REFRESH_CLIENT_ON_RECONNECT:
+                        self.client = await self._build_realtime_client()
+                    # exponential backoff with jitter
+                    base_delay = 2 ** (attempt - 1)  # 1s, 2s, 4s, 8s, etc.
+                    jitter = random.uniform(0, 0.5)
+                    delay = base_delay + jitter
+                    logger.info("Retrying in %.1f seconds...", delay)
+                    await asyncio.sleep(delay)
+                    continue
+                raise
+            finally:
+                # never keep a stale reference
+                self.connection = None
+                try:
+                    self._connected_event.clear()
+                except Exception:
+                    pass
+
+    async def _restart_session(self) -> None:
+        """Force-close the current session and start a fresh one in background.
+
+        Does not block the caller while the new session is establishing.
+        """
+        try:
+            if self.connection is not None:
+                try:
+                    await self.connection.close()
+                except Exception:
+                    pass
+                finally:
+                    self.connection = None
+
+            # Ensure we have a client (start_up must have run once)
+            if getattr(self, "client", None) is None:
+                logger.warning("Cannot restart: realtime client not initialized yet.")
+                return
+
+            # Fire-and-forget new session and wait briefly for connection
+            try:
+                self._connected_event.clear()
+            except Exception:
+                pass
+            if self.REFRESH_CLIENT_ON_RECONNECT:
+                self.client = await self._build_realtime_client()
+            asyncio.create_task(self._run_realtime_session(), name="realtime-session-restart")
+            try:
+                await asyncio.wait_for(self._connected_event.wait(), timeout=5.0)
+                logger.info("Realtime session restarted and connected.")
+            except asyncio.TimeoutError:
+                logger.warning("Realtime session restart timed out; continuing in background.")
+        except Exception as e:
+            logger.warning("_restart_session failed: %s", e)
+
+    async def _safe_response_create(self, **kwargs: Any) -> None:
+        """Enqueue a response.create() kwargs for the sender worker _response_sender_loop().
+
+        This method never blocks the caller.
+        """
+        await self._pending_responses.put(kwargs)
+
+    async def _response_sender_loop(self) -> None:
+        """Dedicated worker that sends ``response.create()`` calls serially.
+
+        This logic was designed to comply with the response.create() docstring specification for event ordering:
+        https://github.com/openai/openai-python/blob/3e0c05b84a2056870abf3bd6a5e7849020209cc3/src/openai/resources/realtime/realtime.py#L649C1-L651C30
+
+        For each queued request the worker:
+        1. Waits until no response is active (_response_done_event).
+        2. Sends response.create().
+        3. Waits until the receiver observes response.created or a rejection.
+        4. Waits for the response cycle to complete (response.done).
+        5. If the server rejected with active_response, retries from step 1.
+        """
+        while self.connection:
+            try:
+                kwargs = await self._pending_responses.get()
+            except asyncio.CancelledError:
+                return
+
+            sent = False
+            max_retries = 5
+            attempts = 0
+            while not sent and self.connection and attempts < max_retries:
+                try:
+                    await asyncio.wait_for(
+                        self._response_done_event.wait(),
+                        timeout=self._response_done_timeout(),
+                    )
+                except asyncio.TimeoutError:
+                    logger.debug("Timed out waiting for previous response to finish; forcing ahead")
+                    self._response_done_event.set()
+
+                if not self.connection:
+                    break
+
+                self._last_response_rejected = False
+                self._response_started_or_rejected_event.clear()
+                try:
+                    await self.connection.response.create(**kwargs)
+                except Exception as e:
+                    logger.debug("_response_sender_loop: send failed: %s", e)
+                    self._response_done_event.set()
+                    break
+
+                try:
+                    await asyncio.wait_for(
+                        self._response_started_or_rejected_event.wait(),
+                        timeout=self._response_done_timeout(),
+                    )
+                except asyncio.TimeoutError:
+                    logger.debug("Timed out waiting for response.created or response rejection")
+
+                # Check if the receiver loop observed an asynchronous rejection.
+                if self._last_response_rejected:
+                    attempts += 1
+                    if attempts >= max_retries:
+                        logger.debug("response.create rejected %d times; giving up", attempts)
+                        break
+                    logger.debug("response.create was rejected; retrying (%d/%d)", attempts, max_retries)
+                    await asyncio.sleep(_RESPONSE_REJECTION_RETRY_DELAY)
+                    continue
+
+                try:
+                    await asyncio.wait_for(
+                        self._response_done_event.wait(),
+                        timeout=self._response_done_timeout(),
+                    )
+                except asyncio.TimeoutError:
+                    logger.debug("Timed out waiting for response.done; assuming response completed")
+                    self._response_done_event.set()
+                    break
+
+                sent = True
+
+    async def _handle_tool_result(self, bg_tool: ToolNotification) -> None:
+        """Process the result of a tool call."""
+        if bg_tool.error is not None:
+            logger.error("Tool '%s' (id=%s) failed with error: %s", bg_tool.tool_name, bg_tool.id, bg_tool.error)
+            tool_result = {"error": bg_tool.error}
+            tool_result_for_model = tool_result
+        elif bg_tool.result is not None:
+            tool_result = bg_tool.result
+            tool_result_for_model = (
+                self._sanitize_tool_result_for_model(bg_tool.tool_name, tool_result)
+                if isinstance(tool_result, dict)
+                else tool_result
+            )
+            logger.info(
+                "Tool '%s' (id=%s) executed successfully.",
+                bg_tool.tool_name,
+                bg_tool.id,
+            )
+            logger.debug("Tool '%s' model-visible result: %s", bg_tool.tool_name, tool_result_for_model)
+        else:
+            logger.warning("Tool '%s' (id=%s) returned no result and no error", bg_tool.tool_name, bg_tool.id)
+            tool_result = {"error": "No result returned from tool execution"}
+            tool_result_for_model = tool_result
+
+        # Connection may have closed while tool was running
+        if not self.connection:
+            logger.warning(
+                "Connection closed during tool '%s' (id=%s) execution; cannot send result back",
+                bg_tool.tool_name,
+                bg_tool.id,
+            )
+            return
+
+        try:
+            self._mark_activity("tool_result_ready")
+            if isinstance(bg_tool.id, str):
+                await self.connection.conversation.item.create(
+                    item={
+                        "type": "function_call_output",
+                        "call_id": bg_tool.id,
+                        "output": json.dumps(tool_result_for_model),
+                    },
+                )
+
+            await self.output_queue.put(
+                AdditionalOutputs(
+                    {
+                        "role": "assistant",
+                        "content": json.dumps(tool_result_for_model),
+                        # Gradio UI metadata.status accept only "pending" and "done". Do not accept bg.tool.status values.
+                        "metadata": {
+                            "title": f"🛠️ Used tool {bg_tool.tool_name}",
+                            "status": "done",
+                        },
+                    },
+                ),
+            )
+
+            if bg_tool.tool_name == "camera" and "b64_im" in tool_result:
+                # use raw base64, don't json.dumps (which adds quotes)
+                b64_im = tool_result["b64_im"]
+                if not isinstance(b64_im, str):
+                    logger.warning("Unexpected type for b64_im: %s", type(b64_im))
+                    b64_im = str(b64_im)
+                image_width = tool_result.get("image_width")
+                image_height = tool_result.get("image_height")
+                jpeg_bytes_value = tool_result.get("jpeg_bytes")
+                jpeg_bytes = jpeg_bytes_value if isinstance(jpeg_bytes_value, int) else (len(b64_im) * 3) // 4
+                await self.connection.conversation.item.create(
+                    item={
+                        "type": "message",
+                        "role": "user",
+                        "content": [
+                            {
+                                "type": "input_image",
+                                "image_url": f"data:image/jpeg;base64,{b64_im}",
+                            },
+                        ],
+                    },
+                )
+                if isinstance(image_width, int) and isinstance(image_height, int):
+                    logger.info(
+                        "Added camera image to conversation frame=%sx%s jpeg_bytes=%s",
+                        image_width,
+                        image_height,
+                        jpeg_bytes,
+                    )
+                else:
+                    logger.info(
+                        "Added camera image to conversation jpeg_bytes=%s",
+                        jpeg_bytes,
+                    )
+
+                if self.deps.camera_worker is not None:
+                    np_img = self.deps.camera_worker.get_latest_frame()
+                    if np_img is not None:
+                        # Camera frames are BGR; reverse channels without requiring OpenCV in core installs.
+                        rgb_frame = np_img[:, :, ::-1].copy() if np_img.ndim == 3 and np_img.shape[-1] == 3 else np_img
+                    else:
+                        rgb_frame = None
+                    img = gr.Image(value=rgb_frame)
+
+                    await self.output_queue.put(
+                        AdditionalOutputs(
+                            {
+                                "role": "assistant",
+                                "content": img,
+                            },
+                        ),
+                    )
+
+            # If this tool call was triggered by an idle signal, don't make the robot speak.
+            # For other tool calls, let the robot reply out loud.
+            if not bg_tool.is_idle_tool_call:
+                await self._safe_response_create(
+                    response=RealtimeResponseCreateParamsParam(
+                        instructions="Use the tool result just returned and answer concisely in speech.",
+                    ),
+                )
+
+        except self._connection_closed_errors():
+            logger.warning("Connection closed while sending tool result")
+            self.connection = None
+            self._response_done_event.set()
+
+    async def _run_realtime_session(self) -> None:
+        """Establish and manage a single realtime session."""
+        tool_specs = self._get_active_tool_specs()
+        logger.info(
+            "Tools to be used in conversation: %s",
+            [tool["name"] for tool in tool_specs],
+        )
+        connect_kwargs: dict[str, Any] = {}
+        if config.MODEL_NAME:
+            connect_kwargs["model"] = config.MODEL_NAME
+        if self._realtime_connect_query:
+            connect_kwargs["extra_query"] = self._realtime_connect_query
+        async with self.client.realtime.connect(**connect_kwargs) as conn:
+            try:
+                session_config = self._get_session_config(tool_specs)
+                await conn.session.update(session=session_config)
+                logger.info(
+                    "Realtime session initialized with profile=%r voice=%r",
+                    getattr(config, "REACHY_MINI_CUSTOM_PROFILE", None),
+                    self.get_current_voice(),
+                )
+                self._persist_credentials_if_needed()
+            except Exception:
+                logger.exception("Realtime session.update failed; aborting startup")
+                raise
+
+            logger.info("Realtime session updated successfully")
+
+            # Reset the partial-transcript accumulator for each new session
+            self.input_transcript_chunks_by_item = InputTranscriptChunksByItem()
+
+            # Manage events received from the realtime server.
+            self.connection = conn
+            try:
+                self._connected_event.set()
+            except Exception:
+                pass
+
+            response_sender_task: asyncio.Task[None] | None = None
+            try:
+                # Start the background tool manager
+                self.tool_manager.start_up(tool_callbacks=[self._handle_tool_result])
+
+                # Start the response sender worker
+                response_sender_task = asyncio.create_task(self._response_sender_loop(), name="response-sender")
+
+                async for event in self.connection:
+                    logger.debug("Realtime event: %s", event.type)
+                    if event.type == "input_audio_buffer.speech_started":
+                        self._mark_activity("user_speech_started")
+                        self._turn_user_done_at = None
+                        self._turn_response_created_at = None
+                        self._turn_first_audio_at = None
+                        if hasattr(self, "_clear_queue") and callable(self._clear_queue):
+                            self._clear_queue()
+                        if self.deps.head_wobbler is not None:
+                            self.deps.head_wobbler.reset()
+                        self.deps.movement_manager.set_listening(True)
+                        logger.debug("User speech started")
+
+                    if event.type == "input_audio_buffer.speech_stopped":
+                        self._mark_activity("user_speech_stopped")
+                        self.deps.movement_manager.set_listening(False)
+                        logger.debug("User speech stopped - server will auto-commit with VAD")
+
+                    if event.type == "response.output_audio.done":
+                        if self.deps.head_wobbler is not None:
+                            self.deps.head_wobbler.request_reset_after_current_audio()
+                        logger.debug("response completed")
+
+                    if event.type == "response.created":
+                        self._mark_activity("response_created")
+                        self._response_done_event.clear()
+                        self._response_started_or_rejected_event.set()
+                        if self._turn_user_done_at is not None and self._turn_response_created_at is None:
+                            self._turn_response_created_at = time.perf_counter()
+                            delta_ms = (self._turn_response_created_at - self._turn_user_done_at) * 1000
+                            logger.info("Turn latency: response.created %.0f ms after user transcript", delta_ms)
+                        logger.debug("Response created (active)")
+
+                    if event.type == "response.done":
+                        # Doesn't mean the audio is done playing
+                        self._response_done_event.set()
+                        self._response_started_or_rejected_event.set()
+                        self.is_idle_tool_call = False
+                        logger.debug("Response done")
+
+                        response = getattr(event, "response", None)
+                        usage = getattr(response, "usage", None) if response else None
+                        if usage:
+                            cost = self._compute_response_cost(usage)
+                            self.cumulative_cost += cost
+                            logger.debug("Cost: $%.4f | Cumulative: $%.4f", cost, self.cumulative_cost)
+                        else:
+                            logger.warning("No usage data available for cost tracking")
+
+                    if event.type == "conversation.item.input_audio_transcription.delta":
+                        self._mark_activity("user_transcription_delta")
+                        logger.debug(f"User partial transcript: {event.delta}")
+
+                        item_id = event.item_id
+                        delta = event.delta or ""
+
+                        input_transcript = self.input_transcript_chunks_by_item
+                        self._record_partial_transcript_delta(input_transcript, item_id, delta)
+
+                        current_partial = "".join(input_transcript.deltas)
+                        sequence_counter = len(input_transcript.deltas) - 1
+
+                        # Cancel previous debounce task if it exists
+                        if self.partial_transcript_task and not self.partial_transcript_task.done():
+                            self.partial_transcript_task.cancel()
+                            try:
+                                await self.partial_transcript_task
+                            except asyncio.CancelledError:
+                                pass
+
+                        # Start new debounce timer with the last delta
+                        self.partial_transcript_task = asyncio.create_task(
+                            self._emit_debounced_partial(current_partial, item_id, sequence_counter)
+                        )
+
+                    # Handle completed transcription (user finished speaking)
+                    if event.type == "conversation.item.input_audio_transcription.completed":
+                        self._mark_activity("user_transcription_completed")
+                        raw_transcript = event.transcript or ""
+                        transcript = raw_transcript.strip()
+                        logger.debug("User transcript: %s", raw_transcript)
+                        self.deps.movement_manager.set_listening(False)
+
+                        # Cancel any pending partial emission
+                        if self.partial_transcript_task and not self.partial_transcript_task.done():
+                            self.partial_transcript_task.cancel()
+                            try:
+                                await self.partial_transcript_task
+                            except asyncio.CancelledError:
+                                pass
+
+                        if not transcript:
+                            logger.debug("Ignoring empty user transcript")
+                            continue
+
+                        self._turn_user_done_at = time.perf_counter()
+                        self._turn_response_created_at = None
+                        self._turn_first_audio_at = None
+
+                        await self.output_queue.put(AdditionalOutputs({"role": "user", "content": transcript}))
+
+                    # Handle assistant transcription
+                    if event.type == "response.output_audio_transcript.done":
+                        self._mark_activity("assistant_transcript_done")
+                        logger.debug(f"Assistant transcript: {event.transcript}")
+                        await self.output_queue.put(
+                            AdditionalOutputs({"role": "assistant", "content": event.transcript})
+                        )
+
+                    # Handle audio delta
+                    if event.type == "response.output_audio.delta":
+                        decoded_pcm_bytes = base64.b64decode(event.delta)
+                        decoded_pcm = np.frombuffer(decoded_pcm_bytes, dtype=np.int16).reshape(1, -1)
+                        if self.gradio_mode and self.deps.head_wobbler is not None:
+                            self.deps.head_wobbler.feed_pcm(decoded_pcm, self.output_sample_rate)
+                        self._mark_activity("assistant_audio_delta")
+                        if self._turn_user_done_at is not None and self._turn_first_audio_at is None:
+                            self._turn_first_audio_at = time.perf_counter()
+                            delta_ms = (self._turn_first_audio_at - self._turn_user_done_at) * 1000
+                            logger.info("Turn latency: first audio delta %.0f ms after user transcript", delta_ms)
+                        await self.output_queue.put(
+                            (
+                                self.output_sample_rate,
+                                decoded_pcm,
+                            ),
+                        )
+                    # ---- tool-calling plumbing ----
+                    if event.type == "response.function_call_arguments.done":
+                        self._mark_activity("tool_call_received")
+                        tool_name = getattr(event, "name", None)
+                        args_json_str = getattr(event, "arguments", None)
+                        call_id: str = str(getattr(event, "call_id", uuid.uuid4()))
+
+                        logger.info(
+                            "Tool call received — tool_name=%r, call_id=%s, is_idle=%s, args=%s",
+                            tool_name,
+                            call_id,
+                            self.is_idle_tool_call,
+                            args_json_str,
+                        )
+
+                        if not isinstance(tool_name, str) or not isinstance(args_json_str, str):
+                            logger.error(
+                                "Invalid tool call: tool_name=%s (type=%s), args=%s (type=%s), call_id=%s",
+                                tool_name,
+                                type(tool_name).__name__,
+                                args_json_str,
+                                type(args_json_str).__name__,
+                                call_id,
+                            )
+                            continue
+
+                        bg_tool = await self.tool_manager.start_tool(
+                            call_id=call_id,
+                            tool_call_routine=ToolCallRoutine(
+                                tool_name=tool_name,
+                                args_json_str=args_json_str,
+                                deps=self.deps,
+                            ),
+                            is_idle_tool_call=self.is_idle_tool_call,
+                        )
+
+                        await self.output_queue.put(
+                            AdditionalOutputs(
+                                {
+                                    "role": "assistant",
+                                    "content": f"🛠️ Used tool {tool_name} with args {args_json_str}. The tool is now running. Tool ID: {bg_tool.tool_id}",
+                                },
+                            ),
+                        )
+                        logger.info(
+                            "Started background tool: %s (id=%s, call_id=%s)", tool_name, bg_tool.tool_id, call_id
+                        )
+
+                    # server error
+                    if event.type == "error":
+                        err = getattr(event, "error", None)
+                        msg = getattr(err, "message", str(err) if err else "unknown error")
+                        code = getattr(err, "code", "") or getattr(err, "type", "")
+
+                        if code == "conversation_already_has_active_response":
+                            # response.create was rejected.  The sender worker
+                            # is waiting on _response_done_event; when the active
+                            # response finishes it will wake up and see this flag.
+                            self._last_response_rejected = True
+                            self._response_started_or_rejected_event.set()
+                            logger.debug("response.create rejected; worker will retry after active response finishes")
+                        else:
+                            self._response_started_or_rejected_event.set()
+                            logger.error("Realtime error [%s]: %s (raw=%s)", code, msg, err)
+
+                        if code == "input_audio_buffer_commit_empty":
+                            self.deps.movement_manager.set_listening(False)
+
+                        # Only show user-facing errors, not internal state errors.
+                        if code not in ("input_audio_buffer_commit_empty", "conversation_already_has_active_response"):
+                            await self.output_queue.put(
+                                AdditionalOutputs({"role": "assistant", "content": f"[error] {msg}"})
+                            )
+            finally:
+                # Stop the response sender worker.
+                if response_sender_task is not None:
+                    response_sender_task.cancel()
+                    try:
+                        await response_sender_task
+                    except asyncio.CancelledError:
+                        pass
+
+                # Stop background tool manager tasks (listener + cleanup) in all paths.
+                await self.tool_manager.shutdown()
+
+    # Microphone receive
+    async def receive(self, frame: Tuple[int, NDArray[np.int16]]) -> None:
+        """Receive audio frame from the microphone and send it to the realtime server.
+
+        Handles both mono and stereo audio formats, converting to the expected
+        mono format for the realtime API. Resamples if the input sample rate differs
+        from the expected rate.
+
+        Args:
+            frame: A tuple containing (sample_rate, audio_data).
+
+        """
+        if not self.connection:
+            return
+
+        input_sample_rate, audio_frame = frame
+
+        # Reshape if needed
+        if audio_frame.ndim == 2:
+            # Scipy channels last convention
+            if audio_frame.shape[1] > audio_frame.shape[0]:
+                audio_frame = audio_frame.T
+            # Multiple channels -> Mono channel
+            if audio_frame.shape[1] > 1:
+                audio_frame = audio_frame[:, 0]
+
+        # Resample if needed
+        if self.input_sample_rate != input_sample_rate:
+            audio_frame = resample(audio_frame, int(len(audio_frame) * self.input_sample_rate / input_sample_rate))
+
+        # Cast if needed
+        audio_frame = audio_to_int16(audio_frame)
+
+        # Send to the realtime input buffer (guard against races during reconnect).
+        try:
+            audio_message = base64.b64encode(audio_frame.tobytes()).decode("utf-8")
+            await self.connection.input_audio_buffer.append(audio=audio_message)
+        except Exception as e:
+            logger.debug("Dropping audio frame: connection not ready (%s)", e)
+            return
+
+    async def emit(self) -> Tuple[int, NDArray[np.int16]] | AdditionalOutputs | None:
+        """Emit audio frame to be played by the speaker."""
+        # Sends output queued by the realtime event handler to the stream.
+        # This is called periodically by the fastrtc Stream
+
+        # Handle idle
+        idle_duration = asyncio.get_event_loop().time() - self.last_activity_time
+        if idle_duration > 180.0 and self._response_done_event.is_set() and self.deps.movement_manager.is_idle():
+            try:
+                await self.send_idle_signal(idle_duration)
+            except Exception as e:
+                logger.warning("Idle signal skipped (connection closed?): %s", e)
+                return None
+
+            self.last_activity_time = asyncio.get_event_loop().time()  # avoid repeated resets
+
+        return await self._wait_for_output_item()
+
+    async def shutdown(self) -> None:
+        """Shutdown the handler."""
+        # Unblock the response sender worker so it can exit
+        self._response_done_event.set()
+
+        # Stop background tool manager tasks (listener + cleanup)
+        await self.tool_manager.shutdown()
+
+        # Cancel any pending debounce task
+        if self.partial_transcript_task and not self.partial_transcript_task.done():
+            self.partial_transcript_task.cancel()
+            try:
+                await self.partial_transcript_task
+            except asyncio.CancelledError:
+                pass
+
+        if self.connection:
+            try:
+                await self.connection.close()
+            except self._connection_closed_errors() as e:
+                logger.debug(f"Connection already closed during shutdown: {e}")
+            except Exception as e:
+                logger.debug(f"connection.close() ignored: {e}")
+            finally:
+                self.connection = None
+
+        # Clear any remaining items in the output queue
+        while not self.output_queue.empty():
+            try:
+                self.output_queue.get_nowait()
+            except asyncio.QueueEmpty:
+                break
+
+    def format_timestamp(self) -> str:
+        """Format current timestamp with date, time, and elapsed seconds."""
+        loop_time = asyncio.get_event_loop().time()  # monotonic
+        elapsed_seconds = loop_time - self.start_time
+        dt = datetime.now()  # wall-clock
+        return f"[{dt.strftime('%Y-%m-%d %H:%M:%S')} | +{elapsed_seconds:.1f}s]"
+
+    async def get_available_voices(self) -> list[str]:
+        """Return available voices for this backend."""
+        return get_available_voices_for_backend(self.BACKEND_PROVIDER)
+
+    @abstractmethod
+    async def _build_realtime_client(self) -> AsyncOpenAI:
+        """Build the realtime SDK client for this backend."""
+
+    async def send_idle_signal(self, idle_duration: float) -> None:
+        """Send an idle signal to the realtime server."""
+        logger.debug("Sending idle signal")
+        self.is_idle_tool_call = True
+        timestamp_msg = f"[Idle time update: {self.format_timestamp()} - No activity for {idle_duration:.1f}s] You've been idle for a while. Feel free to get creative - dance, show an emotion, look around, call idle_do_nothing to stay still and silent, or just be yourself!"
+        if not self.connection:
+            logger.debug("No connection, cannot send idle signal")
+            return
+        await self.connection.conversation.item.create(
+            item={
+                "type": "message",
+                "role": "user",
+                "content": [{"type": "input_text", "text": timestamp_msg}],
+            },
+        )
+        await self._safe_response_create(
+            response=RealtimeResponseCreateParamsParam(
+                instructions="You MUST respond with function calls only - no speech or text. Choose appropriate actions for idle behavior. Use idle_do_nothing only if you intentionally want no movement or sound during this idle turn.",
+                tool_choice="required",
+            ),
+        )

src/reachy_mini_conversation_app/config.py

@@ -2,6 +2,8 @@ import os
 import sys
 import logging
 from pathlib import Path
+from dataclasses import dataclass
+from urllib.parse import urlsplit, parse_qsl, urlunsplit
 from importlib.resources import files
 
 from dotenv import find_dotenv, load_dotenv
@@ -56,6 +58,20 @@ AVAILABLE_VOICES: list[str] = [
     "shimmer",
     "verse",
 ]
+OPENAI_DEFAULT_VOICE = "cedar"
+
+# Qwen3-TTS CustomVoice speaker catalog from the deployed Hugging Face backend.
+HF_AVAILABLE_VOICES: list[str] = [
+    "Aiden",
+    "Ryan",
+    "Dylan",
+    "Eric",
+    "Ono_Anna",
+    "Serena",
+    "Sohee",
+    "Uncle_Fu",
+    "Vivian",
+]
 
 # Voices supported by the Gemini Live API
 GEMINI_AVAILABLE_VOICES: list[str] = [
@@ -71,14 +87,45 @@ GEMINI_AVAILABLE_VOICES: list[str] = [
 
 OPENAI_BACKEND = "openai"
 GEMINI_BACKEND = "gemini"
-DEFAULT_BACKEND_PROVIDER = OPENAI_BACKEND
+HF_BACKEND = "huggingface"
+DEFAULT_BACKEND_PROVIDER = HF_BACKEND
+HF_REALTIME_CONNECTION_MODE_ENV = "HF_REALTIME_CONNECTION_MODE"
+HF_REALTIME_WS_URL_ENV = "HF_REALTIME_WS_URL"
+HF_LOCAL_CONNECTION_MODE = "local"
+HF_DEPLOYED_CONNECTION_MODE = "deployed"
+HF_REALTIME_SESSION_PROXY_URL = "https://pollen-robotics-reachy-mini-realtime-url.hf.space/session"
+
+
+@dataclass(frozen=True)
+class HFBackendDefaults:
+    """Defaults for the Hugging Face realtime backend."""
+
+    connection_mode: str = HF_DEPLOYED_CONNECTION_MODE
+    # App-managed Hugging Face Space proxy. The Space forwards to the current
+    # session allocator, so allocator changes do not require app releases.
+    # Users who need a custom target should use HF_REALTIME_CONNECTION_MODE=local
+    # with HF_REALTIME_WS_URL.
+    session_url: str = HF_REALTIME_SESSION_PROXY_URL
+    voice: str = "Aiden"
+    model_name: str = ""
+    direct_port: int = 8765
+
+
+HF_DEFAULTS = HFBackendDefaults()
 DEFAULT_MODEL_NAME_BY_BACKEND = {
     OPENAI_BACKEND: "gpt-realtime",
     GEMINI_BACKEND: "gemini-3.1-flash-live-preview",
+    HF_BACKEND: HF_DEFAULTS.model_name,
+}
+BACKEND_LABEL_BY_PROVIDER = {
+    OPENAI_BACKEND: "OpenAI Realtime",
+    GEMINI_BACKEND: "Gemini Live",
+    HF_BACKEND: "Hugging Face",
 }
 DEFAULT_VOICE_BY_BACKEND = {
-    OPENAI_BACKEND: "cedar",
+    OPENAI_BACKEND: OPENAI_DEFAULT_VOICE,
     GEMINI_BACKEND: "Kore",
+    HF_BACKEND: HF_DEFAULTS.voice,
 }
 
 logger = logging.getLogger(__name__)
@@ -94,10 +141,13 @@ def _normalize_backend_provider(
     backend_provider: str | None = None,
     model_name: str | None = None,
 ) -> str:
-    """Normalize backend selection, falling back to MODEL_NAME for compatibility."""
+    """Normalize the configured backend provider."""
     candidate = (backend_provider or "").strip().lower()
     if candidate in DEFAULT_MODEL_NAME_BY_BACKEND:
         return candidate
+    if candidate:
+        expected = ", ".join(sorted(DEFAULT_MODEL_NAME_BY_BACKEND))
+        raise ValueError(f"Invalid BACKEND_PROVIDER={backend_provider!r}. Expected one of: {expected}.")
     return GEMINI_BACKEND if _is_gemini_model_name(model_name) else DEFAULT_BACKEND_PROVIDER
 
 
@@ -107,11 +157,14 @@ def _resolve_model_name(
 ) -> str:
     """Return a model name that matches the selected backend provider."""
     normalized_backend = _normalize_backend_provider(backend_provider, model_name)
+    if normalized_backend == HF_BACKEND:
+        return DEFAULT_MODEL_NAME_BY_BACKEND[HF_BACKEND]
+
     candidate = (model_name or "").strip()
     if candidate:
         if normalized_backend == GEMINI_BACKEND and _is_gemini_model_name(candidate):
             return candidate
-        if normalized_backend == OPENAI_BACKEND and not _is_gemini_model_name(candidate):
+        if normalized_backend != GEMINI_BACKEND and not _is_gemini_model_name(candidate):
             return candidate
         logger.warning(
             "MODEL_NAME=%r does not match BACKEND_PROVIDER=%r, using default %r",
@@ -142,6 +195,92 @@ def _env_flag(name: str, default: bool = False) -> bool:
     return default
 
 
+def _normalize_hf_connection_mode(value: str | None) -> str | None:
+    """Normalize the Hugging Face connection mode, if explicitly configured."""
+    candidate = (value or "").strip().lower()
+    if not candidate:
+        return None
+
+    if candidate not in {HF_LOCAL_CONNECTION_MODE, HF_DEPLOYED_CONNECTION_MODE}:
+        logger.warning(
+            "Invalid %s=%r. Expected local or deployed.",
+            HF_REALTIME_CONNECTION_MODE_ENV,
+            value,
+        )
+        return None
+    return candidate
+
+
+@dataclass(frozen=True)
+class HFConnectionSelection:
+    """Resolved Hugging Face connection mode and target availability."""
+
+    mode: str
+    has_target: bool
+    session_url: str | None = None
+    direct_ws_url: str | None = None
+
+
+@dataclass(frozen=True)
+class HFRealtimeURLParts:
+    """Parsed Hugging Face realtime URL components used by UI and client setup."""
+
+    base_url: str
+    websocket_base_url: str
+    connect_query: dict[str, str]
+    host: str | None
+    port: int | None
+    has_realtime_path: bool
+
+
+def parse_hf_realtime_url(realtime_url: str) -> HFRealtimeURLParts:
+    """Parse a Hugging Face realtime URL into OpenAI-compatible client endpoints."""
+    parsed = urlsplit(realtime_url)
+    scheme = parsed.scheme.lower()
+    if scheme not in {"ws", "wss", "http", "https"}:
+        raise ValueError(
+            "Expected Hugging Face realtime URL to start with ws://, wss://, http://, or https://, "
+            f"got: {realtime_url}"
+        )
+
+    path = parsed.path.rstrip("/")
+    has_realtime_path = path.endswith("/realtime")
+    if has_realtime_path:
+        base_path = path[: -len("/realtime")]
+    else:
+        base_path = path
+
+    connect_query = {key: value for key, value in parse_qsl(parsed.query, keep_blank_values=True) if key != "model"}
+    http_scheme = "https" if scheme in {"wss", "https"} else "http"
+    websocket_scheme = "wss" if scheme in {"wss", "https"} else "ws"
+    base_url = urlunsplit((http_scheme, parsed.netloc, base_path, "", ""))
+    websocket_base_url = urlunsplit((websocket_scheme, parsed.netloc, base_path, "", ""))
+    return HFRealtimeURLParts(
+        base_url=base_url,
+        websocket_base_url=websocket_base_url,
+        connect_query=connect_query,
+        host=parsed.hostname,
+        port=parsed.port or HF_DEFAULTS.direct_port,
+        has_realtime_path=has_realtime_path,
+    )
+
+
+def parse_hf_direct_target(ws_url: str | None) -> tuple[str | None, int | None]:
+    """Extract host and port from a direct Hugging Face realtime URL."""
+    if not ws_url:
+        return None, None
+    try:
+        parsed = parse_hf_realtime_url(ws_url)
+        return parsed.host, parsed.port
+    except Exception:
+        return None, None
+
+
+def build_hf_direct_ws_url(host: str, port: int) -> str:
+    """Build the direct Hugging Face realtime websocket URL used by the app."""
+    return f"ws://{host}:{port}/v1/realtime"
+
+
 def _collect_profile_names(profiles_root: Path) -> set[str]:
     """Return profile folder names from a profiles root directory."""
     if not profiles_root.exists() or not profiles_root.is_dir():
@@ -218,14 +357,23 @@ class Config:
         os.getenv("MODEL_NAME"),
     )
     MODEL_NAME = _resolve_model_name(BACKEND_PROVIDER, os.getenv("MODEL_NAME"))
+    HF_REALTIME_CONNECTION_MODE = (
+        _normalize_hf_connection_mode(os.getenv(HF_REALTIME_CONNECTION_MODE_ENV)) or HF_DEFAULTS.connection_mode
+    )
+    # Deliberately ignore HF_REALTIME_SESSION_URL from the environment; the app-managed proxy is HF_DEFAULTS.session_url.
+    HF_REALTIME_SESSION_URL = HF_DEFAULTS.session_url
+    HF_REALTIME_WS_URL = os.getenv(HF_REALTIME_WS_URL_ENV)
     HF_HOME = os.getenv("HF_HOME", "./cache")
     LOCAL_VISION_MODEL = os.getenv("LOCAL_VISION_MODEL", "HuggingFaceTB/SmolVLM2-2.2B-Instruct")
     HF_TOKEN = os.getenv("HF_TOKEN")  # Optional, falls back to hf auth login if not set
 
     logger.debug(
-        "Backend provider: %s, Model: %s, HF_HOME: %s, Vision Model: %s",
+        "Backend provider: %s, Model: %s, HF mode: %s, HF session URL set: %s, HF direct URL set: %s, HF_HOME: %s, Vision Model: %s",
         BACKEND_PROVIDER,
         MODEL_NAME,
+        HF_REALTIME_CONNECTION_MODE,
+        bool(HF_REALTIME_SESSION_URL and HF_REALTIME_SESSION_URL.strip()),
+        bool(HF_REALTIME_WS_URL and HF_REALTIME_WS_URL.strip()),
         HF_HOME,
         LOCAL_VISION_MODEL,
     )
@@ -312,6 +460,15 @@ def refresh_runtime_config_from_env() -> None:
         os.getenv("MODEL_NAME"),
     )
     config.MODEL_NAME = _resolve_model_name(config.BACKEND_PROVIDER, os.getenv("MODEL_NAME"))
+    config.HF_REALTIME_CONNECTION_MODE = (
+        _normalize_hf_connection_mode(os.getenv(HF_REALTIME_CONNECTION_MODE_ENV)) or HF_DEFAULTS.connection_mode
+    )
+    # Deliberately ignore HF_REALTIME_SESSION_URL from the environment; the app-managed proxy is HF_DEFAULTS.session_url.
+    config.HF_REALTIME_SESSION_URL = HF_DEFAULTS.session_url
+    config.HF_REALTIME_WS_URL = os.getenv(HF_REALTIME_WS_URL_ENV)
+    config.HF_HOME = os.getenv("HF_HOME", "./cache")
+    config.LOCAL_VISION_MODEL = os.getenv("LOCAL_VISION_MODEL", "HuggingFaceTB/SmolVLM2-2.2B-Instruct")
+    config.HF_TOKEN = os.getenv("HF_TOKEN")
     config.REACHY_MINI_CUSTOM_PROFILE = LOCKED_PROFILE or os.getenv("REACHY_MINI_CUSTOM_PROFILE")
 
 
@@ -327,11 +484,19 @@ def get_model_name_for_backend(backend: str) -> str:
     return DEFAULT_MODEL_NAME_BY_BACKEND[_normalize_backend_provider(backend)]
 
 
+def get_backend_label(backend: str | None = None) -> str:
+    """Return a human-readable label for a backend selector value."""
+    normalized_backend = get_backend_choice() if backend is None else _normalize_backend_provider(backend)
+    return BACKEND_LABEL_BY_PROVIDER[normalized_backend]
+
+
 def get_available_voices_for_backend(backend: str | None = None) -> list[str]:
     """Return the curated voice list for a backend selector value."""
     normalized_backend = get_backend_choice() if backend is None else _normalize_backend_provider(backend)
     if normalized_backend == GEMINI_BACKEND:
         return list(GEMINI_AVAILABLE_VOICES)
+    if normalized_backend == HF_BACKEND:
+        return list(HF_AVAILABLE_VOICES)
     return list(AVAILABLE_VOICES)
 
 
@@ -341,6 +506,41 @@ def get_default_voice_for_backend(backend: str | None = None) -> str:
     return DEFAULT_VOICE_BY_BACKEND[normalized_backend]
 
 
+def get_hf_session_url() -> str | None:
+    """Return the built-in Hugging Face session proxy URL, if any."""
+    value = (getattr(config, "HF_REALTIME_SESSION_URL", None) or "").strip()
+    return value or None
+
+
+def get_hf_direct_ws_url() -> str | None:
+    """Return the configured direct Hugging Face realtime URL, if any."""
+    value = (getattr(config, "HF_REALTIME_WS_URL", None) or "").strip()
+    return value or None
+
+
+def get_hf_connection_selection() -> HFConnectionSelection:
+    """Resolve the selected Hugging Face connection mode and whether it is usable."""
+    session_url = get_hf_session_url()
+    direct_ws_url = get_hf_direct_ws_url()
+    mode = _normalize_hf_connection_mode(getattr(config, "HF_REALTIME_CONNECTION_MODE", None))
+    if mode is None:
+        raise RuntimeError(f"{HF_REALTIME_CONNECTION_MODE_ENV} must be set to local or deployed.")
+
+    target = direct_ws_url if mode == HF_LOCAL_CONNECTION_MODE else session_url
+
+    return HFConnectionSelection(
+        mode=mode,
+        has_target=bool(target),
+        session_url=session_url,
+        direct_ws_url=direct_ws_url,
+    )
+
+
+def has_hf_realtime_target() -> bool:
+    """Return whether Hugging Face has a target for the selected mode."""
+    return get_hf_connection_selection().has_target
+
+
 def is_gemini_model() -> bool:
     """Return True if the configured MODEL_NAME is a Gemini Live model."""
     return get_backend_choice() == GEMINI_BACKEND

src/reachy_mini_conversation_app/console.py

@@ -24,24 +24,31 @@ from scipy.signal import resample
 from reachy_mini import ReachyMini
 from reachy_mini.media.media_manager import MediaBackend
 from reachy_mini_conversation_app.config import (
+    HF_BACKEND,
     GEMINI_BACKEND,
     LOCKED_PROFILE,
     OPENAI_BACKEND,
+    HF_REALTIME_WS_URL_ENV,
+    HF_LOCAL_CONNECTION_MODE,
+    HF_DEPLOYED_CONNECTION_MODE,
+    HF_REALTIME_CONNECTION_MODE_ENV,
     config,
     get_backend_choice,
+    get_hf_session_url,
+    get_hf_direct_ws_url,
+    build_hf_direct_ws_url,
+    has_hf_realtime_target,
+    parse_hf_direct_target,
     get_model_name_for_backend,
+    get_hf_connection_selection,
     refresh_runtime_config_from_env,
 )
-from reachy_mini_conversation_app.openai_realtime import OpenaiRealtimeHandler
+from reachy_mini_conversation_app.startup_settings import read_startup_settings, write_startup_settings
+from reachy_mini_conversation_app.audio.startup_config import apply_audio_startup_config
+from reachy_mini_conversation_app.conversation_handler import ConversationHandler
 from reachy_mini_conversation_app.headless_personality_ui import mount_personality_routes
 
 
-try:
-    from reachy_mini_conversation_app.gemini_live import GeminiLiveHandler
-except ImportError:
-    GeminiLiveHandler = None  # type: ignore[misc,assignment]
-
-
 try:
     # FastAPI is provided by the Reachy Mini Apps runtime
     from fastapi import FastAPI, Response
@@ -58,6 +65,17 @@ except Exception:  # pragma: no cover - only loaded when settings_app is used
 
 logger = logging.getLogger(__name__)
 
+LOCAL_PLAYER_BACKEND = (
+    getattr(MediaBackend, "LOCAL", None)
+    or getattr(MediaBackend, "GSTREAMER", None)
+    or getattr(MediaBackend, "DEFAULT", None)
+)
+
+LEGACY_STARTUP_ENV_NAMES = (
+    "REACHY_MINI_CUSTOM_PROFILE",
+    "REACHY_MINI_VOICE_OVERRIDE",
+)
+
 
 def _estimate_pending_playback_seconds(robot: ReachyMini) -> float:
     """Best-effort estimate of audio still queued in the local player."""
@@ -84,13 +102,13 @@ class LocalStream:
 
     def __init__(
         self,
-        handler: "OpenaiRealtimeHandler | GeminiLiveHandler",
+        handler: ConversationHandler,
         robot: ReachyMini,
         *,
         settings_app: Optional[FastAPI] = None,
         instance_path: Optional[str] = None,
     ):
-        """Initialize the stream with an OpenAI realtime handler and pipelines.
+        """Initialize the stream with a realtime handler and pipelines.
 
         - ``settings_app``: the Reachy Mini Apps FastAPI to attach settings endpoints.
         - ``instance_path``: directory where per-instance ``.env`` should be stored.
@@ -105,6 +123,7 @@ class LocalStream:
         self._instance_path: Optional[str] = instance_path
         self._settings_initialized = False
         self._asyncio_loop = None
+        self._active_backend_name = get_backend_choice()
 
     # ---- Settings UI ----
     def _read_env_lines(self, env_path: Path) -> list[str]:
@@ -143,8 +162,7 @@ class LocalStream:
 
     def _active_backend(self) -> str:
         """Return the backend family of the currently running handler."""
-        handler_name = type(self.handler).__name__.lower()
-        return GEMINI_BACKEND if "gemini" in handler_name else OPENAI_BACKEND
+        return self._active_backend_name
 
     @staticmethod
     def _has_key(value: Optional[str]) -> bool:
@@ -155,8 +173,19 @@ class LocalStream:
         """Return whether the requested backend has its required credential."""
         if backend == GEMINI_BACKEND:
             return self._has_key(config.GEMINI_API_KEY)
+        if backend == HF_BACKEND:
+            return has_hf_realtime_target()
         return self._has_key(config.OPENAI_API_KEY)
 
+    @staticmethod
+    def _requirement_name(backend: str) -> str:
+        """Return the env var users need for a backend, if any."""
+        if backend == GEMINI_BACKEND:
+            return "GEMINI_API_KEY"
+        if backend == HF_BACKEND:
+            return HF_REALTIME_WS_URL_ENV
+        return "OPENAI_API_KEY"
+
     def _persist_env_value(self, env_name: str, value: str) -> None:
         """Persist a non-empty environment value in memory and in the instance `.env`."""
         self._persist_env_values({env_name: value})
@@ -204,6 +233,48 @@ class LocalStream:
         except Exception as e:
             logger.warning("Failed to persist %s: %s", ", ".join(sorted(normalized_updates)), e)
 
+    def _remove_persisted_env_values(self, env_names: tuple[str, ...]) -> None:
+        """Remove keys from the instance `.env` without mutating the current runtime."""
+        normalized_names = tuple(sorted({name.strip() for name in env_names if name and name.strip()}))
+        if not normalized_names or not self._instance_path:
+            return
+
+        env_path = Path(self._instance_path) / ".env"
+        if not env_path.exists():
+            return
+
+        try:
+            lines = env_path.read_text(encoding="utf-8").splitlines()
+            filtered_lines = [
+                line
+                for line in lines
+                if not any(line.strip().startswith(f"{env_name}=") for env_name in normalized_names)
+            ]
+            if filtered_lines == lines:
+                return
+
+            final_text = "\n".join(filtered_lines)
+            if final_text:
+                final_text += "\n"
+            env_path.write_text(final_text, encoding="utf-8")
+            logger.info("Removed %s from %s", ", ".join(normalized_names), env_path)
+        except Exception as e:
+            logger.warning("Failed to remove %s: %s", ", ".join(normalized_names), e)
+
+    def _persist_hf_direct_connection(self, host: str, port: int) -> None:
+        """Persist a direct Hugging Face websocket target."""
+        self._persist_env_values(
+            {
+                HF_REALTIME_CONNECTION_MODE_ENV: HF_LOCAL_CONNECTION_MODE,
+                HF_REALTIME_WS_URL_ENV: build_hf_direct_ws_url(host, port),
+            }
+        )
+
+    def _persist_hf_allocator_connection(self) -> None:
+        """Persist the deployed Hugging Face allocator mode."""
+        self._persist_env_value(HF_REALTIME_CONNECTION_MODE_ENV, HF_DEPLOYED_CONNECTION_MODE)
+        self._remove_persisted_env_values(("HF_REALTIME_SESSION_URL",))
+
     def _persist_api_key(self, key: str) -> None:
         """Persist OPENAI_API_KEY to environment and instance `.env`."""
         self._persist_env_value("OPENAI_API_KEY", key)
@@ -217,17 +288,28 @@ class LocalStream:
         current_backend = get_backend_choice()
         current_model_name = (os.getenv("MODEL_NAME") or "").strip()
         updates = {"BACKEND_PROVIDER": backend}
+        if backend == HF_BACKEND:
+            self._persist_env_values(updates)
+            try:
+                os.environ.pop("MODEL_NAME", None)
+            except Exception:
+                pass
+            self._remove_persisted_env_values(("MODEL_NAME",))
+            refresh_runtime_config_from_env()
+            return
+
         if current_model_name and current_model_name != get_model_name_for_backend(current_backend):
             updates["MODEL_NAME"] = current_model_name
         else:
             updates["MODEL_NAME"] = get_model_name_for_backend(backend)
         self._persist_env_values(updates)
 
-    def _persist_personality(self, profile: Optional[str]) -> None:
-        """Persist the startup personality to the instance .env and config."""
+    def _persist_personality(self, profile: Optional[str], voice_override: Optional[str] = None) -> None:
+        """Persist startup profile and voice in instance-local UI settings."""
         if LOCKED_PROFILE is not None:
             return
         selection = (profile or "").strip() or None
+        normalized_voice_override = (voice_override or "").strip() or None
         try:
             from reachy_mini_conversation_app.config import set_custom_profile
 
@@ -238,48 +320,19 @@ class LocalStream:
         if not self._instance_path:
             return
         try:
-            env_path = Path(self._instance_path) / ".env"
-            lines = self._read_env_lines(env_path)
-            replaced = False
-            for i, ln in enumerate(list(lines)):
-                if ln.strip().startswith("REACHY_MINI_CUSTOM_PROFILE="):
-                    if selection:
-                        lines[i] = f"REACHY_MINI_CUSTOM_PROFILE={selection}"
-                    else:
-                        lines.pop(i)
-                    replaced = True
-                    break
-            if selection and not replaced:
-                lines.append(f"REACHY_MINI_CUSTOM_PROFILE={selection}")
-            if selection is None and not env_path.exists():
-                return
-            final_text = "\n".join(lines) + "\n"
-            env_path.write_text(final_text, encoding="utf-8")
-            logger.info("Persisted startup personality to %s", env_path)
-            try:
-                from dotenv import load_dotenv
-
-                load_dotenv(dotenv_path=str(env_path), override=True)
-            except Exception:
-                pass
+            write_startup_settings(
+                self._instance_path,
+                profile=selection,
+                voice=normalized_voice_override,
+            )
+            self._remove_persisted_env_values(LEGACY_STARTUP_ENV_NAMES)
+            logger.info("Persisted startup personality settings to %s", Path(self._instance_path))
         except Exception as e:
-            logger.warning("Failed to persist REACHY_MINI_CUSTOM_PROFILE: %s", e)
+            logger.warning("Failed to persist startup personality settings: %s", e)
 
     def _read_persisted_personality(self) -> Optional[str]:
-        """Read persisted startup personality from instance .env (if any)."""
-        if not self._instance_path:
-            return None
-        env_path = Path(self._instance_path) / ".env"
-        try:
-            if env_path.exists():
-                for ln in env_path.read_text(encoding="utf-8").splitlines():
-                    if ln.strip().startswith("REACHY_MINI_CUSTOM_PROFILE="):
-                        _, _, val = ln.partition("=")
-                        v = val.strip()
-                        return v or None
-        except Exception:
-            pass
-        return None
+        """Read the saved startup personality from instance-local UI settings."""
+        return read_startup_settings(self._instance_path).profile
 
     def _init_settings_ui_if_needed(self) -> None:
         """Attach minimal settings UI to the settings app.
@@ -308,14 +361,26 @@ class LocalStream:
         class BackendPayload(BaseModel):
             backend: str
             api_key: Optional[str] = None
+            hf_mode: Optional[str] = None
+            hf_host: Optional[str] = None
+            hf_port: Optional[int] = None
 
         def _status_payload() -> dict[str, object]:
             backend_provider = get_backend_choice()
             active_backend = self._active_backend()
             has_openai_key = self._has_required_key(OPENAI_BACKEND)
             has_gemini_key = self._has_required_key(GEMINI_BACKEND)
+            hf_session_url = get_hf_session_url()
+            hf_ws_url = get_hf_direct_ws_url()
+            hf_direct_host, hf_direct_port = parse_hf_direct_target(hf_ws_url)
+            has_hf_session_url = bool(hf_session_url)
+            has_hf_ws_url = bool(hf_ws_url)
+            hf_connection_selection = get_hf_connection_selection()
+            hf_connection_mode = hf_connection_selection.mode
+            has_hf_connection = hf_connection_selection.has_target
             can_proceed_with_openai = has_openai_key
             can_proceed_with_gemini = has_gemini_key
+            can_proceed_with_hf = has_hf_connection
             can_proceed = self._has_required_key(active_backend)
             requires_restart = backend_provider != active_backend
             return {
@@ -324,9 +389,16 @@ class LocalStream:
                 "has_key": can_proceed,
                 "has_openai_key": has_openai_key,
                 "has_gemini_key": has_gemini_key,
+                "has_hf_session_url": has_hf_session_url,
+                "has_hf_ws_url": has_hf_ws_url,
+                "has_hf_connection": has_hf_connection,
+                "hf_connection_mode": hf_connection_mode,
+                "hf_direct_host": hf_direct_host,
+                "hf_direct_port": hf_direct_port,
                 "can_proceed": can_proceed,
                 "can_proceed_with_openai": can_proceed_with_openai,
                 "can_proceed_with_gemini": can_proceed_with_gemini,
+                "can_proceed_with_hf": can_proceed_with_hf,
                 "requires_restart": requires_restart,
             }
 
@@ -367,7 +439,7 @@ class LocalStream:
         @self._settings_app.post("/backend_config")
         def _set_backend(payload: BackendPayload) -> JSONResponse:
             backend = payload.backend.strip().lower()
-            if backend not in {OPENAI_BACKEND, GEMINI_BACKEND}:
+            if backend not in {OPENAI_BACKEND, GEMINI_BACKEND, HF_BACKEND}:
                 return JSONResponse({"ok": False, "error": "invalid_backend"}, status_code=400)
 
             api_key = (payload.api_key or "").strip()
@@ -378,6 +450,28 @@ class LocalStream:
                 self._persist_api_key(api_key)
             if backend == GEMINI_BACKEND and api_key:
                 self._persist_gemini_api_key(api_key)
+            if backend == HF_BACKEND:
+                hf_selection = get_hf_connection_selection()
+                hf_mode = (payload.hf_mode or hf_selection.mode).strip().lower()
+                if hf_mode == HF_LOCAL_CONNECTION_MODE:
+                    existing_host, existing_port = parse_hf_direct_target(hf_selection.direct_ws_url)
+                    host = (payload.hf_host or "").strip() or existing_host or ""
+                    if not host:
+                        return JSONResponse({"ok": False, "error": "empty_hf_host"}, status_code=400)
+                    if "://" in host or "/" in host or "?" in host or "#" in host:
+                        return JSONResponse({"ok": False, "error": "invalid_hf_host"}, status_code=400)
+
+                    port = payload.hf_port if payload.hf_port is not None else existing_port or 8765
+                    if port < 1 or port > 65535:
+                        return JSONResponse({"ok": False, "error": "invalid_hf_port"}, status_code=400)
+
+                    self._persist_hf_direct_connection(host, port)
+                elif hf_mode == HF_DEPLOYED_CONNECTION_MODE:
+                    if not bool(get_hf_session_url()):
+                        return JSONResponse({"ok": False, "error": "missing_hf_session_url"}, status_code=400)
+                    self._persist_hf_allocator_connection()
+                else:
+                    return JSONResponse({"ok": False, "error": "invalid_hf_mode"}, status_code=400)
 
             self._persist_backend_choice(backend)
             payload_data = _status_payload()
@@ -443,29 +537,20 @@ class LocalStream:
 
         active_backend = self._active_backend()
 
-        # If key is still missing, try to download one from HuggingFace (OpenAI only)
-        if active_backend == OPENAI_BACKEND and not self._has_required_key(active_backend):
-            logger.info("OPENAI_API_KEY not set, attempting to download from HuggingFace...")
-            try:
-                from gradio_client import Client
-
-                client = Client("HuggingFaceM4/gradium_setup", verbose=False)
-                key, _ = client.predict(api_name="/claim_b_key")
-                if key and key.strip():
-                    logger.info("Successfully downloaded API key from HuggingFace")
-                    # Persist it immediately
-                    self._persist_api_key(key)
-            except Exception as e:
-                logger.warning(f"Failed to download API key from HuggingFace: {e}")
-
         # Always expose settings UI if a settings app is available
-        # (do this AFTER loading/downloading the key so status endpoint sees the right value)
+        # (do this AFTER loading the instance .env so status endpoint sees the right value)
         self._init_settings_ui_if_needed()
 
         # If key is still missing -> wait until provided via the settings UI
         if not self._has_required_key(active_backend):
-            key_name = "GEMINI_API_KEY" if active_backend == GEMINI_BACKEND else "OPENAI_API_KEY"
-            logger.warning("%s not found. Open the app settings page to enter it.", key_name)
+            requirement_name = self._requirement_name(active_backend)
+            if active_backend == HF_BACKEND:
+                logger.error(
+                    "%s not found. Set it in the app .env before starting the Hugging Face backend.", requirement_name
+                )
+                return
+            else:
+                logger.warning("%s not found. Open the app settings page to enter it.", requirement_name)
             # Poll until the key becomes available (set via the settings UI)
             try:
                 while not self._has_required_key(active_backend):
@@ -478,6 +563,7 @@ class LocalStream:
         self._robot.media.start_recording()
         self._robot.media.start_playing()
         time.sleep(1)  # give some time to the pipelines to start
+        apply_audio_startup_config(self._robot, logger=logger)
 
         async def runner() -> None:
             # Capture loop for cross-thread personality actions
@@ -546,7 +632,12 @@ class LocalStream:
         backend = getattr(self._robot.media, "backend", None)
         audio = getattr(self._robot.media, "audio", None)
         if audio is not None:
-            if backend == MediaBackend.LOCAL and hasattr(audio, "clear_player") and callable(audio.clear_player):
+            if (
+                LOCAL_PLAYER_BACKEND is not None
+                and backend == LOCAL_PLAYER_BACKEND
+                and hasattr(audio, "clear_player")
+                and callable(audio.clear_player)
+            ):
                 audio.clear_player()
             elif (
                 backend == MediaBackend.WEBRTC

src/reachy_mini_conversation_app/conversation_handler.py

@@ -0,0 +1,69 @@
+from __future__ import annotations
+import asyncio
+from abc import ABC, abstractmethod
+from typing import TypeAlias
+from collections.abc import Callable
+
+import numpy as np
+from fastrtc import AdditionalOutputs, AsyncStreamHandler
+from numpy.typing import NDArray
+
+from reachy_mini_conversation_app.tools.core_tools import ToolDependencies
+
+
+AudioFrame: TypeAlias = tuple[int, NDArray[np.int16]]
+HandlerOutput: TypeAlias = AudioFrame | AdditionalOutputs | None
+QueueItem: TypeAlias = AudioFrame | AdditionalOutputs
+
+
+class ConversationHandler(AsyncStreamHandler, ABC):
+    """Shared app handler contract for realtime conversation backends."""
+
+    deps: ToolDependencies
+    output_queue: asyncio.Queue[QueueItem]
+    _clear_queue: Callable[[], None] | None
+
+    @abstractmethod
+    def copy(self) -> ConversationHandler:
+        """Create a copy of the handler."""
+        ...
+
+    @abstractmethod
+    async def start_up(self) -> None:
+        """Start the realtime handler."""
+        ...
+
+    @abstractmethod
+    async def shutdown(self) -> None:
+        """Shut down the realtime handler."""
+        ...
+
+    @abstractmethod
+    async def receive(self, frame: AudioFrame) -> None:
+        """Receive an input audio frame."""
+        ...
+
+    @abstractmethod
+    async def emit(self) -> HandlerOutput:
+        """Emit the next output item."""
+        ...
+
+    @abstractmethod
+    async def apply_personality(self, profile: str | None) -> str:
+        """Apply a personality profile."""
+        ...
+
+    @abstractmethod
+    async def get_available_voices(self) -> list[str]:
+        """Return voices available for the active backend."""
+        ...
+
+    @abstractmethod
+    def get_current_voice(self) -> str:
+        """Return the current voice."""
+        ...
+
+    @abstractmethod
+    async def change_voice(self, voice: str) -> str:
+        """Change the current voice."""
+        ...

src/reachy_mini_conversation_app/gemini_live.py

@@ -20,7 +20,7 @@ from datetime import datetime
 import numpy as np
 import gradio as gr
 from google import genai
-from fastrtc import AdditionalOutputs, AsyncStreamHandler, wait_for_item, audio_to_int16
+from fastrtc import AdditionalOutputs, wait_for_item, audio_to_int16
 from google.genai import types
 from numpy.typing import NDArray
 from scipy.signal import resample
@@ -34,8 +34,9 @@ from reachy_mini_conversation_app.config import (
 from reachy_mini_conversation_app.prompts import get_session_voice, get_session_instructions
 from reachy_mini_conversation_app.tools.core_tools import (
     ToolDependencies,
-    get_tool_specs,
+    get_active_tool_specs,
 )
+from reachy_mini_conversation_app.conversation_handler import ConversationHandler
 from reachy_mini_conversation_app.camera_frame_encoding import encode_bgr_frame_as_jpeg
 from reachy_mini_conversation_app.tools.background_tool_manager import (
     ToolCallRoutine,
@@ -121,10 +122,32 @@ def _resolve_gemini_voice(profile_voice: str) -> str:
     return voice_map.get(profile_voice.lower(), DEFAULT_VOICE_BY_BACKEND[GEMINI_BACKEND])
 
 
-class GeminiLiveHandler(AsyncStreamHandler):
+def _resolve_gemini_startup_voice(voice: str | None) -> str | None:
+    """Return a valid persisted Gemini startup voice or None."""
+    if voice is None:
+        return None
+
+    voice_map = {candidate.lower(): candidate for candidate in GEMINI_AVAILABLE_VOICES}
+    resolved = voice_map.get(voice.lower())
+    if resolved is None:
+        logger.warning(
+            "Ignoring persisted Gemini startup voice %r; expected one of %s",
+            voice,
+            GEMINI_AVAILABLE_VOICES,
+        )
+    return resolved
+
+
+class GeminiLiveHandler(ConversationHandler):
     """Gemini Live API handler for fastrtc Stream."""
 
-    def __init__(self, deps: ToolDependencies, gradio_mode: bool = False, instance_path: Optional[str] = None):
+    def __init__(
+        self,
+        deps: ToolDependencies,
+        gradio_mode: bool = False,
+        instance_path: Optional[str] = None,
+        startup_voice: Optional[str] = None,
+    ):
         """Initialize the handler."""
         super().__init__(
             expected_layout="mono",
@@ -135,7 +158,7 @@ class GeminiLiveHandler(AsyncStreamHandler):
         self.deps = deps
         self.gradio_mode = gradio_mode
         self.instance_path = instance_path
-        self._voice_override: str | None = None
+        self._voice_override: str | None = _resolve_gemini_startup_voice(startup_voice)
 
         self.session: Any = None  # google.genai live session
         self.output_queue: "asyncio.Queue[Tuple[int, NDArray[np.int16]] | AdditionalOutputs]" = asyncio.Queue()
@@ -162,7 +185,12 @@ class GeminiLiveHandler(AsyncStreamHandler):
 
     def copy(self) -> "GeminiLiveHandler":
         """Create a copy of the handler."""
-        return GeminiLiveHandler(self.deps, self.gradio_mode, self.instance_path)
+        return GeminiLiveHandler(
+            self.deps,
+            self.gradio_mode,
+            self.instance_path,
+            startup_voice=self._voice_override,
+        )
 
     def _set_listening_state(self, listening: bool) -> None:
         """Avoid queueing redundant listening-state updates."""
@@ -217,7 +245,6 @@ class GeminiLiveHandler(AsyncStreamHandler):
             from reachy_mini_conversation_app.config import set_custom_profile
 
             set_custom_profile(profile)
-            self._voice_override = None
             logger.info("Set custom profile to %r", profile)
 
             try:
@@ -341,7 +368,11 @@ class GeminiLiveHandler(AsyncStreamHandler):
         voice = _resolve_gemini_voice(self._voice_override or get_session_voice())
 
         # Convert OpenAI-style tool specs to Gemini function declarations
-        tool_specs = get_tool_specs()
+        tool_specs = get_active_tool_specs(self.deps)
+        logger.info(
+            "Tools to be used in conversation: %s",
+            [tool["name"] for tool in tool_specs],
+        )
         function_declarations = _openai_tool_specs_to_gemini(tool_specs)
 
         tools_config: List[Dict[str, Any]] = []
@@ -700,7 +731,7 @@ class GeminiLiveHandler(AsyncStreamHandler):
         timestamp_msg = (
             f"[Idle time update: {self.format_timestamp()} - No activity for {idle_duration:.1f}s] "
             "You've been idle for a while. Feel free to get creative - dance, show an emotion, "
-            "look around, do nothing, or just be yourself!"
+            "look around, call idle_do_nothing to stay still and silent, or just be yourself!"
         )
         if not self.session:
             logger.debug("No session, cannot send idle signal")

src/reachy_mini_conversation_app/gradio_personality.py

@@ -79,6 +79,44 @@ class PersonalityUI:
         except Exception as e:
             return f"Could not load instructions: {e}"
 
+    def _read_tools_for(self, name: str) -> str:
+        try:
+            profile_name = "default" if name == self.DEFAULT_OPTION else name
+            target = self._resolve_profile_dir(profile_name) / "tools.txt"
+            if target.exists():
+                return target.read_text(encoding="utf-8")
+        except Exception:
+            pass
+        return ""
+
+    def _available_tools_for(self, selected: str) -> tuple[list[str], list[str]]:
+        shared: list[str] = []
+        try:
+            for py in self._tools_dir.glob("*.py"):
+                if py.stem in {"__init__", "core_tools"}:
+                    continue
+                shared.append(py.stem)
+        except Exception:
+            pass
+        local: list[str] = []
+        try:
+            if selected != self.DEFAULT_OPTION:
+                for py in (self._profiles_root / selected).glob("*.py"):
+                    local.append(py.stem)
+        except Exception:
+            pass
+        return sorted(shared), sorted(local)
+
+    @staticmethod
+    def _parse_enabled_tools(text: str) -> list[str]:
+        enabled: list[str] = []
+        for line in text.splitlines():
+            s = line.strip()
+            if not s or s.startswith("#"):
+                continue
+            enabled.append(s)
+        return enabled
+
     @staticmethod
     def _sanitize_name(name: str) -> str:
         import re
@@ -101,6 +139,10 @@ class PersonalityUI:
             current_value = config.REACHY_MINI_CUSTOM_PROFILE or self.DEFAULT_OPTION
             dropdown_label = "Select personality"
             dropdown_choices = [self.DEFAULT_OPTION, *(self._list_personalities())]
+        initial_tools_txt = self._read_tools_for(current_value)
+        shared_tools, local_tools = self._available_tools_for(current_value)
+        initial_available_tools = sorted(set(shared_tools + local_tools))
+        initial_enabled_tools = self._parse_enabled_tools(initial_tools_txt)
 
         self.personalities_dropdown = gr.Dropdown(
             label=dropdown_label,
@@ -113,7 +155,12 @@ class PersonalityUI:
         self.preview_md = gr.Markdown(value=self._read_instructions_for(current_value))
         self.person_name_tb = gr.Textbox(label="Personality name", interactive=not is_locked)
         self.person_instr_ta = gr.TextArea(label="Personality instructions", lines=10, interactive=not is_locked)
-        self.tools_txt_ta = gr.TextArea(label="tools.txt", lines=10, interactive=not is_locked)
+        self.tools_txt_ta = gr.TextArea(
+            label="tools.txt",
+            value=initial_tools_txt,
+            lines=10,
+            interactive=not is_locked,
+        )
         self.voice_dropdown = gr.Dropdown(
             label="Voice",
             choices=get_available_voices_for_backend(),
@@ -122,7 +169,10 @@ class PersonalityUI:
         )
         self.new_personality_btn = gr.Button("New personality", interactive=not is_locked)
         self.available_tools_cg = gr.CheckboxGroup(
-            label="Available tools (helper)", choices=[], value=[], interactive=not is_locked
+            label="Available tools (helper)",
+            choices=initial_available_tools,
+            value=initial_enabled_tools,
+            interactive=not is_locked,
         )
         self.save_btn = gr.Button("Save personality (instructions + tools)", interactive=not is_locked)
 
@@ -183,43 +233,12 @@ class PersonalityUI:
                     value=get_default_voice_for_backend(),
                 )
 
-        def _available_tools_for(selected: str) -> tuple[list[str], list[str]]:
-            shared: list[str] = []
-            try:
-                for py in self._tools_dir.glob("*.py"):
-                    if py.stem in {"__init__", "core_tools"}:
-                        continue
-                    shared.append(py.stem)
-            except Exception:
-                pass
-            local: list[str] = []
-            try:
-                if selected != self.DEFAULT_OPTION:
-                    for py in (self._profiles_root / selected).glob("*.py"):
-                        local.append(py.stem)
-            except Exception:
-                pass
-            return sorted(shared), sorted(local)
-
-        def _parse_enabled_tools(text: str) -> list[str]:
-            enabled: list[str] = []
-            for line in text.splitlines():
-                s = line.strip()
-                if not s or s.startswith("#"):
-                    continue
-                enabled.append(s)
-            return enabled
-
         def _load_profile_for_edit(selected: str) -> tuple[dict[str, Any], dict[str, Any], dict[str, Any], str]:
             instr = self._read_instructions_for(selected)
-            tools_txt = ""
-            if selected != self.DEFAULT_OPTION:
-                tp = self._resolve_profile_dir(selected) / "tools.txt"
-                if tp.exists():
-                    tools_txt = tp.read_text(encoding="utf-8")
-            shared, local = _available_tools_for(selected)
+            tools_txt = self._read_tools_for(selected)
+            shared, local = self._available_tools_for(selected)
             all_tools = sorted(set(shared + local))
-            enabled = _parse_enabled_tools(tools_txt)
+            enabled = self._parse_enabled_tools(tools_txt)
             status_text = f"Loaded profile '{selected}'."
             return (
                 gr.update(value=instr),
@@ -239,7 +258,7 @@ class PersonalityUI:
                     gr.update(value=""),
                     gr.update(value=instr_val),
                     gr.update(value=tools_txt_val),
-                    gr.update(choices=sorted(_available_tools_for(self.DEFAULT_OPTION)[0]), value=[]),
+                    gr.update(choices=sorted(self._available_tools_for(self.DEFAULT_OPTION)[0]), value=[]),
                     "Fill in a name, instructions and (optional) tools, then Save.",
                     gr.update(value=get_default_voice_for_backend()),
                 )

src/reachy_mini_conversation_app/headless_personality.py

@@ -11,7 +11,7 @@ from __future__ import annotations
 from typing import List
 from pathlib import Path
 
-from .config import DEFAULT_PROFILES_DIRECTORY
+from .config import DEFAULT_PROFILES_DIRECTORY, get_default_voice_for_backend
 
 
 DEFAULT_OPTION = "(built-in default)"
@@ -76,6 +76,16 @@ def read_instructions_for(name: str) -> str:
         return f"Could not load instructions: {e}"
 
 
+def read_tools_for(name: str) -> str:
+    """Read the tools.txt content for the given profile name."""
+    try:
+        profile_name = "default" if name == DEFAULT_OPTION else name
+        target = resolve_profile_dir(profile_name) / "tools.txt"
+        return target.read_text(encoding="utf-8") if target.exists() else ""
+    except Exception:
+        return ""
+
+
 def available_tools_for(selected: str) -> List[str]:
     """List available tool modules for the given profile selection."""
     shared: List[str] = []
@@ -96,9 +106,10 @@ def available_tools_for(selected: str) -> List[str]:
     return sorted(set(shared + local))
 
 
-def _write_profile(name_s: str, instructions: str, tools_text: str, voice: str = "cedar") -> None:
+def _write_profile(name_s: str, instructions: str, tools_text: str, voice: str | None = None) -> None:
+    default_voice = get_default_voice_for_backend()
     target_dir = _profiles_root() / "user_personalities" / name_s
     target_dir.mkdir(parents=True, exist_ok=True)
     (target_dir / "instructions.txt").write_text(instructions.strip() + "\n", encoding="utf-8")
     (target_dir / "tools.txt").write_text((tools_text or "").strip() + "\n", encoding="utf-8")
-    (target_dir / "voice.txt").write_text((voice or "cedar").strip() + "\n", encoding="utf-8")
+    (target_dir / "voice.txt").write_text((voice or default_voice).strip() + "\n", encoding="utf-8")

src/reachy_mini_conversation_app/headless_personality_ui.py

@@ -9,7 +9,7 @@ callable to avoid cross-thread issues.
 from __future__ import annotations
 import asyncio
 import logging
-from typing import TYPE_CHECKING, Any, Callable, Optional
+from typing import Any, Callable, Optional
 
 from fastapi import Query, FastAPI, Request
 
@@ -19,15 +19,12 @@ from .config import (
     get_default_voice_for_backend,
     get_available_voices_for_backend,
 )
-from .openai_realtime import OpenaiRealtimeHandler
-
-
-if TYPE_CHECKING:
-    from .gemini_live import GeminiLiveHandler
+from .conversation_handler import ConversationHandler
 from .headless_personality import (
     DEFAULT_OPTION,
     _sanitize_name,
     _write_profile,
+    read_tools_for,
     list_personalities,
     available_tools_for,
     resolve_profile_dir,
@@ -40,10 +37,10 @@ logger = logging.getLogger(__name__)
 
 def mount_personality_routes(
     app: FastAPI,
-    handler: OpenaiRealtimeHandler | GeminiLiveHandler,
+    handler: ConversationHandler,
     get_loop: Callable[[], asyncio.AbstractEventLoop | None],
     *,
-    persist_personality: Callable[[Optional[str]], None] | None = None,
+    persist_personality: Callable[[Optional[str], Optional[str]], None] | None = None,
     get_persisted_personality: Callable[[], Optional[str]] | None = None,
 ) -> None:
     """Register personality management endpoints on a FastAPI app."""
@@ -92,14 +89,11 @@ def mount_personality_routes(
     @app.get("/personalities/load")
     def _load(name: str) -> dict:  # type: ignore
         instr = read_instructions_for(name)
-        tools_txt = ""
+        tools_txt = read_tools_for(name)
         voice = get_default_voice_for_backend()
         uses_default_voice = True
         if name != DEFAULT_OPTION:
             pdir = resolve_profile_dir(name)
-            tp = pdir / "tools.txt"
-            if tp.exists():
-                tools_txt = tp.read_text(encoding="utf-8")
             vf = pdir / "voice.txt"
             if vf.exists():
                 v = vf.read_text(encoding="utf-8").strip()
@@ -258,19 +252,21 @@ def mount_personality_routes(
         if not sel_name:
             sel_name = DEFAULT_OPTION
 
-        async def _do_apply() -> str:
+        async def _do_apply() -> tuple[str, Optional[str]]:
             sel = None if sel_name == DEFAULT_OPTION else sel_name
             status = await handler.apply_personality(sel)
-            return status
+            get_current_voice = getattr(handler, "get_current_voice", None)
+            voice_override = get_current_voice() if callable(get_current_voice) else None
+            return status, voice_override
 
         try:
             logger.info("Headless apply: requested name=%r", sel_name)
             fut = asyncio.run_coroutine_threadsafe(_do_apply(), loop)
-            status = fut.result(timeout=10)
+            status, voice_override = fut.result(timeout=10)
             persisted_choice = _startup_choice()
             if persist_flag and persist_personality is not None:
                 try:
-                    persist_personality(None if sel_name == DEFAULT_OPTION else sel_name)
+                    persist_personality(None if sel_name == DEFAULT_OPTION else sel_name, voice_override)
                     persisted_choice = _startup_choice()
                 except Exception as e:
                     logger.warning("Failed to persist startup personality: %s", e)

src/reachy_mini_conversation_app/huggingface_realtime.py

@@ -0,0 +1,160 @@
+import logging
+from typing import Any
+
+import httpx
+from openai import AsyncOpenAI
+from typing_extensions import Literal, TypedDict
+from openai.types.realtime import (
+    AudioTranscriptionParam,
+    RealtimeAudioConfigParam,
+    RealtimeAudioConfigInputParam,
+    RealtimeAudioConfigOutputParam,
+    RealtimeSessionCreateRequestParam,
+)
+from openai.types.realtime.realtime_audio_input_turn_detection_param import ServerVad
+
+from reachy_mini_conversation_app.config import (
+    HF_BACKEND,
+    HF_LOCAL_CONNECTION_MODE,
+    config,
+    get_hf_direct_ws_url,
+    parse_hf_realtime_url,
+    get_hf_connection_selection,
+)
+from reachy_mini_conversation_app.prompts import get_session_voice, get_session_instructions
+from reachy_mini_conversation_app.base_realtime import (
+    BaseRealtimeHandler,
+    InputTranscriptChunksByItem,
+    to_realtime_tools_config,
+)
+from reachy_mini_conversation_app.tools.core_tools import get_active_tool_specs
+
+
+logger = logging.getLogger(__name__)
+
+
+def _build_openai_compatible_client_from_realtime_url(
+    realtime_url: str,
+    bearer_token: str | None,
+) -> tuple[AsyncOpenAI, dict[str, str]]:
+    """Build an OpenAI-compatible realtime client from a direct websocket/base URL."""
+    parsed = parse_hf_realtime_url(realtime_url)
+    client = AsyncOpenAI(
+        api_key=bearer_token or "DUMMY",
+        base_url=parsed.base_url,
+        websocket_base_url=parsed.websocket_base_url,
+    )
+    return client, parsed.connect_query
+
+
+class HFNativeRateAudioPCM(TypedDict):
+    """Hugging Face extension for native-rate PCM audio."""
+
+    type: Literal["audio/pcm"]
+    rate: None
+
+
+def _native_rate_audio_pcm() -> HFNativeRateAudioPCM:
+    """Return the Hugging Face native-rate PCM config."""
+    return {"type": "audio/pcm", "rate": None}
+
+
+class HuggingFaceRealtimeHandler(BaseRealtimeHandler):
+    """Realtime handler for Hugging Face endpoints."""
+
+    BACKEND_PROVIDER = HF_BACKEND
+    SAMPLE_RATE = 16000
+    REFRESH_CLIENT_ON_RECONNECT = True
+    AUDIO_INPUT_COST_PER_1M = 0.0
+    AUDIO_OUTPUT_COST_PER_1M = 0.0
+    TEXT_INPUT_COST_PER_1M = 0.0
+    TEXT_OUTPUT_COST_PER_1M = 0.0
+    IMAGE_INPUT_COST_PER_1M = 0.0
+
+    def _get_session_instructions(self) -> str:
+        """Return Hugging Face session instructions."""
+        return get_session_instructions()
+
+    def _get_session_voice(self, default: str | None = None) -> str:
+        """Return the configured Hugging Face session voice."""
+        return get_session_voice(default)
+
+    def _get_active_tool_specs(self) -> list[dict[str, Any]]:
+        """Return active tool specs for the current session dependencies."""
+        return get_active_tool_specs(self.deps)
+
+    def _get_session_config(self, tool_specs: list[dict[str, Any]]) -> RealtimeSessionCreateRequestParam:
+        """Return the Hugging Face OpenAI-compatible session config."""
+        return RealtimeSessionCreateRequestParam(
+            type="realtime",
+            instructions=self._get_session_instructions(),
+            audio=RealtimeAudioConfigParam(
+                input=RealtimeAudioConfigInputParam(
+                    # The OpenAI SDK type only includes 24 kHz PCM, but the HF
+                    # compatible server uses rate=None for native 16 kHz mode.
+                    format=_native_rate_audio_pcm(),  # type: ignore[typeddict-item]
+                    transcription=AudioTranscriptionParam(model="gpt-4o-transcribe", language="en"),
+                    turn_detection=ServerVad(type="server_vad", interrupt_response=True),
+                ),
+                output=RealtimeAudioConfigOutputParam(
+                    format=_native_rate_audio_pcm(),  # type: ignore[typeddict-item]
+                    voice=self.get_current_voice(),
+                ),
+            ),
+            tools=to_realtime_tools_config(tool_specs),
+            tool_choice="auto",
+        )
+
+    def _record_partial_transcript_delta(
+        self,
+        input_transcript: InputTranscriptChunksByItem,
+        item_id: str,
+        delta: str,
+    ) -> None:
+        """Record a Hugging Face partial transcript snapshot."""
+        input_transcript.item_id = item_id
+        input_transcript.deltas = [delta]
+
+    async def _build_realtime_client(self) -> AsyncOpenAI:
+        """Build the Hugging Face OpenAI-compatible realtime client."""
+        bearer_token = (config.HF_TOKEN or "").strip()
+        connection_selection = get_hf_connection_selection()
+        direct_realtime_url = get_hf_direct_ws_url()
+        if connection_selection.mode == HF_LOCAL_CONNECTION_MODE:
+            if not direct_realtime_url:
+                raise RuntimeError("HF_REALTIME_WS_URL must be set when HF_REALTIME_CONNECTION_MODE=local")
+            client, connect_query = _build_openai_compatible_client_from_realtime_url(
+                direct_realtime_url,
+                bearer_token,
+            )
+            self._realtime_connect_query = connect_query
+            logger.info("Using direct Hugging Face realtime endpoint %s", direct_realtime_url)
+            return client
+
+        session_url = connection_selection.session_url
+        if not session_url:
+            raise RuntimeError("Built-in Hugging Face session proxy URL is unavailable")
+        if direct_realtime_url:
+            logger.info("HF_REALTIME_CONNECTION_MODE=deployed; ignoring HF_REALTIME_WS_URL.")
+
+        allocator_headers = {"Authorization": f"Bearer {bearer_token}"} if bearer_token else None
+        async with httpx.AsyncClient(timeout=10.0) as http_client:
+            response = await http_client.post(session_url, headers=allocator_headers)
+            response.raise_for_status()
+            payload = response.json()
+
+        connect_url = payload.get("connect_url")
+        if not isinstance(connect_url, str) or not connect_url:
+            raise RuntimeError(f"Session allocator response did not contain a valid connect_url: {payload!r}")
+
+        parsed_connect_url = parse_hf_realtime_url(connect_url)
+        if not parsed_connect_url.has_realtime_path:
+            raise ValueError(f"Expected realtime connect URL ending with /realtime, got: {connect_url}")
+
+        logger.info("Allocated realtime session %s", payload.get("session_id") or "<unknown>")
+        client, connect_query = _build_openai_compatible_client_from_realtime_url(
+            connect_url,
+            bearer_token,
+        )
+        self._realtime_connect_query = connect_query
+        return client

src/reachy_mini_conversation_app/main.py

@@ -46,10 +46,25 @@ def run(
     """Run the Reachy Mini conversation app."""
     # Putting these dependencies here makes the dashboard faster to load when the conversation app is installed
     from reachy_mini_conversation_app.moves import MovementManager
-    from reachy_mini_conversation_app.config import config, is_gemini_model, refresh_runtime_config_from_env
+    from reachy_mini_conversation_app.config import (
+        HF_BACKEND,
+        GEMINI_BACKEND,
+        OPENAI_BACKEND,
+        HF_LOCAL_CONNECTION_MODE,
+        config,
+        is_gemini_model,
+        get_backend_label,
+        get_hf_connection_selection,
+        refresh_runtime_config_from_env,
+    )
+    from reachy_mini_conversation_app.startup_settings import (
+        StartupSettings,
+        load_startup_settings_into_runtime,
+    )
 
     logger = setup_logger(args.debug)
     logger.info("Starting Reachy Mini Conversation App")
+    startup_settings = StartupSettings()
 
     if instance_path is not None:
         try:
@@ -63,6 +78,26 @@ def run(
         except Exception as e:
             logger.warning("Failed to load instance configuration: %s", e)
 
+        try:
+            startup_settings = load_startup_settings_into_runtime(instance_path)
+        except Exception as e:
+            logger.warning("Failed to load startup settings: %s", e)
+
+    if config.BACKEND_PROVIDER == HF_BACKEND:
+        logger.info(
+            "Configured backend provider: %s (%s), connection mode: %s",
+            config.BACKEND_PROVIDER,
+            get_backend_label(config.BACKEND_PROVIDER),
+            get_hf_connection_selection().mode,
+        )
+    else:
+        logger.info(
+            "Configured backend provider: %s (%s), model: %s",
+            config.BACKEND_PROVIDER,
+            get_backend_label(config.BACKEND_PROVIDER),
+            config.MODEL_NAME,
+        )
+
     from reachy_mini_conversation_app.console import LocalStream
     from reachy_mini_conversation_app.tools.core_tools import ToolDependencies
     from reachy_mini_conversation_app.audio.head_wobbler import HeadWobbler
@@ -144,40 +179,75 @@ def run(
     if is_gemini_model():
         from reachy_mini_conversation_app.gemini_live import GeminiLiveHandler
 
-        logger.info("Using Gemini Live handler for model: %s", config.MODEL_NAME)
-        handler = GeminiLiveHandler(deps, gradio_mode=args.gradio, instance_path=instance_path)
+        logger.info(
+            "Using %s via GeminiLiveHandler",
+            get_backend_label(config.BACKEND_PROVIDER),
+        )
+        handler = GeminiLiveHandler(
+            deps,
+            gradio_mode=args.gradio,
+            instance_path=instance_path,
+            startup_voice=startup_settings.voice,
+        )
+    elif config.BACKEND_PROVIDER == HF_BACKEND:
+        from reachy_mini_conversation_app.huggingface_realtime import HuggingFaceRealtimeHandler
+
+        hf_connection_selection = get_hf_connection_selection()
+        transport_label = (
+            "Hugging Face direct websocket"
+            if hf_connection_selection.mode == HF_LOCAL_CONNECTION_MODE and hf_connection_selection.has_target
+            else "Hugging Face session proxy"
+        )
+        logger.info(
+            "Using %s via Hugging Face realtime handler (%s)",
+            get_backend_label(config.BACKEND_PROVIDER),
+            transport_label,
+        )
+        handler = HuggingFaceRealtimeHandler(
+            deps,
+            gradio_mode=args.gradio,
+            instance_path=instance_path,
+            startup_voice=startup_settings.voice,
+        )  # type: ignore[assignment]
     else:
         from reachy_mini_conversation_app.openai_realtime import OpenaiRealtimeHandler
 
-        logger.info("Using OpenAI Realtime handler for model: %s", config.MODEL_NAME)
-        handler = OpenaiRealtimeHandler(deps, gradio_mode=args.gradio, instance_path=instance_path)  # type: ignore[assignment]
+        logger.info(
+            "Using %s via OpenAI realtime handler (OpenAI Realtime API)",
+            get_backend_label(config.BACKEND_PROVIDER),
+        )
+        handler = OpenaiRealtimeHandler(
+            deps,
+            gradio_mode=args.gradio,
+            instance_path=instance_path,
+            startup_voice=startup_settings.voice,
+        )  # type: ignore[assignment]
 
     stream_manager: gr.Blocks | LocalStream | None = None
 
     if args.gradio:
-        uses_gemini_backend = is_gemini_model()
-        api_key_textbox = gr.Textbox(
-            label="GEMINI_API_KEY" if uses_gemini_backend else "OPENAI API Key",
-            type="password",
-            value=(os.getenv("GEMINI_API_KEY") if uses_gemini_backend else os.getenv("OPENAI_API_KEY"))
-            if not get_space()
-            else "",
-        )
-
         from reachy_mini_conversation_app.gradio_personality import PersonalityUI
 
         personality_ui = PersonalityUI()
         personality_ui.create_components()
+        additional_inputs: list[Any] = [chatbot, *personality_ui.additional_inputs_ordered()]
+
+        if config.BACKEND_PROVIDER in {OPENAI_BACKEND, GEMINI_BACKEND}:
+            uses_gemini_backend = is_gemini_model()
+            api_key_textbox = gr.Textbox(
+                label="GEMINI_API_KEY" if uses_gemini_backend else "OPENAI API Key",
+                type="password",
+                value=(os.getenv("GEMINI_API_KEY") if uses_gemini_backend else os.getenv("OPENAI_API_KEY"))
+                if not get_space()
+                else "",
+            )
+            additional_inputs.insert(1, api_key_textbox)
 
         stream = Stream(
             handler=handler,
             mode="send-receive",
             modality="audio",
-            additional_inputs=[
-                chatbot,
-                api_key_textbox,
-                *personality_ui.additional_inputs_ordered(),
-            ],
+            additional_inputs=additional_inputs,
             additional_outputs=[chatbot],
             additional_outputs_handler=update_chatbot,
             ui_args={"title": "Talk with Reachy Mini"},

src/reachy_mini_conversation_app/openai_realtime.py

@@ -1,830 +1,167 @@
-import json
-import uuid
-import base64
-import random
-import asyncio
 import logging
-from typing import Any, Final, Tuple, Literal, Optional
+from typing import Any, Literal
 from pathlib import Path
-from datetime import datetime
 
-import numpy as np
-import gradio as gr
 from openai import AsyncOpenAI
-from fastrtc import AdditionalOutputs, AsyncStreamHandler, wait_for_item, audio_to_int16
-from pydantic import Field, BaseModel
-from numpy.typing import NDArray
-from scipy.signal import resample
 from openai.types.realtime import (
     AudioTranscriptionParam,
     RealtimeAudioConfigParam,
     RealtimeAudioConfigInputParam,
     RealtimeAudioConfigOutputParam,
-    RealtimeResponseCreateParamsParam,
     RealtimeSessionCreateRequestParam,
 )
-from websockets.exceptions import ConnectionClosedError
-from openai.resources.realtime.realtime import AsyncRealtimeConnection
 from openai.types.realtime.realtime_audio_formats_param import AudioPCM
 from openai.types.realtime.realtime_audio_input_turn_detection_param import ServerVad
 
-from reachy_mini_conversation_app.config import AVAILABLE_VOICES, config
+from reachy_mini_conversation_app.config import OPENAI_BACKEND, config, get_default_voice_for_backend
 from reachy_mini_conversation_app.prompts import get_session_voice, get_session_instructions
-from reachy_mini_conversation_app.tools.core_tools import (
-    ToolDependencies,
-    get_tool_specs,
-)
-from reachy_mini_conversation_app.tools.background_tool_manager import (
-    ToolCallRoutine,
-    ToolNotification,
-    BackgroundToolManager,
-)
+from reachy_mini_conversation_app.base_realtime import BaseRealtimeHandler, to_realtime_tools_config
+from reachy_mini_conversation_app.tools.core_tools import ToolDependencies, get_active_tool_specs
 
 
 logger = logging.getLogger(__name__)
 
-OPEN_AI_INPUT_SAMPLE_RATE: Final[Literal[24000]] = 24000
-OPEN_AI_OUTPUT_SAMPLE_RATE: Final[Literal[24000]] = 24000
-
-# Cost tracking from usage data (pricing as of Feb 2026 https://openai.com/api/pricing/)
-AUDIO_INPUT_COST_PER_1M = 32.0
-AUDIO_OUTPUT_COST_PER_1M = 64.0
-TEXT_INPUT_COST_PER_1M = 4.0
-TEXT_OUTPUT_COST_PER_1M = 16.0
-IMAGE_INPUT_COST_PER_1M = 5.0
-
-_RESPONSE_DONE_TIMEOUT: Final[float] = 30.0
-
-
-class InputTranscriptChunksByItem(BaseModel):
-    """Current item_id and its accumulated deltas. Only one item at a time."""
-
-    item_id: str | None = None
-    deltas: list[str] = Field(default_factory=list)
-
-
-def _compute_response_cost(usage: Any) -> float:
-    """Compute dollar cost from a response usage object."""
-    inp = getattr(usage, "input_token_details", None)
-    out = getattr(usage, "output_token_details", None)
-    cost = 0.0
-    if inp:
-        cost += (getattr(inp, "audio_tokens", 0) or 0) * AUDIO_INPUT_COST_PER_1M / 1e6
-        cost += (getattr(inp, "text_tokens", 0) or 0) * TEXT_INPUT_COST_PER_1M / 1e6
-        cost += (getattr(inp, "image_tokens", 0) or 0) * IMAGE_INPUT_COST_PER_1M / 1e6
-    if out:
-        cost += (getattr(out, "audio_tokens", 0) or 0) * AUDIO_OUTPUT_COST_PER_1M / 1e6
-        cost += (getattr(out, "text_tokens", 0) or 0) * TEXT_OUTPUT_COST_PER_1M / 1e6
-    return cost
-
-
-class OpenaiRealtimeHandler(AsyncStreamHandler):
-    """An OpenAI realtime handler for fastrtc Stream."""
-
-    def __init__(self, deps: ToolDependencies, gradio_mode: bool = False, instance_path: Optional[str] = None):
-        """Initialize the handler."""
-        super().__init__(
-            expected_layout="mono",
-            output_sample_rate=OPEN_AI_OUTPUT_SAMPLE_RATE,
-            input_sample_rate=OPEN_AI_INPUT_SAMPLE_RATE,
-        )
-
-        # Override typing of the sample rates to match OpenAI's requirements
-        self.output_sample_rate: Literal[24000] = self.output_sample_rate
-        self.input_sample_rate: Literal[24000] = self.input_sample_rate
-
-        self.deps = deps
-
-        # Override type annotations for OpenAI strict typing (only for values used in API)
-        self.output_sample_rate = OPEN_AI_OUTPUT_SAMPLE_RATE
-        self.input_sample_rate = OPEN_AI_INPUT_SAMPLE_RATE
-
-        self.connection: AsyncRealtimeConnection | None = None
-        self.output_queue: "asyncio.Queue[Tuple[int, NDArray[np.int16]] | AdditionalOutputs]" = asyncio.Queue()
-
-        self.last_activity_time = asyncio.get_event_loop().time()
-        self.start_time = asyncio.get_event_loop().time()
-        self.is_idle_tool_call = False
-        self.gradio_mode = gradio_mode
-        self.instance_path = instance_path
-        self._voice_override: str | None = None
-        # Track how the API key was provided (env vs textbox) and its value
+__all__ = ["OpenaiRealtimeHandler"]
+
+
+class OpenaiRealtimeHandler(BaseRealtimeHandler):
+    """Realtime handler for the direct OpenAI Realtime API."""
+
+    BACKEND_PROVIDER = OPENAI_BACKEND
+    SAMPLE_RATE = 24000
+    REFRESH_CLIENT_ON_RECONNECT = False
+    AUDIO_INPUT_COST_PER_1M = 32.0
+    AUDIO_OUTPUT_COST_PER_1M = 64.0
+    TEXT_INPUT_COST_PER_1M = 4.0
+    TEXT_OUTPUT_COST_PER_1M = 16.0
+    IMAGE_INPUT_COST_PER_1M = 5.0
+
+    def __init__(
+        self,
+        deps: ToolDependencies,
+        gradio_mode: bool = False,
+        instance_path: str | None = None,
+        startup_voice: str | None = None,
+    ) -> None:
+        """Initialize OpenAI-specific credential state."""
+        super().__init__(deps, gradio_mode, instance_path, startup_voice)
         self._key_source: Literal["env", "textbox"] = "env"
         self._provided_api_key: str | None = None
 
-        # Debouncing for partial transcripts
-        self.partial_transcript_task: asyncio.Task[None] | None = None
-        self.partial_debounce_delay = 0.5  # seconds
-        self.input_transcript_chunks_by_item = InputTranscriptChunksByItem()
-
-        # Internal lifecycle flags
-        self._connected_event: asyncio.Event = asyncio.Event()
-
-        # Background tool manager
-        self.tool_manager = BackgroundToolManager()
-
-        # Cost tracking
-        self.cumulative_cost: float = 0.0
-
-        # Response-in-progress guard: the Realtime API only allows one active
-        # response per conversation at a time.  A dedicated worker task
-        # (_response_sender_loop) dequeues and sends one request at a time
-        self._pending_responses: asyncio.Queue[dict[str, Any]] = asyncio.Queue()
-        self._response_done_event: asyncio.Event = asyncio.Event()
-        self._response_done_event.set()
-        self._last_response_rejected: bool = False
-
-    def copy(self) -> "OpenaiRealtimeHandler":
-        """Create a copy of the handler."""
-        return OpenaiRealtimeHandler(self.deps, self.gradio_mode, self.instance_path)
-
-    async def apply_personality(self, profile: str | None) -> str:
-        """Apply a new personality (profile) at runtime if possible.
-
-        - Updates the global config's selected profile for subsequent calls.
-        - If a realtime connection is active, sends a session.update with the
-          freshly resolved instructions so the change takes effect immediately.
-
-        Returns a short status message for UI feedback.
-        """
-        try:
-            # Update the in-process config value and env
-            from reachy_mini_conversation_app.config import config as _config
-            from reachy_mini_conversation_app.config import set_custom_profile
-
-            set_custom_profile(profile)
-            self._voice_override = None
-            logger.info(
-                "Set custom profile to %r (config=%r)", profile, getattr(_config, "REACHY_MINI_CUSTOM_PROFILE", None)
-            )
-
-            try:
-                instructions = get_session_instructions()
-                voice = get_session_voice()
-            except BaseException as e:  # catch SystemExit from prompt loader without crashing
-                logger.error("Failed to resolve personality content: %s", e)
-                return f"Failed to apply personality: {e}"
-
-            # Attempt a live update first, then force a full restart to ensure it sticks
-            if self.connection is not None:
-                try:
-                    await self.connection.session.update(
-                        session=RealtimeSessionCreateRequestParam(
-                            type="realtime",
-                            instructions=instructions,
-                            audio=RealtimeAudioConfigParam(
-                                output=RealtimeAudioConfigOutputParam(
-                                    voice=voice,
-                                ),
-                            ),
-                        ),
-                    )
-                    logger.info("Applied personality via live update: %s", profile or "built-in default")
-                except Exception as e:
-                    logger.warning("Live update failed; will restart session: %s", e)
-
-                # Force a real restart to guarantee the new instructions/voice
-                try:
-                    await self._restart_session()
-                    return "Applied personality and restarted realtime session."
-                except Exception as e:
-                    logger.warning("Failed to restart session after apply: %s", e)
-                    return "Applied personality. Will take effect on next connection."
-            else:
-                logger.info(
-                    "Applied personality recorded: %s (no live connection; will apply on next session)",
-                    profile or "built-in default",
-                )
-                return "Applied personality. Will take effect on next connection."
-        except Exception as e:
-            logger.error("Error applying personality '%s': %s", profile, e)
-            return f"Failed to apply personality: {e}"
-
-    async def change_voice(self, voice: str) -> str:
-        """Change only the voice and restart the session."""
-        self._voice_override = voice
-        if getattr(self, "client", None) is not None:
-            try:
-                await self._restart_session()
-                return f"Voice changed to {voice}."
-            except Exception as e:
-                logger.warning("Failed to restart session for voice change: %s", e)
-                return "Voice change failed. Will take effect on next connection."
-        return "Voice changed. Will take effect on next connection."
-
-    def get_current_voice(self) -> str:
-        """Return the voice currently selected for this handler."""
-        return self._voice_override or get_session_voice()
-
-    async def _emit_debounced_partial(self, transcript: str, item_id: str, sequence_counter: int) -> None:
-        """Emit partial transcript after debounce delay."""
-        try:
-            await asyncio.sleep(self.partial_debounce_delay)
-
-            input_transcript = self.input_transcript_chunks_by_item
-            if input_transcript.item_id == item_id and len(input_transcript.deltas) - 1 == sequence_counter:
-                await self.output_queue.put(AdditionalOutputs({"role": "user_partial", "content": transcript}))
-                logger.debug(f"Debounced partial emitted: {transcript}")
-        except asyncio.CancelledError:
-            logger.debug("Debounced partial cancelled")
-            raise
-
-    async def start_up(self) -> None:
-        """Start the handler with minimal retries on unexpected websocket closure."""
+    async def _prepare_startup_credentials(self) -> None:
+        """Collect an OpenAI API key from Gradio input when needed."""
         openai_api_key = config.OPENAI_API_KEY
-        if self.gradio_mode and not openai_api_key:
-            # api key was not found in .env or in the environment variables
-            await self.wait_for_args()  # type: ignore[no-untyped-call]
-            args = list(self.latest_args)
-            textbox_api_key = args[3] if len(args[3]) > 0 else None
-            if textbox_api_key is not None:
-                openai_api_key = textbox_api_key
-                self._key_source = "textbox"
-                self._provided_api_key = textbox_api_key
-            else:
-                openai_api_key = config.OPENAI_API_KEY
-        else:
-            if not openai_api_key or not openai_api_key.strip():
-                # In headless console mode, LocalStream now blocks startup until the key is provided.
-                # However, unit tests may invoke this handler directly with a stubbed client.
-                # To keep tests hermetic without requiring a real key, fall back to a placeholder.
-                logger.warning("OPENAI_API_KEY missing. Proceeding with a placeholder (tests/offline).")
-                openai_api_key = "DUMMY"
+        if not self.gradio_mode or openai_api_key:
+            return
 
-        self.client = AsyncOpenAI(api_key=openai_api_key)
+        await self.wait_for_args()  # type: ignore[no-untyped-call]
+        args = list(self.latest_args)
+        textbox_api_key = args[3] if len(args) > 3 and len(args[3]) > 0 else None
+        if textbox_api_key is not None:
+            self._key_source = "textbox"
+            self._provided_api_key = textbox_api_key
 
-        max_attempts = 3
-        for attempt in range(1, max_attempts + 1):
-            try:
-                await self._run_realtime_session()
-                # Normal exit from the session, stop retrying
+    def _persist_credentials_if_needed(self) -> None:
+        """Persist a textbox-provided OpenAI API key into the instance `.env`."""
+        try:
+            if not self.gradio_mode:
+                logger.warning("Not in Gradio mode; skipping OpenAI API key persistence.")
                 return
-            except ConnectionClosedError as e:
-                # Abrupt close (e.g., "no close frame received or sent") → retry
-                logger.warning("Realtime websocket closed unexpectedly (attempt %d/%d): %s", attempt, max_attempts, e)
-                if attempt < max_attempts:
-                    # exponential backoff with jitter
-                    base_delay = 2 ** (attempt - 1)  # 1s, 2s, 4s, 8s, etc.
-                    jitter = random.uniform(0, 0.5)
-                    delay = base_delay + jitter
-                    logger.info("Retrying in %.1f seconds...", delay)
-                    await asyncio.sleep(delay)
-                    continue
-                raise
-            finally:
-                # never keep a stale reference
-                self.connection = None
-                try:
-                    self._connected_event.clear()
-                except Exception:
-                    pass
 
-    async def _restart_session(self) -> None:
-        """Force-close the current session and start a fresh one in background.
-
-        Does not block the caller while the new session is establishing.
-        """
-        try:
-            if self.connection is not None:
-                try:
-                    await self.connection.close()
-                except Exception:
-                    pass
-                finally:
-                    self.connection = None
+            if self._key_source != "textbox":
+                logger.info("OpenAI API key not provided via textbox; skipping persistence.")
+                return
 
-            # Ensure we have a client (start_up must have run once)
-            if getattr(self, "client", None) is None:
-                logger.warning("Cannot restart: OpenAI client not initialized yet.")
+            key = (self._provided_api_key or "").strip()
+            if not key:
+                logger.warning("No OpenAI API key provided via textbox; skipping persistence.")
+                return
+            if self.instance_path is None:
+                logger.warning("Instance path is None; cannot persist OpenAI API key.")
                 return
 
-            # Fire-and-forget new session and wait briefly for connection
-            try:
-                self._connected_event.clear()
-            except Exception:
-                pass
-            asyncio.create_task(self._run_realtime_session(), name="openai-realtime-restart")
+            # Update the current process environment for downstream consumers.
             try:
-                await asyncio.wait_for(self._connected_event.wait(), timeout=5.0)
-                logger.info("Realtime session restarted and connected.")
-            except asyncio.TimeoutError:
-                logger.warning("Realtime session restart timed out; continuing in background.")
-        except Exception as e:
-            logger.warning("_restart_session failed: %s", e)
-
-    async def _safe_response_create(self, **kwargs: Any) -> None:
-        """Enqueue a response.create() kwargs for the sender worker _response_sender_loop().
-
-        This method never blocks the caller.
-        """
-        await self._pending_responses.put(kwargs)
-
-    async def _response_sender_loop(self) -> None:
-        """Dedicated worker that sends ``response.create()`` calls serially.
+                import os
 
-        This logic was designed to comply with the response.create() docstring specification for event ordering:
-        https://github.com/openai/openai-python/blob/3e0c05b84a2056870abf3bd6a5e7849020209cc3/src/openai/resources/realtime/realtime.py#L649C1-L651C30
+                os.environ["OPENAI_API_KEY"] = key
+            except Exception:  # best-effort
+                pass
 
-        For each queued request the worker:
-        1. Waits until no response is active (_response_done_event).
-        2. Sends response.create().
-        3. Waits for the response cycle to complete (response.done).
-        4. If the server rejected with active_response, retries from step 1.
-        """
-        while self.connection:
-            try:
-                kwargs = await self._pending_responses.get()
-            except asyncio.CancelledError:
+            target_dir = Path(self.instance_path)
+            env_path = target_dir / ".env"
+            if env_path.exists():
+                # Respect existing user configuration.
+                logger.info(".env already exists at %s; not overwriting.", env_path)
                 return
 
-            sent = False
-            max_retries = 5
-            attempts = 0
-            while not sent and self.connection and attempts < max_retries:
-                try:
-                    await asyncio.wait_for(self._response_done_event.wait(), timeout=_RESPONSE_DONE_TIMEOUT)
-                except asyncio.TimeoutError:
-                    logger.debug("Timed out waiting for previous response to finish; forcing ahead")
-                    self._response_done_event.set()
-
-                if not self.connection:
-                    break
-
-                self._last_response_rejected = False
+            example_path = target_dir / ".env.example"
+            content_lines: list[str] = []
+            if example_path.exists():
                 try:
-                    await self.connection.response.create(**kwargs)
+                    content = example_path.read_text(encoding="utf-8")
+                    content_lines = content.splitlines()
                 except Exception as e:
-                    logger.debug("_response_sender_loop: send failed: %s", e)
-                    self._response_done_event.set()
-                    break
+                    logger.warning("Failed to read .env.example at %s: %s", example_path, e)
 
-                try:
-                    await asyncio.wait_for(self._response_done_event.wait(), timeout=_RESPONSE_DONE_TIMEOUT)
-                except asyncio.TimeoutError:
-                    logger.debug("Timed out waiting for response.done; assuming response completed")
-                    self._response_done_event.set()
+            replaced = False
+            for i, line in enumerate(content_lines):
+                if line.strip().startswith("OPENAI_API_KEY="):
+                    content_lines[i] = f"OPENAI_API_KEY={key}"
+                    replaced = True
                     break
+            if not replaced:
+                content_lines.append(f"OPENAI_API_KEY={key}")
 
-                # Check if we were rejected
-                if self._last_response_rejected:
-                    attempts += 1
-                    if attempts >= max_retries:
-                        logger.debug("response.create rejected %d times; giving up", attempts)
-                        break
-                    logger.debug("response.create was rejected; retrying (%d/%d)", attempts, max_retries)
-                    continue
-
-                sent = True
-
-    async def _handle_tool_result(self, bg_tool: ToolNotification) -> None:
-        """Process the result of a tool call."""
-        if bg_tool.error is not None:
-            logger.error("Tool '%s' (id=%s) failed with error: %s", bg_tool.tool_name, bg_tool.id, bg_tool.error)
-            tool_result = {"error": bg_tool.error}
-        elif bg_tool.result is not None:
-            tool_result = bg_tool.result
-            logger.info(
-                "Tool '%s' (id=%s) executed successfully.",
-                bg_tool.tool_name,
-                bg_tool.id,
-            )
-            logger.debug("Tool '%s' full result: %s", bg_tool.tool_name, tool_result)
-        else:
-            logger.warning("Tool '%s' (id=%s) returned no result and no error", bg_tool.tool_name, bg_tool.id)
-            tool_result = {"error": "No result returned from tool execution"}
-
-        # Connection may have closed while tool was running
-        if not self.connection:
-            logger.warning(
-                "Connection closed during tool '%s' (id=%s) execution; cannot send result back",
-                bg_tool.tool_name,
-                bg_tool.id,
-            )
-            return
-
-        try:
-            # TODO: refactor this since it's repeated here, in the camera branch below, and in send_idle_signal
-            if isinstance(bg_tool.id, str):
-                await self.connection.conversation.item.create(
-                    item={
-                        "type": "function_call_output",
-                        "call_id": bg_tool.id,
-                        "output": json.dumps(tool_result),
-                    },
-                )
-
-            await self.output_queue.put(
-                AdditionalOutputs(
-                    {
-                        "role": "assistant",
-                        "content": json.dumps(tool_result),
-                        # Gradio UI metadata.status accept only "pending" and "done". Do not accept bg.tool.status values.
-                        "metadata": {
-                            "title": f"🛠️ Used tool {bg_tool.tool_name}",
-                            "status": "done",
-                        },
-                    },
-                ),
-            )
-
-            if bg_tool.tool_name == "camera" and "b64_im" in tool_result:
-                # use raw base64, don't json.dumps (which adds quotes)
-                b64_im = tool_result["b64_im"]
-                if not isinstance(b64_im, str):
-                    logger.warning("Unexpected type for b64_im: %s", type(b64_im))
-                    b64_im = str(b64_im)
-                await self.connection.conversation.item.create(
-                    item={
-                        "type": "message",
-                        "role": "user",
-                        "content": [
-                            {
-                                "type": "input_image",
-                                "image_url": f"data:image/jpeg;base64,{b64_im}",
-                            },
-                        ],
-                    },
-                )
-                logger.info("Added camera image to conversation")
-
-                if self.deps.camera_worker is not None:
-                    np_img = self.deps.camera_worker.get_latest_frame()
-                    if np_img is not None:
-                        rgb_frame = np.ascontiguousarray(np_img[..., ::-1])
-                    else:
-                        rgb_frame = None
-                    img = gr.Image(value=rgb_frame)
-
-                    await self.output_queue.put(
-                        AdditionalOutputs(
-                            {
-                                "role": "assistant",
-                                "content": img,
-                            },
-                        ),
-                    )
-
-            # If this tool call was triggered by an idle signal, don't make the robot speak.
-            # For other tool calls, let the robot reply out loud.
-            if not bg_tool.is_idle_tool_call:
-                await self._safe_response_create(
-                    response=RealtimeResponseCreateParamsParam(
-                        instructions="Use the tool result just returned and answer concisely in speech.",
-                    ),
-                )
-
-        except ConnectionClosedError:
-            logger.warning("Connection closed while sending tool result")
-            self.connection = None
-            self._response_done_event.set()
-
-    async def _run_realtime_session(self) -> None:
-        """Establish and manage a single realtime session."""
-        async with self.client.realtime.connect(model=config.MODEL_NAME) as conn:
-            try:
-                session_config = RealtimeSessionCreateRequestParam(
-                    type="realtime",
-                    instructions=get_session_instructions(),
-                    audio=RealtimeAudioConfigParam(
-                        input=RealtimeAudioConfigInputParam(
-                            format=AudioPCM(type="audio/pcm", rate=self.input_sample_rate),
-                            transcription=AudioTranscriptionParam(model="gpt-4o-transcribe", language="en"),
-                            turn_detection=ServerVad(type="server_vad", interrupt_response=True),
-                        ),
-                        output=RealtimeAudioConfigOutputParam(
-                            format=AudioPCM(type="audio/pcm", rate=self.output_sample_rate),
-                            voice=self._voice_override or get_session_voice(),
-                        ),
-                    ),
-                    tools=get_tool_specs(),  # type: ignore[typeddict-item]
-                    tool_choice="auto",
-                )
-                await conn.session.update(session=session_config)
-                logger.info(
-                    "Realtime session initialized with profile=%r voice=%r",
-                    getattr(config, "REACHY_MINI_CUSTOM_PROFILE", None),
-                    self._voice_override or get_session_voice(),
-                )
-                # If we reached here, the session update succeeded which implies the API key worked.
-                # Persist the key to a newly created .env (copied from .env.example) if needed.
-                self._persist_api_key_if_needed()
-            except Exception:
-                logger.exception("Realtime session.update failed; aborting startup")
-                return
-
-            logger.info("Realtime session updated successfully")
-
-            # Reset the partial-transcript accumulator for each new session
-            self.input_transcript_chunks_by_item = InputTranscriptChunksByItem()
-
-            # Manage event received from the openai server
-            self.connection = conn
-            try:
-                self._connected_event.set()
-            except Exception:
-                pass
-
-            response_sender_task: asyncio.Task[None] | None = None
-            try:
-                # Start the background tool manager
-                self.tool_manager.start_up(tool_callbacks=[self._handle_tool_result])
-
-                # Start the response sender worker
-                response_sender_task = asyncio.create_task(self._response_sender_loop(), name="response-sender")
-
-                async for event in self.connection:
-                    logger.debug(f"OpenAI event: {event.type}")
-                    if event.type == "input_audio_buffer.speech_started":
-                        if hasattr(self, "_clear_queue") and callable(self._clear_queue):
-                            self._clear_queue()
-                        if self.deps.head_wobbler is not None:
-                            self.deps.head_wobbler.reset()
-                        self.deps.movement_manager.set_listening(True)
-                        logger.debug("User speech started")
-
-                    if event.type == "input_audio_buffer.speech_stopped":
-                        self.deps.movement_manager.set_listening(False)
-                        logger.debug("User speech stopped - server will auto-commit with VAD")
-
-                    if event.type == "response.output_audio.done":
-                        if self.deps.head_wobbler is not None:
-                            self.deps.head_wobbler.request_reset_after_current_audio()
-                        logger.debug("response completed")
-
-                    if event.type == "response.created":
-                        self._response_done_event.clear()
-                        logger.debug("Response created (active)")
-
-                    if event.type == "response.done":
-                        # Doesn't mean the audio is done playing
-                        self._response_done_event.set()
-                        logger.debug("Response done")
-
-                        response = getattr(event, "response", None)
-                        usage = getattr(response, "usage", None) if response else None
-                        if usage:
-                            cost = _compute_response_cost(usage)
-                            self.cumulative_cost += cost
-                            logger.debug("Cost: $%.4f | Cumulative: $%.4f", cost, self.cumulative_cost)
-                        else:
-                            logger.warning("No usage data available for cost tracking")
-
-                    if event.type == "conversation.item.input_audio_transcription.delta":
-                        logger.debug(f"User partial transcript: {event.delta}")
-
-                        item_id = event.item_id
-                        delta = event.delta or ""
-
-                        input_transcript = self.input_transcript_chunks_by_item
-                        if input_transcript.item_id != item_id:
-                            input_transcript.item_id = item_id
-                            input_transcript.deltas = [delta]
-                        else:
-                            input_transcript.deltas.append(delta)
-
-                        sequence_counter = len(input_transcript.deltas) - 1
-
-                        # Cancel previous debounce task if it exists
-                        if self.partial_transcript_task and not self.partial_transcript_task.done():
-                            self.partial_transcript_task.cancel()
-                            try:
-                                await self.partial_transcript_task
-                            except asyncio.CancelledError:
-                                pass
-
-                        # Start new debounce timer with the last delta
-                        self.partial_transcript_task = asyncio.create_task(
-                            self._emit_debounced_partial("".join(input_transcript.deltas), item_id, sequence_counter)
-                        )
-
-                    # Handle completed transcription (user finished speaking)
-                    if event.type == "conversation.item.input_audio_transcription.completed":
-                        logger.debug(f"User transcript: {event.transcript}")
-
-                        # Cancel any pending partial emission
-                        if self.partial_transcript_task and not self.partial_transcript_task.done():
-                            self.partial_transcript_task.cancel()
-                            try:
-                                await self.partial_transcript_task
-                            except asyncio.CancelledError:
-                                pass
-
-                        await self.output_queue.put(AdditionalOutputs({"role": "user", "content": event.transcript}))
-
-                    # Handle assistant transcription
-                    if event.type == "response.output_audio_transcript.done":
-                        logger.debug(f"Assistant transcript: {event.transcript}")
-                        await self.output_queue.put(
-                            AdditionalOutputs({"role": "assistant", "content": event.transcript})
-                        )
-
-                    # Handle audio delta
-                    if event.type == "response.output_audio.delta":
-                        if self.gradio_mode and self.deps.head_wobbler is not None:
-                            self.deps.head_wobbler.feed(event.delta)
-                        self.last_activity_time = asyncio.get_event_loop().time()
-                        logger.debug("last activity time updated to %s", self.last_activity_time)
-                        await self.output_queue.put(
-                            (
-                                self.output_sample_rate,
-                                np.frombuffer(base64.b64decode(event.delta), dtype=np.int16).reshape(1, -1),
-                            ),
-                        )
-
-                    # ---- tool-calling plumbing ----
-                    if event.type == "response.function_call_arguments.done":
-                        tool_name = getattr(event, "name", None)
-                        args_json_str = getattr(event, "arguments", None)
-                        call_id: str = str(getattr(event, "call_id", uuid.uuid4()))
-
-                        logger.info(
-                            "Tool call received — tool_name=%r, call_id=%s, is_idle=%s, args=%s",
-                            tool_name,
-                            call_id,
-                            self.is_idle_tool_call,
-                            args_json_str,
-                        )
-
-                        if not isinstance(tool_name, str) or not isinstance(args_json_str, str):
-                            logger.error(
-                                "Invalid tool call: tool_name=%s (type=%s), args=%s (type=%s), call_id=%s",
-                                tool_name,
-                                type(tool_name).__name__,
-                                args_json_str,
-                                type(args_json_str).__name__,
-                                call_id,
-                            )
-                            continue
-
-                        bg_tool = await self.tool_manager.start_tool(
-                            call_id=call_id,
-                            tool_call_routine=ToolCallRoutine(
-                                tool_name=tool_name,
-                                args_json_str=args_json_str,
-                                deps=self.deps,
-                            ),
-                            is_idle_tool_call=self.is_idle_tool_call,
-                        )
-
-                        await self.output_queue.put(
-                            AdditionalOutputs(
-                                {
-                                    "role": "assistant",
-                                    "content": f"🛠️ Used tool {tool_name} with args {args_json_str}. The tool is now running. Tool ID: {bg_tool.tool_id}",
-                                },
-                            ),
-                        )
-
-                        if self.is_idle_tool_call:
-                            self.is_idle_tool_call = False
-
-                        logger.info(
-                            "Started background tool: %s (id=%s, call_id=%s)", tool_name, bg_tool.tool_id, call_id
-                        )
-
-                    # server error
-                    if event.type == "error":
-                        err = getattr(event, "error", None)
-                        msg = getattr(err, "message", str(err) if err else "unknown error")
-                        code = getattr(err, "code", "")
-
-                        if code == "conversation_already_has_active_response":
-                            # response.create was rejected.  The sender worker
-                            # is waiting on _response_done_event; when the active
-                            # response finishes it will wake up and see this flag.
-                            self._last_response_rejected = True
-                            logger.debug("response.create rejected; worker will retry after active response finishes")
-                        else:
-                            logger.error("Realtime error [%s]: %s (raw=%s)", code, msg, err)
-
-                        # Only show user-facing errors, not internal state errors
-                        if code not in ("input_audio_buffer_commit_empty",):
-                            await self.output_queue.put(
-                                AdditionalOutputs({"role": "assistant", "content": f"[error] {msg}"})
-                            )
-            finally:
-                # Stop the response sender worker.
-                if response_sender_task is not None:
-                    response_sender_task.cancel()
-                    try:
-                        await response_sender_task
-                    except asyncio.CancelledError:
-                        pass
-
-                # Stop background tool manager tasks (listener + cleanup) in all patus.
-                await self.tool_manager.shutdown()
-
-    # Microphone receive
-    async def receive(self, frame: Tuple[int, NDArray[np.int16]]) -> None:
-        """Receive audio frame from the microphone and send it to the OpenAI server.
-
-        Handles both mono and stereo audio formats, converting to the expected
-        mono format for OpenAI's API. Resamples if the input sample rate differs
-        from the expected rate.
-
-        Args:
-            frame: A tuple containing (sample_rate, audio_data).
-
-        """
-        if not self.connection:
-            return
-
-        input_sample_rate, audio_frame = frame
-
-        # Reshape if needed
-        if audio_frame.ndim == 2:
-            # Scipy channels last convention
-            if audio_frame.shape[1] > audio_frame.shape[0]:
-                audio_frame = audio_frame.T
-            # Multiple channels -> Mono channel
-            if audio_frame.shape[1] > 1:
-                audio_frame = audio_frame[:, 0]
-
-        # Resample if needed
-        if self.input_sample_rate != input_sample_rate:
-            audio_frame = resample(audio_frame, int(len(audio_frame) * self.input_sample_rate / input_sample_rate))
-
-        # Cast if needed
-        audio_frame = audio_to_int16(audio_frame)
-
-        # Send to OpenAI (guard against races during reconnect)
-        try:
-            audio_message = base64.b64encode(audio_frame.tobytes()).decode("utf-8")
-            await self.connection.input_audio_buffer.append(audio=audio_message)
+            final_text = "\n".join(content_lines) + "\n"
+            env_path.write_text(final_text, encoding="utf-8")
+            logger.info("Created %s and stored OPENAI_API_KEY for future runs.", env_path)
         except Exception as e:
-            logger.debug("Dropping audio frame: connection not ready (%s)", e)
-            return
-
-    async def emit(self) -> Tuple[int, NDArray[np.int16]] | AdditionalOutputs | None:
-        """Emit audio frame to be played by the speaker."""
-        # sends to the stream the stuff put in the output queue by the openai event handler
-        # This is called periodically by the fastrtc Stream
-
-        # Handle idle
-        idle_duration = asyncio.get_event_loop().time() - self.last_activity_time
-        if idle_duration > 15.0 and self.deps.movement_manager.is_idle():
-            try:
-                await self.send_idle_signal(idle_duration)
-            except Exception as e:
-                logger.warning("Idle signal skipped (connection closed?): %s", e)
-                return None
-
-            self.last_activity_time = asyncio.get_event_loop().time()  # avoid repeated resets
-
-        return await wait_for_item(self.output_queue)  # type: ignore[no-any-return]
-
-    async def shutdown(self) -> None:
-        """Shutdown the handler."""
-        # Unblock the response sender worker so it can exit
-        self._response_done_event.set()
-
-        # Stop background tool manager tasks (listener + cleanup)
-        await self.tool_manager.shutdown()
-
-        # Cancel any pending debounce task
-        if self.partial_transcript_task and not self.partial_transcript_task.done():
-            self.partial_transcript_task.cancel()
-            try:
-                await self.partial_transcript_task
-            except asyncio.CancelledError:
-                pass
-
-        if self.connection:
-            try:
-                await self.connection.close()
-            except ConnectionClosedError as e:
-                logger.debug(f"Connection already closed during shutdown: {e}")
-            except Exception as e:
-                logger.debug(f"connection.close() ignored: {e}")
-            finally:
-                self.connection = None
-
-        # Clear any remaining items in the output queue
-        while not self.output_queue.empty():
-            try:
-                self.output_queue.get_nowait()
-            except asyncio.QueueEmpty:
-                break
+            # Never crash the app for QoL persistence; just log.
+            logger.warning("Could not persist OPENAI_API_KEY to .env: %s", e)
 
-    def format_timestamp(self) -> str:
-        """Format current timestamp with date, time, and elapsed seconds."""
-        loop_time = asyncio.get_event_loop().time()  # monotonic
-        elapsed_seconds = loop_time - self.start_time
-        dt = datetime.now()  # wall-clock
-        return f"[{dt.strftime('%Y-%m-%d %H:%M:%S')} | +{elapsed_seconds:.1f}s]"
+    def _get_session_instructions(self) -> str:
+        """Return OpenAI session instructions."""
+        return get_session_instructions()
+
+    def _get_session_voice(self, default: str | None = None) -> str:
+        """Return the configured OpenAI session voice."""
+        return get_session_voice(default)
+
+    def _get_active_tool_specs(self) -> list[dict[str, Any]]:
+        """Return active tool specs for the current session dependencies."""
+        return get_active_tool_specs(self.deps)
+
+    def _get_session_config(self, tool_specs: list[dict[str, Any]]) -> RealtimeSessionCreateRequestParam:
+        """Return the OpenAI Realtime session config."""
+        return RealtimeSessionCreateRequestParam(
+            type="realtime",
+            instructions=self._get_session_instructions(),
+            audio=RealtimeAudioConfigParam(
+                input=RealtimeAudioConfigInputParam(
+                    format=AudioPCM(type="audio/pcm", rate=24000),
+                    transcription=AudioTranscriptionParam(model="gpt-4o-transcribe", language="en"),
+                    turn_detection=ServerVad(type="server_vad", interrupt_response=True),
+                ),
+                output=RealtimeAudioConfigOutputParam(
+                    format=AudioPCM(type="audio/pcm", rate=24000),
+                    voice=self.get_current_voice(),
+                ),
+            ),
+            tools=to_realtime_tools_config(tool_specs),
+            tool_choice="auto",
+        )
 
     async def get_available_voices(self) -> list[str]:
-        """Try to discover available voices for the configured realtime model.
+        """Try to discover available voices for the configured OpenAI realtime model.
 
         Attempts to retrieve model metadata from the OpenAI Models API and look
         for any keys that might contain voice names. Falls back to a curated
         list known to work with realtime if discovery fails.
         """
-        fallback = list(AVAILABLE_VOICES)
+        fallback = await super().get_available_voices()
         try:
-            # Best effort discovery; safe-guarded for unexpected shapes
             model = await self.client.models.retrieve(config.MODEL_NAME)
-            # Try common serialization paths
             raw = None
             for attr in ("model_dump", "to_dict"):
                 fn = getattr(model, attr, None)
@@ -839,123 +176,46 @@ class OpenaiRealtimeHandler(AsyncStreamHandler):
                     raw = dict(model)
                 except Exception:
                     raw = None
-            # Scan for voice candidates
+
             candidates: set[str] = set()
 
             def _collect(obj: object) -> None:
                 try:
                     if isinstance(obj, dict):
-                        for k, v in obj.items():
-                            kl = str(k).lower()
-                            if "voice" in kl and isinstance(v, (list, tuple)):
-                                for item in v:
+                        for key, value in obj.items():
+                            key_lower = str(key).lower()
+                            if "voice" in key_lower and isinstance(value, (list, tuple)):
+                                for item in value:
                                     if isinstance(item, str):
                                         candidates.add(item)
-                                    elif isinstance(item, dict) and "name" in item and isinstance(item["name"], str):
+                                    elif isinstance(item, dict) and isinstance(item.get("name"), str):
                                         candidates.add(item["name"])
                             else:
-                                _collect(v)
+                                _collect(value)
                     elif isinstance(obj, (list, tuple)):
-                        for it in obj:
-                            _collect(it)
+                        for item in obj:
+                            _collect(item)
                 except Exception:
                     pass
 
             if isinstance(raw, dict):
                 _collect(raw)
-            # Ensure default present and stable order
+
             voices = sorted(candidates) if candidates else fallback
-            if "cedar" not in voices:
-                voices = ["cedar", *[v for v in voices if v != "cedar"]]
+            default_voice = get_default_voice_for_backend(self.BACKEND_PROVIDER)
+            if default_voice not in voices:
+                voices = [default_voice, *[voice for voice in voices if voice != default_voice]]
             return voices
         except Exception:
             return fallback
 
-    async def send_idle_signal(self, idle_duration: float) -> None:
-        """Send an idle signal to the openai server."""
-        logger.debug("Sending idle signal")
-        self.is_idle_tool_call = True
-        timestamp_msg = f"[Idle time update: {self.format_timestamp()} - No activity for {idle_duration:.1f}s] You've been idle for a while. Feel free to get creative - dance, show an emotion, look around, do nothing, or just be yourself!"
-        if not self.connection:
-            logger.debug("No connection, cannot send idle signal")
-            return
-        await self.connection.conversation.item.create(
-            item={
-                "type": "message",
-                "role": "user",
-                "content": [{"type": "input_text", "text": timestamp_msg}],
-            },
-        )
-        await self._safe_response_create(
-            response=RealtimeResponseCreateParamsParam(
-                instructions="You MUST respond with function calls only - no speech or text. Choose appropriate actions for idle behavior.",
-                tool_choice="required",
-            ),
-        )
-
-    def _persist_api_key_if_needed(self) -> None:
-        """Persist the API key into `.env` inside `instance_path/` when appropriate.
-
-        - Only runs in Gradio mode when key came from the textbox and is non-empty.
-        - Only saves if `self.instance_path` is not None.
-        - Writes `.env` to `instance_path/.env` (does not overwrite if it already exists).
-        - If `instance_path/.env.example` exists, copies its contents while overriding OPENAI_API_KEY.
-        """
-        try:
-            if not self.gradio_mode:
-                logger.warning("Not in Gradio mode; skipping API key persistence.")
-                return
-
-            if self._key_source != "textbox":
-                logger.info("API key not provided via textbox; skipping persistence.")
-                return
-
-            key = (self._provided_api_key or "").strip()
-            if not key:
-                logger.warning("No API key provided via textbox; skipping persistence.")
-                return
-            if self.instance_path is None:
-                logger.warning("Instance path is None; cannot persist API key.")
-                return
-
-            # Update the current process environment for downstream consumers
-            try:
-                import os
-
-                os.environ["OPENAI_API_KEY"] = key
-            except Exception:  # best-effort
-                pass
-
-            target_dir = Path(self.instance_path)
-            env_path = target_dir / ".env"
-            if env_path.exists():
-                # Respect existing user configuration
-                logger.info(".env already exists at %s; not overwriting.", env_path)
-                return
-
-            example_path = target_dir / ".env.example"
-            content_lines: list[str] = []
-            if example_path.exists():
-                try:
-                    content = example_path.read_text(encoding="utf-8")
-                    content_lines = content.splitlines()
-                except Exception as e:
-                    logger.warning("Failed to read .env.example at %s: %s", example_path, e)
-
-            # Replace or append the OPENAI_API_KEY line
-            replaced = False
-            for i, line in enumerate(content_lines):
-                if line.strip().startswith("OPENAI_API_KEY="):
-                    content_lines[i] = f"OPENAI_API_KEY={key}"
-                    replaced = True
-                    break
-            if not replaced:
-                content_lines.append(f"OPENAI_API_KEY={key}")
-
-            # Ensure file ends with newline
-            final_text = "\n".join(content_lines) + "\n"
-            env_path.write_text(final_text, encoding="utf-8")
-            logger.info("Created %s and stored OPENAI_API_KEY for future runs.", env_path)
-        except Exception as e:
-            # Never crash the app for QoL persistence; just log.
-            logger.warning("Could not persist OPENAI_API_KEY to .env: %s", e)
+    async def _build_realtime_client(self) -> AsyncOpenAI:
+        """Build the OpenAI realtime SDK client."""
+        self._realtime_connect_query = {}
+        resolved_api_key = (self._provided_api_key or config.OPENAI_API_KEY or "").strip()
+        if not resolved_api_key:
+            # In headless console mode, LocalStream blocks startup until the key is provided.
+            # Unit tests may invoke this handler directly with a stubbed client.
+            logger.warning("OPENAI_API_KEY missing. Proceeding with a placeholder (tests/offline).")
+            resolved_api_key = "DUMMY"
+        return AsyncOpenAI(api_key=resolved_api_key)

src/reachy_mini_conversation_app/startup_settings.py

@@ -0,0 +1,106 @@
+"""Helpers for persisting UI-selected startup profile and voice settings."""
+
+from __future__ import annotations
+import os
+import json
+import logging
+from pathlib import Path
+from dataclasses import dataclass
+
+
+logger = logging.getLogger(__name__)
+
+STARTUP_SETTINGS_FILENAME = "startup_settings.json"
+
+
+@dataclass(frozen=True)
+class StartupSettings:
+    """Instance-local startup profile/voice settings selected from the UI."""
+
+    profile: str | None = None
+    voice: str | None = None
+
+
+def _normalize_optional_text(value: object) -> str | None:
+    """Return a stripped string or None for empty/non-string values."""
+    if not isinstance(value, str):
+        return None
+    normalized = value.strip()
+    return normalized or None
+
+
+def _startup_settings_path(instance_path: str | Path | None) -> Path | None:
+    """Return the startup settings JSON path for an instance directory."""
+    if instance_path is None:
+        return None
+    return Path(instance_path) / STARTUP_SETTINGS_FILENAME
+
+
+def read_startup_settings(instance_path: str | Path | None) -> StartupSettings:
+    """Read startup settings from an instance-local JSON file."""
+    settings_path = _startup_settings_path(instance_path)
+    if settings_path is None or not settings_path.exists():
+        return StartupSettings()
+
+    try:
+        payload = json.loads(settings_path.read_text(encoding="utf-8"))
+    except Exception as exc:
+        logger.warning("Failed to read startup settings from %s: %s", settings_path, exc)
+        return StartupSettings()
+
+    if not isinstance(payload, dict):
+        logger.warning("Ignoring invalid startup settings payload from %s: %r", settings_path, payload)
+        return StartupSettings()
+
+    return StartupSettings(
+        profile=_normalize_optional_text(payload.get("profile")),
+        voice=_normalize_optional_text(payload.get("voice")),
+    )
+
+
+def write_startup_settings(
+    instance_path: str | Path | None,
+    *,
+    profile: str | None,
+    voice: str | None,
+) -> None:
+    """Persist startup settings in an instance-local JSON file."""
+    settings_path = _startup_settings_path(instance_path)
+    if settings_path is None:
+        return
+
+    settings = StartupSettings(
+        profile=_normalize_optional_text(profile),
+        voice=_normalize_optional_text(voice),
+    )
+    if settings.profile is None and settings.voice is None:
+        try:
+            settings_path.unlink()
+        except FileNotFoundError:
+            return
+        return
+
+    payload: dict[str, str] = {}
+    if settings.profile is not None:
+        payload["profile"] = settings.profile
+    if settings.voice is not None:
+        payload["voice"] = settings.voice
+
+    settings_path.write_text(f"{json.dumps(payload, indent=2, sort_keys=True)}\n", encoding="utf-8")
+
+
+def load_startup_settings_into_runtime(instance_path: str | Path | None) -> StartupSettings:
+    """Load instance-local startup settings when no explicit profile override is set."""
+    from reachy_mini_conversation_app.config import LOCKED_PROFILE, set_custom_profile
+
+    if LOCKED_PROFILE is not None:
+        return StartupSettings()
+
+    settings_path = _startup_settings_path(instance_path)
+    settings = read_startup_settings(instance_path)
+    if settings_path is None or not settings_path.exists():
+        if os.getenv("REACHY_MINI_CUSTOM_PROFILE"):
+            return StartupSettings(voice=settings.voice)
+
+    set_custom_profile(settings.profile)
+    return settings

src/reachy_mini_conversation_app/static/index.html

@@ -16,18 +16,18 @@
       <header class="hero">
         <div class="pill">Headless control</div>
         <h1>Reachy Mini Conversation</h1>
-        <p class="subtitle">Choose a realtime backend, add a Gemini token if needed, and tweak personalities without the full UI.</p>
+        <p class="subtitle">Choose a realtime backend, add credentials only when needed, and tweak personalities without the full UI.</p>
       </header>
 
       <aside class="privacy-notice" role="note" aria-label="Privacy notice">
         <p>
           <strong>Privacy notice:</strong>
-          Speech and camera data are sent to your selected realtime backend for inference. See OpenAI's
+          Speech and camera data are sent to your selected realtime backend for inference. Hugging Face won't store any speech or images sent to our server. See OpenAI's
           <a href="https://platform.openai.com/docs/guides/your-data" target="_blank" rel="noopener">
             data usage policy
           </a>
           and Google's
-          <a href="https://ai.google.dev/gemini-api/docs/data-usage" target="_blank" rel="noopener">
+          <a href="https://ai.google.dev/gemini-api/terms#data-use-unpaid" target="_blank" rel="noopener">
             Gemini API data usage policy
           </a>
           for details.
@@ -37,11 +37,13 @@
           <summary>Learn more</summary>
 
           <p>
-            In privacy-sensitive environments, use <code>--local-vision</code> to process images on-device. See the
+            In privacy-sensitive environments, use your own local version of
+            <a href="https://github.com/huggingface/speech-to-speech" target="_blank" rel="noopener">speech-to-speech</a>,
+            or combine any backend with <code>--local-vision</code> to process images on-device. See the
             <a href="https://github.com/pollen-robotics/reachy_mini_conversation_app/blob/main/README.md" target="_blank" rel="noopener">
               project README
             </a>
-            for full setup instructions. An open-source approach is in the works – stay tuned for its release.
+            for full setup instructions.
           </p>
         </details>
       </aside>
@@ -55,17 +57,24 @@
           <span id="backend-chip" class="chip">Required</span>
         </div>
         <p id="backend-note" class="muted backend-note">
-          OpenAI Realtime uses the distributed OpenAI key. Gemini Live needs your own <code>GEMINI_API_KEY</code>.
+          OpenAI Realtime needs your own <code>OPENAI_API_KEY</code>. Gemini Live needs your own <code>GEMINI_API_KEY</code>. Hugging Face uses either the built-in server or a direct realtime websocket endpoint.
         </p>
         <p class="muted backend-note">
           Backend changes are saved here, then applied after you restart Reachy Mini Conversation from the dashboard or desktop app.
         </p>
         <div class="backend-choice-grid" role="radiogroup" aria-label="Conversation backend">
+          <label class="backend-choice" data-backend-card="huggingface">
+            <input type="radio" name="backend" value="huggingface" />
+            <span class="backend-choice-body">
+              <span class="backend-choice-title">Hugging Face</span>
+              <span class="backend-choice-copy">Use the built-in Hugging Face server or your own local endpoint.</span>
+            </span>
+          </label>
           <label class="backend-choice is-selected" data-backend-card="openai">
             <input type="radio" name="backend" value="openai" checked />
             <span class="backend-choice-body">
               <span class="backend-choice-title">OpenAI Realtime</span>
-              <span class="backend-choice-copy">Start quickly with the bundled OpenAI access.</span>
+              <span class="backend-choice-copy">Bring your own <code>OPENAI_API_KEY</code> to use OpenAI Realtime.</span>
             </span>
           </label>
           <label class="backend-choice" data-backend-card="gemini">
@@ -103,8 +112,43 @@
           <span class="chip">Required</span>
         </div>
         <p id="form-copy" class="muted">Paste your API key once and we will store it locally for the headless conversation loop.</p>
-        <label id="api-key-label" for="api-key">OpenAI API Key</label>
-        <input id="api-key" type="password" placeholder="sk-..." autocomplete="off" />
+        <div id="api-key-fields">
+          <label id="api-key-label" for="api-key">OpenAI API Key</label>
+          <input id="api-key" type="password" placeholder="sk-..." autocomplete="off" />
+        </div>
+        <div id="hf-fields" class="hidden">
+          <label for="hf-mode">Connection mode</label>
+          <select id="hf-mode">
+            <option value="local">Local</option>
+            <option value="deployed">Hugging Face server</option>
+          </select>
+
+          <div id="hf-direct-fields" class="hf-config-grid">
+            <p class="muted hf-help hf-local-note">
+              Use your own Hugging Face backend powered by
+              <a href="https://github.com/huggingface/speech-to-speech" target="_blank" rel="noopener">speech-to-speech</a>.
+              Run it either on the same machine, locally on your network, or hosted on Hugging Face.
+            </p>
+            <div class="hf-field">
+              <label for="hf-host-preset">Host</label>
+              <select id="hf-host-preset">
+                <option value="localhost">localhost</option>
+                <option value="custom">Custom IP or hostname</option>
+              </select>
+            </div>
+            <div class="hf-field">
+              <label for="hf-port">Port</label>
+              <input id="hf-port" type="text" inputmode="numeric" value="8765" placeholder="8765" autocomplete="off" />
+            </div>
+          </div>
+
+          <div id="hf-host-custom-wrap" class="hidden">
+            <label for="hf-host-custom">Custom host or IP</label>
+            <input id="hf-host-custom" type="text" placeholder="192.168.1.20" autocomplete="off" />
+          </div>
+
+          <p id="hf-preview" class="status"></p>
+        </div>
         <div class="actions">
           <button id="save-btn">Save key</button>
           <p id="status" class="status" role="status" aria-live="polite" aria-atomic="true"></p>

src/reachy_mini_conversation_app/static/main.js

@@ -1,5 +1,9 @@
 const OPENAI_BACKEND = "openai";
 const GEMINI_BACKEND = "gemini";
+const HF_BACKEND = "huggingface";
+const DEFAULT_BACKEND = HF_BACKEND;
+const HF_DEFAULT_HOST = "localhost";
+const HF_DEFAULT_PORT = 8765;
 const BACKEND_META = {
   [OPENAI_BACKEND]: {
     label: "OpenAI Realtime",
@@ -9,10 +13,10 @@ const BACKEND_META = {
     saveButton: "Save key",
     changeButton: "Change OpenAI key",
     readyTitle: "OpenAI Realtime ready",
-    readyCopy: "OpenAI Realtime is configured. You can jump straight to personalities.",
-    formCopy: "OpenAI Realtime uses the distributed key when available. Paste your own key if you want an override or need a fallback.",
-    requiredCredentialsCopy: "OpenAI Realtime usually uses the distributed key. If it is unavailable here, paste your own OpenAI key to continue.",
-    note: "OpenAI Realtime uses the distributed OpenAI key. You can still paste your own key if you want to override it.",
+    readyCopy: "OpenAI Realtime is configured. Your saved OpenAI key is ready to use.",
+    formCopy: "Paste your OPENAI_API_KEY once and we will store it locally for the headless conversation loop.",
+    requiredCredentialsCopy: "OpenAI Realtime requires your own OPENAI_API_KEY before you can switch.",
+    note: "OpenAI Realtime requires your own OPENAI_API_KEY.",
   },
   [GEMINI_BACKEND]: {
     label: "Gemini Live",
@@ -25,12 +29,27 @@ const BACKEND_META = {
     readyCopy: "Gemini Live is configured. Your saved Gemini token is ready to use.",
     formCopy: "Paste your GEMINI_API_KEY once and we will store it locally for the headless conversation loop.",
     requiredCredentialsCopy: "Gemini Live requires your own GEMINI_API_KEY before you can switch.",
-    note: "OpenAI Realtime uses the distributed OpenAI key. Gemini Live needs your own GEMINI_API_KEY.",
+    note: "OpenAI Realtime requires OPENAI_API_KEY. Gemini Live needs GEMINI_API_KEY.",
+  },
+  [HF_BACKEND]: {
+    label: "Hugging Face",
+    formTitle: "Configure Hugging Face",
+    inputLabel: "",
+    placeholder: "",
+    saveButton: "Save connection",
+    changeButton: "Edit connection",
+    readyTitle: "Hugging Face ready",
+    readyCopy: "Hugging Face is configured. You can jump straight to personalities.",
+    formCopy: "Choose where Reachy should connect for Hugging Face.",
+    requiredCredentialsCopy: "Set up the Hugging Face connection details before switching.",
+    note: "Hugging Face can use the built-in server or your own local realtime websocket.",
   },
 };
 
 function backendHasCredentials(status, backend) {
-  return backend === GEMINI_BACKEND ? !!status.has_gemini_key : !!status.has_openai_key;
+  if (backend === GEMINI_BACKEND) return !!status.has_gemini_key;
+  if (backend === HF_BACKEND) return !!(status.has_hf_connection ?? (status.has_hf_session_url || status.has_hf_ws_url));
+  return !!status.has_openai_key;
 }
 
 function backendCanProceed(status, backend) {
@@ -39,17 +58,24 @@ function backendCanProceed(status, backend) {
       ? !!status.can_proceed_with_gemini
       : backendHasCredentials(status, backend);
   }
+  if (backend === HF_BACKEND) {
+    return status.can_proceed_with_hf !== undefined
+      ? !!status.can_proceed_with_hf
+      : backendHasCredentials(status, backend);
+  }
   return status.can_proceed_with_openai !== undefined
     ? !!status.can_proceed_with_openai
     : backendHasCredentials(status, backend);
 }
 
 function backendMeta(backend) {
-  return BACKEND_META[backend] || BACKEND_META[OPENAI_BACKEND];
+  return BACKEND_META[backend] || BACKEND_META[DEFAULT_BACKEND];
 }
 
 function formatBackendNote(text) {
-  return text.replace("GEMINI_API_KEY", "<code>GEMINI_API_KEY</code>");
+  return text
+    .replace("GEMINI_API_KEY", "<code>GEMINI_API_KEY</code>")
+    .replace("HF_REALTIME_WS_URL", "<code>HF_REALTIME_WS_URL</code>");
 }
 
 const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
@@ -113,8 +139,13 @@ async function validateKey(key) {
   return data;
 }
 
-async function saveBackendConfig(backend, key = "") {
+async function saveBackendConfig(backend, { key = "", hfMode = "", hfHost = "", hfPort = null } = {}) {
   const body = { backend, api_key: key };
+  if (backend === HF_BACKEND) {
+    if (hfMode) body.hf_mode = hfMode;
+    if (hfHost) body.hf_host = hfHost;
+    if (hfPort !== null && hfPort !== undefined) body.hf_port = hfPort;
+  }
   const resp = await fetch("/backend_config", {
     method: "POST",
     headers: { "Content-Type": "application/json" },
@@ -246,6 +277,22 @@ function setStatusMessage(el, text, tone = "") {
   el.setAttribute("aria-atomic", "true");
 }
 
+function describeHFConfiguration(status) {
+  if (status.hf_connection_mode === "local") {
+    const host = status.hf_direct_host || HF_DEFAULT_HOST;
+    const port = status.hf_direct_port || HF_DEFAULT_PORT;
+    return `Hugging Face will connect directly to ${host}:${port}.`;
+  }
+  if (status.has_hf_session_url) {
+    return "Hugging Face will use the built-in server.";
+  }
+  return "Choose the Hugging Face server or a local realtime endpoint.";
+}
+
+function isLocalHFHost(host) {
+  return !host || host === "localhost" || host === "127.0.0.1";
+}
+
 async function init() {
   const loading = document.getElementById("loading");
   show(loading, true);
@@ -263,10 +310,19 @@ async function init() {
   const personalityPanel = document.getElementById("personality-panel");
   const formTitle = document.getElementById("form-title");
   const formCopy = document.getElementById("form-copy");
+  const apiKeyFields = document.getElementById("api-key-fields");
   const apiKeyLabel = document.getElementById("api-key-label");
   const saveBtn = document.getElementById("save-btn");
   const changeKeyBtn = document.getElementById("change-key-btn");
   const input = document.getElementById("api-key");
+  const hfFields = document.getElementById("hf-fields");
+  const hfMode = document.getElementById("hf-mode");
+  const hfDirectFields = document.getElementById("hf-direct-fields");
+  const hfHostPreset = document.getElementById("hf-host-preset");
+  const hfHostCustomWrap = document.getElementById("hf-host-custom-wrap");
+  const hfHostCustom = document.getElementById("hf-host-custom");
+  const hfPort = document.getElementById("hf-port");
+  const hfPreview = document.getElementById("hf-preview");
 
   // Personality elements
   const pSelect = document.getElementById("personality-select");
@@ -287,11 +343,51 @@ async function init() {
     dance: ["stop_dance"],
     play_emotion: ["stop_emotion"],
   };
-  let selectedBackend = OPENAI_BACKEND;
+  let selectedBackend = DEFAULT_BACKEND;
   let editingCredentials = false;
 
+  function resolveHFHost() {
+    return hfHostPreset.value === "custom" ? hfHostCustom.value.trim() : HF_DEFAULT_HOST;
+  }
+
+  function updateHFControls() {
+    const localMode = hfMode.value !== "deployed";
+    const customHost = hfHostPreset.value === "custom";
+    show(hfDirectFields, localMode);
+    show(hfHostCustomWrap, localMode && customHost);
+
+    if (!localMode) {
+      setStatusMessage(hfPreview, "Hugging Face will use the built-in server.");
+      return;
+    }
+
+    const host = resolveHFHost() || "<host>";
+    const port = (hfPort.value || String(HF_DEFAULT_PORT)).trim();
+    setStatusMessage(hfPreview, `Will save ws://${host}:${port}/v1/realtime`);
+  }
+
+  function populateHFFields(status) {
+    const mode = status.hf_connection_mode
+      || (status.has_hf_session_url ? "deployed" : "local");
+    const existingHost = status.hf_direct_host || HF_DEFAULT_HOST;
+    const existingPort = status.hf_direct_port || HF_DEFAULT_PORT;
+
+    hfMode.value = mode;
+    if (isLocalHFHost(existingHost)) {
+      hfHostPreset.value = "localhost";
+      hfHostCustom.value = "";
+    } else {
+      hfHostPreset.value = "custom";
+      hfHostCustom.value = existingHost;
+    }
+    hfPort.value = String(existingPort);
+    updateHFControls();
+  }
+
   function setSelectedBackend(backend) {
-    selectedBackend = backend === GEMINI_BACKEND ? GEMINI_BACKEND : OPENAI_BACKEND;
+    selectedBackend = [OPENAI_BACKEND, GEMINI_BACKEND, HF_BACKEND].includes(backend)
+      ? backend
+      : DEFAULT_BACKEND;
     backendInputs.forEach((radio) => {
       radio.checked = radio.value === selectedBackend;
     });
@@ -301,31 +397,42 @@ async function init() {
   }
 
   function renderCredentialPanels(status) {
-    const persistedBackend = status.backend_provider || OPENAI_BACKEND;
+    const persistedBackend = status.backend_provider || DEFAULT_BACKEND;
     const activeBackend = status.active_backend || persistedBackend;
     const requiresRestart = !!status.requires_restart;
     const meta = backendMeta(selectedBackend);
     const canProceedWithSelectedBackend = backendCanProceed(status, selectedBackend);
     const selectedMatchesPersisted = selectedBackend === persistedBackend;
     const selectedMatchesActive = selectedBackend === activeBackend;
+    const usesApiKeyForm = selectedBackend === OPENAI_BACKEND || selectedBackend === GEMINI_BACKEND;
+    const usesHFForm = selectedBackend === HF_BACKEND;
+    const supportsForm = usesApiKeyForm || usesHFForm;
 
     backendChip.textContent = selectedBackend === persistedBackend ? "Saved" : "Selected";
     backendNote.innerHTML = formatBackendNote(meta.note);
 
     configuredTitle.textContent = meta.readyTitle;
-    configuredCopy.textContent = meta.readyCopy;
+    configuredCopy.textContent = usesHFForm ? describeHFConfiguration(status) : meta.readyCopy;
     formTitle.textContent = meta.formTitle;
-    formCopy.textContent = canProceedWithSelectedBackend ? meta.formCopy : meta.requiredCredentialsCopy;
+    formCopy.textContent = usesHFForm
+      ? meta.formCopy
+      : canProceedWithSelectedBackend
+        ? meta.formCopy
+        : meta.requiredCredentialsCopy;
     apiKeyLabel.textContent = meta.inputLabel;
     input.placeholder = meta.placeholder;
     saveBtn.textContent = meta.saveButton;
     changeKeyBtn.textContent = meta.changeButton;
 
     show(configuredPanel, canProceedWithSelectedBackend && !editingCredentials);
-    show(formPanel, editingCredentials || !canProceedWithSelectedBackend);
+    show(formPanel, supportsForm && (editingCredentials || !canProceedWithSelectedBackend));
+    show(apiKeyFields, usesApiKeyForm);
+    show(hfFields, usesHFForm);
+    if (usesHFForm) updateHFControls();
+    show(changeKeyBtn, supportsForm && canProceedWithSelectedBackend && !editingCredentials);
     show(
       backendSaveBtn,
-      canProceedWithSelectedBackend && !selectedMatchesPersisted,
+      canProceedWithSelectedBackend && !selectedMatchesPersisted && !editingCredentials,
     );
     backendSaveBtn.textContent = `Use ${meta.label}`;
 
@@ -356,17 +463,25 @@ async function init() {
   show(personalityPanel, false);
 
   const st = (await waitForStatus()) || {
-    active_backend: OPENAI_BACKEND,
-    backend_provider: OPENAI_BACKEND,
+    active_backend: DEFAULT_BACKEND,
+    backend_provider: DEFAULT_BACKEND,
     has_key: false,
     has_openai_key: false,
     has_gemini_key: false,
+    has_hf_session_url: false,
+    has_hf_ws_url: false,
+    has_hf_connection: false,
+    hf_connection_mode: "local",
+    hf_direct_host: HF_DEFAULT_HOST,
+    hf_direct_port: HF_DEFAULT_PORT,
     can_proceed: false,
     can_proceed_with_openai: false,
     can_proceed_with_gemini: false,
+    can_proceed_with_hf: false,
     requires_restart: false,
   };
-  setSelectedBackend(st.backend_provider || OPENAI_BACKEND);
+  populateHFFields(st);
+  setSelectedBackend(st.backend_provider || DEFAULT_BACKEND);
   statusEl.textContent = "";
   renderCredentialPanels(st);
 
@@ -382,6 +497,23 @@ async function init() {
   input.addEventListener("input", () => {
     input.classList.remove("error");
   });
+  hfHostCustom.addEventListener("input", () => {
+    hfHostCustom.classList.remove("error");
+    updateHFControls();
+  });
+  hfPort.addEventListener("input", () => {
+    hfPort.classList.remove("error");
+    updateHFControls();
+  });
+  hfMode.addEventListener("change", () => {
+    hfHostCustom.classList.remove("error");
+    hfPort.classList.remove("error");
+    updateHFControls();
+  });
+  hfHostPreset.addEventListener("change", () => {
+    hfHostCustom.classList.remove("error");
+    updateHFControls();
+  });
 
   backendInputs.forEach((radio) => {
     radio.addEventListener("change", () => {
@@ -404,6 +536,59 @@ async function init() {
   });
 
   saveBtn.addEventListener("click", async () => {
+    if (selectedBackend === HF_BACKEND) {
+      const localMode = hfMode.value !== "deployed";
+      setStatusMessage(statusEl, "Saving connection...");
+      hfHostCustom.classList.remove("error");
+      hfPort.classList.remove("error");
+
+      try {
+        if (localMode) {
+          const host = resolveHFHost();
+          const port = Number.parseInt((hfPort.value || "").trim(), 10);
+          if (!host) {
+            hfHostCustom.classList.add("error");
+            setStatusMessage(statusEl, "Enter a valid host or IP address.", "warn");
+            return;
+          }
+          if (!Number.isInteger(port) || port < 1 || port > 65535) {
+            hfPort.classList.add("error");
+            setStatusMessage(statusEl, "Enter a valid port between 1 and 65535.", "warn");
+            return;
+          }
+
+          await saveBackendConfig(selectedBackend, {
+            hfMode: "local",
+            hfHost: host,
+            hfPort: port,
+          });
+        } else {
+          await saveBackendConfig(selectedBackend, {
+            hfMode: "deployed",
+          });
+        }
+        setStatusMessage(statusEl, "Saved. Reloading…", "ok");
+        window.location.reload();
+      } catch (e) {
+        if (e.message === "missing_hf_session_url") {
+          setStatusMessage(
+            statusEl,
+            "The built-in Hugging Face server URL is unavailable. Restart the app and try again.",
+            "error",
+          );
+        } else if (e.message === "empty_hf_host" || e.message === "invalid_hf_host") {
+          hfHostCustom.classList.add("error");
+          setStatusMessage(statusEl, "Enter a valid host or IP address.", "error");
+        } else if (e.message === "invalid_hf_port") {
+          hfPort.classList.add("error");
+          setStatusMessage(statusEl, "Enter a valid port between 1 and 65535.", "error");
+        } else {
+          setStatusMessage(statusEl, "Failed to save the Hugging Face connection.", "error");
+        }
+      }
+      return;
+    }
+
     const key = input.value.trim();
     if (!key) {
       setStatusMessage(statusEl, "Please enter a valid key.", "warn");
@@ -424,7 +609,7 @@ async function init() {
       } else {
         setStatusMessage(statusEl, "Saving Gemini token...", "ok");
       }
-      await saveBackendConfig(selectedBackend, key);
+      await saveBackendConfig(selectedBackend, { key });
       setStatusMessage(statusEl, "Saved. Reloading…", "ok");
       window.location.reload();
     } catch (e) {
@@ -443,7 +628,7 @@ async function init() {
     }
   });
 
-  if (!(st.can_proceed ?? backendCanProceed(st, st.backend_provider || OPENAI_BACKEND)) || st.requires_restart) {
+  if (!(st.can_proceed ?? backendCanProceed(st, st.backend_provider || DEFAULT_BACKEND)) || st.requires_restart) {
     show(loading, false);
     return;
   }

src/reachy_mini_conversation_app/static/style.css

@@ -274,7 +274,7 @@ body {
   border-color: rgba(76, 224, 179, 0.4);
 }
 
-.hidden { display: none; }
+.hidden { display: none !important; }
 label {
   display: block;
   margin: 8px 0 6px;
@@ -350,6 +350,28 @@ button.ghost:hover { border-color: rgba(94, 240, 193, 0.4); }
 .status.ok { color: var(--ok); }
 .status.warn { color: var(--warn); }
 .status.error { color: var(--error); }
+.hf-help {
+  margin: 8px 0 0;
+  font-size: 13px;
+  line-height: 1.5;
+}
+.hf-help a {
+  color: var(--accent);
+  text-underline-offset: 2px;
+}
+.hf-config-grid {
+  display: grid;
+  grid-template-columns: repeat(2, minmax(0, 1fr));
+  gap: 12px;
+  margin-top: 10px;
+}
+.hf-local-note {
+  grid-column: 1 / -1;
+  margin: 0;
+}
+.hf-field {
+  min-width: 0;
+}
 
 /* Personality layout */
 .row {
@@ -439,6 +461,7 @@ button.ghost:hover { border-color: rgba(94, 240, 193, 0.4); }
   .hero h1 { font-size: 26px; }
   .privacy-notice { padding: 10px 12px; }
   .backend-choice-grid { grid-template-columns: 1fr; }
+  .hf-config-grid { grid-template-columns: 1fr; }
   .row { grid-template-columns: 1fr; }
   #personality-panel .row:first-of-type { grid-template-columns: 1fr; }
   button { width: 100%; justify-content: center; }

src/reachy_mini_conversation_app/tools/core_tools.py

@@ -27,14 +27,6 @@ if TYPE_CHECKING:
 logger = logging.getLogger(__name__)
 
 
-if not logger.handlers:
-    handler = logging.StreamHandler()
-    formatter = logging.Formatter("%(asctime)s %(levelname)s %(name)s:%(lineno)d | %(message)s")
-    handler.setFormatter(formatter)
-    logger.addHandler(handler)
-    logger.setLevel(logging.INFO)
-
-
 ALL_TOOLS: Dict[str, "Tool"] = {}
 ALL_TOOL_SPECS: List[Dict[str, Any]] = []
 _TOOLS_INITIALIZED = False
@@ -290,6 +282,14 @@ def get_tool_specs(exclusion_list: list[str] = []) -> list[Dict[str, Any]]:
     return [spec for spec in ALL_TOOL_SPECS if spec.get("name") not in exclusion_list]
 
 
+def get_active_tool_specs(deps: ToolDependencies) -> list[Dict[str, Any]]:
+    """Get tool specs filtered by what the current session deps support."""
+    exclusion_list: list[str] = []
+    if not (deps.camera_worker and deps.camera_worker.head_tracker):
+        exclusion_list.append("head_tracking")
+    return get_tool_specs(exclusion_list)
+
+
 # Dispatcher
 def _safe_load_obj(args_json: str) -> Dict[str, Any]:
     try:

src/reachy_mini_conversation_app/tools/dance.py

@@ -1,3 +1,4 @@
+import random
 import logging
 from typing import Any, Dict
 
@@ -18,6 +19,21 @@ except ImportError as e:
     DANCE_AVAILABLE = False
 
 
+def get_available_dances_and_descriptions() -> str:
+    """Get formatted list of available dances with descriptions."""
+    if not DANCE_AVAILABLE:
+        return "Moves not available."
+
+    if not AVAILABLE_MOVES:  # if AVAILABLE_MOVES is empty
+        return "Moves not available."
+
+    output = ""
+    for move_name, (func, params, metadata) in AVAILABLE_MOVES.items():
+        description = metadata.get("description", "No description available.")
+        output += f"{move_name}: {description}\n"
+    return output
+
+
 class Dance(Tool):
     """Play a named or random dance move once (or repeat). Non-blocking."""
 
@@ -28,28 +44,11 @@ class Dance(Tool):
         "properties": {
             "move": {
                 "type": "string",
-                "description": """Name of the move; use 'random' or omit for random.
-                                    Here is a list of the available moves:
-                                        simple_nod: A simple, continuous up-and-down nodding motion.
-                                        head_tilt_roll: A continuous side-to-side head roll (ear to shoulder).
-                                        side_to_side_sway: A smooth, side-to-side sway of the entire head.
-                                        dizzy_spin: A circular 'dizzy' head motion combining roll and pitch.
-                                        stumble_and_recover: A simulated stumble and recovery with multiple axis movements. Good vibes
-                                        interwoven_spirals: A complex spiral motion using three axes at different frequencies.
-                                        sharp_side_tilt: A sharp, quick side-to-side tilt using a triangle waveform.
-                                        side_peekaboo: A multi-stage peekaboo performance, hiding and peeking to each side.
-                                        yeah_nod: An emphatic two-part yeah nod using transient motions.
-                                        uh_huh_tilt: A combined roll-and-pitch uh-huh gesture of agreement.
-                                        neck_recoil: A quick, transient backward recoil of the neck.
-                                        chin_lead: A forward motion led by the chin, combining translation and pitch.
-                                        groovy_sway_and_roll: A side-to-side sway combined with a corresponding roll for a groovy effect.
-                                        chicken_peck: A sharp, forward, chicken-like pecking motion.
-                                        side_glance_flick: A quick glance to the side that holds, then returns.
-                                        polyrhythm_combo: A 3-beat sway and a 2-beat nod create a polyrhythmic feel.
-                                        grid_snap: A robotic, grid-snapping motion using square waveforms.
-                                        pendulum_swing: A simple, smooth pendulum-like swing using a roll motion.
-                                        jackson_square: Traces a rectangle via a 5-point path, with sharp twitches on arrival at each checkpoint.
-                """,
+                "enum": list(AVAILABLE_MOVES.keys() if DANCE_AVAILABLE else []),
+                "description": f"""Name of the moves and their descriptions; omit for random.
+                                Here is a list of the available moves, you MUST only choose from these: \n
+                                {get_available_dances_and_descriptions()}
+                                """,
             },
             "repeat": {
                 "type": "integer",
@@ -64,14 +63,15 @@ class Dance(Tool):
         if not DANCE_AVAILABLE:
             return {"error": "Dance system not available"}
 
+        if not AVAILABLE_MOVES:  # if AVAILABLE_MOVES is empty
+            return {"error": "No moves currently available"}
+
         move_name = kwargs.get("move")
         repeat = int(kwargs.get("repeat", 1))
 
         logger.info("Tool call: dance move=%s repeat=%d", move_name, repeat)
 
-        if not move_name or move_name == "random":
-            import random
-
+        if not move_name:
             move_name = random.choice(list(AVAILABLE_MOVES.keys()))
 
         if move_name not in AVAILABLE_MOVES:

src/reachy_mini_conversation_app/tools/{do_nothing.py → idle_do_nothing.py}

@@ -7,24 +7,27 @@ from reachy_mini_conversation_app.tools.core_tools import Tool, ToolDependencies
 logger = logging.getLogger(__name__)
 
 
-class DoNothing(Tool):
-    """Choose to do nothing - stay still and silent. Use when you want to be contemplative or just chill."""
+class IdleDoNothing(Tool):
+    """Explicitly choose no action during an idle turn."""
 
-    name = "do_nothing"
-    description = "Choose to do nothing - stay still and silent. Use when you want to be contemplative or just chill."
+    name = "idle_do_nothing"
+    description = (
+        "Use only in response to an idle time update when you intentionally want Reachy to stay still and silent "
+        "instead of choosing another idle action."
+    )
     parameters_schema = {
         "type": "object",
         "properties": {
             "reason": {
                 "type": "string",
-                "description": "Optional reason for doing nothing (e.g., 'contemplating existence', 'saving energy', 'being mysterious')",
+                "description": "Optional reason for staying idle during this idle turn.",
             },
         },
         "required": [],
     }
 
     async def __call__(self, deps: ToolDependencies, **kwargs: Any) -> Dict[str, Any]:
-        """Do nothing - stay still and silent."""
-        reason = kwargs.get("reason", "just chilling")
-        logger.info("Tool call: do_nothing reason=%s", reason)
-        return {"status": "doing nothing", "reason": reason}
+        """Stay still and silent for the current idle turn."""
+        reason = kwargs.get("reason", "idle turn")
+        logger.info("Tool call: idle_do_nothing reason=%s", reason)
+        return {"status": "idle", "reason": reason}

src/reachy_mini_conversation_app/tools/play_emotion.py

@@ -1,3 +1,4 @@
+import random
 import logging
 from typing import Any, Dict
 
@@ -27,6 +28,9 @@ def get_available_emotions_and_descriptions() -> str:
 
     try:
         emotion_names = RECORDED_MOVES.list_moves()
+        if not emotion_names:
+            return "No emotions currently available"
+
         output = "Available emotions:\n"
         for name in emotion_names:
             description = RECORDED_MOVES.get(name).description
@@ -46,13 +50,14 @@ class PlayEmotion(Tool):
         "properties": {
             "emotion": {
                 "type": "string",
-                "description": f"""Name of the emotion to play.
-                                    Here is a list of the available emotions:
+                "enum": list(RECORDED_MOVES.list_moves()) if EMOTION_AVAILABLE else [],
+                "description": f"""Name of the emotion to play; omit for random.
+                                    Here is a list of the available emotions, you MUST only choose from these: \n
                                     {get_available_emotions_and_descriptions()}
                                     """,
             },
         },
-        "required": ["emotion"],
+        "required": [],
     }
 
     async def __call__(self, deps: ToolDependencies, **kwargs: Any) -> Dict[str, Any]:
@@ -61,14 +66,18 @@ class PlayEmotion(Tool):
             return {"error": "Emotion system not available"}
 
         emotion_name = kwargs.get("emotion")
-        if not emotion_name:
-            return {"error": "Emotion name is required"}
 
         logger.info("Tool call: play_emotion emotion=%s", emotion_name)
 
         # Check if emotion exists
         try:
             emotion_names = RECORDED_MOVES.list_moves()
+            if not emotion_names:
+                return {"error": "No emotions currently available"}
+
+            if not emotion_name:
+                emotion_name = random.choice(emotion_names)
+
             if emotion_name not in emotion_names:
                 return {"error": f"Unknown emotion '{emotion_name}'. Available: {emotion_names}"}
 

src/reachy_mini_conversation_app/vision/head_tracking/yolo.py

@@ -1,5 +1,7 @@
 from __future__ import annotations
 import logging
+from typing import Protocol, cast
+from collections.abc import Sequence
 
 import numpy as np
 from numpy.typing import NDArray
@@ -9,7 +11,7 @@ from reachy_mini_conversation_app.vision.head_tracking import HeadTrackerResult
 
 try:
     from supervision import Detections
-    from ultralytics import YOLO  # type: ignore
+    from ultralytics import YOLO  # type: ignore[attr-defined]
 except ImportError as e:
     raise ImportError(
         "To use YOLO head tracker, please install the extra dependencies: pip install '.[yolo_vision]'",
@@ -20,6 +22,14 @@ from huggingface_hub import hf_hub_download
 logger = logging.getLogger(__name__)
 
 
+class _YoloModel(Protocol):
+    """Minimal YOLO model interface used by the head tracker."""
+
+    def __call__(self, source: NDArray[np.uint8], **kwargs: object) -> Sequence[object]: ...
+
+    def to(self, device: str) -> _YoloModel: ...
+
+
 class YoloHeadTracker:
     """Lightweight head tracker using YOLO for face detection."""
 
@@ -35,7 +45,7 @@ class YoloHeadTracker:
 
         try:
             model_path = hf_hub_download(repo_id=model_repo, filename=model_filename)
-            self.model = YOLO(model_path).to(device)
+            self.model = cast(_YoloModel, YOLO(model_path).to(device))
             logger.info("YOLO face detection model loaded from %s", model_repo)
         except Exception as e:
             logger.error("Failed to load YOLO model: %s", e)

src/reachy_mini_conversation_app/vision/head_tracking/yolo_process.py

@@ -137,6 +137,7 @@ class YoloHeadTrackerProcess:
         self._messages: queue.Queue[tuple[str, object | None]] = queue.Queue()
         self._next_request_id = 0
         self._timed_out_request_id: int | None = None
+        self._recovery_call_pending = False
         self._tracker_name = "yolo"
 
         module_path = "reachy_mini_conversation_app.vision.head_tracking.yolo_process"
@@ -304,8 +305,14 @@ class YoloHeadTrackerProcess:
         request_id: int | None = None
         try:
             with self._send_lock:
-                # Avoid writing another frame while the child is still finishing a timed-out one.
-                if not self._drain_timed_out_reply():
+                # Reserve the immediate next call after a timeout for recovery
+                # only. Later calls may drain a delayed reply and continue with
+                # a fresh request in the same turn.
+                if self._recovery_call_pending:
+                    self._recovery_call_pending = False
+                    self._drain_timed_out_reply()
+                    return None, None
+                if self._timed_out_request_id is not None and not self._drain_timed_out_reply():
                     return None, None
 
                 request_id = self._next_request_id
@@ -315,6 +322,7 @@ class YoloHeadTrackerProcess:
         except TimeoutError as exc:
             if request_id is not None:
                 self._timed_out_request_id = request_id
+                self._recovery_call_pending = True
             logger.warning("Head tracker %s communication timed out: %s", self._tracker_name, exc)
             return None, None
         except Exception as exc:

src/reachy_mini_conversation_app/vision/local_vision.py

@@ -1,13 +1,16 @@
+from __future__ import annotations
 import os
 import time
 import logging
+from typing import Any, Protocol, cast
 from dataclasses import dataclass
+from collections.abc import Mapping, Sequence
 
 import numpy as np
 import torch
 from PIL import Image
 from numpy.typing import NDArray
-from transformers import AutoProcessor, ProcessorMixin, AutoModelForImageTextToText
+from transformers import AutoProcessor, AutoModelForImageTextToText
 from huggingface_hub import snapshot_download
 
 from reachy_mini_conversation_app.config import config
@@ -23,6 +26,48 @@ LOCAL_VISION_RESPONSE_INSTRUCTIONS = (
 )
 
 
+class _VisionInputs(Mapping[str, object]):
+    """Tokenized processor inputs that can be moved to the inference device."""
+
+    def to(self, device: str) -> _VisionInputs:
+        """Move inputs to the selected inference device."""
+        raise NotImplementedError
+
+
+class _VisionTokenizer(Protocol):
+    """Tokenizer attributes used by generation."""
+
+    eos_token_id: int | None
+
+
+class _VisionProcessor(Protocol):
+    """Small interface required from the Hugging Face image-text processor."""
+
+    tokenizer: _VisionTokenizer
+
+    def apply_chat_template(
+        self,
+        conversation: object,
+        *,
+        add_generation_prompt: bool,
+        tokenize: bool,
+        return_dict: bool,
+        return_tensors: str,
+    ) -> _VisionInputs: ...
+
+    def batch_decode(self, sequences: object, *, skip_special_tokens: bool) -> list[str]: ...
+
+
+class _VisionModel(Protocol):
+    """Small interface required from the Hugging Face image-text model."""
+
+    def to(self, device: str) -> _VisionModel: ...
+
+    def eval(self) -> object: ...
+
+    def generate(self, **kwargs: object) -> Any: ...
+
+
 @dataclass
 class VisionConfig:
     """Configuration for vision processing."""
@@ -41,8 +86,8 @@ class VisionProcessor:
         """Initialize the vision processor."""
         self.vision_config = vision_config or VisionConfig()
         self.device = self._determine_device()
-        self.processor: ProcessorMixin | None = None
-        self.model: torch.nn.Module | None = None
+        self.processor: _VisionProcessor | None = None
+        self.model: _VisionModel | None = None
         self._initialized = False
 
     def _determine_device(self) -> str:
@@ -62,15 +107,21 @@ class VisionProcessor:
     def initialize(self) -> None:
         """Load model and processor onto the selected device."""
         logger.info("Loading SmolVLM2 model on %s (HF_HOME=%s)", self.device, config.HF_HOME)
-        processor: ProcessorMixin = AutoProcessor.from_pretrained(self.vision_config.model_path)  # type: ignore[no-untyped-call]
+        processor = cast(
+            _VisionProcessor,
+            AutoProcessor.from_pretrained(self.vision_config.model_path),  # type: ignore[no-untyped-call]
+        )
 
         model_kwargs: dict[str, object] = {
             "dtype": torch.bfloat16 if self.device == "cuda" else torch.float32,
         }
 
-        model: torch.nn.Module = AutoModelForImageTextToText.from_pretrained(
-            self.vision_config.model_path,
-            **model_kwargs,
+        model = cast(
+            _VisionModel,
+            AutoModelForImageTextToText.from_pretrained(
+                self.vision_config.model_path,
+                **model_kwargs,
+            ),
         )
         model = model.to(self.device)
 
@@ -112,25 +163,21 @@ class VisionProcessor:
         for attempt in range(self.vision_config.max_retries):
             try:
                 inputs = processor.apply_chat_template(
-                    messages,  # type: ignore[arg-type]
+                    messages,
                     add_generation_prompt=True,
                     tokenize=True,
                     return_dict=True,
                     return_tensors="pt",
                 )
-                inputs = inputs.to(self.device)  # type: ignore[attr-defined]
-                prompt_len = None
-                input_ids = inputs.get("input_ids")
-                input_shape = getattr(input_ids, "shape", None)
-                if input_shape:
-                    prompt_len = int(input_shape[-1])
+                inputs = inputs.to(self.device)
+                prompt_len = _last_shape_dim(inputs.get("input_ids"))
 
                 with torch.inference_mode():
-                    generated_ids = model.generate(  # type: ignore[operator]
+                    generated_ids = model.generate(
                         **inputs,
                         do_sample=False,
                         max_new_tokens=self.vision_config.max_new_tokens,
-                        pad_token_id=processor.tokenizer.eos_token_id,  # type: ignore[attr-defined]
+                        pad_token_id=processor.tokenizer.eos_token_id,
                     )
 
                 # Decode only the newly generated tokens, skipping the prompt
@@ -140,7 +187,7 @@ class VisionProcessor:
                     new_token_ids = generated_ids[:, prompt_len:]
                 else:
                     new_token_ids = [token_ids[prompt_len:] for token_ids in generated_ids]
-                response = processor.batch_decode(  # type: ignore[no-untyped-call]
+                response = processor.batch_decode(
                     new_token_ids,
                     skip_special_tokens=True,
                 )[0]
@@ -167,6 +214,14 @@ class VisionProcessor:
         return f"Vision processing error after {self.vision_config.max_retries} attempts"
 
 
+def _last_shape_dim(value: object) -> int | None:
+    """Return the last dimension from tensor-like objects that expose a shape."""
+    shape = getattr(value, "shape", None)
+    if not isinstance(shape, Sequence) or not shape:
+        return None
+    return int(shape[-1])
+
+
 def initialize_vision_processor() -> VisionProcessor:
     """Download the vision model and return an initialized VisionProcessor."""
     try:

tests/audio/test_head_wobbler.py

@@ -8,6 +8,7 @@ from typing import Any, List, Tuple
 from collections.abc import Callable
 
 import numpy as np
+from numpy.typing import NDArray
 
 from reachy_mini_conversation_app.audio.head_wobbler import HeadWobbler
 
@@ -18,7 +19,11 @@ def _make_audio_chunk(duration_s: float = 0.3, frequency_hz: float = 220.0) -> s
     return base64.b64encode(pcm.tobytes()).decode("ascii")
 
 
-def _make_pcm(duration_s: float = 0.3, frequency_hz: float = 220.0, sample_rate: int = 24000) -> np.ndarray:
+def _make_pcm(
+    duration_s: float = 0.3,
+    frequency_hz: float = 220.0,
+    sample_rate: int = 24000,
+) -> NDArray[np.int16]:
     """Generate a mono PCM16 sine wave at the requested sample rate."""
     sample_count = int(sample_rate * duration_s)
     t = np.linspace(0, duration_s, sample_count, endpoint=False)

tests/audio/test_startup_config.py

@@ -0,0 +1,95 @@
+"""Tests for Reachy Mini audio startup configuration."""
+
+from __future__ import annotations
+from types import SimpleNamespace
+
+from reachy_mini_conversation_app.audio.startup_config import (
+    AUDIO_STARTUP_CONFIG,
+    WRITE_SETTLE_SECONDS,
+    apply_audio_startup_config,
+)
+
+
+class FakeAudio:
+    """Fake SDK audio wrapper."""
+
+    def __init__(self, *, result: bool = True, error: Exception | None = None) -> None:
+        """Initialize the fake audio wrapper."""
+        self.result = result
+        self.error = error
+        self.calls: list[tuple[object, bool, float]] = []
+
+    def apply_audio_config(
+        self,
+        config: object,
+        *,
+        verify: bool = True,
+        write_settle_seconds: float = WRITE_SETTLE_SECONDS,
+    ) -> bool:
+        """Record SDK audio config calls."""
+        if self.error is not None:
+            raise self.error
+        self.calls.append((config, verify, write_settle_seconds))
+        return self.result
+
+
+def test_apply_audio_startup_config_uses_sdk_audio_config_api() -> None:
+    """Startup config should delegate writes and verification to the SDK audio API."""
+    audio = FakeAudio()
+    robot = SimpleNamespace(media=SimpleNamespace(audio=audio))
+
+    applied = apply_audio_startup_config(robot)
+
+    assert applied is True
+    assert audio.calls == [(AUDIO_STARTUP_CONFIG, True, WRITE_SETTLE_SECONDS)]
+
+
+def test_apply_audio_startup_config_forwards_sdk_options() -> None:
+    """SDK verification options should stay configurable for tests and callers."""
+    audio = FakeAudio()
+    robot = SimpleNamespace(media=SimpleNamespace(audio=audio))
+
+    applied = apply_audio_startup_config(robot, verify=False, write_settle_seconds=0)
+
+    assert applied is True
+    assert audio.calls == [(AUDIO_STARTUP_CONFIG, False, 0)]
+
+
+def test_apply_audio_startup_config_returns_false_without_audio() -> None:
+    """Startup should continue when the SDK audio object is unavailable."""
+    robot = SimpleNamespace(media=SimpleNamespace(audio=None))
+
+    applied = apply_audio_startup_config(robot)
+
+    assert applied is False
+
+
+def test_apply_audio_startup_config_returns_false_without_sdk_api() -> None:
+    """Startup should continue when the installed SDK does not expose audio config helpers."""
+    robot = SimpleNamespace(media=SimpleNamespace(audio=object()))
+
+    applied = apply_audio_startup_config(robot)
+
+    assert applied is False
+
+
+def test_apply_audio_startup_config_returns_false_when_sdk_returns_false() -> None:
+    """SDK application failures should be reported without raising."""
+    audio = FakeAudio(result=False)
+    robot = SimpleNamespace(media=SimpleNamespace(audio=audio))
+
+    applied = apply_audio_startup_config(robot)
+
+    assert applied is False
+    assert audio.calls == [(AUDIO_STARTUP_CONFIG, True, WRITE_SETTLE_SECONDS)]
+
+
+def test_apply_audio_startup_config_returns_false_when_sdk_raises() -> None:
+    """Unexpected SDK audio config errors should not prevent app startup."""
+    audio = FakeAudio(error=RuntimeError("audio board unavailable"))
+    robot = SimpleNamespace(media=SimpleNamespace(audio=audio))
+
+    applied = apply_audio_startup_config(robot)
+
+    assert applied is False
+    assert audio.calls == []

tests/test_config_name_collisions.py

@@ -59,3 +59,86 @@ def test_config_raises_when_selected_external_profile_is_missing(
 
     with pytest.raises(RuntimeError, match="Selected profile 'missing_profile' was not found"):
         config_mod.Config()
+
+
+def test_backend_provider_defaults_to_hf_when_unset() -> None:
+    """Non-Gemini models should default to the Hugging Face backend."""
+    assert config_mod._normalize_backend_provider(None, None) == config_mod.HF_BACKEND
+    assert config_mod._normalize_backend_provider("", None) == config_mod.HF_BACKEND
+    assert config_mod._normalize_backend_provider(None, "gpt-realtime") == config_mod.HF_BACKEND
+    assert config_mod._normalize_backend_provider(None, "gemini-3.1-flash-live-preview") == config_mod.GEMINI_BACKEND
+
+
+def test_backend_provider_rejects_explicit_unknown_backend() -> None:
+    """An explicit backend typo should fail instead of falling through to the default backend."""
+    with pytest.raises(ValueError, match="Invalid BACKEND_PROVIDER='openia'"):
+        config_mod._normalize_backend_provider("openia", None)
+
+
+def test_huggingface_backend_does_not_resolve_model_name() -> None:
+    """Hugging Face should rely on the server's model selection."""
+    assert config_mod._resolve_model_name(config_mod.HF_BACKEND, None) == ""
+    assert config_mod._resolve_model_name(config_mod.HF_BACKEND, "gpt-realtime") == ""
+
+
+def test_hf_default_session_url_uses_stable_space_proxy() -> None:
+    """The app should not embed the raw, replaceable Inference Endpoint allocator URL."""
+    assert config_mod.HF_DEFAULTS.session_url == "https://pollen-robotics-reachy-mini-realtime-url.hf.space/session"
+    assert ".aws.endpoints.huggingface.cloud" not in config_mod.HF_DEFAULTS.session_url
+
+
+def test_refresh_runtime_config_reloads_hf_runtime_fields(monkeypatch: pytest.MonkeyPatch) -> None:
+    """Instance-local .env reloads should update every env-backed Hugging Face runtime field."""
+    monkeypatch.setenv("HF_TOKEN", "hf-runtime-token")
+    monkeypatch.setenv("HF_HOME", "/tmp/reachy-hf-cache")
+    monkeypatch.setenv("LOCAL_VISION_MODEL", "test/local-vision-model")
+
+    monkeypatch.setattr(config_mod.config, "HF_TOKEN", None)
+    monkeypatch.setattr(config_mod.config, "HF_HOME", "./old-cache")
+    monkeypatch.setattr(config_mod.config, "LOCAL_VISION_MODEL", "old/model")
+
+    config_mod.refresh_runtime_config_from_env()
+
+    assert config_mod.config.HF_TOKEN == "hf-runtime-token"
+    assert config_mod.config.HF_HOME == "/tmp/reachy-hf-cache"
+    assert config_mod.config.LOCAL_VISION_MODEL == "test/local-vision-model"
+
+
+@pytest.mark.parametrize(
+    ("configured_mode", "session_url", "direct_ws_url", "expected_mode", "expected_has_target"),
+    [
+        ("local", "https://hf.example.test/session", None, "local", False),
+        ("deployed", "https://hf.example.test/session", "ws://127.0.0.1:8765/v1/realtime", "deployed", True),
+        ("local", None, "ws://127.0.0.1:8765/v1/realtime", "local", True),
+        ("deployed", None, "ws://127.0.0.1:8765/v1/realtime", "deployed", False),
+    ],
+)
+def test_hf_connection_selection_uses_explicit_mode_for_target(
+    monkeypatch: pytest.MonkeyPatch,
+    configured_mode: str | None,
+    session_url: str | None,
+    direct_ws_url: str | None,
+    expected_mode: str,
+    expected_has_target: bool,
+) -> None:
+    """Hugging Face selection should use the configured mode without inferring from URLs."""
+    monkeypatch.setattr(config_mod.config, "HF_REALTIME_CONNECTION_MODE", configured_mode)
+    monkeypatch.setattr(config_mod.config, "HF_REALTIME_SESSION_URL", session_url)
+    monkeypatch.setattr(config_mod.config, "HF_REALTIME_WS_URL", direct_ws_url)
+
+    selection = config_mod.get_hf_connection_selection()
+
+    assert selection.mode == expected_mode
+    assert selection.has_target is expected_has_target
+    assert selection.session_url == session_url
+    assert selection.direct_ws_url == direct_ws_url
+
+
+def test_hf_connection_selection_requires_mode(monkeypatch: pytest.MonkeyPatch) -> None:
+    """Hugging Face selection should fail instead of inferring a missing mode."""
+    monkeypatch.setattr(config_mod.config, "HF_REALTIME_CONNECTION_MODE", None)
+    monkeypatch.setattr(config_mod.config, "HF_REALTIME_SESSION_URL", "https://hf.example.test/session")
+    monkeypatch.setattr(config_mod.config, "HF_REALTIME_WS_URL", "ws://127.0.0.1:8765/v1/realtime")
+
+    with pytest.raises(RuntimeError, match="HF_REALTIME_CONNECTION_MODE must be set"):
+        config_mod.get_hf_connection_selection()

tests/test_console.py

@@ -1,18 +1,26 @@
 """Tests for the headless console stream."""
 
+import sys
 import asyncio
 import threading
 from types import SimpleNamespace
+from typing import Any
+from pathlib import Path
 from unittest.mock import AsyncMock, MagicMock
 
 import numpy as np
 import pytest
 from fastapi import FastAPI
+from numpy.typing import NDArray
 from fastapi.testclient import TestClient
 
 from reachy_mini.media.media_manager import MediaBackend
 from reachy_mini_conversation_app.config import GEMINI_AVAILABLE_VOICES, config
-from reachy_mini_conversation_app.console import LocalStream
+from reachy_mini_conversation_app.console import LOCAL_PLAYER_BACKEND, LocalStream
+from reachy_mini_conversation_app.startup_settings import (
+    StartupSettings,
+    load_startup_settings_into_runtime,
+)
 from reachy_mini_conversation_app.headless_personality_ui import mount_personality_routes
 
 
@@ -23,7 +31,7 @@ def test_clear_audio_queue_prefers_clear_player_when_available() -> None:
         clear_player=MagicMock(),
         clear_output_buffer=MagicMock(),
     )
-    robot = SimpleNamespace(media=SimpleNamespace(audio=audio, backend=MediaBackend.LOCAL))
+    robot = SimpleNamespace(media=SimpleNamespace(audio=audio, backend=LOCAL_PLAYER_BACKEND))
     stream = LocalStream(handler, robot)
 
     stream.clear_audio_queue()
@@ -75,10 +83,10 @@ async def test_play_loop_feeds_head_wobbler_with_local_playback_delay() -> None:
     class Handler:
         def __init__(self) -> None:
             self.deps = SimpleNamespace(head_wobbler=head_wobbler)
-            self.output_queue = asyncio.Queue()
+            self.output_queue: asyncio.Queue[Any] = asyncio.Queue()
             self._emitted = False
 
-        async def emit(self):
+        async def emit(self) -> tuple[int, NDArray[np.int16]] | None:
             if not self._emitted:
                 self._emitted = True
                 return (24000, chunk.copy())
@@ -90,7 +98,7 @@ async def test_play_loop_feeds_head_wobbler_with_local_playback_delay() -> None:
     )
     media = SimpleNamespace(
         audio=audio,
-        backend=MediaBackend.LOCAL,
+        backend=LOCAL_PLAYER_BACKEND,
         get_output_audio_samplerate=lambda: 24000,
         push_audio_sample=MagicMock(),
     )
@@ -117,8 +125,8 @@ async def test_play_loop_feeds_head_wobbler_with_local_playback_delay() -> None:
 
 
 def test_backend_config_persists_gemini_selection_and_status(
-    tmp_path,
-    monkeypatch,
+    tmp_path: Path,
+    monkeypatch: pytest.MonkeyPatch,
 ) -> None:
     """Settings API should persist Gemini backend choice and token."""
     monkeypatch.setattr(config, "BACKEND_PROVIDER", "openai")
@@ -171,8 +179,8 @@ def test_backend_config_persists_gemini_selection_and_status(
 
 
 def test_backend_config_preserves_explicit_model_override_when_saving_key(
-    tmp_path,
-    monkeypatch,
+    tmp_path: Path,
+    monkeypatch: pytest.MonkeyPatch,
 ) -> None:
     """Saving credentials should not reset a custom model override."""
     custom_model = "gpt-4o-realtime-preview-2025-06-03"
@@ -211,7 +219,219 @@ def test_backend_config_preserves_explicit_model_override_when_saving_key(
     assert "OPENAI_API_KEY=openai-test-key" in env_text
 
 
-def test_headless_personality_routes_return_gemini_voices_when_backend_selected(monkeypatch) -> None:
+def test_backend_config_persists_local_hf_selection_and_status(
+    tmp_path: Path,
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """Settings API should persist a direct Hugging Face websocket target."""
+    monkeypatch.setattr(config, "BACKEND_PROVIDER", "openai")
+    monkeypatch.setattr(config, "MODEL_NAME", "gpt-realtime")
+    monkeypatch.setattr(config, "HF_REALTIME_CONNECTION_MODE", "deployed")
+    monkeypatch.setattr(config, "HF_REALTIME_SESSION_URL", None)
+    monkeypatch.setattr(config, "HF_REALTIME_WS_URL", None)
+    monkeypatch.setenv("BACKEND_PROVIDER", "openai")
+    monkeypatch.setenv("MODEL_NAME", "gpt-realtime")
+    monkeypatch.delenv("HF_REALTIME_CONNECTION_MODE", raising=False)
+    monkeypatch.delenv("HF_REALTIME_SESSION_URL", raising=False)
+    monkeypatch.delenv("HF_REALTIME_WS_URL", raising=False)
+
+    app = FastAPI()
+    robot = SimpleNamespace(media=SimpleNamespace(audio=None, backend=None))
+    stream = LocalStream(MagicMock(), robot, settings_app=app, instance_path=str(tmp_path))
+    stream._init_settings_ui_if_needed()
+
+    client = TestClient(app)
+    response = client.post(
+        "/backend_config",
+        json={
+            "backend": "huggingface",
+            "hf_mode": "local",
+            "hf_host": "localhost",
+            "hf_port": 8765,
+        },
+    )
+
+    assert response.status_code == 200
+    data = response.json()
+    assert data["ok"] is True
+    assert data["backend_provider"] == "huggingface"
+    assert data["active_backend"] == "openai"
+    assert data["has_hf_ws_url"] is True
+    assert data["has_hf_connection"] is True
+    assert data["hf_connection_mode"] == "local"
+    assert data["hf_direct_host"] == "localhost"
+    assert data["hf_direct_port"] == 8765
+    assert data["requires_restart"] is True
+
+    env_text = (tmp_path / ".env").read_text(encoding="utf-8")
+    env_lines = env_text.splitlines()
+    assert "BACKEND_PROVIDER=huggingface" in env_text
+    assert "HF_REALTIME_CONNECTION_MODE=local" in env_text
+    assert "HF_REALTIME_WS_URL=ws://localhost:8765/v1/realtime" in env_text
+    assert not any(line.startswith("MODEL_NAME=") for line in env_lines)
+
+
+def test_backend_config_persists_deployed_mode_without_clearing_local_hf_ws_url(
+    tmp_path: Path,
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """Saving deployed mode should make env selection explicit and remove stale allocator URLs."""
+    env_path = tmp_path / ".env"
+    env_path.write_text(
+        "BACKEND_PROVIDER=huggingface\n"
+        "HF_REALTIME_SESSION_URL=https://lb.example.test/session\n"
+        "HF_REALTIME_WS_URL=ws://localhost:8765/v1/realtime\n",
+        encoding="utf-8",
+    )
+
+    monkeypatch.setattr(config, "BACKEND_PROVIDER", "huggingface")
+    monkeypatch.setattr(config, "MODEL_NAME", "gpt-realtime")
+    monkeypatch.setattr(config, "HF_REALTIME_CONNECTION_MODE", "deployed")
+    monkeypatch.setattr(config, "HF_REALTIME_SESSION_URL", "https://lb.example.test/session")
+    monkeypatch.setattr(config, "HF_REALTIME_WS_URL", "ws://localhost:8765/v1/realtime")
+    monkeypatch.setenv("BACKEND_PROVIDER", "huggingface")
+    monkeypatch.setenv("MODEL_NAME", "gpt-realtime")
+    monkeypatch.delenv("HF_REALTIME_CONNECTION_MODE", raising=False)
+    monkeypatch.setenv("HF_REALTIME_SESSION_URL", "https://lb.example.test/session")
+    monkeypatch.setenv("HF_REALTIME_WS_URL", "ws://localhost:8765/v1/realtime")
+
+    app = FastAPI()
+    robot = SimpleNamespace(media=SimpleNamespace(audio=None, backend=None))
+    stream = LocalStream(MagicMock(), robot, settings_app=app, instance_path=str(tmp_path))
+    stream._init_settings_ui_if_needed()
+
+    client = TestClient(app)
+    response = client.post(
+        "/backend_config",
+        json={
+            "backend": "huggingface",
+            "hf_mode": "deployed",
+        },
+    )
+
+    assert response.status_code == 200
+    data = response.json()
+    assert data["ok"] is True
+    assert data["has_hf_session_url"] is True
+    assert data["has_hf_ws_url"] is True
+    assert data["hf_connection_mode"] == "deployed"
+
+    env_text = env_path.read_text(encoding="utf-8")
+    assert "HF_REALTIME_CONNECTION_MODE=deployed" in env_text
+    assert "HF_REALTIME_SESSION_URL=" not in env_text
+    assert "HF_REALTIME_WS_URL=ws://localhost:8765/v1/realtime" in env_text
+
+
+def test_backend_config_switches_to_saved_local_hf_connection_without_payload_target(
+    tmp_path: Path,
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """Switching back to a saved local Hugging Face backend should reuse the persisted target."""
+    env_path = tmp_path / ".env"
+    env_path.write_text(
+        "BACKEND_PROVIDER=openai\n"
+        "MODEL_NAME=gpt-realtime\n"
+        "HF_REALTIME_CONNECTION_MODE=local\n"
+        "HF_REALTIME_WS_URL=ws://192.168.1.42:8766/v1/realtime\n",
+        encoding="utf-8",
+    )
+
+    monkeypatch.setattr(config, "BACKEND_PROVIDER", "openai")
+    monkeypatch.setattr(config, "MODEL_NAME", "gpt-realtime")
+    monkeypatch.setattr(config, "HF_REALTIME_CONNECTION_MODE", "local")
+    monkeypatch.setattr(config, "HF_REALTIME_SESSION_URL", None)
+    monkeypatch.setattr(config, "HF_REALTIME_WS_URL", "ws://192.168.1.42:8766/v1/realtime")
+    monkeypatch.setenv("BACKEND_PROVIDER", "openai")
+    monkeypatch.setenv("MODEL_NAME", "gpt-realtime")
+    monkeypatch.setenv("HF_REALTIME_CONNECTION_MODE", "local")
+    monkeypatch.setenv("HF_REALTIME_WS_URL", "ws://192.168.1.42:8766/v1/realtime")
+
+    app = FastAPI()
+    robot = SimpleNamespace(media=SimpleNamespace(audio=None, backend=None))
+    stream = LocalStream(MagicMock(), robot, settings_app=app, instance_path=str(tmp_path))
+    stream._init_settings_ui_if_needed()
+
+    client = TestClient(app)
+    response = client.post(
+        "/backend_config",
+        json={"backend": "huggingface"},
+    )
+
+    assert response.status_code == 200
+    data = response.json()
+    assert data["ok"] is True
+    assert data["backend_provider"] == "huggingface"
+    assert data["hf_connection_mode"] == "local"
+    assert data["hf_direct_host"] == "192.168.1.42"
+    assert data["hf_direct_port"] == 8766
+
+    env_text = env_path.read_text(encoding="utf-8")
+    assert "BACKEND_PROVIDER=huggingface" in env_text
+    assert "HF_REALTIME_CONNECTION_MODE=local" in env_text
+    assert "HF_REALTIME_WS_URL=ws://192.168.1.42:8766/v1/realtime" in env_text
+
+
+def test_backend_config_rejects_invalid_hf_port_zero(
+    tmp_path: Path,
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """Settings API should reject invalid local Hugging Face ports from direct callers."""
+    monkeypatch.setattr(config, "BACKEND_PROVIDER", "huggingface")
+    monkeypatch.setattr(config, "HF_REALTIME_CONNECTION_MODE", "deployed")
+    monkeypatch.setattr(config, "HF_REALTIME_SESSION_URL", None)
+    monkeypatch.setattr(config, "HF_REALTIME_WS_URL", None)
+
+    app = FastAPI()
+    robot = SimpleNamespace(media=SimpleNamespace(audio=None, backend=None))
+    stream = LocalStream(MagicMock(), robot, settings_app=app, instance_path=str(tmp_path))
+    stream._init_settings_ui_if_needed()
+
+    client = TestClient(app)
+    response = client.post(
+        "/backend_config",
+        json={
+            "backend": "huggingface",
+            "hf_mode": "local",
+            "hf_host": "localhost",
+            "hf_port": 0,
+        },
+    )
+
+    assert response.status_code == 400
+    assert response.json()["error"] == "invalid_hf_port"
+
+
+def test_status_reports_direct_hf_ws_url_as_ready(
+    tmp_path: Path,
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """Settings API should treat a direct Hugging Face websocket as a valid configuration."""
+    monkeypatch.setattr(config, "BACKEND_PROVIDER", "huggingface")
+    monkeypatch.setattr(config, "HF_REALTIME_CONNECTION_MODE", "local")
+    monkeypatch.setattr(config, "HF_REALTIME_SESSION_URL", None)
+    monkeypatch.setattr(config, "HF_REALTIME_WS_URL", "ws://127.0.0.1:8765/v1/realtime")
+
+    app = FastAPI()
+    robot = SimpleNamespace(media=SimpleNamespace(audio=None, backend=None))
+    stream = LocalStream(MagicMock(), robot, settings_app=app, instance_path=str(tmp_path))
+    stream._init_settings_ui_if_needed()
+
+    client = TestClient(app)
+    response = client.get("/status")
+
+    assert response.status_code == 200
+    data = response.json()
+    assert data["backend_provider"] == "huggingface"
+    assert data["has_hf_session_url"] is False
+    assert data["has_hf_ws_url"] is True
+    assert data["has_hf_connection"] is True
+    assert data["hf_connection_mode"] == "local"
+    assert data["can_proceed_with_hf"] is True
+
+
+def test_headless_personality_routes_return_gemini_voices_when_backend_selected(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
     """Headless personality UI should expose Gemini voices when Gemini is selected."""
     monkeypatch.setattr(config, "BACKEND_PROVIDER", "gemini")
     monkeypatch.setattr(config, "MODEL_NAME", "gemini-3.1-flash-live-preview")
@@ -227,6 +447,22 @@ def test_headless_personality_routes_return_gemini_voices_when_backend_selected(
     assert response.json() == GEMINI_AVAILABLE_VOICES
 
 
+def test_headless_personality_routes_load_builtin_default_tools() -> None:
+    """Headless personality UI should expose built-in default tools on initial load."""
+    app = FastAPI()
+    handler = MagicMock()
+    mount_personality_routes(app, handler, lambda: None)
+
+    client = TestClient(app)
+    response = client.get("/personalities/load", params={"name": "(built-in default)"})
+
+    assert response.status_code == 200
+    data = response.json()
+    assert data["tools_text"]
+    assert "dance" in data["enabled_tools"]
+    assert "camera" in data["enabled_tools"]
+
+
 def test_headless_personality_routes_apply_voice_accepts_query_param() -> None:
     """Headless personality UI should apply a voice change from a POST query param."""
     app = FastAPI()
@@ -258,3 +494,116 @@ def test_headless_personality_routes_apply_voice_accepts_query_param() -> None:
         loop.call_soon_threadsafe(loop.stop)
         thread.join(timeout=1.0)
         loop.close()
+
+
+def test_headless_personality_routes_persist_startup_with_voice_override() -> None:
+    """Saving a startup personality should persist the active manual voice override."""
+    app = FastAPI()
+    handler = MagicMock()
+    handler.apply_personality = AsyncMock(return_value="Applied personality and restarted realtime session.")
+    handler.get_current_voice = MagicMock(return_value="shimmer")
+    persist_personality = MagicMock()
+
+    loop = asyncio.new_event_loop()
+    started = threading.Event()
+
+    def _run_loop() -> None:
+        asyncio.set_event_loop(loop)
+        started.set()
+        loop.run_forever()
+
+    thread = threading.Thread(target=_run_loop, daemon=True)
+    thread.start()
+    started.wait(timeout=1.0)
+
+    try:
+        mount_personality_routes(app, handler, lambda: loop, persist_personality=persist_personality)
+
+        client = TestClient(app)
+        response = client.post("/personalities/apply?name=sorry_bro&persist=1")
+
+        assert response.status_code == 200
+        assert response.json()["ok"] is True
+        handler.apply_personality.assert_awaited_once_with("sorry_bro")
+        persist_personality.assert_called_once_with("sorry_bro", "shimmer")
+    finally:
+        loop.call_soon_threadsafe(loop.stop)
+        thread.join(timeout=1.0)
+        loop.close()
+
+
+def test_local_stream_persist_personality_stores_voice_override(tmp_path) -> None:
+    """Persisting startup settings should write both profile and voice override."""
+    stream = LocalStream(MagicMock(), MagicMock(), instance_path=str(tmp_path))
+
+    stream._persist_personality("sorry_bro", "shimmer")
+
+    settings_path = tmp_path / "startup_settings.json"
+    assert settings_path.exists()
+    assert settings_path.read_text(encoding="utf-8") == '{\n  "profile": "sorry_bro",\n  "voice": "shimmer"\n}\n'
+    assert stream._read_persisted_personality() == "sorry_bro"
+
+
+def test_local_stream_persist_personality_clears_legacy_startup_env_overrides(tmp_path, monkeypatch) -> None:
+    """Saving startup settings should remove legacy `.env` profile and voice overrides."""
+    env_path = tmp_path / ".env"
+    env_path.write_text(
+        "OPENAI_API_KEY=test-key\n"
+        "REACHY_MINI_CUSTOM_PROFILE=mad_scientist_assistant\n"
+        "REACHY_MINI_VOICE_OVERRIDE=shimmer\n",
+        encoding="utf-8",
+    )
+    stream = LocalStream(MagicMock(), MagicMock(), instance_path=str(tmp_path))
+
+    stream._persist_personality(None, "Aiden")
+
+    env_text = env_path.read_text(encoding="utf-8")
+    assert "OPENAI_API_KEY=test-key" in env_text
+    assert "REACHY_MINI_CUSTOM_PROFILE=" not in env_text
+    assert "REACHY_MINI_VOICE_OVERRIDE=" not in env_text
+
+    applied_profiles: list[str | None] = []
+    monkeypatch.delenv("REACHY_MINI_CUSTOM_PROFILE", raising=False)
+    monkeypatch.setattr(
+        "reachy_mini_conversation_app.config.set_custom_profile",
+        lambda profile: applied_profiles.append(profile),
+    )
+
+    settings = load_startup_settings_into_runtime(tmp_path)
+
+    assert settings == StartupSettings(voice="Aiden")
+    assert applied_profiles == [None]
+
+
+def test_local_stream_launch_waits_for_manual_openai_key_without_download(
+    tmp_path: Path,
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """OpenAI startup should wait for settings input instead of claiming a bundled key."""
+    monkeypatch.setattr(config, "BACKEND_PROVIDER", "openai")
+    monkeypatch.setattr(config, "OPENAI_API_KEY", None)
+    monkeypatch.setenv("BACKEND_PROVIDER", "openai")
+    monkeypatch.delenv("OPENAI_API_KEY", raising=False)
+
+    fake_client_ctor = MagicMock(side_effect=AssertionError("launch() should not try to download an OpenAI key"))
+    monkeypatch.setitem(sys.modules, "gradio_client", SimpleNamespace(Client=fake_client_ctor))
+
+    media = SimpleNamespace(
+        start_recording=MagicMock(),
+        start_playing=MagicMock(),
+    )
+    robot = SimpleNamespace(media=media)
+    stream = LocalStream(MagicMock(), robot, settings_app=FastAPI(), instance_path=str(tmp_path))
+    stream._active_backend_name = "openai"
+
+    init_settings_ui = MagicMock()
+    monkeypatch.setattr(stream, "_init_settings_ui_if_needed", init_settings_ui)
+    monkeypatch.setattr(stream, "_has_required_key", MagicMock(side_effect=[False, False]))
+    monkeypatch.setattr("reachy_mini_conversation_app.console.time.sleep", MagicMock(side_effect=KeyboardInterrupt))
+
+    stream.launch()
+
+    fake_client_ctor.assert_not_called()
+    init_settings_ui.assert_called_once()
+    media.start_recording.assert_not_called()
+    media.start_playing.assert_not_called()

tests/test_gemini_live.py

@@ -3,6 +3,7 @@
 import base64
 import asyncio
 from types import SimpleNamespace
+from typing import Any, Callable, AsyncIterator
 from unittest.mock import AsyncMock, MagicMock, call
 
 import numpy as np
@@ -10,13 +11,14 @@ import pytest
 from fastrtc import AdditionalOutputs
 
 import reachy_mini_conversation_app.gemini_live as gemini_mod
+import reachy_mini_conversation_app.tools.core_tools as ct_mod
 from reachy_mini_conversation_app.gemini_live import GeminiLiveHandler
 from reachy_mini_conversation_app.tools.core_tools import ToolDependencies
 from reachy_mini_conversation_app.tools.tool_constants import ToolState
 from reachy_mini_conversation_app.tools.background_tool_manager import ToolNotification
 
 
-def _server_content(**kwargs):
+def _server_content(**kwargs: Any) -> SimpleNamespace:
     defaults = {
         "model_turn": None,
         "turn_complete": None,
@@ -33,11 +35,11 @@ def _server_content(**kwargs):
     return SimpleNamespace(**defaults)
 
 
-def _response(server_content=None, tool_call=None):
+def _response(server_content: Any = None, tool_call: Any = None) -> SimpleNamespace:
     return SimpleNamespace(server_content=server_content, tool_call=tool_call)
 
 
-async def _wait_for(predicate, timeout: float = 1.0) -> None:
+async def _wait_for(predicate: Callable[[], bool], timeout: float = 1.0) -> None:
     deadline = asyncio.get_running_loop().time() + timeout
     while asyncio.get_running_loop().time() < deadline:
         if predicate():
@@ -47,24 +49,24 @@ async def _wait_for(predicate, timeout: float = 1.0) -> None:
 
 
 class _FakeSession:
-    def __init__(self, batches, stop_event: asyncio.Event):
+    def __init__(self, batches: list[list[SimpleNamespace]], stop_event: asyncio.Event) -> None:
         self._batches = list(batches)
         self._stop_event = stop_event
-        self.realtime_inputs = []
-        self.tool_responses = []
+        self.realtime_inputs: list[dict[str, Any]] = []
+        self.tool_responses: list[dict[str, Any]] = []
 
     async def close(self) -> None:
         self._stop_event.set()
 
-    async def send_realtime_input(self, **kwargs) -> None:
+    async def send_realtime_input(self, **kwargs: Any) -> None:
         self.realtime_inputs.append(kwargs)
         return None
 
-    async def send_tool_response(self, **kwargs) -> None:
+    async def send_tool_response(self, **kwargs: Any) -> None:
         self.tool_responses.append(kwargs)
         return None
 
-    async def receive(self):
+    async def receive(self) -> AsyncIterator[SimpleNamespace]:
         if self._batches:
             for response in self._batches.pop(0):
                 yield response
@@ -82,21 +84,23 @@ class _FakeConnectContext:
     async def __aenter__(self) -> _FakeSession:
         return self._session
 
-    async def __aexit__(self, *_args) -> bool:
+    async def __aexit__(self, *_args: object) -> bool:
         return False
 
 
 class _FakeLiveClient:
-    def __init__(self, session: _FakeSession):
+    def __init__(self, session: _FakeSession) -> None:
         self.aio = SimpleNamespace(live=SimpleNamespace(connect=lambda **_kwargs: _FakeConnectContext(session)))
 
 
 @pytest.mark.asyncio
-async def test_gemini_turn_buffers_transcripts_and_schedules_motion_reset(monkeypatch) -> None:
+async def test_gemini_turn_buffers_transcripts_and_schedules_motion_reset(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
     """Gemini turns should emit one transcript per role and let the wobbler reset after speech."""
     monkeypatch.setattr(gemini_mod, "get_session_instructions", lambda: "test")
     monkeypatch.setattr(gemini_mod, "get_session_voice", lambda: "Kore")
-    monkeypatch.setattr(gemini_mod, "get_tool_specs", lambda: [])
+    monkeypatch.setattr(gemini_mod, "get_active_tool_specs", lambda _: [])
 
     movement_manager = MagicMock()
     movement_manager.is_idle.return_value = False
@@ -108,8 +112,8 @@ async def test_gemini_turn_buffers_transcripts_and_schedules_motion_reset(monkey
         head_wobbler=head_wobbler,
     )
     handler = GeminiLiveHandler(deps)
-    object.__setattr__(handler.tool_manager, "start_up", MagicMock())
-    object.__setattr__(handler.tool_manager, "shutdown", AsyncMock())
+    monkeypatch.setattr(type(handler.tool_manager), "start_up", MagicMock())
+    monkeypatch.setattr(type(handler.tool_manager), "shutdown", AsyncMock())
 
     audio_bytes = b"\x00\x00\x10\x00" * 256
     session = _FakeSession(
@@ -235,3 +239,80 @@ async def test_gemini_camera_tool_sends_snapshot_and_returns_json_result() -> No
             "metadata": {"title": "🛠️ Used tool camera", "status": "done"},
         }
     ]
+
+
+@pytest.mark.asyncio
+async def test_apply_personality_preserves_manual_voice_override(monkeypatch) -> None:
+    """Applying a profile should keep a manually selected Gemini voice active."""
+    monkeypatch.setattr(gemini_mod, "get_session_instructions", lambda: "test")
+    monkeypatch.setattr(gemini_mod, "get_session_voice", lambda: "Kore")
+    monkeypatch.setattr("reachy_mini_conversation_app.config.set_custom_profile", lambda _profile: None)
+
+    handler = GeminiLiveHandler(ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock()))
+    handler.session = object()
+    handler._voice_override = "Orus"
+    restart = AsyncMock()
+    monkeypatch.setattr(handler, "_restart_session", restart)
+
+    status = await handler.apply_personality("example")
+
+    assert status == "Applied personality and restarted Gemini session."
+    assert handler.get_current_voice() == "Orus"
+    restart.assert_awaited_once()
+
+
+def test_handler_uses_startup_voice_at_startup() -> None:
+    """Gemini handler startup should restore a persisted startup voice."""
+    handler = GeminiLiveHandler(
+        ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock()),
+        startup_voice="Orus",
+    )
+
+    assert handler.get_current_voice() == "Orus"
+
+
+def test_copy_preserves_current_voice_override() -> None:
+    """Copied Gemini handlers should keep the current voice override."""
+    handler = GeminiLiveHandler(
+        ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock()),
+        startup_voice="Orus",
+    )
+    handler._voice_override = "Zephyr"
+
+    copied_handler = handler.copy()
+
+    assert copied_handler.get_current_voice() == "Zephyr"
+
+
+def test_gemini_excludes_head_tracking_when_no_head_tracker(monkeypatch) -> None:
+    """head_tracking tool must not appear in Gemini session config when head_tracker is not active."""
+    monkeypatch.setattr(gemini_mod, "get_session_instructions", lambda: "test")
+    monkeypatch.setattr(gemini_mod, "get_session_voice", lambda: "Kore")
+
+    # mock ALL_TOOL_SPECS to include at least head_tracking and one other tool, to verify that only head_tracking is excluded, not all tools
+    monkeypatch.setattr(
+        ct_mod,
+        "ALL_TOOL_SPECS",
+        [
+            {"type": "function", "name": "head_tracking", "description": "head_tracking", "parameters": {}},
+            {"type": "function", "name": "fake_tool", "description": "fake_tool", "parameters": {}},
+        ],
+    )
+
+    # case 1: no camera at all, --no-camera flag passed
+    deps = ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock(), camera_worker=None)
+    handler = GeminiLiveHandler(deps)
+    live_config = handler._build_live_config()
+    tool_names = [fd.name for fd in live_config.tools[0].function_declarations] if live_config.tools else []
+    assert "head_tracking" not in tool_names, "case 1 failed: camera_worker=None"
+    assert "fake_tool" in tool_names, "case 1 failed: a non-head-tracking tool was unexpectedly excluded"
+
+    # case 2: camera is running but --head-tracker flag was not passed
+    camera_worker = MagicMock()
+    camera_worker.head_tracker = None
+    deps = ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock(), camera_worker=camera_worker)
+    handler = GeminiLiveHandler(deps)
+    live_config = handler._build_live_config()
+    tool_names = [fd.name for fd in live_config.tools[0].function_declarations] if live_config.tools else []
+    assert "head_tracking" not in tool_names, "case 2 failed: camera_worker.head_tracker=None"
+    assert "fake_tool" in tool_names, "case 2 failed: a non-head-tracking tool was unexpectedly excluded"

tests/test_huggingface_realtime.py

@@ -0,0 +1,628 @@
+import asyncio
+from typing import Any
+from unittest.mock import AsyncMock, MagicMock
+
+import pytest
+
+import reachy_mini_conversation_app.base_realtime as base_rt_mod
+import reachy_mini_conversation_app.huggingface_realtime as hf_mod
+from reachy_mini_conversation_app.config import HF_BACKEND, config, get_default_voice_for_backend
+from reachy_mini_conversation_app.tools.core_tools import ToolDependencies
+from reachy_mini_conversation_app.huggingface_realtime import HuggingFaceRealtimeHandler
+
+
+HF_DEFAULT_VOICE = get_default_voice_for_backend(HF_BACKEND)
+
+
+def _make_usage(
+    audio_in: int | None = 100,
+    text_in: int | None = 200,
+    image_in: int | None = 300,
+    audio_out: int | None = 400,
+    text_out: int | None = 500,
+    has_input: bool = True,
+    has_output: bool = True,
+) -> MagicMock:
+    """Build a fake usage object matching the OpenAI-compatible response.usage shape."""
+    usage = MagicMock()
+    if has_input:
+        inp = MagicMock()
+        inp.audio_tokens = audio_in
+        inp.text_tokens = text_in
+        inp.image_tokens = image_in
+        usage.input_token_details = inp
+    else:
+        usage.input_token_details = None
+    if has_output:
+        out = MagicMock()
+        out.audio_tokens = audio_out
+        out.text_tokens = text_out
+        usage.output_token_details = out
+    else:
+        usage.output_token_details = None
+    return usage
+
+
+@pytest.mark.asyncio
+async def test_partial_transcription_uses_latest_snapshot(monkeypatch: Any) -> None:
+    """Partial transcription snapshots should replace older snapshots for the same item."""
+    monkeypatch.setattr(hf_mod, "get_session_instructions", lambda: "test")
+    monkeypatch.setattr(hf_mod, "get_session_voice", lambda default=HF_DEFAULT_VOICE: "Aiden")
+    monkeypatch.setattr(hf_mod, "get_active_tool_specs", lambda _: [])
+
+    class FakeEvent:
+        def __init__(self, etype: str, **kwargs: Any) -> None:
+            self.type = etype
+            for key, value in kwargs.items():
+                setattr(self, key, value)
+
+    class FakeSession:
+        async def update(self, **_kw: Any) -> None:
+            pass
+
+    class FakeInputAudioBuffer:
+        async def append(self, **_kw: Any) -> None:
+            pass
+
+    class FakeItem:
+        async def create(self, **_kw: Any) -> None:
+            pass
+
+    class FakeConversation:
+        item = FakeItem()
+
+    class FakeResponse:
+        async def create(self, **_kw: Any) -> None:
+            pass
+
+        async def cancel(self, **_kw: Any) -> None:
+            pass
+
+    class FakeConn:
+        session = FakeSession()
+        input_audio_buffer = FakeInputAudioBuffer()
+        conversation = FakeConversation()
+        response = FakeResponse()
+
+        def __init__(self) -> None:
+            self._events = iter(
+                [
+                    FakeEvent("conversation.item.input_audio_transcription.delta", item_id="item-1", delta="Hey"),
+                    FakeEvent(
+                        "conversation.item.input_audio_transcription.delta",
+                        item_id="item-1",
+                        delta="Hey, how are you?",
+                    ),
+                ]
+            )
+
+        async def __aenter__(self) -> "FakeConn":
+            return self
+
+        async def __aexit__(self, *_args: Any) -> bool:
+            return False
+
+        async def close(self) -> None:
+            pass
+
+        def __aiter__(self) -> "FakeConn":
+            return self
+
+        async def __anext__(self) -> FakeEvent:
+            try:
+                return next(self._events)
+            except StopIteration:
+                raise StopAsyncIteration
+
+    class FakeRealtime:
+        def connect(self, **_kw: Any) -> FakeConn:
+            return FakeConn()
+
+    class FakeClient:
+        def __init__(self) -> None:
+            self.realtime = FakeRealtime()
+
+    deps = ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock())
+    handler = HuggingFaceRealtimeHandler(deps)
+    fake_client = FakeClient()
+    handler.client = fake_client
+
+    start_up = MagicMock()
+    shutdown = AsyncMock()
+    monkeypatch.setattr(type(handler.tool_manager), "start_up", start_up)
+    monkeypatch.setattr(type(handler.tool_manager), "shutdown", shutdown)
+
+    await handler._run_realtime_session()
+
+    assert handler.input_transcript_chunks_by_item.item_id == "item-1"
+    assert handler.input_transcript_chunks_by_item.deltas == ["Hey, how are you?"]
+
+
+@pytest.mark.asyncio
+async def test_output_audio_delta_passes_output_sample_rate_to_head_wobbler(monkeypatch: Any) -> None:
+    """Assistant audio deltas should propagate the realtime output sample rate to the head wobbler."""
+    monkeypatch.setattr(hf_mod, "get_session_instructions", lambda: "test")
+    monkeypatch.setattr(hf_mod, "get_session_voice", lambda default=HF_DEFAULT_VOICE: "Aiden")
+    monkeypatch.setattr(hf_mod, "get_active_tool_specs", lambda _: [])
+    monkeypatch.setattr(config, "BACKEND_PROVIDER", "huggingface")
+
+    audio_delta = "AAABAAIAAwA="
+
+    class FakeEvent:
+        def __init__(self, etype: str, **kwargs: Any) -> None:
+            self.type = etype
+            for key, value in kwargs.items():
+                setattr(self, key, value)
+
+    class FakeSession:
+        async def update(self, **_kw: Any) -> None:
+            pass
+
+    class FakeInputAudioBuffer:
+        async def append(self, **_kw: Any) -> None:
+            pass
+
+    class FakeItem:
+        async def create(self, **_kw: Any) -> None:
+            pass
+
+    class FakeConversation:
+        item = FakeItem()
+
+    class FakeResponse:
+        async def create(self, **_kw: Any) -> None:
+            pass
+
+        async def cancel(self, **_kw: Any) -> None:
+            pass
+
+    class FakeConn:
+        session = FakeSession()
+        input_audio_buffer = FakeInputAudioBuffer()
+        conversation = FakeConversation()
+        response = FakeResponse()
+
+        def __init__(self) -> None:
+            self._events = iter(
+                [
+                    FakeEvent("response.created"),
+                    FakeEvent("response.output_audio.delta", delta=audio_delta),
+                    FakeEvent("response.output_audio.done"),
+                ]
+            )
+
+        async def __aenter__(self) -> "FakeConn":
+            return self
+
+        async def __aexit__(self, *_args: Any) -> bool:
+            return False
+
+        async def close(self) -> None:
+            pass
+
+        def __aiter__(self) -> "FakeConn":
+            return self
+
+        async def __anext__(self) -> FakeEvent:
+            try:
+                return next(self._events)
+            except StopIteration:
+                raise StopAsyncIteration
+
+    class FakeRealtime:
+        def connect(self, **_kw: Any) -> FakeConn:
+            return FakeConn()
+
+    class FakeClient:
+        def __init__(self) -> None:
+            self.realtime = FakeRealtime()
+
+    head_wobbler = MagicMock()
+    deps = ToolDependencies(
+        reachy_mini=MagicMock(),
+        movement_manager=MagicMock(),
+        head_wobbler=head_wobbler,
+    )
+    handler = HuggingFaceRealtimeHandler(deps, gradio_mode=True)
+    handler.client = FakeClient()
+
+    start_up = MagicMock()
+    shutdown = AsyncMock()
+    monkeypatch.setattr(type(handler.tool_manager), "start_up", start_up)
+    monkeypatch.setattr(type(handler.tool_manager), "shutdown", shutdown)
+
+    await handler._run_realtime_session()
+
+    head_wobbler.feed_pcm.assert_called_once()
+    assert head_wobbler.feed_pcm.call_args.args[1] == handler.output_sample_rate
+    head_wobbler.request_reset_after_current_audio.assert_called_once()
+
+
+@pytest.mark.asyncio
+async def test_emit_skips_idle_signal_while_response_active(monkeypatch: Any) -> None:
+    """Idle tools should not trigger while a response is still active."""
+    movement_manager = MagicMock()
+    movement_manager.is_idle.return_value = True
+    deps = ToolDependencies(reachy_mini=MagicMock(), movement_manager=movement_manager)
+    handler = HuggingFaceRealtimeHandler(deps)
+    handler.last_activity_time = asyncio.get_running_loop().time() - 60.0
+    handler._response_done_event.clear()
+
+    send_idle_signal = AsyncMock()
+    monkeypatch.setattr(handler, "send_idle_signal", send_idle_signal)
+    monkeypatch.setattr(base_rt_mod, "wait_for_item", AsyncMock(return_value=None))
+
+    result = await handler.emit()
+
+    assert result is None
+    send_idle_signal.assert_not_awaited()
+
+
+def test_handler_uses_hf_startup_voice_at_startup(monkeypatch: Any) -> None:
+    """Hugging Face startup should restore persisted HF voices."""
+    monkeypatch.setattr(config, "BACKEND_PROVIDER", "huggingface")
+
+    handler = HuggingFaceRealtimeHandler(
+        ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock()),
+        startup_voice="Aiden",
+    )
+
+    assert handler.get_current_voice() == "Aiden"
+
+
+@pytest.mark.asyncio
+async def test_start_up_hf_gradio_does_not_wait_for_api_key(monkeypatch: Any) -> None:
+    """Hugging Face backend should not wait for gradio key input."""
+    monkeypatch.setattr(config, "BACKEND_PROVIDER", "huggingface")
+    monkeypatch.setattr(config, "OPENAI_API_KEY", "sk-openai-secret")
+
+    deps = ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock())
+    handler = hf_mod.HuggingFaceRealtimeHandler(deps, gradio_mode=True)
+
+    build_client = AsyncMock(return_value=MagicMock())
+    run_realtime_session = AsyncMock(return_value=None)
+    wait_for_args = AsyncMock(side_effect=AssertionError("wait_for_args should not be called"))
+
+    monkeypatch.setattr(handler, "_build_realtime_client", build_client)
+    monkeypatch.setattr(handler, "_run_realtime_session", run_realtime_session)
+    monkeypatch.setattr(handler, "wait_for_args", wait_for_args)
+
+    await handler.start_up()
+
+    wait_for_args.assert_not_awaited()
+    build_client.assert_awaited_once_with()
+    run_realtime_session.assert_awaited_once()
+
+
+@pytest.mark.asyncio
+async def test_run_realtime_session_uses_default_voice_for_lb_allocated_sessions(monkeypatch: Any) -> None:
+    """Use the backend default speaker when no profile voice is selected for the hf LB."""
+    monkeypatch.setattr(hf_mod, "get_session_instructions", lambda: "test")
+    monkeypatch.setattr(hf_mod, "get_session_voice", lambda default=HF_DEFAULT_VOICE: default)
+    monkeypatch.setattr(hf_mod, "get_active_tool_specs", lambda _: [])
+    monkeypatch.setattr(config, "BACKEND_PROVIDER", "huggingface")
+    monkeypatch.setattr(config, "HF_REALTIME_SESSION_URL", "https://lb.example.test/session")
+
+    captured_update: dict[str, Any] = {}
+
+    class FakeSession:
+        async def update(self, **kwargs: Any) -> None:
+            captured_update.update(kwargs)
+
+    class FakeInputAudioBuffer:
+        async def append(self, **_kw: Any) -> None:
+            pass
+
+    class FakeItem:
+        async def create(self, **_kw: Any) -> None:
+            pass
+
+    class FakeConversation:
+        item = FakeItem()
+
+    class FakeResponse:
+        async def create(self, **_kw: Any) -> None:
+            pass
+
+        async def cancel(self, **_kw: Any) -> None:
+            pass
+
+    class FakeConn:
+        session = FakeSession()
+        input_audio_buffer = FakeInputAudioBuffer()
+        conversation = FakeConversation()
+        response = FakeResponse()
+
+        async def __aenter__(self) -> "FakeConn":
+            return self
+
+        async def __aexit__(self, *_args: Any) -> bool:
+            return False
+
+        async def close(self) -> None:
+            pass
+
+        def __aiter__(self) -> "FakeConn":
+            return self
+
+        async def __anext__(self) -> Any:
+            raise StopAsyncIteration
+
+    class FakeRealtime:
+        def connect(self, **_kw: Any) -> FakeConn:
+            return FakeConn()
+
+    class FakeClient:
+        def __init__(self) -> None:
+            self.realtime = FakeRealtime()
+
+    deps = ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock())
+    handler = HuggingFaceRealtimeHandler(deps)
+    fake_client = FakeClient()
+    handler.client = fake_client
+
+    await handler._run_realtime_session()
+
+    session = captured_update["session"]
+    # HF at 16 kHz passes None so the backend uses its optimal default (16 kHz).
+    assert session["audio"]["input"]["format"]["rate"] is None
+    assert session["audio"]["output"]["format"]["rate"] is None
+    output = session["audio"]["output"]
+    assert output["voice"] == HF_DEFAULT_VOICE
+
+
+@pytest.mark.asyncio
+async def test_run_realtime_session_passes_allocated_session_query(monkeypatch: Any) -> None:
+    """Hugging Face sessions must forward the allocated session token to the websocket connect call."""
+    monkeypatch.setattr(hf_mod, "get_session_instructions", lambda: "test")
+    monkeypatch.setattr(hf_mod, "get_session_voice", lambda default=HF_DEFAULT_VOICE: default)
+    monkeypatch.setattr(hf_mod, "get_active_tool_specs", lambda _: [])
+
+    captured_connect: dict[str, Any] = {}
+
+    class FakeSession:
+        async def update(self, **_kw: Any) -> None:
+            pass
+
+    class FakeInputAudioBuffer:
+        async def append(self, **_kw: Any) -> None:
+            pass
+
+    class FakeItem:
+        async def create(self, **_kw: Any) -> None:
+            pass
+
+    class FakeConversation:
+        item = FakeItem()
+
+    class FakeResponse:
+        async def create(self, **_kw: Any) -> None:
+            pass
+
+        async def cancel(self, **_kw: Any) -> None:
+            pass
+
+    class FakeConn:
+        session = FakeSession()
+        input_audio_buffer = FakeInputAudioBuffer()
+        conversation = FakeConversation()
+        response = FakeResponse()
+
+        async def __aenter__(self) -> "FakeConn":
+            return self
+
+        async def __aexit__(self, *_args: Any) -> bool:
+            return False
+
+        async def close(self) -> None:
+            pass
+
+        def __aiter__(self) -> "FakeConn":
+            return self
+
+        async def __anext__(self) -> Any:
+            raise StopAsyncIteration
+
+    class FakeRealtime:
+        def connect(self, **kwargs: Any) -> FakeConn:
+            captured_connect.update(kwargs)
+            return FakeConn()
+
+    class FakeClient:
+        def __init__(self) -> None:
+            self.realtime = FakeRealtime()
+
+    handler = HuggingFaceRealtimeHandler(ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock()))
+    fake_client = FakeClient()
+    handler.client = fake_client
+    handler._realtime_connect_query = {"session_token": "abc123"}
+
+    await handler._run_realtime_session()
+
+    assert "model" not in captured_connect
+    assert captured_connect["extra_query"] == {"session_token": "abc123"}
+
+
+@pytest.mark.asyncio
+async def test_build_realtime_client_uses_direct_hf_ws_url(monkeypatch: Any) -> None:
+    """Hugging Face direct websocket mode should bypass the session allocator."""
+    captured_client_kwargs: dict[str, Any] = {}
+
+    class FakeClient:
+        def __init__(self, **kwargs: Any) -> None:
+            captured_client_kwargs.update(kwargs)
+
+    def _unexpected_async_client(*_args: Any, **_kwargs: Any) -> Any:
+        raise AssertionError("session allocator should not be called in direct websocket mode")
+
+    monkeypatch.setattr(hf_mod, "AsyncOpenAI", FakeClient)
+    monkeypatch.setattr(hf_mod.httpx, "AsyncClient", _unexpected_async_client)
+    monkeypatch.setattr(config, "BACKEND_PROVIDER", "huggingface")
+    monkeypatch.setattr(config, "HF_REALTIME_CONNECTION_MODE", "local")
+    monkeypatch.setattr(config, "HF_REALTIME_SESSION_URL", "https://lb.example.test/session")
+    monkeypatch.setattr(config, "OPENAI_API_KEY", "sk-openai-secret")
+    monkeypatch.setattr(config, "HF_TOKEN", None)
+    monkeypatch.setattr(
+        config,
+        "HF_REALTIME_WS_URL",
+        "ws://127.0.0.1:8765/v1/realtime?session_token=abc123&model=ignored-by-sdk",
+    )
+
+    handler = HuggingFaceRealtimeHandler(ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock()))
+
+    client = await handler._build_realtime_client()
+
+    assert client is not None
+    assert captured_client_kwargs["api_key"] == "DUMMY"
+    assert captured_client_kwargs["base_url"] == "http://127.0.0.1:8765/v1"
+    assert captured_client_kwargs["websocket_base_url"] == "ws://127.0.0.1:8765/v1"
+    assert handler._realtime_connect_query == {"session_token": "abc123"}
+
+
+@pytest.mark.asyncio
+async def test_build_realtime_client_uses_deployed_mode_even_when_direct_hf_ws_url_is_saved(
+    monkeypatch: Any,
+) -> None:
+    """Explicit deployed mode should let .env recover from a stale local websocket URL."""
+    captured_client_kwargs: dict[str, Any] = {}
+    requested_session_urls: list[str] = []
+    requested_session_headers: list[dict[str, str] | None] = []
+
+    class FakeClient:
+        def __init__(self, **kwargs: Any) -> None:
+            captured_client_kwargs.update(kwargs)
+
+    class FakeResponse:
+        def raise_for_status(self) -> None:
+            pass
+
+        def json(self) -> dict[str, str]:
+            return {
+                "session_id": "session-123",
+                "connect_url": "wss://hf.example.test/v1/realtime?session_token=allocated",
+            }
+
+    class FakeAsyncClient:
+        def __init__(self, **_kwargs: Any) -> None:
+            pass
+
+        async def __aenter__(self) -> "FakeAsyncClient":
+            return self
+
+        async def __aexit__(self, *_args: Any) -> bool:
+            return False
+
+        async def post(self, url: str, headers: dict[str, str] | None = None) -> FakeResponse:
+            requested_session_urls.append(url)
+            requested_session_headers.append(headers)
+            return FakeResponse()
+
+    monkeypatch.setattr(hf_mod, "AsyncOpenAI", FakeClient)
+    monkeypatch.setattr(hf_mod.httpx, "AsyncClient", FakeAsyncClient)
+    monkeypatch.setattr(config, "BACKEND_PROVIDER", "huggingface")
+    monkeypatch.setattr(config, "HF_REALTIME_CONNECTION_MODE", "deployed")
+    monkeypatch.setattr(config, "HF_REALTIME_SESSION_URL", "https://lb.example.test/session")
+    monkeypatch.setattr(config, "HF_REALTIME_WS_URL", "ws://127.0.0.1:8765/v1/realtime")
+    monkeypatch.setattr(config, "OPENAI_API_KEY", "sk-openai-secret")
+    monkeypatch.setattr(config, "HF_TOKEN", "hf-secret")
+
+    handler = HuggingFaceRealtimeHandler(ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock()))
+
+    client = await handler._build_realtime_client()
+
+    assert client is not None
+    assert requested_session_urls == ["https://lb.example.test/session"]
+    assert requested_session_headers == [{"Authorization": "Bearer hf-secret"}]
+    assert captured_client_kwargs["api_key"] == "hf-secret"
+    assert captured_client_kwargs["base_url"] == "https://hf.example.test/v1"
+    assert captured_client_kwargs["websocket_base_url"] == "wss://hf.example.test/v1"
+    assert handler._realtime_connect_query == {"session_token": "allocated"}
+
+
+@pytest.mark.asyncio
+async def test_build_realtime_client_does_not_send_openai_key_to_hf_allocator(monkeypatch: Any) -> None:
+    """Hugging Face allocator auth should use HF_TOKEN only."""
+    captured_client_kwargs: dict[str, Any] = {}
+    requested_session_headers: list[dict[str, str] | None] = []
+
+    class FakeClient:
+        def __init__(self, **kwargs: Any) -> None:
+            captured_client_kwargs.update(kwargs)
+
+    class FakeResponse:
+        def raise_for_status(self) -> None:
+            pass
+
+        def json(self) -> dict[str, str]:
+            return {
+                "session_id": "session-123",
+                "connect_url": "wss://hf.example.test/v1/realtime?session_token=allocated",
+            }
+
+    class FakeAsyncClient:
+        def __init__(self, **_kwargs: Any) -> None:
+            pass
+
+        async def __aenter__(self) -> "FakeAsyncClient":
+            return self
+
+        async def __aexit__(self, *_args: Any) -> bool:
+            return False
+
+        async def post(self, _url: str, headers: dict[str, str] | None = None) -> FakeResponse:
+            requested_session_headers.append(headers)
+            return FakeResponse()
+
+    monkeypatch.setattr(hf_mod, "AsyncOpenAI", FakeClient)
+    monkeypatch.setattr(hf_mod.httpx, "AsyncClient", FakeAsyncClient)
+    monkeypatch.setattr(config, "BACKEND_PROVIDER", "huggingface")
+    monkeypatch.setattr(config, "HF_REALTIME_CONNECTION_MODE", "deployed")
+    monkeypatch.setattr(config, "HF_REALTIME_SESSION_URL", "https://lb.example.test/session")
+    monkeypatch.setattr(config, "HF_REALTIME_WS_URL", None)
+    monkeypatch.setattr(config, "OPENAI_API_KEY", "sk-openai-secret")
+    monkeypatch.setattr(config, "HF_TOKEN", None)
+
+    handler = HuggingFaceRealtimeHandler(ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock()))
+
+    client = await handler._build_realtime_client()
+
+    assert client is not None
+    assert requested_session_headers == [None]
+    assert captured_client_kwargs["api_key"] == "DUMMY"
+
+
+@pytest.mark.asyncio
+async def test_apply_personality_uses_selected_voice_for_lb_allocated_sessions(monkeypatch: Any) -> None:
+    """Live personality updates should honor the selected Qwen CustomVoice speaker."""
+    monkeypatch.setattr(hf_mod, "get_session_instructions", lambda: "new instructions")
+    monkeypatch.setattr(hf_mod, "get_session_voice", lambda default=HF_DEFAULT_VOICE: "Serena")
+    monkeypatch.setattr(config, "BACKEND_PROVIDER", "huggingface")
+    monkeypatch.setattr(config, "HF_REALTIME_SESSION_URL", "https://lb.example.test/session")
+
+    captured_update: dict[str, Any] = {}
+
+    class FakeSession:
+        async def update(self, **kwargs: Any) -> None:
+            captured_update.update(kwargs)
+
+    class FakeConnection:
+        session = FakeSession()
+
+    handler = HuggingFaceRealtimeHandler(ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock()))
+    handler.connection = FakeConnection()
+    monkeypatch.setattr(handler, "_restart_session", AsyncMock(return_value=None))
+
+    result = await handler.apply_personality("example")
+
+    assert "restarted realtime session" in result.lower()
+    session = captured_update["session"]
+    assert session["instructions"] == "new instructions"
+    assert session["audio"]["output"]["voice"] == "Serena"
+
+
+def test_huggingface_response_cost_defaults_to_zero() -> None:
+    """Hugging Face should not inherit OpenAI pricing from the shared base handler."""
+    usage = _make_usage(audio_in=1000, text_in=2000, image_in=500, audio_out=800, text_out=300)
+    handler = HuggingFaceRealtimeHandler(ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock()))
+
+    assert handler._compute_response_cost(usage) == 0.0

tests/test_openai_realtime.py

@@ -1,4 +1,3 @@
-import base64
 import random
 import asyncio
 import logging
@@ -8,26 +7,114 @@ from datetime import datetime, timezone
 from unittest.mock import AsyncMock, MagicMock
 
 import pytest
+from fastrtc import AdditionalOutputs
 
+import reachy_mini_conversation_app.base_realtime as base_rt_mod
 import reachy_mini_conversation_app.openai_realtime as rt_mod
+import reachy_mini_conversation_app.tools.core_tools as ct_mod
 import reachy_mini_conversation_app.tools.background_tool_manager as btm_mod
-from reachy_mini_conversation_app.openai_realtime import OpenaiRealtimeHandler, _compute_response_cost
+from reachy_mini_conversation_app.config import OPENAI_BACKEND, config, get_default_voice_for_backend
+from reachy_mini_conversation_app.openai_realtime import OpenaiRealtimeHandler
 from reachy_mini_conversation_app.tools.core_tools import ToolDependencies
 from reachy_mini_conversation_app.tools.background_tool_manager import ToolCallRoutine
 
 
+OPENAI_DEFAULT_VOICE = get_default_voice_for_backend(OPENAI_BACKEND)
+
+
 def _build_handler(loop: asyncio.AbstractEventLoop) -> OpenaiRealtimeHandler:
     asyncio.set_event_loop(loop)
     deps = ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock())
     return OpenaiRealtimeHandler(deps)
 
 
+async def _run_openai_handler_with_events(
+    monkeypatch: Any,
+    events: list[Any],
+    *,
+    movement_manager: MagicMock | None = None,
+) -> OpenaiRealtimeHandler:
+    """Run an OpenAI realtime handler against a fixed event sequence."""
+    monkeypatch.setattr(rt_mod, "get_session_instructions", lambda: "test")
+    monkeypatch.setattr(rt_mod, "get_session_voice", lambda default=OPENAI_DEFAULT_VOICE: "alloy")
+    monkeypatch.setattr(rt_mod, "get_active_tool_specs", lambda _: [])
+
+    class FakeSession:
+        async def update(self, **_kw: Any) -> None:
+            pass
+
+    class FakeInputAudioBuffer:
+        async def append(self, **_kw: Any) -> None:
+            pass
+
+    class FakeItem:
+        async def create(self, **_kw: Any) -> None:
+            pass
+
+    class FakeConversation:
+        item = FakeItem()
+
+    class FakeResponse:
+        async def create(self, **_kw: Any) -> None:
+            pass
+
+        async def cancel(self, **_kw: Any) -> None:
+            pass
+
+    class FakeConn:
+        session = FakeSession()
+        input_audio_buffer = FakeInputAudioBuffer()
+        conversation = FakeConversation()
+        response = FakeResponse()
+
+        def __init__(self) -> None:
+            self._events = iter(events)
+
+        async def __aenter__(self) -> "FakeConn":
+            return self
+
+        async def __aexit__(self, *_args: Any) -> bool:
+            return False
+
+        async def close(self) -> None:
+            pass
+
+        def __aiter__(self) -> "FakeConn":
+            return self
+
+        async def __anext__(self) -> Any:
+            try:
+                return next(self._events)
+            except StopIteration:
+                raise StopAsyncIteration
+
+    class FakeRealtime:
+        def connect(self, **_kw: Any) -> FakeConn:
+            return FakeConn()
+
+    class FakeClient:
+        def __init__(self) -> None:
+            self.realtime = FakeRealtime()
+
+    deps = ToolDependencies(reachy_mini=MagicMock(), movement_manager=movement_manager or MagicMock())
+    handler = OpenaiRealtimeHandler(deps)
+    handler.client = FakeClient()
+
+    start_up = MagicMock()
+    shutdown = AsyncMock()
+    monkeypatch.setattr(type(handler.tool_manager), "start_up", start_up)
+    monkeypatch.setattr(type(handler.tool_manager), "shutdown", shutdown)
+
+    await handler._run_realtime_session()
+    return handler
+
+
 @pytest.mark.asyncio
 async def test_tool_completion_does_not_reset_head_wobbler(monkeypatch: Any) -> None:
     """Tool completion should not interrupt ongoing speech wobble."""
     monkeypatch.setattr(rt_mod, "get_session_instructions", lambda: "test")
-    monkeypatch.setattr(rt_mod, "get_session_voice", lambda: "alloy")
-    monkeypatch.setattr(rt_mod, "get_tool_specs", lambda: [])
+    monkeypatch.setattr(rt_mod, "get_session_voice", lambda default=OPENAI_DEFAULT_VOICE: "alloy")
+    monkeypatch.setattr(rt_mod, "get_active_tool_specs", lambda _: [])
 
     async def _fake_dispatch(tool_name: str, args_json: str, deps: Any, **_kw: Any) -> dict[str, Any]:
         return {"image_description": "A person in front of a door.", "tool": tool_name}
@@ -104,7 +191,7 @@ async def test_tool_completion_does_not_reset_head_wobbler(monkeypatch: Any) ->
         head_wobbler=head_wobbler,
     )
     handler = OpenaiRealtimeHandler(deps)
-    fake_client: Any = FakeClient()
+    fake_client = FakeClient()
     handler.client = fake_client
 
     session_task = asyncio.create_task(handler._run_realtime_session())
@@ -132,8 +219,8 @@ async def test_tool_completion_does_not_reset_head_wobbler(monkeypatch: Any) ->
 async def test_non_idle_tool_call_does_not_queue_progress_response(monkeypatch: Any) -> None:
     """Tool-call startup should not enqueue a second speech response."""
     monkeypatch.setattr(rt_mod, "get_session_instructions", lambda: "test")
-    monkeypatch.setattr(rt_mod, "get_session_voice", lambda: "alloy")
-    monkeypatch.setattr(rt_mod, "get_tool_specs", lambda: [])
+    monkeypatch.setattr(rt_mod, "get_session_voice", lambda default=OPENAI_DEFAULT_VOICE: "alloy")
+    monkeypatch.setattr(rt_mod, "get_active_tool_specs", lambda _: [])
 
     class FakeEvent:
         def __init__(self, etype: str, **kwargs: Any) -> None:
@@ -209,16 +296,16 @@ async def test_non_idle_tool_call_does_not_queue_progress_response(monkeypatch:
 
     deps = ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock())
     handler = OpenaiRealtimeHandler(deps)
-    fake_client: Any = FakeClient()
+    fake_client = FakeClient()
     handler.client = fake_client
     safe_response_create = AsyncMock()
-    object.__setattr__(handler, "_safe_response_create", safe_response_create)
+    monkeypatch.setattr(handler, "_safe_response_create", safe_response_create)
     start_up = MagicMock()
     shutdown = AsyncMock()
     start_tool = AsyncMock(return_value=MagicMock(tool_id="camera-call_camera_1-0"))
-    object.__setattr__(handler.tool_manager, "start_up", start_up)
-    object.__setattr__(handler.tool_manager, "shutdown", shutdown)
-    object.__setattr__(handler.tool_manager, "start_tool", start_tool)
+    monkeypatch.setattr(type(handler.tool_manager), "start_up", start_up)
+    monkeypatch.setattr(type(handler.tool_manager), "shutdown", shutdown)
+    monkeypatch.setattr(type(handler.tool_manager), "start_tool", start_tool)
 
     await handler._run_realtime_session()
 
@@ -227,11 +314,11 @@ async def test_non_idle_tool_call_does_not_queue_progress_response(monkeypatch:
 
 
 @pytest.mark.asyncio
-async def test_output_audio_done_schedules_head_wobbler_reset(monkeypatch: Any) -> None:
-    """OpenAI speech completion should let the wobbler reset itself after queued audio."""
+async def test_user_speech_events_reset_idle_timer(monkeypatch: Any) -> None:
+    """User speech/transcription events should postpone idle behavior."""
     monkeypatch.setattr(rt_mod, "get_session_instructions", lambda: "test")
-    monkeypatch.setattr(rt_mod, "get_session_voice", lambda: "alloy")
-    monkeypatch.setattr(rt_mod, "get_tool_specs", lambda: [])
+    monkeypatch.setattr(rt_mod, "get_session_voice", lambda default=OPENAI_DEFAULT_VOICE: "alloy")
+    monkeypatch.setattr(rt_mod, "get_active_tool_specs", lambda _: [])
 
     class FakeEvent:
         def __init__(self, etype: str, **kwargs: Any) -> None:
@@ -270,12 +357,8 @@ async def test_output_audio_done_schedules_head_wobbler_reset(monkeypatch: Any)
         def __init__(self) -> None:
             self._events = iter(
                 [
-                    FakeEvent("response.created"),
-                    FakeEvent(
-                        "response.output_audio.delta",
-                        delta=base64.b64encode(b"\x00\x00\x10\x00").decode("ascii"),
-                    ),
-                    FakeEvent("response.output_audio.done"),
+                    FakeEvent("input_audio_buffer.speech_started"),
+                    FakeEvent("conversation.item.input_audio_transcription.completed", transcript="hello there"),
                 ]
             )
 
@@ -305,28 +388,121 @@ async def test_output_audio_done_schedules_head_wobbler_reset(monkeypatch: Any)
         def __init__(self) -> None:
             self.realtime = FakeRealtime()
 
-    head_wobbler = MagicMock()
-    audio = SimpleNamespace()
-    reachy_mini = SimpleNamespace(media=SimpleNamespace(audio=audio))
-    deps = ToolDependencies(
-        reachy_mini=reachy_mini,
-        movement_manager=MagicMock(),
-        head_wobbler=head_wobbler,
-    )
-    handler = OpenaiRealtimeHandler(deps, gradio_mode=True)
-    handler.client = FakeClient()
-    object.__setattr__(handler.tool_manager, "start_up", MagicMock())
-    object.__setattr__(handler.tool_manager, "shutdown", AsyncMock())
+    movement_manager = MagicMock()
+    deps = ToolDependencies(reachy_mini=MagicMock(), movement_manager=movement_manager)
+    handler = OpenaiRealtimeHandler(deps)
+    fake_client = FakeClient()
+    handler.client = fake_client
+    handler.last_activity_time = asyncio.get_running_loop().time() - 60.0
 
+    start_up = MagicMock()
+    shutdown = AsyncMock()
+    monkeypatch.setattr(type(handler.tool_manager), "start_up", start_up)
+    monkeypatch.setattr(type(handler.tool_manager), "shutdown", shutdown)
+
+    previous_activity_time = handler.last_activity_time
     await handler._run_realtime_session()
 
-    head_wobbler.feed.assert_called_once()
-    head_wobbler.request_reset_after_current_audio.assert_called_once()
-    head_wobbler.reset.assert_not_called()
+    assert handler.last_activity_time > previous_activity_time
+    movement_manager.set_listening.assert_any_call(True)
+
+
+@pytest.mark.asyncio
+async def test_empty_user_transcript_exits_listening_without_chat_message(monkeypatch: Any) -> None:
+    """Blank VAD commits should not leave listening motion frozen."""
+    movement_manager = MagicMock()
+
+    handler = await _run_openai_handler_with_events(
+        monkeypatch,
+        [
+            SimpleNamespace(type="input_audio_buffer.speech_started"),
+            SimpleNamespace(type="conversation.item.input_audio_transcription.completed", transcript="   "),
+        ],
+        movement_manager=movement_manager,
+    )
+
+    assert [call.args[0] for call in movement_manager.set_listening.call_args_list] == [True, False]
+    assert handler.output_queue.empty()
+    assert handler._turn_user_done_at is None
+
+
+@pytest.mark.asyncio
+async def test_empty_audio_buffer_error_exits_listening_without_chat_error(monkeypatch: Any) -> None:
+    """Empty audio-buffer commits are internal and should restore listening state."""
+    movement_manager = MagicMock()
+
+    handler = await _run_openai_handler_with_events(
+        monkeypatch,
+        [
+            SimpleNamespace(type="input_audio_buffer.speech_started"),
+            SimpleNamespace(
+                type="error",
+                error=SimpleNamespace(code="input_audio_buffer_commit_empty", message="empty audio buffer"),
+            ),
+        ],
+        movement_manager=movement_manager,
+    )
+
+    assert [call.args[0] for call in movement_manager.set_listening.call_args_list] == [True, False]
+    assert handler.output_queue.empty()
+
+
+@pytest.mark.asyncio
+async def test_apply_personality_preserves_manual_voice_override(monkeypatch: Any) -> None:
+    """Applying a profile should not discard a voice manually selected in the current session."""
+    monkeypatch.setattr(rt_mod, "get_session_instructions", lambda: "test")
+    monkeypatch.setattr(rt_mod, "get_session_voice", lambda: "cedar")
+    monkeypatch.setattr("reachy_mini_conversation_app.config.set_custom_profile", lambda _profile: None)
+
+    handler = OpenaiRealtimeHandler(ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock()))
+    update = AsyncMock()
+    handler.connection = SimpleNamespace(session=SimpleNamespace(update=update))
+    handler._voice_override = "marin"
+    restart = AsyncMock()
+    monkeypatch.setattr(handler, "_restart_session", restart)
+
+    status = await handler.apply_personality("example")
+
+    assert status == "Applied personality and restarted realtime session."
+    assert handler.get_current_voice() == "marin"
+    restart.assert_awaited_once()
+    session = update.await_args.kwargs["session"]
+    assert session["audio"]["output"]["voice"] == "marin"
+
+
+def test_handler_uses_startup_voice_at_startup(monkeypatch: Any) -> None:
+    """OpenAI handler startup should restore a persisted startup voice."""
+    monkeypatch.setattr(config, "BACKEND_PROVIDER", "openai")
+
+    handler = OpenaiRealtimeHandler(
+        ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock()),
+        startup_voice="shimmer",
+    )
+
+    assert handler.get_current_voice() == "shimmer"
+
+
+def test_copy_preserves_current_voice_override(monkeypatch: Any) -> None:
+    """Copied OpenAI handlers should keep the current voice override."""
+    monkeypatch.setattr(config, "BACKEND_PROVIDER", "openai")
+
+    handler = OpenaiRealtimeHandler(
+        ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock()),
+        startup_voice="shimmer",
+    )
+    handler._voice_override = "marin"
+
+    copied_handler = handler.copy()
+
+    assert copied_handler.get_current_voice() == "marin"
 
 
 def test_format_timestamp_uses_wall_clock() -> None:
     """Test that format_timestamp uses wall clock time."""
+    try:
+        previous_loop = asyncio.get_event_loop()
+    except RuntimeError:
+        previous_loop = asyncio.new_event_loop()
     loop = asyncio.new_event_loop()
     try:
         print("Testing format_timestamp...")
@@ -334,8 +510,8 @@ def test_format_timestamp_uses_wall_clock() -> None:
         formatted = handler.format_timestamp()
         print(f"Formatted timestamp: {formatted}")
     finally:
-        asyncio.set_event_loop(None)
         loop.close()
+        asyncio.set_event_loop(previous_loop)
 
     # Extract year from "[YYYY-MM-DD ...]"
     year = int(formatted[1:5])
@@ -351,9 +527,9 @@ async def test_start_up_retries_on_abrupt_close(monkeypatch: Any, caplog: Any) -
     """
     caplog.set_level(logging.WARNING)
 
-    # Use a local Exception as the module's ConnectionClosedError to avoid ws dependency
+    # Use a local Exception as the base module's ConnectionClosedError to avoid ws dependency.
     FakeCCE = type("FakeCCE", (Exception,), {})
-    monkeypatch.setattr(rt_mod, "ConnectionClosedError", FakeCCE)
+    monkeypatch.setattr(base_rt_mod, "ConnectionClosedError", FakeCCE)
 
     # Make asyncio.sleep return immediately (for backoff)
     _real_sleep = asyncio.sleep
@@ -431,6 +607,7 @@ async def test_start_up_retries_on_abrupt_close(monkeypatch: Any, caplog: Any) -
 
     # Patch the OpenAI client used by the handler
     monkeypatch.setattr(rt_mod, "AsyncOpenAI", FakeClient)
+    monkeypatch.setattr(config, "BACKEND_PROVIDER", "openai")
 
     # Build handler with minimal deps
     deps = ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock())
@@ -448,6 +625,78 @@ async def test_start_up_retries_on_abrupt_close(monkeypatch: Any, caplog: Any) -
     assert len(warnings) == 1
 
 
+@pytest.mark.asyncio
+async def test_start_up_openai_gradio_collects_textbox_api_key(monkeypatch: Any) -> None:
+    """OpenAI should own Gradio textbox credential collection."""
+    monkeypatch.setattr(config, "BACKEND_PROVIDER", "openai")
+    monkeypatch.setattr(config, "OPENAI_API_KEY", None)
+
+    deps = ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock())
+    handler = rt_mod.OpenaiRealtimeHandler(deps, gradio_mode=True)
+    handler.latest_args = ["profile", "voice", "unused", "sk-textbox-secret"]
+
+    build_client = AsyncMock(return_value=MagicMock())
+    run_realtime_session = AsyncMock(return_value=None)
+    wait_for_args = AsyncMock(return_value=None)
+
+    monkeypatch.setattr(handler, "_build_realtime_client", build_client)
+    monkeypatch.setattr(handler, "_run_realtime_session", run_realtime_session)
+    monkeypatch.setattr(handler, "wait_for_args", wait_for_args)
+
+    await handler.start_up()
+
+    wait_for_args.assert_awaited_once()
+    build_client.assert_awaited_once_with()
+    run_realtime_session.assert_awaited_once()
+    assert handler._provided_api_key == "sk-textbox-secret"
+
+
+@pytest.mark.asyncio
+async def test_run_realtime_session_propagates_session_update_failure(monkeypatch: Any) -> None:
+    """A failed session.update must abort startup instead of looking like a clean session exit."""
+    monkeypatch.setattr(rt_mod, "get_session_instructions", lambda: "test")
+    monkeypatch.setattr(rt_mod, "get_session_voice", lambda default=OPENAI_DEFAULT_VOICE: "alloy")
+    monkeypatch.setattr(rt_mod, "get_active_tool_specs", lambda _: [])
+
+    class FakeSession:
+        async def update(self, **_kw: Any) -> None:
+            raise RuntimeError("invalid session config")
+
+    class FakeConn:
+        session = FakeSession()
+
+        async def __aenter__(self) -> "FakeConn":
+            return self
+
+        async def __aexit__(self, *_args: Any) -> bool:
+            return False
+
+    class FakeRealtime:
+        def connect(self, **_kw: Any) -> FakeConn:
+            return FakeConn()
+
+    class FakeClient:
+        def __init__(self) -> None:
+            self.realtime = FakeRealtime()
+
+    handler = rt_mod.OpenaiRealtimeHandler(ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock()))
+    handler.client = FakeClient()
+
+    with pytest.raises(RuntimeError, match="invalid session config"):
+        await handler._run_realtime_session()
+
+
+@pytest.mark.asyncio
+async def test_handler_uses_openai_sample_rate_for_openai_backend(monkeypatch: Any) -> None:
+    """OpenAI backend should keep the 24 kHz realtime audio configuration."""
+    monkeypatch.setattr(config, "BACKEND_PROVIDER", "openai")
+
+    handler = OpenaiRealtimeHandler(ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock()))
+
+    assert handler.input_sample_rate == 24000
+    assert handler.output_sample_rate == 24000
+
+
 # ---- Cost calculation tests ----
 
 
@@ -495,9 +744,10 @@ def _make_usage(
     ids=["normal", "all_none", "mixed", "missing_details"],
 )
 def test_compute_response_cost(usage_kwargs: dict[str, Any], expect_positive: bool) -> None:
-    """Verify _compute_response_cost handles various token combinations without crashing."""
+    """Verify handler cost computation handles various token combinations without crashing."""
     usage = _make_usage(**usage_kwargs)
-    cost = _compute_response_cost(usage)
+    handler = OpenaiRealtimeHandler(ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock()))
+    cost = handler._compute_response_cost(usage)
     if expect_positive:
         assert cost > 0
     else:
@@ -507,6 +757,138 @@ def test_compute_response_cost(usage_kwargs: dict[str, Any], expect_positive: bo
 # ---- Stress test: response.create rejection + retry ----
 
 
+@pytest.mark.asyncio
+async def test_response_sender_retries_when_active_response_error_uses_type_only(
+    monkeypatch: Any,
+    caplog: Any,
+) -> None:
+    """Retry active-response rejections even when the server omits ``error.code``.
+
+    Some backends only populate ``error.type=conversation_already_has_active_response``.
+    That should still take the retry path and must not be surfaced as a user-facing error.
+    """
+    caplog.set_level(logging.DEBUG)
+    monkeypatch.setattr(base_rt_mod, "_RESPONSE_REJECTION_RETRY_DELAY", 0.01)
+    monkeypatch.setattr(rt_mod, "get_session_instructions", lambda: "test")
+    monkeypatch.setattr(rt_mod, "get_session_voice", lambda default=OPENAI_DEFAULT_VOICE: "alloy")
+    monkeypatch.setattr(rt_mod, "get_active_tool_specs", lambda _: [])
+
+    class FakeError:
+        def __init__(self, message: str) -> None:
+            self.message = message
+            self.code = None
+            self.type = "conversation_already_has_active_response"
+            self.event_id = None
+            self.param = None
+
+        def __repr__(self) -> str:
+            return f"RealtimeError(message='{self.message}', type='{self.type}', code=None, event_id=None, param=None)"
+
+    class FakeEvent:
+        def __init__(self, etype: str, **kwargs: Any) -> None:
+            self.type = etype
+            for key, value in kwargs.items():
+                setattr(self, key, value)
+
+    event_queue: asyncio.Queue[FakeEvent | None] = asyncio.Queue()
+
+    class FakeSession:
+        async def update(self, **_kw: Any) -> None:
+            pass
+
+    class FakeInputAudioBuffer:
+        async def append(self, **_kw: Any) -> None:
+            pass
+
+    class FakeItem:
+        async def create(self, **_kw: Any) -> None:
+            pass
+
+    class FakeConversation:
+        item = FakeItem()
+
+    class FakeResponseAPI:
+        def __init__(self) -> None:
+            self.call_count = 0
+
+        async def create(self, **_kw: Any) -> None:
+            self.call_count += 1
+            if self.call_count == 1:
+                event_queue.put_nowait(
+                    FakeEvent(
+                        "error",
+                        error=FakeError("Cannot create response while another response is in progress."),
+                    )
+                )
+            else:
+                event_queue.put_nowait(FakeEvent("response.created"))
+                event_queue.put_nowait(FakeEvent("response.done", response=MagicMock()))
+
+        async def cancel(self, **_kw: Any) -> None:
+            pass
+
+    fake_response_api = FakeResponseAPI()
+
+    class FakeConn:
+        session = FakeSession()
+        input_audio_buffer = FakeInputAudioBuffer()
+        conversation = FakeConversation()
+        response = fake_response_api
+
+        async def __aenter__(self) -> "FakeConn":
+            return self
+
+        async def __aexit__(self, *_args: Any) -> bool:
+            return False
+
+        async def close(self) -> None:
+            pass
+
+        def __aiter__(self) -> "FakeConn":
+            return self
+
+        async def __anext__(self) -> FakeEvent:
+            event = await event_queue.get()
+            if event is None:
+                raise StopAsyncIteration
+            return event
+
+    class FakeRealtime:
+        def connect(self, **_kw: Any) -> FakeConn:
+            return FakeConn()
+
+    class FakeClient:
+        def __init__(self) -> None:
+            self.realtime = FakeRealtime()
+
+    handler = rt_mod.OpenaiRealtimeHandler(ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock()))
+    handler.client = FakeClient()
+
+    session_task = asyncio.create_task(handler._run_realtime_session())
+    await asyncio.sleep(0)
+    await handler._safe_response_create(instructions="req")
+    await asyncio.sleep(0.1)
+    await event_queue.put(None)
+    await asyncio.wait_for(session_task, timeout=2.0)
+
+    assert fake_response_api.call_count == 2
+    assert not any(
+        record.levelname == "ERROR" and "Realtime error" in record.getMessage() for record in caplog.records
+    )
+    assert any("worker will retry after active response finishes" in record.getMessage() for record in caplog.records)
+    queued_outputs = []
+    while not handler.output_queue.empty():
+        queued_outputs.append(handler.output_queue.get_nowait())
+    queued_messages = [
+        message
+        for output in queued_outputs
+        if isinstance(output, AdditionalOutputs)
+        for message in output.args
+        if isinstance(message, dict)
+    ]
+    assert not any(str(message.get("content", "")).startswith("[error]") for message in queued_messages)
+
+
 @pytest.mark.asyncio
 async def test_response_sender_retries_on_active_response_rejection(monkeypatch: Any, caplog: Any) -> None:
     """Stress test: response.create rejection + retry via real event processing.
@@ -521,20 +903,20 @@ async def test_response_sender_retries_on_active_response_rejection(monkeypatch:
     processing, not mocked out.
     """
     caplog.set_level(logging.DEBUG)
+    monkeypatch.setattr(base_rt_mod, "_RESPONSE_REJECTION_RETRY_DELAY", 0.01)
 
     FakeCCE = type("FakeCCE", (Exception,), {})
-    monkeypatch.setattr(rt_mod, "ConnectionClosedError", FakeCCE)
+    monkeypatch.setattr(base_rt_mod, "ConnectionClosedError", FakeCCE)
     monkeypatch.setattr(rt_mod, "get_session_instructions", lambda: "test")
-    monkeypatch.setattr(rt_mod, "get_session_voice", lambda: "alloy")
-    monkeypatch.setattr(rt_mod, "get_tool_specs", lambda: [])
+    monkeypatch.setattr(rt_mod, "get_session_voice", lambda default=OPENAI_DEFAULT_VOICE: "alloy")
+    monkeypatch.setattr(rt_mod, "get_active_tool_specs", lambda _: [])
 
     N_TOOL_RESULTS = 400
     REJECT_CALL_NUMBERS = {1, 3, 5, 10, 25, 50, 75, 100, 150, 200, 300, 399}
     EXPECTED_TOTAL_CALLS = N_TOOL_RESULTS + len(REJECT_CALL_NUMBERS)
 
-    event_queue: asyncio.Queue[Any] = asyncio.Queue()
     response_create_log: list[tuple[int, dict[str, Any]]] = []
-    handler_ref: list[Any] = []
+    handler_ref: list[rt_mod.OpenaiRealtimeHandler] = []
 
     # ---- Fake event / error objects mirroring the OpenAI SDK shapes ----
 
@@ -558,6 +940,8 @@ async def test_response_sender_retries_on_active_response_rejection(monkeypatch:
             for k, v in kwargs.items():
                 setattr(self, k, v)
 
+    event_queue: asyncio.Queue[FakeEvent | None] = asyncio.Queue()
+
     # ---- Fake connection components ----
 
     class FakeResponseAPI:
@@ -660,7 +1044,7 @@ async def test_response_sender_retries_on_active_response_rejection(monkeypatch:
             return self
 
         async def __anext__(self) -> FakeEvent:
-            event: FakeEvent = await event_queue.get()
+            event = await event_queue.get()
             if event is None:  # sentinel → end iteration
                 raise StopAsyncIteration
             return event
@@ -674,6 +1058,7 @@ async def test_response_sender_retries_on_active_response_rejection(monkeypatch:
             self.realtime = FakeRealtime()
 
     monkeypatch.setattr(rt_mod, "AsyncOpenAI", FakeClient)
+    monkeypatch.setattr(config, "BACKEND_PROVIDER", "openai")
 
     # Patch dispatch_tool_call so tools complete with a result.
     async def _fake_dispatch(tool_name: str, args_json: str, deps: Any, **_kw: Any) -> dict[str, Any]:
@@ -704,10 +1089,13 @@ async def test_response_sender_retries_on_active_response_rejection(monkeypatch:
             is_idle_tool_call=False,
         )
 
-    # Yield so spawned tool tasks, the listener, and the sender can drain.
-    # This stress test queues hundreds of serialized response.create calls, so
-    # slower CI runners need a wider drain window before teardown.
-    await asyncio.sleep(10)
+    # Wait until spawned tool tasks, the listener, and the sender have drained.
+    # This stress test queues hundreds of serialized response.create calls; a
+    # condition-based wait avoids racing slower CI runners while still failing
+    # promptly if the sender stops making progress.
+    deadline = asyncio.get_event_loop().time() + 25.0
+    while fake_response_api._call_count < EXPECTED_TOTAL_CALLS and asyncio.get_event_loop().time() < deadline:
+        await asyncio.sleep(0.05)
 
     # ---- Tear down ----
 
@@ -760,7 +1148,7 @@ async def test_response_sender_loop_times_out_waiting_for_response_done(
     """
     caplog.set_level(logging.DEBUG)
 
-    monkeypatch.setattr(rt_mod, "_RESPONSE_DONE_TIMEOUT", 0.3)
+    monkeypatch.setattr(base_rt_mod, "_RESPONSE_DONE_TIMEOUT", 0.3)
 
     deps = ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock())
     handler = rt_mod.OpenaiRealtimeHandler(deps)
@@ -812,7 +1200,7 @@ async def test_response_sender_loop_times_out_waiting_for_previous_response(
     """
     caplog.set_level(logging.DEBUG)
 
-    monkeypatch.setattr(rt_mod, "_RESPONSE_DONE_TIMEOUT", 0.3)
+    monkeypatch.setattr(base_rt_mod, "_RESPONSE_DONE_TIMEOUT", 0.3)
 
     deps = ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock())
     handler = rt_mod.OpenaiRealtimeHandler(deps)
@@ -848,3 +1236,104 @@ async def test_response_sender_loop_times_out_waiting_for_previous_response(
 
     timeout_logs = [r for r in caplog.records if "Timed out waiting for previous response" in r.getMessage()]
     assert len(timeout_logs) == 1, f"Expected 1 pre-condition timeout warning, got {len(timeout_logs)}"
+
+
+@pytest.mark.asyncio
+async def test_openai_excludes_head_tracking_when_no_head_tracker(monkeypatch: Any) -> None:
+    """head_tracking tool must not appear in OpenAI session config when head_tracker is not active."""
+    monkeypatch.setattr(rt_mod, "get_session_instructions", lambda: "test")
+    monkeypatch.setattr(rt_mod, "get_session_voice", lambda default=None: "alloy")
+
+    # mock ALL_TOOL_SPECS to include at least head_tracking and one other tool, to verify that only head_tracking is excluded, not all tools
+    monkeypatch.setattr(
+        ct_mod,
+        "ALL_TOOL_SPECS",
+        [
+            {"type": "function", "name": "head_tracking", "description": "head_tracking", "parameters": {}},
+            {"type": "function", "name": "fake_tool", "description": "fake_tool", "parameters": {}},
+        ],
+    )
+
+    session_kwargs: dict = {}
+
+    class FakeSession:
+        async def update(self, **kwargs: Any) -> None:
+            session_kwargs["session"] = kwargs.get("session")
+
+    class FakeInputAudioBuffer:
+        async def append(self, **_kw: Any) -> None:
+            pass
+
+    class FakeItem:
+        async def create(self, **_kw: Any) -> None:
+            pass
+
+    class FakeConversation:
+        item = FakeItem()
+
+    class FakeResponse:
+        async def create(self, **_kw: Any) -> None:
+            pass
+
+        async def cancel(self, **_kw: Any) -> None:
+            pass
+
+    class FakeConn:
+        session = FakeSession()
+        input_audio_buffer = FakeInputAudioBuffer()
+        conversation = FakeConversation()
+        response = FakeResponse()
+
+        async def __aenter__(self) -> "FakeConn":
+            return self
+
+        async def __aexit__(self, *_: Any) -> bool:
+            return False
+
+        async def close(self) -> None:
+            pass
+
+        def __aiter__(self) -> "FakeConn":
+            return self
+
+        async def __anext__(self) -> Any:
+            raise StopAsyncIteration
+
+    class FakeRealtime:
+        def connect(self, **_kw: Any) -> FakeConn:
+            return FakeConn()
+
+    class FakeClient:
+        def __init__(self) -> None:
+            self.realtime = FakeRealtime()
+
+    # case 1: no camera at all, --no-camera flag passed
+    deps = ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock(), camera_worker=None)
+    handler = OpenaiRealtimeHandler(deps)
+    handler.client = FakeClient()
+    monkeypatch.setattr(type(handler.tool_manager), "start_up", MagicMock())
+    monkeypatch.setattr(type(handler.tool_manager), "shutdown", AsyncMock())
+
+    await handler._run_realtime_session()
+
+    session_tools = session_kwargs.get("session", {}).get("tools", [])
+    tool_names = [t["name"] for t in session_tools]
+    assert "head_tracking" not in tool_names, "case 1 failed: camera_worker=None"
+    assert "fake_tool" in tool_names, "case 1 failed: a non-head-tracking tool was unexpectedly excluded"
+
+    # case 2: camera is running but --head-tracker flag was not passed
+    session_kwargs.clear()
+    camera_worker = MagicMock()
+    camera_worker.head_tracker = None
+    deps = ToolDependencies(reachy_mini=MagicMock(), movement_manager=MagicMock(), camera_worker=camera_worker)
+    handler = OpenaiRealtimeHandler(deps)
+    handler.client = FakeClient()
+    monkeypatch.setattr(type(handler.tool_manager), "start_up", MagicMock())
+    monkeypatch.setattr(type(handler.tool_manager), "shutdown", AsyncMock())
+
+    await handler._run_realtime_session()
+
+    session_tools = session_kwargs.get("session", {}).get("tools", [])
+    tool_names = [t["name"] for t in session_tools]
+    assert "head_tracking" not in tool_names, "case 2 failed: camera_worker.head_tracker=None"
+    assert "fake_tool" in tool_names, "case 2 failed: a non-head-tracking tool was unexpectedly excluded"

tests/test_profile_paths.py

@@ -7,8 +7,12 @@ import pytest
 
 import reachy_mini_conversation_app.config as config_mod
 import reachy_mini_conversation_app.prompts as prompts_mod
+import reachy_mini_conversation_app.headless_personality as headless_mod
 from reachy_mini_conversation_app.config import DEFAULT_PROFILES_DIRECTORY, config
+from reachy_mini_conversation_app.gradio_personality import PersonalityUI
 from reachy_mini_conversation_app.headless_personality import (
+    DEFAULT_OPTION,
+    read_tools_for,
     resolve_profile_dir,
     read_instructions_for,
 )
@@ -81,6 +85,29 @@ def test_prompts_load_from_compact_builtin_profile(monkeypatch: pytest.MonkeyPat
     assert read_instructions_for("mad_scientist_assistant") == expected
 
 
+def test_builtin_default_profile_tools_load_for_ui() -> None:
+    """The UI should read built-in default tools from the packaged default profile."""
+    expected = (DEFAULT_PROFILES_DIRECTORY / "default" / "tools.txt").read_text(encoding="utf-8")
+
+    assert read_tools_for(DEFAULT_OPTION) == expected
+
+
+def test_gradio_personality_ui_prefills_builtin_default_tools(monkeypatch: pytest.MonkeyPatch) -> None:
+    """Gradio should show the built-in default profile tools on first render."""
+    monkeypatch.setattr(config, "REACHY_MINI_CUSTOM_PROFILE", None)
+
+    ui = PersonalityUI()
+    ui.create_components()
+
+    expected_tools = read_tools_for(ui.DEFAULT_OPTION)
+    expected_enabled = [
+        line.strip() for line in expected_tools.splitlines() if line.strip() and not line.strip().startswith("#")
+    ]
+
+    assert ui.tools_txt_ta.value == expected_tools
+    assert sorted(ui.available_tools_cg.value) == sorted(expected_enabled)
+
+
 def test_session_voice_defaults_follow_selected_backend(monkeypatch: pytest.MonkeyPatch) -> None:
     """Session voice should fall back to the active backend default."""
     monkeypatch.setattr(config, "BACKEND_PROVIDER", "gemini")
@@ -90,6 +117,21 @@ def test_session_voice_defaults_follow_selected_backend(monkeypatch: pytest.Monk
     assert prompts_mod.get_session_voice() == "Kore"
 
 
+def test_headless_profile_write_defaults_voice_at_call_time(
+    tmp_path: Path,
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """New headless profiles should use the currently selected backend default voice."""
+    monkeypatch.setattr(config, "BACKEND_PROVIDER", "gemini")
+    monkeypatch.setattr(config, "MODEL_NAME", "gemini-3.1-flash-live-preview")
+    monkeypatch.setattr(headless_mod, "_profiles_root", lambda: tmp_path)
+
+    headless_mod._write_profile("runtime_voice_default", "test instructions", "")
+
+    voice_file = tmp_path / "user_personalities" / "runtime_voice_default" / "voice.txt"
+    assert voice_file.read_text(encoding="utf-8") == "Kore\n"
+
+
 def test_packaged_profiles_win_outside_source_checkout(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
     """Installed builds should use packaged profiles, not an unrelated sibling folder."""
     unrelated_profiles = tmp_path / "profiles"

tests/test_startup_settings.py

@@ -0,0 +1,80 @@
+"""Tests for persisted instance-local startup settings."""
+
+from reachy_mini_conversation_app.startup_settings import (
+    StartupSettings,
+    read_startup_settings,
+    write_startup_settings,
+    load_startup_settings_into_runtime,
+)
+
+
+def test_write_and_read_startup_settings(tmp_path) -> None:
+    """Startup settings should round-trip through startup_settings.json."""
+    write_startup_settings(tmp_path, profile="sorry_bro", voice="shimmer")
+
+    assert read_startup_settings(tmp_path) == StartupSettings(profile="sorry_bro", voice="shimmer")
+
+
+def test_load_startup_settings_into_runtime_applies_profile_when_no_env(monkeypatch, tmp_path) -> None:
+    """Startup settings should seed the runtime profile when no explicit env override exists."""
+    write_startup_settings(tmp_path, profile="sorry_bro", voice="shimmer")
+    applied_profiles: list[str | None] = []
+    monkeypatch.delenv("REACHY_MINI_CUSTOM_PROFILE", raising=False)
+    monkeypatch.setattr(
+        "reachy_mini_conversation_app.config.set_custom_profile",
+        lambda profile: applied_profiles.append(profile),
+    )
+
+    settings = load_startup_settings_into_runtime(tmp_path)
+
+    assert settings == StartupSettings(profile="sorry_bro", voice="shimmer")
+    assert applied_profiles == ["sorry_bro"]
+
+
+def test_load_startup_settings_into_runtime_saved_settings_override_instance_env(monkeypatch, tmp_path) -> None:
+    """Saved startup settings should override an instance-local profile env value."""
+    write_startup_settings(tmp_path, profile="sorry_bro", voice="shimmer")
+    applied_profiles: list[str | None] = []
+    monkeypatch.setenv("REACHY_MINI_CUSTOM_PROFILE", "env_profile")
+    monkeypatch.setattr(
+        "reachy_mini_conversation_app.config.set_custom_profile",
+        lambda profile: applied_profiles.append(profile),
+    )
+
+    settings = load_startup_settings_into_runtime(tmp_path)
+
+    assert settings == StartupSettings(profile="sorry_bro", voice="shimmer")
+    assert applied_profiles == ["sorry_bro"]
+
+
+def test_load_startup_settings_into_runtime_saved_settings_override_inherited_env(monkeypatch, tmp_path) -> None:
+    """Saved startup settings should override a profile inherited from another `.env`."""
+    write_startup_settings(tmp_path, profile="nature_documentarian", voice="cedar")
+    applied_profiles: list[str | None] = []
+    monkeypatch.setenv("REACHY_MINI_CUSTOM_PROFILE", "example")
+    monkeypatch.setattr(
+        "reachy_mini_conversation_app.config.set_custom_profile",
+        lambda profile: applied_profiles.append(profile),
+    )
+
+    settings = load_startup_settings_into_runtime(tmp_path)
+
+    assert settings == StartupSettings(profile="nature_documentarian", voice="cedar")
+    assert applied_profiles == ["nature_documentarian"]
+
+
+def test_load_startup_settings_into_runtime_preserves_inherited_env_without_saved_settings(
+    monkeypatch, tmp_path
+) -> None:
+    """Inherited env config should still apply when no startup settings have been saved."""
+    applied_profiles: list[str | None] = []
+    monkeypatch.setenv("REACHY_MINI_CUSTOM_PROFILE", "example")
+    monkeypatch.setattr(
+        "reachy_mini_conversation_app.config.set_custom_profile",
+        lambda profile: applied_profiles.append(profile),
+    )
+
+    settings = load_startup_settings_into_runtime(tmp_path)
+
+    assert settings == StartupSettings()
+    assert applied_profiles == []