NERDDISCO commited on
Commit
b7ce6b9
·
unverified ·
1 Parent(s): 696222f

feat: record (#14)

Browse files

* feat: moved record-demo to own view

* feat: simplified recording; interactive how to

* feat: add visual progress indicators to recording how-to; fix keyboard teleop not to hijack inputs or browser shortcuts; handle only teleop keys with normalized casing; allow editing camera names while teleop is active; add per-camera source selector on card via camera icon with fixed-width inline select; switch streams and persist camera config; disable source changes during active recording

* feat: improved performance of webcam while teleop is active

* feat: select supported video format

* fix: attach camera to recorder

* docs: best pratcise when recording video in the web

* fix: make video work

* fix: each episode has its own video

* feat(record): making sure that we use version 2.1 of the spec

* refactor: move upload logic from library to demo, unify on record() api

remove cloud storage uploaders (LeRobotHFUploader, LeRobotS3Uploader) from
@lerobot/web library, reducing dependencies by ~700kb (@huggingface/hub,
@aws-sdk/client-s3, @aws-sdk/lib-storage).

add dataset-uploader.ts utility to demo for handling HuggingFace and S3 uploads,
allowing applications to implement custom upload strategies.

update RecordConfig to support videoStreams parameter for multi-camera recording.
enhance RecordProcess to expose underlying recorder instance for advanced use cases
(episode persistence, dynamic camera management, custom metadata).

update cyberpunk demo to use record() function with recorder access for all
advanced features, eliminating direct LeRobotDatasetRecorder instantiation.

update package.json, README, index.ts exports, and type definitions accordingly.

BREAKING CHANGE: exportForLeRobot('huggingface') and exportForLeRobot('s3')
removed from library (users should export 'blobs' and handle upload in their apps)

* feat(record): add flexible recordprocess api with runtime management

expand RecordConfig to support robotType metadata for upfront configuration.

enhance RecordProcess with runtime management methods:
- getEpisodeCount() / getEpisodes() - episode introspection
- clearEpisodes() - delete all recorded episodes
- nextEpisode() - create new episode segment
- restoreEpisodes() - restore persisted episodes
- addCamera() / removeCamera() - dynamic camera management

remove direct recorder access requirement from demo. update cyberpunk example
to use clean RecordProcess API instead of accessing internal recorder instance.

support both upfront configuration (videos, robot type, options) and runtime
operations (add cameras, manage episodes, export), providing maximum flexibility
for both simple and complex recording workflows.

update recorder.tsx to use new RecordProcess methods:
- record() call with robotType metadata
- episode management via getEpisodeCount() instead of direct access
- camera management via addCamera() instead of direct recorder access
- episode display via getEpisodes() for clean data access

this eliminates need for 'as any' casts and direct recorder access in demo,
maintaining clean API surface while supporting all advanced recording features.

* docs(record): simplify readme to show unified record() api only

remove references to LeRobotDatasetRecorder class and advanced use cases.
show single, clean record() function with all features available through
unified RecordProcess API.

simplify example to show:
- record() creation with config (teleoperator, videoStreams, robotType, options)
- recording control (start, stop, getState)
- episode management (getEpisodeCount, nextEpisode, clearEpisodes)
- dynamic camera management (addCamera)
- export options (zip-download, blobs)

users dont need to know about internal classes - record() handles everything.

* docs(readme): remove duplicate record() section

the 'dataset recording and export' section was a duplicate of the
record() documentation already in the core api section. remove it
to avoid confusion - one section, one api.

* docs(demo): add record() api documentation to docs section

add complete record() api reference to the demo's docs section including:
- usage example with teleoperation and video streams
- complete options and return types
- episode management methods (getEpisodeCount, nextEpisode, clearEpisodes)
- dynamic camera management (addCamera)
- export formats (blobs, zip, zip-download)

keeps documentation in sync with the library api.

* docs(demo): simplify record() example, remove setTimeout

remove unnecessary setTimeout from docs example and show cleaner,
more straightforward usage pattern. add nextEpisode() call to demonstrate
runtime episode management feature.

* docs(readme): fix record() example and api reference

remove setTimeout and show clean, straightforward usage.
add videoStreams, robotType to example and options documentation.
update returns to document all RecordProcess methods:
- episode management (getEpisodeCount, nextEpisode, clearEpisodes)
- dynamic cameras (addCamera)
- export (exportForLeRobot)

* chore: add changeset for @lerobot/web minor release

record api enhancements including flexible runtime management,
video stream support, and episode/camera management methods.

* chore(demo): mark record() as completed on roadmap

move record from in_progress to completed status.
reorder roadmap so record appears before SO-100 leader arm.
keep train and eval as planned features.

.changeset/odd-mice-eat.md ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ "@lerobot/web": minor
3
+ ---
4
+
5
+ enhance record() api with flexible runtime management and video stream support.
6
+
7
+ add videoStreams and robotType to RecordConfig for upfront configuration.
8
+ expose episode management methods on RecordProcess:
9
+ - getEpisodeCount() / getEpisodes() - introspect recorded episodes
10
+ - clearEpisodes() - delete all episodes
11
+ - nextEpisode() - create new episode segment
12
+ - restoreEpisodes() - restore persisted episodes
13
+
14
+ expose dynamic camera management methods:
15
+ - addCamera(name, stream) - add camera during recording
16
+ - removeCamera(name) - remove camera
17
+
18
+ all functionality previously requiring direct LeRobotDatasetRecorder access
19
+ now available through clean, type-safe RecordProcess interface. supports both
20
+ upfront configuration and runtime operations for maximum flexibility.
21
+
22
+ update demo to use unified record() api exclusively, removing direct recorder
23
+ access and supporting all advanced features (episode management, dynamic cameras,
24
+ custom metadata) through consistent api surface.
docs/conventions.md CHANGED
@@ -441,6 +441,26 @@ const response = await new Promise((resolve, reject) => {
441
  }
442
  ```
443
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
444
  #### Web Implementation Blockers Solved
445
 
446
  **These blockers were identified during SO-100 web calibration development:**
 
441
  }
442
  ```
443
 
444
+ ##### 6. Media Recording Workflow (Web Cameras)
445
+
446
+ **Required order of operations for video export to work correctly:**
447
+
448
+ - Attach camera streams before recording: call `recorder.addVideoStream(key, stream)` on the same `LeRobotDatasetRecorder` instance before `startRecording()`.
449
+ - Do not add/replace streams while recording: changing streams during an active recording is unsupported and will throw.
450
+ - Stop before export: always stop recording before calling any export method so `MediaRecorder` can finalize the video blob.
451
+
452
+ **Implementation notes:**
453
+
454
+ - Use a separate preview stream and clone for recording (e.g., `const recordingStream = previewStream.clone()`) to avoid preview interruptions.
455
+ - If your UI allows configuring cameras before enabling control, attach those streams to the recorder immediately after the recorder is created.
456
+ - The recorder selects a supported container/mime per browser; exported files are placed under `videos/chunk-000/observation.images.<cameraName>/episode_000000.<ext>`.
457
+
458
+ **Anti-patterns (avoid):**
459
+
460
+ - Creating the recorder, starting recording, then adding cameras.
461
+ - Mutating `videoStreams` during recording.
462
+ - Exporting while recording is still active.
463
+
464
  #### Web Implementation Blockers Solved
465
 
466
  **These blockers were identified during SO-100 web calibration development:**
docs/dataset/v2.1.md ADDED
@@ -0,0 +1,209 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # LeRobot Dataset Format v2.1 — Complete Spec
2
+
3
+ This document describes the **exact structure and syntax** of a dataset repository following the **LeRobot 2.1 standard**.
4
+ It is intended as an implementation guide — follow it exactly to ensure 100% compatibility with the dataset viewer and `LeRobotDataset` loader.
5
+
6
+ ---
7
+
8
+ ## 1) Repository Layout (typical)
9
+
10
+ ```
11
+ <dataset-root>/
12
+ ├─ data/
13
+ │ ├─ chunk-000/
14
+ │ │ ├─ episode_000000.parquet
15
+ │ │ └─ episode_000001.parquet
16
+ │ └─ chunk-001/
17
+ │ └─ ...
18
+ ├─ videos/
19
+ │ ├─ chunk-000/
20
+ │ │ ├─ observation.images.front/
21
+ │ │ │ ├─ episode_000000.mp4
22
+ │ │ │ └─ episode_000001.mp4
23
+ │ │ └─ observation.images.wrist/
24
+ │ │ ├─ episode_000000.mp4
25
+ │ │ └─ episode_000001.mp4
26
+ │ └─ chunk-001/
27
+ │ └─ ...
28
+ ├─ meta/
29
+ │ ├─ info.json
30
+ │ ├─ episodes.jsonl
31
+ │ ├─ tasks.jsonl
32
+ │ └─ episodes_stats.jsonl # (v2.1); older v2.0 used stats.json
33
+ └─ README.md
34
+ ```
35
+
36
+ ---
37
+
38
+ ## 2) `data/`
39
+
40
+ - Contains **one Parquet file per episode**.
41
+ - Path pattern (from `info.json.data_path`):
42
+ ```
43
+ data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet
44
+ ```
45
+
46
+ Each Parquet file stores synchronized timesteps with:
47
+
48
+ - `timestamp`
49
+ - `observations` (sensor values, images, poses, etc.)
50
+ - `actions` (target joint commands, end-effector states, etc.)
51
+
52
+ ---
53
+
54
+ ## 3) `videos/`
55
+
56
+ - Contains **one MP4 video per episode per camera key**.
57
+ - Path pattern (from `info.json.video_path`):
58
+ ```
59
+ videos/chunk-{episode_chunk:03d}/{video_key}/episode_{episode_index:06d}.mp4
60
+ ```
61
+ - `{video_key}` MUST exactly match a feature key in `info.json.features` under `observation.images.*`.
62
+
63
+ **Requirements:**
64
+
65
+ - Codec: H.264 (`avc1`)
66
+ - FPS matches `info.json.fps`
67
+ - Frame count == number of timesteps in corresponding Parquet
68
+
69
+ ---
70
+
71
+ ## 4) `meta/`
72
+
73
+ ### `info.json`
74
+
75
+ Top-level dataset metadata and templates.
76
+
77
+ ```json
78
+ {
79
+ "version": "2.1",
80
+ "fps": 30,
81
+ "chunks_size": 1000,
82
+ "data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
83
+ "video_path": "videos/chunk-{episode_chunk:03d}/{video_key}/episode_{episode_index:06d}.mp4",
84
+ "features": {
85
+ "observation.images.front": {
86
+ "dtype": "video",
87
+ "shape": [240, 320, 3],
88
+ "names": ["height", "width", "channel"],
89
+ "video_info": { "video.fps": 30.0, "video.codec": "avc1" }
90
+ }
91
+ }
92
+ }
93
+ ```
94
+
95
+ ---
96
+
97
+ ### `episodes.jsonl`
98
+
99
+ Line-delimited JSON, one entry per episode.
100
+
101
+ ```json
102
+ {"episode_id":"000000","task":"pick_and_place","length":243,"chunk":0}
103
+ {"episode_id":"000001","task":"pick_and_place","length":198,"chunk":0}
104
+ ```
105
+
106
+ Fields:
107
+
108
+ - `episode_id` → must match filenames (`episode_000000`)
109
+ - `task` → must exist in `tasks.jsonl`
110
+ - `length` → number of timesteps (and frames)
111
+ - `chunk` → numeric index (0,1,2…) matching folder
112
+
113
+ ---
114
+
115
+ ### `tasks.jsonl`
116
+
117
+ Defines available tasks.
118
+
119
+ ```json
120
+ {"task":"pick_and_place","description":"Pick an object and place it"}
121
+ {"task":"push_button","description":"Push the button until LED lights up"}
122
+ ```
123
+
124
+ ---
125
+
126
+ ### `episodes_stats.jsonl`
127
+
128
+ Dataset statistics for analysis & normalization.
129
+
130
+ ```json
131
+ {
132
+ "total_frames": 25873,
133
+ "episode_lengths": { "min": 120, "max": 450, "mean": 215 },
134
+ "action_stats": {
135
+ "x": { "min": -0.12, "max": 0.12, "mean": 0.0, "std": 0.05 },
136
+ "y": { "min": -0.1, "max": 0.1, "mean": 0.0, "std": 0.04 },
137
+ "z": { "min": 0.05, "max": 0.35, "mean": 0.2, "std": 0.06 }
138
+ }
139
+ }
140
+ ```
141
+
142
+ ---
143
+
144
+ ## 5) Chunking explained
145
+
146
+ - **Purpose:** purely structural; keeps folders manageable and enables fast lookups.
147
+ - **Decided by dataset builder:** choose `chunks_size` (often 1000).
148
+ - **Rule:**
149
+ ```
150
+ episode_chunk = episode_index // chunks_size
151
+ ```
152
+
153
+ ### Examples
154
+
155
+ - Small dataset (N ≤ 1000, chunks_size=1000) → all in `chunk-000`
156
+ - Large dataset (N=2000, chunks_size=1000):
157
+ - episodes 0–999 → `chunk-000`
158
+ - episodes 1000–1999 → `chunk-001`
159
+
160
+ ### Naming
161
+
162
+ - Chunk folder must be `chunk-XXX` with zero padding.
163
+
164
+ ---
165
+
166
+ ## 6) Synchronization rules
167
+
168
+ - Parquet rows = MP4 frames = `episodes.jsonl.length`
169
+ - Episode IDs consistent across parquet, mp4, and JSONL
170
+ - Video folder name = exact feature key (`observation.images.front`)
171
+ - Codecs = H.264 (`avc1`)
172
+ - Forward slashes in all paths
173
+
174
+ ---
175
+
176
+ ## 7) Common pitfalls
177
+
178
+ 1. Wrong folder naming (`front/` instead of `observation.images.front/`)
179
+ 2. Skipping `chunk-000` entirely (must exist, even for small datasets)
180
+ 3. Filename mismatch (`000001.mp4` vs `episode_000001.mp4`)
181
+ 4. Nested JSON instead of JSONL
182
+ 5. Using unsupported codec (HEVC/AV1)
183
+
184
+ ---
185
+
186
+ ## 8) Quick implementation checklist
187
+
188
+ - [ ] `info.json.version == "2.1"`
189
+ - [ ] `chunks_size` defined (e.g., 1000)
190
+ - [ ] `data_path` and `video_path` templates correct
191
+ - [ ] All `video_key` entries match `features` keys under `observation.images.*`
192
+ - [ ] Episode IDs zero‑padded: `episode_{:06d}`
193
+ - [ ] `episodes.jsonl` one JSON per line
194
+ - [ ] Parquet rows == MP4 frames == `length`
195
+ - [ ] Codec = H.264 (`avc1`), FPS correct
196
+ - [ ] All paths use `/` slashes
197
+
198
+ ---
199
+
200
+ ## 9) Minimal valid dataset (≤1000 episodes)
201
+
202
+ ```
203
+ my_dataset/
204
+ ├─ data/chunk-000/episode_000000.parquet
205
+ ├─ data/chunk-000/episode_000001.parquet
206
+ ├─ videos/chunk-000/observation.images.front/episode_000000.mp4
207
+ ├─ videos/chunk-000/observation.images.front/episode_000001.mp4
208
+ └─ meta/{info.json, episodes.jsonl, tasks.jsonl, episodes_stats.jsonl}
209
+ ```
docs/planning/{007_record.md → 007_refactor_record.md} RENAMED
@@ -4,76 +4,126 @@
4
 
5
  **As a** robotics developer building teleoperation recording systems
6
  **I want** to record robot motor positions and control data using a clean `record()` function API
7
- **So that** I can capture teleoperation sessions for training AI models, analysis, and replay without dealing with complex class-based APIs or mixed concerns
8
 
9
  ## Background
10
 
11
- A community contributor has provided a recording implementation in this PR branch, which includes a comprehensive `LeRobotDatasetRecorder` class with video recording, data export, and LeRobot dataset format support. However, the current implementation violates several of our core conventions and doesn't match the clean API patterns established by `calibrate()`, `teleoperate()`, and `findPort()`.
12
 
13
- ### Current Implementation Problems
14
 
15
- The existing `LeRobotDatasetRecorder` implementation has several architectural issues:
16
 
17
- - **Missing Standard Library Pattern**: Uses class instantiation instead of simple function call like our other APIs
18
- - **Library vs Demo Separation Violation**: Mixes hardware recording (library concern) with video streams, export formats, and UI (demo concerns)
19
- - **Teleoperator Integration Issues**: Recording logic deeply embedded in `BaseWebTeleoperator` with complex state management
20
- - **Complex Constructor Anti-Pattern**: Requires pre-configured teleoperators and video streams, violating our "direct library usage" principle
21
- - **Export API Complexity**: ZIP, HuggingFace, and S3 upload belong in demo code, not standard library
22
- - **No Clean Process API**: Doesn't follow our consistent `start()/stop()/result` pattern
23
- - **Redundant Event System**: Uses `dispatchMotorPositionChanged` events that aren't consumed and duplicate callback functionality
24
- - **Artificial Polling**: 100ms polling in teleoperate instead of immediate callbacks when motors change
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
25
 
26
  ### Convention Alignment Needed
27
 
28
  Our established patterns from `calibrate()`, `teleoperate()`, and `findPort()` follow these principles:
29
 
30
- - **Simple Function API**: `const process = await record(config)`
31
- - **Clean Process Objects**: Consistent `start()`, `stop()`, `getState()`, `result` interface
32
- - **Hardware-Only Library**: Standard library handles only robotics hardware, not UI/storage/export
33
  - **Demo Handles UI**: Examples handle video, export formats, browser storage, file downloads
34
- - **Immediate Callbacks**: Real-time updates via callbacks, not polling or unused events
35
  - **Direct Usage**: End users call library functions directly without complex setup
36
 
37
  ## Acceptance Criteria
38
 
39
  ### Core Functionality
40
 
41
- - [ ] **Standard Library API**: Clean `record(config)` function matching our established patterns
42
- - [ ] **Process Object Interface**: Consistent `RecordProcess` with `start()`, `stop()`, `getState()`, `result` methods
43
- - [ ] **Hardware-Only Recording**: Library captures only robot motor positions and teleoperation data
44
- - [ ] **Real-Time Callbacks**: Immediate `onDataUpdate` and `onStateUpdate` callbacks, no polling
45
- - [ ] **Device-Agnostic**: Works with any robot type through configuration, not hardcoded values
46
- - [ ] **Clean Teleoperator Integration**: Recording subscribes to teleoperation changes without embedding in teleoperator classes
47
 
48
  ### User Experience
49
 
50
  - [ ] **Simple Integration**: Easy to add recording to existing teleoperation workflows
51
  - [ ] **Consistent API**: Same patterns as `calibrate()` and `teleoperate()` for familiar developer experience
52
- - [ ] **Immediate Feedback**: Real-time recording state and data updates for responsive UI
53
  - [ ] **Error Handling**: Clear error messages for recording failures or invalid configurations
54
  - [ ] **Resource Management**: Proper cleanup of recording resources on stop/disconnect
55
 
56
  ### Technical Requirements
57
 
58
- - [ ] **Library/Demo Separation**: Move video, export, and storage logic to examples/demo layer
59
- - [ ] **Remove Event System**: Eliminate unused `dispatchMotorPositionChanged` events, use callbacks only
60
- - [ ] **Extract from Teleoperators**: Remove recording state and logic from `BaseWebTeleoperator`
61
  - [ ] **TypeScript**: Fully typed with proper interfaces for recording configuration and data
62
- - [ ] **No Code Duplication**: Reuse existing teleoperation and motor communication infrastructure
63
- - [ ] **Performance**: Immediate callbacks when data changes, no unnecessary polling
64
 
65
  ## Expected User Flow
66
 
67
- ### Basic Robot Recording
68
 
69
  ```typescript
70
- import { record } from "@lerobot/web";
71
 
72
- // Clean API matching our conventions
73
- const recordProcess = await record({
74
  robot: connectedRobot,
 
 
 
 
 
 
 
75
  options: {
76
  fps: 30,
 
77
  onDataUpdate: (data) => {
78
  // Real-time recording data for UI feedback
79
  console.log(`Recorded ${data.frameCount} frames`);
@@ -87,26 +137,57 @@ const recordProcess = await record({
87
  },
88
  });
89
 
90
- // Consistent process interface
 
91
  recordProcess.start();
92
 
93
- // Recording runs automatically while teleoperation is active
94
  setTimeout(() => {
95
  recordProcess.stop();
96
  }, 30000);
97
 
98
- // Get pure robot recording data
99
  const robotData = await recordProcess.result;
100
  console.log("Episodes:", robotData.episodes);
101
  console.log("Metadata:", robotData.metadata);
102
  ```
103
 
104
- ### Recording with Teleoperation
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
105
 
106
  ```typescript
107
  import { teleoperate, record } from "@lerobot/web";
108
 
109
- // Start teleoperation
110
  const teleoperationProcess = await teleoperate({
111
  robot: connectedRobot,
112
  teleop: { type: "keyboard" },
@@ -116,21 +197,23 @@ const teleoperationProcess = await teleoperate({
116
  },
117
  });
118
 
119
- // Add recording to existing teleoperation
120
  const recordProcess = await record({
121
- robot: connectedRobot,
122
  options: {
 
 
123
  onDataUpdate: (data) => {
124
  console.log(`Recording frame ${data.frameCount}`);
125
  },
126
  },
127
  });
128
 
129
- // Both run independently
130
  teleoperationProcess.start();
131
  recordProcess.start();
132
 
133
- // Control independently
134
  setTimeout(() => {
135
  recordProcess.stop(); // Stop recording, keep teleoperation
136
  }, 60000);
@@ -140,12 +223,19 @@ setTimeout(() => {
140
  }, 120000);
141
  ```
142
 
143
- ### Demo-Layer Dataset Export
 
 
 
 
 
 
 
144
 
145
  ```typescript
146
  // In examples/demo - NOT in standard library
147
  import { record } from "@lerobot/web";
148
- import { DatasetExporter } from "./dataset-exporter"; // Demo code
149
 
150
  const recordProcess = await record({ robot, options });
151
  recordProcess.start();
@@ -153,21 +243,35 @@ recordProcess.start();
153
  // ... recording session ...
154
 
155
  recordProcess.stop();
156
- const robotData = await recordProcess.result;
157
 
158
- // Demo handles complex export logic
159
  const exporter = new DatasetExporter({
160
  robotData,
161
  videoStreams: cameraStreams, // Demo manages video
162
  taskDescription: "Pick and place task",
163
  });
164
 
165
- // Export options handled by demo
166
  await exporter.downloadZip();
167
  await exporter.uploadToHuggingFace({ apiKey, repoName });
168
  await exporter.uploadToS3({ credentials });
169
  ```
170
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
171
  ### Component Integration
172
 
173
  ```typescript
@@ -203,45 +307,61 @@ const handleStopRecording = async () => {
203
 
204
  ## Implementation Details
205
 
206
- ### File Structure Refactoring
207
 
208
  ```
209
  packages/web/src/
210
- ├── record.ts # NEW: Clean record() function
 
211
  ├── types/
212
- │ └── recording.ts # NEW: Recording-specific types
213
- ├── utils/
214
- │ └── recording-manager.ts # NEW: Internal recording logic
215
  ├── teleoperators/
216
- │ └── base-teleoperator.ts # UPDATED: Remove recording logic
217
- └── [MOVED TO EXAMPLES]
218
- ├── LeRobotDatasetRecorder.ts # Complex export logic
219
- ├── dataset-exporter.ts # Video + export functionality
220
- └── upload-handlers.ts # HuggingFace, S3 upload logic
221
  ```
222
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
223
  ### Key Dependencies
224
 
225
- #### No New Dependencies for Standard Library
226
 
227
- - **Existing**: Reuse all current dependencies (motor communication, teleoperation integration)
228
- - **Architecture Only**: Pure refactoring to clean up existing functionality
229
 
230
- #### Demo Dependencies (Moved)
231
 
232
- - **Video/Export**: `parquet-wasm`, `apache-arrow`, `jszip` - moved to examples
233
- - **Upload**: `@huggingface/hub`, AWS SDK - moved to examples
 
234
 
235
  ### Core Functions to Implement
236
 
237
- #### Clean Record API
238
 
239
  ```typescript
240
- // record.ts - New clean API
241
  interface RecordConfig {
242
- robot: RobotConnection;
243
  options?: {
244
  fps?: number; // Default: 30
 
245
  onDataUpdate?: (data: RecordingData) => void;
246
  onStateUpdate?: (state: RecordingState) => void;
247
  };
@@ -265,11 +385,11 @@ interface RecordingState {
265
  interface RecordingData {
266
  frameCount: number;
267
  currentEpisode: number;
268
- recentFrames: MotorPositionFrame[]; // Last few frames for UI
269
  }
270
 
271
  interface RobotRecordingData {
272
- episodes: MotorPositionFrame[][]; // Pure motor data only
273
  metadata: {
274
  fps: number;
275
  robotType: string;
@@ -280,106 +400,80 @@ interface RobotRecordingData {
280
  };
281
  }
282
 
283
- // Main function - matches our conventions
 
284
  export async function record(config: RecordConfig): Promise<RecordProcess>;
285
  ```
286
 
287
- #### Recording Manager (Internal)
288
-
289
- ```typescript
290
- // utils/recording-manager.ts - Internal implementation
291
- class RecordingManager {
292
- private robot: RobotConnection;
293
- private isActive: boolean = false;
294
- private episodes: MotorPositionFrame[][] = [];
295
- private currentEpisode: MotorPositionFrame[] = [];
296
- private startTime: number = 0;
297
- private frameCount: number = 0;
298
-
299
- constructor(
300
- robot: RobotConnection,
301
- private options: RecordOptions,
302
- private onDataUpdate?: (data: RecordingData) => void,
303
- private onStateUpdate?: (state: RecordingState) => void
304
- ) {
305
- this.robot = robot;
306
- }
307
-
308
- start(): void {
309
- if (this.isActive) return;
310
 
311
- this.isActive = true;
312
- this.startTime = Date.now();
313
 
314
- // Subscribe to teleoperation changes (NO events, just callbacks)
315
- this.subscribeToRobotChanges();
316
-
317
- this.notifyStateUpdate();
318
- }
319
-
320
- stop(): void {
321
- if (!this.isActive) return;
322
-
323
- this.isActive = false;
324
- this.finishCurrentEpisode();
325
- this.unsubscribeFromRobotChanges();
326
-
327
- this.notifyStateUpdate();
328
- }
329
-
330
- private subscribeToRobotChanges(): void {
331
- // Listen to existing teleoperation callbacks - no new events needed
332
- // This integrates with the existing onStateUpdate mechanism
333
- }
334
-
335
- private recordFrame(motorConfigs: MotorConfig[]): void {
336
- const frame: MotorPositionFrame = {
337
- timestamp: Date.now() - this.startTime,
338
- motorPositions: motorConfigs.map((config) => ({
339
- id: config.id,
340
- name: config.name,
341
- position: config.currentPosition,
342
- })),
343
- frameIndex: this.frameCount++,
344
- };
345
-
346
- this.currentEpisode.push(frame);
347
-
348
- if (this.onDataUpdate) {
349
- this.onDataUpdate({
350
- frameCount: this.frameCount,
351
- currentEpisode: this.episodes.length,
352
- recentFrames: this.currentEpisode.slice(-10), // Last 10 frames
353
- });
354
- }
355
- }
356
 
357
- getState(): RecordingState {
358
- return {
359
- isActive: this.isActive,
360
- frameCount: this.frameCount,
361
- episodeCount: this.episodes.length,
362
- duration: this.isActive ? Date.now() - this.startTime : 0,
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
363
  lastUpdate: Date.now(),
364
- };
365
- }
366
-
367
- async getResult(): Promise<RobotRecordingData> {
368
- return {
369
- episodes: [...this.episodes],
370
  metadata: {
371
- fps: this.options.fps || 30,
372
- robotType: this.robot.robotType || "unknown",
373
- startTime: this.startTime,
374
  endTime: Date.now(),
375
- totalFrames: this.frameCount,
376
- totalEpisodes: this.episodes.length,
377
  },
378
- };
379
- }
380
  }
381
  ```
382
 
 
 
 
 
 
 
 
383
  #### Updated Teleoperate Integration
384
 
385
  ```typescript
@@ -468,16 +562,29 @@ class AdvancedDatasetExporter extends DatasetExporter {
468
 
469
  ## Definition of Done
470
 
471
- - [ ] **Clean Record API**: `record(config)` function implemented matching our established patterns
 
 
472
  - [ ] **Process Interface**: `RecordProcess` with consistent `start()`, `stop()`, `getState()`, `result` methods
473
- - [ ] **Hardware-Only Library**: Standard library captures only robot motor data, no video/export complexity
474
- - [ ] **Demo Separation**: Video recording, export formats, and UI logic moved to examples layer
475
- - [ ] **Remove Events**: `dispatchMotorPositionChanged` events eliminated, callbacks used exclusively
476
- - [ ] **Fix Polling**: 100ms artificial polling replaced with immediate callbacks when motors change
477
- - [ ] **Clean Teleoperators**: Recording logic extracted from `BaseWebTeleoperator` and teleoperator classes
478
- - [ ] **TypeScript Coverage**: Full type safety with proper interfaces for all recording functionality
479
- - [ ] **Performance**: Immediate, event-driven updates with no unnecessary polling or unused listeners
480
- - [ ] **Integration**: Easy integration with existing teleoperation workflows using familiar patterns
481
- - [ ] **Example Updates**: Cyberpunk demo updated to use new clean API with demo-layer export features
482
- - [ ] **No Regression**: All existing recording functionality preserved through demo layer
483
- - [ ] **Documentation**: Clear examples showing standard library vs demo separation
 
 
 
 
 
 
 
 
 
 
 
 
4
 
5
  **As a** robotics developer building teleoperation recording systems
6
  **I want** to record robot motor positions and control data using a clean `record()` function API
7
+ **So that** I can capture teleoperation sessions for training AI models, analysis, and replay with the same simple patterns as other LeRobot.js functions
8
 
9
  ## Background
10
 
11
+ A community contributor has successfully implemented comprehensive recording functionality, including a `LeRobotDatasetRecorder` class with video recording, data export, and LeRobot dataset format support. The implementation is functional and well-integrated, but doesn't follow our established simple function API patterns from `calibrate()`, `teleoperate()`, and `findPort()`.
12
 
13
+ ### Current Recording Implementation (From README)
14
 
15
+ The existing system works as documented in the web package README:
16
 
17
+ ```typescript
18
+ import { LeRobotDatasetRecorder } from "@lerobot/web";
19
+
20
+ // Create a recorder with teleoperator and video streams
21
+ const recorder = new LeRobotDatasetRecorder(
22
+ [teleoperator], // Array of teleoperators to record
23
+ { main: videoStream }, // Video streams by camera key
24
+ 30, // Target FPS
25
+ "Pick and place task" // Task description
26
+ );
27
+
28
+ // Start recording
29
+ await recorder.startRecording();
30
+ // ... robot performs task ...
31
+ const recordingData = await recorder.stopRecording();
32
+
33
+ // Export the dataset in various formats
34
+ await recorder.exportForLeRobot("zip-download");
35
+ await recorder.exportForLeRobot("huggingface", { repoName, accessToken });
36
+ await recorder.exportForLeRobot("s3", { bucketName, credentials });
37
+ ```
38
+
39
+ This implementation has **excellent architecture** - the explicit teleoperator dependency makes it clear, testable, and flexible.
40
+
41
+ ### Current Implementation Status
42
+
43
+ The existing recording system is **fully functional** with these components:
44
+
45
+ ✅ **Already Working Well:**
46
+
47
+ - `LeRobotDatasetRecorder` class with complete functionality
48
+ - Proper callback-based integration with teleoperators (no polling issues)
49
+ - Full LeRobot dataset format support with Parquet export
50
+ - Video recording and synchronization capabilities
51
+ - Complete cyberpunk example integration with camera management
52
+ - Clean separation between recording logic and teleoperator classes
53
+ - Export to ZIP, Hugging Face, and S3
54
+
55
+ ### What Works Well (Keep This)
56
+
57
+ The current implementation has **excellent architectural decisions**:
58
+
59
+ - **Explicit Teleoperator Dependency**: `LeRobotDatasetRecorder([teleoperator], ...)` makes dependencies clear and predictable
60
+ - **Clean Separation**: Recording subscribes to teleoperator callbacks without tight coupling
61
+ - **Flexible Architecture**: Can record from any teleoperator, multiple teleoperators, or no teleoperator at all
62
+
63
+ ### Areas for Improvement
64
+
65
+ The only issues are **API consistency** and **UI organization**:
66
+
67
+ - **Missing Simple Function API**: Users must instantiate `LeRobotDatasetRecorder` class directly instead of calling `record()` like other functions
68
+ - **UI Integration Pattern**: Recording is embedded within teleoperation view instead of being its own separate component/page
69
+ - **Library vs Demo Boundary**: Complex export functionality (video processing, cloud uploads) should be in demo layer, not standard library
70
 
71
  ### Convention Alignment Needed
72
 
73
  Our established patterns from `calibrate()`, `teleoperate()`, and `findPort()` follow these principles:
74
 
75
+ - **Simple Function API**: `const process = await record(config)` (currently requires class instantiation)
76
+ - **Clean Process Objects**: Consistent `start()`, `stop()`, `getState()`, `result` interface (class has `startRecording()`, `stopRecording()`)
77
+ - **Hardware-Only Library**: Standard library handles only robotics hardware (currently includes video/export)
78
  - **Demo Handles UI**: Examples handle video, export formats, browser storage, file downloads
 
79
  - **Direct Usage**: End users call library functions directly without complex setup
80
 
81
  ## Acceptance Criteria
82
 
83
  ### Core Functionality
84
 
85
+ - [ ] **Standard Library API**: Clean `record(config)` function matching our established patterns (wrap existing `LeRobotDatasetRecorder`)
86
+ - [ ] **Process Object Interface**: Consistent `RecordProcess` with `start()`, `stop()`, `getState()`, `result` methods (adapt existing methods)
87
+ - [ ] **Hardware-Only Recording**: Library captures only robot motor positions and teleoperation data (move video/export to demo)
88
+ - [ ] **Clean Teleoperator Integration**: Recording uses existing callback system in `BaseWebTeleoperator`
89
+ - [ ] **Preserve Advanced Features**: Keep `LeRobotDatasetRecorder` class for users who need full control
 
90
 
91
  ### User Experience
92
 
93
  - [ ] **Simple Integration**: Easy to add recording to existing teleoperation workflows
94
  - [ ] **Consistent API**: Same patterns as `calibrate()` and `teleoperate()` for familiar developer experience
95
+ - [ ] **Separate UI Component**: Move recording from teleoperation view to its own dedicated page/section in cyberpunk example
96
  - [ ] **Error Handling**: Clear error messages for recording failures or invalid configurations
97
  - [ ] **Resource Management**: Proper cleanup of recording resources on stop/disconnect
98
 
99
  ### Technical Requirements
100
 
101
+ - [ ] **Library/Demo Separation**: Move video recording, complex export logic (HF, S3) to examples/demo layer
102
+ - [ ] **Wrapper Function**: Create simple `record()` function that wraps existing `LeRobotDatasetRecorder`
103
+ - [ ] **Preserve Existing Integration**: Keep current callback system in `BaseWebTeleoperator` (it works well)
104
  - [ ] **TypeScript**: Fully typed with proper interfaces for recording configuration and data
105
+ - [ ] **No Breaking Changes**: Existing `LeRobotDatasetRecorder` class should remain available for advanced users
 
106
 
107
  ## Expected User Flow
108
 
109
+ ### Basic Robot Recording (Proposed Simple API)
110
 
111
  ```typescript
112
+ import { teleoperate, record } from "@lerobot/web";
113
 
114
+ // 1. Create teleoperation first (existing pattern)
115
+ const teleoperationProcess = await teleoperate({
116
  robot: connectedRobot,
117
+ teleop: { type: "keyboard" },
118
+ calibrationData: calibrationData,
119
+ });
120
+
121
+ // 2. NEW: Clean API with explicit teleoperator dependency
122
+ const recordProcess = await record({
123
+ teleoperator: teleoperationProcess.teleoperator, // ← Explicit dependency
124
  options: {
125
  fps: 30,
126
+ taskDescription: "Pick and place task",
127
  onDataUpdate: (data) => {
128
  // Real-time recording data for UI feedback
129
  console.log(`Recorded ${data.frameCount} frames`);
 
137
  },
138
  });
139
 
140
+ // 3. Start both processes
141
+ teleoperationProcess.start();
142
  recordProcess.start();
143
 
144
+ // 4. Recording captures teleoperation automatically via callbacks
145
  setTimeout(() => {
146
  recordProcess.stop();
147
  }, 30000);
148
 
149
+ // 5. Get pure robot recording data (no video/export complexity)
150
  const robotData = await recordProcess.result;
151
  console.log("Episodes:", robotData.episodes);
152
  console.log("Metadata:", robotData.metadata);
153
  ```
154
 
155
+ ### Current Implementation (Works Well, Just Different API Style)
156
+
157
+ ```typescript
158
+ import { LeRobotDatasetRecorder } from "@lerobot/web";
159
+
160
+ // CURRENT: Class-based API with explicit dependencies (good architecture!)
161
+ const recorder = new LeRobotDatasetRecorder(
162
+ [teleoperator], // ← GOOD: Explicit teleoperator dependency
163
+ { main: videoStream }, // Video complexity in library (to be moved)
164
+ 30, // fps
165
+ "Pick and place task" // Task description
166
+ );
167
+
168
+ // Different method names than our conventions (but functional)
169
+ await recorder.startRecording();
170
+ // ... robot performs task ...
171
+ const result = await recorder.stopRecording();
172
+
173
+ // Complex export in standard library (should be demo-only)
174
+ await recorder.exportForLeRobot("zip-download");
175
+ await recorder.exportForLeRobot("huggingface", { repoName, accessToken });
176
+ ```
177
+
178
+ **What's Good About Current Implementation:**
179
+
180
+ - ✅ **Explicit Dependencies**: Clear what the recorder needs to work
181
+ - ✅ **Clean Architecture**: Recording subscribes to teleoperator via callbacks
182
+ - ✅ **Full Functionality**: Complete LeRobot dataset format support
183
+ - ✅ **Flexible**: Can record from any teleoperator instance
184
+
185
+ ### Recording with Teleoperation (Proposed Simple API)
186
 
187
  ```typescript
188
  import { teleoperate, record } from "@lerobot/web";
189
 
190
+ // 1. Start teleoperation (existing pattern)
191
  const teleoperationProcess = await teleoperate({
192
  robot: connectedRobot,
193
  teleop: { type: "keyboard" },
 
197
  },
198
  });
199
 
200
+ // 2. NEW: Add recording with explicit teleoperator dependency
201
  const recordProcess = await record({
202
+ teleoperator: teleoperationProcess.teleoperator, // ← Explicit dependency (good!)
203
  options: {
204
+ fps: 30,
205
+ taskDescription: "Pick and place task",
206
  onDataUpdate: (data) => {
207
  console.log(`Recording frame ${data.frameCount}`);
208
  },
209
  },
210
  });
211
 
212
+ // 3. Both run independently
213
  teleoperationProcess.start();
214
  recordProcess.start();
215
 
216
+ // 4. Control independently
217
  setTimeout(() => {
218
  recordProcess.stop(); // Stop recording, keep teleoperation
219
  }, 60000);
 
223
  }, 120000);
224
  ```
225
 
226
+ **Why Explicit Teleoperator Dependency is Good:**
227
+
228
+ - 🎯 **Clear**: You know exactly what gets recorded
229
+ - 🔧 **Flexible**: Can record from any teleoperator
230
+ - 🧪 **Testable**: Easy to mock teleoperator for testing
231
+ - 📦 **Reusable**: Same teleoperator can serve multiple recorders
232
+
233
+ ### Demo-Layer Dataset Export (Proposed Architecture)
234
 
235
  ```typescript
236
  // In examples/demo - NOT in standard library
237
  import { record } from "@lerobot/web";
238
+ import { DatasetExporter } from "./dataset-exporter"; // MOVE complex logic here
239
 
240
  const recordProcess = await record({ robot, options });
241
  recordProcess.start();
 
243
  // ... recording session ...
244
 
245
  recordProcess.stop();
246
+ const robotData = await recordProcess.result; // Pure motor data only
247
 
248
+ // MOVE TO DEMO: Complex export logic with video/cloud features
249
  const exporter = new DatasetExporter({
250
  robotData,
251
  videoStreams: cameraStreams, // Demo manages video
252
  taskDescription: "Pick and place task",
253
  });
254
 
255
+ // MOVE TO DEMO: Export options
256
  await exporter.downloadZip();
257
  await exporter.uploadToHuggingFace({ apiKey, repoName });
258
  await exporter.uploadToS3({ credentials });
259
  ```
260
 
261
+ ### Current Cyberpunk Example Integration
262
+
263
+ ```typescript
264
+ // CURRENT: Recording embedded in teleoperation view
265
+ // examples/cyberpunk-standalone/src/components/teleoperation-view.tsx
266
+ <TeleoperationView robot={robot} />
267
+ // ^ Contains embedded <Recorder /> component
268
+
269
+ // PROPOSED: Separate recording page/component
270
+ // examples/cyberpunk-standalone/src/components/recording-view.tsx
271
+ <RecordingView robot={robot} />
272
+ // ^ Dedicated component with full recording interface
273
+ ```
274
+
275
  ### Component Integration
276
 
277
  ```typescript
 
307
 
308
  ## Implementation Details
309
 
310
+ ### File Structure Changes
311
 
312
  ```
313
  packages/web/src/
314
+ ├── record.ts # UPDATE: Add simple record() function wrapper
315
+ ├── record-class.ts # RENAME: Move LeRobotDatasetRecorder here
316
  ├── types/
317
+ │ └── recording.ts # NEW: Recording-specific types for simple API
 
 
318
  ├── teleoperators/
319
+ │ └── base-teleoperator.ts # KEEP: Current callback system works well
320
+ └── [MOVE TO EXAMPLES]
321
+ ├── dataset-exporter.ts # Video recording + export functionality
322
+ ├── hf_uploader.ts # HuggingFace upload logic
323
+ └── s3_uploader.ts # S3 upload logic
324
  ```
325
 
326
+ ### Current vs Proposed Architecture
327
+
328
+ **Current (Working):**
329
+
330
+ - `LeRobotDatasetRecorder` class with full functionality
331
+ - Integrated in cyberpunk example within teleoperation view
332
+ - Video, HF, S3 export in standard library
333
+
334
+ **Proposed (Convention-Aligned):**
335
+
336
+ - Simple `record()` function wrapping existing class
337
+ - Separate recording component/page in cyberpunk example
338
+ - Video, HF, S3 export moved to demo layer
339
+ - Keep existing class available for advanced users
340
+
341
  ### Key Dependencies
342
 
343
+ #### Standard Library (Minimal Changes)
344
 
345
+ - **Keep Existing**: All current dependencies for core recording functionality
346
+ - **Wrapper Only**: Simple `record()` function is just a wrapper, no new dependencies
347
 
348
+ #### Demo Dependencies (To Be Moved)
349
 
350
+ - **Video/Export**: `parquet-wasm`, `apache-arrow`, `jszip` - move to examples
351
+ - **Upload**: `@huggingface/hub`, AWS SDK - move to examples
352
+ - **Keep Available**: Advanced users can still import `LeRobotDatasetRecorder` for full features
353
 
354
  ### Core Functions to Implement
355
 
356
+ #### Simple Record API (Wrapper)
357
 
358
  ```typescript
359
+ // record.ts - Simple wrapper around existing LeRobotDatasetRecorder
360
  interface RecordConfig {
361
+ teleoperator: WebTeleoperator; // ← Explicit dependency (keep this!)
362
  options?: {
363
  fps?: number; // Default: 30
364
+ taskDescription?: string;
365
  onDataUpdate?: (data: RecordingData) => void;
366
  onStateUpdate?: (state: RecordingState) => void;
367
  };
 
385
  interface RecordingData {
386
  frameCount: number;
387
  currentEpisode: number;
388
+ recentFrames: any[]; // Simplified for basic API
389
  }
390
 
391
  interface RobotRecordingData {
392
+ episodes: any[]; // Pure motor data only (no video)
393
  metadata: {
394
  fps: number;
395
  robotType: string;
 
400
  };
401
  }
402
 
403
+ // Simple wrapper function - internally uses LeRobotDatasetRecorder
404
+ // Preserves the excellent explicit dependency architecture
405
  export async function record(config: RecordConfig): Promise<RecordProcess>;
406
  ```
407
 
408
+ #### Implementation Strategy
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
409
 
410
+ **Phase 1: Simple Wrapper (Preserves Current Architecture)**
 
411
 
412
+ ```typescript
413
+ // record.ts - Simple wrapper implementation
414
+ import { LeRobotDatasetRecorder } from "./record-class.js";
415
+
416
+ export async function record(config: RecordConfig): Promise<RecordProcess> {
417
+ // Use the provided teleoperator (explicit dependency - good!)
418
+ const recorder = new LeRobotDatasetRecorder(
419
+ [config.teleoperator], // ← Use explicit teleoperator dependency
420
+ {}, // No video streams in simple API (move to demo)
421
+ config.options?.fps || 30,
422
+ config.options?.taskDescription || "Robot recording"
423
+ );
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
424
 
425
+ return {
426
+ start: () => {
427
+ recorder.startRecording();
428
+ if (config.options?.onStateUpdate) {
429
+ // Set up state update polling for simple API
430
+ const updateLoop = () => {
431
+ if (recorder.isRecording) {
432
+ config.options.onStateUpdate!({
433
+ isActive: recorder.isRecording,
434
+ frameCount: recorder.teleoperatorData.length,
435
+ episodeCount: recorder.teleoperatorData.length,
436
+ duration: Date.now() - (recorder as any).startTime,
437
+ lastUpdate: Date.now(),
438
+ });
439
+ setTimeout(updateLoop, 100);
440
+ }
441
+ };
442
+ updateLoop();
443
+ }
444
+ },
445
+ stop: () => {
446
+ return recorder.stopRecording();
447
+ },
448
+ getState: () => ({
449
+ isActive: recorder.isRecording,
450
+ frameCount: recorder.teleoperatorData.length,
451
+ episodeCount: recorder.teleoperatorData.length,
452
+ duration: 0, // Calculate from recorder
453
  lastUpdate: Date.now(),
454
+ }),
455
+ result: recorder.stopRecording().then(() => ({
456
+ episodes: recorder.episodes, // Pure motor data
 
 
 
457
  metadata: {
458
+ fps: config.options?.fps || 30,
459
+ robotType: "unknown", // Get from teleoperator if possible
460
+ startTime: Date.now(),
461
  endTime: Date.now(),
462
+ totalFrames: recorder.teleoperatorData.length,
463
+ totalEpisodes: recorder.teleoperatorData.length,
464
  },
465
+ })),
466
+ };
467
  }
468
  ```
469
 
470
+ **Key Benefits of This Approach:**
471
+
472
+ - ✅ **Preserves Explicit Dependencies**: Keeps the excellent `teleoperator` parameter
473
+ - ✅ **Minimal Changes**: Just wraps existing `LeRobotDatasetRecorder`
474
+ - ✅ **No Breaking Changes**: Current class remains available
475
+ - ✅ **Consistent API**: Follows `start()`, `stop()`, `getState()`, `result` pattern
476
+
477
  #### Updated Teleoperate Integration
478
 
479
  ```typescript
 
562
 
563
  ## Definition of Done
564
 
565
+ ### Phase 1: Simple Function API (Priority)
566
+
567
+ - [ ] **Clean Record API**: `record(config)` function implemented as wrapper around existing `LeRobotDatasetRecorder`
568
  - [ ] **Process Interface**: `RecordProcess` with consistent `start()`, `stop()`, `getState()`, `result` methods
569
+ - [ ] **Hardware-Only Simple API**: Simple `record()` function captures only robot motor data (no video)
570
+ - [ ] **Preserve Advanced Features**: Keep existing `LeRobotDatasetRecorder` class available for full functionality
571
+ - [ ] **TypeScript Coverage**: Full type safety with proper interfaces for simple recording API
572
+
573
+ ### Phase 2: UI Separation (Secondary)
574
+
575
+ - [ ] **Separate Recording Component**: Move recording from teleoperation view to dedicated component/page
576
+ - [ ] **Clean Navigation**: Add recording as separate section in cyberpunk example navigation
577
+ - [ ] **No Breaking Changes**: Existing functionality continues to work during transition
578
+
579
+ ### Phase 3: Library/Demo Boundary (Future)
580
+
581
+ - [ ] **Demo Separation**: Video recording, export formats moved to examples layer (optional enhancement)
582
+ - [ ] **Advanced Export Demo**: Create demo showing complex export features using `LeRobotDatasetRecorder`
583
+ - [ ] **Documentation**: Clear examples showing simple API vs advanced class usage
584
+
585
+ ### Success Criteria
586
+
587
+ - [ ] **API Consistency**: `record()` function follows same patterns as `calibrate()` and `teleoperate()`
588
+ - [ ] **No Regression**: All existing recording functionality preserved and working
589
+ - [ ] **Easy Migration**: Users can easily switch between simple API and advanced class
590
+ - [ ] **Clean Example**: Recording has its own dedicated UI section in cyberpunk demo
examples/cyberpunk-standalone/src/App.tsx CHANGED
@@ -13,6 +13,7 @@ import { EditRobotDialog } from "@/components/edit-robot-dialog";
13
  import { DeviceDashboard } from "@/components/device-dashboard";
14
  import { CalibrationView } from "@/components/calibration-view";
15
  import { TeleoperationView } from "@/components/teleoperation-view";
 
16
  import { SetupCards } from "@/components/setup-cards";
17
  import { DocsSection } from "@/components/docs-section";
18
  import { RoadmapSection } from "@/components/roadmap-section";
@@ -225,31 +226,17 @@ function App() {
225
  };
226
 
227
  const handleCalibrate = (robot: RobotConnection) => {
228
- if (!robot.isConnected) {
229
- toast({
230
- title: "Robot Not Connected",
231
- description: "Please connect the robot before calibrating",
232
- variant: "destructive",
233
- });
234
- return;
235
- }
236
-
237
  navigate(`/device/${robot.serialNumber}/calibrate`);
238
  };
239
 
240
  const handleTeleoperate = (robot: RobotConnection) => {
241
- if (!robot.isConnected) {
242
- toast({
243
- title: "Robot Not Connected",
244
- description: "Please connect the robot before teleoperating",
245
- variant: "destructive",
246
- });
247
- return;
248
- }
249
-
250
  navigate(`/device/${robot.serialNumber}/control`);
251
  };
252
 
 
 
 
 
253
  const handleBackToDashboard = () => {
254
  navigate("/");
255
  };
@@ -270,6 +257,7 @@ function App() {
270
  robots={robots}
271
  onCalibrate={handleCalibrate}
272
  onTeleoperate={handleTeleoperate}
 
273
  onRemove={handleRemoveRobot}
274
  onEdit={setEditingRobot}
275
  onFindNew={handleFindNewRobots}
@@ -297,6 +285,15 @@ function App() {
297
  />
298
  }
299
  />
 
 
 
 
 
 
 
 
 
300
  </Routes>
301
  <EditRobotDialog
302
  robot={editingRobot}
@@ -316,6 +313,7 @@ function DashboardPage({
316
  robots,
317
  onCalibrate,
318
  onTeleoperate,
 
319
  onRemove,
320
  onEdit,
321
  onFindNew,
@@ -326,6 +324,7 @@ function DashboardPage({
326
  robots: RobotConnection[];
327
  onCalibrate: (robot: RobotConnection) => void;
328
  onTeleoperate: (robot: RobotConnection) => void;
 
329
  onRemove: (robot: RobotConnection) => void;
330
  onEdit: (robot: RobotConnection | null) => void;
331
  onFindNew: () => void;
@@ -341,6 +340,7 @@ function DashboardPage({
341
  robots={robots}
342
  onCalibrate={onCalibrate}
343
  onTeleoperate={onTeleoperate}
 
344
  onRemove={onRemove}
345
  onEdit={onEdit}
346
  onFindNew={onFindNew}
@@ -442,6 +442,43 @@ function ControlPage({
442
  );
443
  }
444
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
445
  // Page Header Component
446
  function PageHeader({
447
  onBackToDashboard,
@@ -454,6 +491,7 @@ function PageHeader({
454
  const isDashboard = location.pathname === "/";
455
  const isCalibrating = location.pathname.includes("/calibrate");
456
  const isTeleoperating = location.pathname.includes("/control");
 
457
 
458
  return (
459
  <div className="flex items-center justify-between mb-12">
@@ -483,6 +521,16 @@ function PageHeader({
483
  {selectedRobot.robotId?.toUpperCase()}
484
  </span>
485
  </h1>
 
 
 
 
 
 
 
 
 
 
486
  ) : (
487
  <h1
488
  className="font-mono text-4xl font-bold text-primary tracking-wider text-glitch uppercase"
 
13
  import { DeviceDashboard } from "@/components/device-dashboard";
14
  import { CalibrationView } from "@/components/calibration-view";
15
  import { TeleoperationView } from "@/components/teleoperation-view";
16
+ import { RecordingView } from "@/components/recording-view";
17
  import { SetupCards } from "@/components/setup-cards";
18
  import { DocsSection } from "@/components/docs-section";
19
  import { RoadmapSection } from "@/components/roadmap-section";
 
226
  };
227
 
228
  const handleCalibrate = (robot: RobotConnection) => {
 
 
 
 
 
 
 
 
 
229
  navigate(`/device/${robot.serialNumber}/calibrate`);
230
  };
231
 
232
  const handleTeleoperate = (robot: RobotConnection) => {
 
 
 
 
 
 
 
 
 
233
  navigate(`/device/${robot.serialNumber}/control`);
234
  };
235
 
236
+ const handleRecord = (robot: RobotConnection) => {
237
+ navigate(`/device/${robot.serialNumber}/record`);
238
+ };
239
+
240
  const handleBackToDashboard = () => {
241
  navigate("/");
242
  };
 
257
  robots={robots}
258
  onCalibrate={handleCalibrate}
259
  onTeleoperate={handleTeleoperate}
260
+ onRecord={handleRecord}
261
  onRemove={handleRemoveRobot}
262
  onEdit={setEditingRobot}
263
  onFindNew={handleFindNewRobots}
 
285
  />
286
  }
287
  />
288
+ <Route
289
+ path="/device/:serialNumber/record"
290
+ element={
291
+ <RecordPage
292
+ robots={robots}
293
+ onBackToDashboard={handleBackToDashboard}
294
+ />
295
+ }
296
+ />
297
  </Routes>
298
  <EditRobotDialog
299
  robot={editingRobot}
 
313
  robots,
314
  onCalibrate,
315
  onTeleoperate,
316
+ onRecord,
317
  onRemove,
318
  onEdit,
319
  onFindNew,
 
324
  robots: RobotConnection[];
325
  onCalibrate: (robot: RobotConnection) => void;
326
  onTeleoperate: (robot: RobotConnection) => void;
327
+ onRecord: (robot: RobotConnection) => void;
328
  onRemove: (robot: RobotConnection) => void;
329
  onEdit: (robot: RobotConnection | null) => void;
330
  onFindNew: () => void;
 
340
  robots={robots}
341
  onCalibrate={onCalibrate}
342
  onTeleoperate={onTeleoperate}
343
+ onRecord={onRecord}
344
  onRemove={onRemove}
345
  onEdit={onEdit}
346
  onFindNew={onFindNew}
 
442
  );
443
  }
444
 
445
+ // Record Page Component
446
+ function RecordPage({
447
+ robots,
448
+ onBackToDashboard,
449
+ }: {
450
+ robots: RobotConnection[];
451
+ onBackToDashboard: () => void;
452
+ }) {
453
+ const { serialNumber } = useParams<{ serialNumber: string }>();
454
+ const selectedRobot = robots.find(
455
+ (robot) => robot.serialNumber === serialNumber
456
+ );
457
+
458
+ if (!selectedRobot) {
459
+ return (
460
+ <div>
461
+ <PageHeader onBackToDashboard={onBackToDashboard} />
462
+ <div className="text-center py-20">
463
+ <p className="text-muted-foreground">
464
+ Device not found or not connected.
465
+ </p>
466
+ </div>
467
+ </div>
468
+ );
469
+ }
470
+
471
+ return (
472
+ <div>
473
+ <PageHeader
474
+ onBackToDashboard={onBackToDashboard}
475
+ selectedRobot={selectedRobot}
476
+ />
477
+ <RecordingView robot={selectedRobot} />
478
+ </div>
479
+ );
480
+ }
481
+
482
  // Page Header Component
483
  function PageHeader({
484
  onBackToDashboard,
 
491
  const isDashboard = location.pathname === "/";
492
  const isCalibrating = location.pathname.includes("/calibrate");
493
  const isTeleoperating = location.pathname.includes("/control");
494
+ const isRecording = location.pathname.includes("/record");
495
 
496
  return (
497
  <div className="flex items-center justify-between mb-12">
 
521
  {selectedRobot.robotId?.toUpperCase()}
522
  </span>
523
  </h1>
524
+ ) : isRecording && selectedRobot ? (
525
+ <h1 className="font-mono text-4xl font-bold tracking-wider">
526
+ <span className="text-muted-foreground uppercase">record:</span>{" "}
527
+ <span
528
+ className="text-primary text-glitch uppercase"
529
+ data-text={selectedRobot.robotId}
530
+ >
531
+ {selectedRobot.robotId?.toUpperCase()}
532
+ </span>
533
+ </h1>
534
  ) : (
535
  <h1
536
  className="font-mono text-4xl font-bold text-primary tracking-wider text-glitch uppercase"
examples/cyberpunk-standalone/src/components/calibration-view.tsx CHANGED
@@ -3,6 +3,7 @@ import { useState, useMemo, useEffect, useCallback } from "react";
3
  import { Download } from "lucide-react";
4
  import { Button } from "@/components/ui/button";
5
  import { Card } from "@/components/ui/card";
 
6
  import {
7
  AlertDialog,
8
  AlertDialogAction,
@@ -28,6 +29,7 @@ import {
28
  type CalibrationMetadata,
29
  } from "@/lib/unified-storage";
30
  import { MotorCalibrationVisual } from "@/components/motor-calibration-visual";
 
31
 
32
  interface CalibrationViewProps {
33
  robot: RobotConnection;
@@ -281,32 +283,50 @@ export function CalibrationView({ robot }: CalibrationViewProps) {
281
  </p>
282
  </div>
283
  </div>
284
- <div className="flex gap-4">
285
- {!isCalibrating ? (
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
286
  <Button
287
- onClick={handleStart}
 
288
  size="lg"
289
- disabled={isPreparing || !robot.isConnected}
290
  >
291
- {isPreparing
292
- ? "Preparing..."
293
- : calibrationResults
294
- ? "Re-calibrate"
295
- : "Start Calibration"}
296
  </Button>
297
- ) : (
298
- <Button onClick={handleFinish} variant="destructive" size="lg">
299
- Finish Recording
300
- </Button>
301
- )}
302
- <Button
303
- onClick={downloadJson}
304
- variant="outline"
305
- size="lg"
306
- disabled={!calibrationResults}
307
- >
308
- <Download className="w-4 h-4 mr-2" /> Download JSON
309
- </Button>
310
  </div>
311
  </div>
312
  <div className="pt-6 p-6">
 
3
  import { Download } from "lucide-react";
4
  import { Button } from "@/components/ui/button";
5
  import { Card } from "@/components/ui/card";
6
+ import { Badge } from "@/components/ui/badge";
7
  import {
8
  AlertDialog,
9
  AlertDialogAction,
 
29
  type CalibrationMetadata,
30
  } from "@/lib/unified-storage";
31
  import { MotorCalibrationVisual } from "@/components/motor-calibration-visual";
32
+ import { cn } from "@/lib/utils";
33
 
34
  interface CalibrationViewProps {
35
  robot: RobotConnection;
 
283
  </p>
284
  </div>
285
  </div>
286
+ <div className="flex items-center gap-4">
287
+ <div className="flex items-center gap-2">
288
+ <span className="text-sm font-mono text-muted-foreground uppercase">
289
+ robot status:
290
+ </span>
291
+ <Badge
292
+ variant="outline"
293
+ className={cn(
294
+ "border-primary/50 bg-primary/20 text-primary font-mono text-xs",
295
+ robot.isConnected
296
+ ? "border-green-500/50 bg-green-500/20 text-green-400"
297
+ : "border-red-500/50 bg-red-500/20 text-red-400"
298
+ )}
299
+ >
300
+ {robot.isConnected ? "ONLINE" : "OFFLINE"}
301
+ </Badge>
302
+ </div>
303
+ <div className="flex gap-4">
304
+ {!isCalibrating ? (
305
+ <Button
306
+ onClick={handleStart}
307
+ size="lg"
308
+ disabled={isPreparing || !robot.isConnected}
309
+ >
310
+ {isPreparing
311
+ ? "Preparing..."
312
+ : calibrationResults
313
+ ? "Re-calibrate"
314
+ : "Start Calibration"}
315
+ </Button>
316
+ ) : (
317
+ <Button onClick={handleFinish} variant="destructive" size="lg">
318
+ Finish Recording
319
+ </Button>
320
+ )}
321
  <Button
322
+ onClick={downloadJson}
323
+ variant="outline"
324
  size="lg"
325
+ disabled={!calibrationResults}
326
  >
327
+ <Download className="w-4 h-4 mr-2" /> Download JSON
 
 
 
 
328
  </Button>
329
+ </div>
 
 
 
 
 
 
 
 
 
 
 
 
330
  </div>
331
  </div>
332
  <div className="pt-6 p-6">
examples/cyberpunk-standalone/src/components/device-dashboard.tsx CHANGED
@@ -8,6 +8,7 @@ import {
8
  Pencil,
9
  Plus,
10
  ExternalLink,
 
11
  } from "lucide-react";
12
  import { Button } from "@/components/ui/button";
13
  import { Card, CardFooter } from "@/components/ui/card";
@@ -31,6 +32,7 @@ interface DeviceDashboardProps {
31
  robots: RobotConnection[];
32
  onCalibrate: (robot: RobotConnection) => void;
33
  onTeleoperate: (robot: RobotConnection) => void;
 
34
  onRemove: (robotId: string) => void;
35
  onEdit: (robot: RobotConnection) => void;
36
  onFindNew: () => void;
@@ -42,6 +44,7 @@ export function DeviceDashboard({
42
  robots,
43
  onCalibrate,
44
  onTeleoperate,
 
45
  onRemove,
46
  onEdit,
47
  onFindNew,
@@ -162,7 +165,9 @@ export function DeviceDashboard({
162
  variant="outline"
163
  className={cn(
164
  "border-primary/50 bg-primary/20 text-primary font-mono text-xs",
165
- robot.isConnected && "animate-pulse-slow"
 
 
166
  )}
167
  >
168
  {robot.isConnected ? "ONLINE" : "OFFLINE"}
@@ -220,6 +225,14 @@ export function DeviceDashboard({
220
  >
221
  <Gamepad2 className="w-3 h-3 mr-0.5" /> control
222
  </Button>
 
 
 
 
 
 
 
 
223
  <Button
224
  variant="destructive"
225
  size="sm"
 
8
  Pencil,
9
  Plus,
10
  ExternalLink,
11
+ Disc,
12
  } from "lucide-react";
13
  import { Button } from "@/components/ui/button";
14
  import { Card, CardFooter } from "@/components/ui/card";
 
32
  robots: RobotConnection[];
33
  onCalibrate: (robot: RobotConnection) => void;
34
  onTeleoperate: (robot: RobotConnection) => void;
35
+ onRecord: (robot: RobotConnection) => void;
36
  onRemove: (robotId: string) => void;
37
  onEdit: (robot: RobotConnection) => void;
38
  onFindNew: () => void;
 
44
  robots,
45
  onCalibrate,
46
  onTeleoperate,
47
+ onRecord,
48
  onRemove,
49
  onEdit,
50
  onFindNew,
 
165
  variant="outline"
166
  className={cn(
167
  "border-primary/50 bg-primary/20 text-primary font-mono text-xs",
168
+ robot.isConnected
169
+ ? "border-green-500/50 bg-green-500/20 text-green-400 animate-pulse-slow"
170
+ : "border-red-500/50 bg-red-500/20 text-red-400"
171
  )}
172
  >
173
  {robot.isConnected ? "ONLINE" : "OFFLINE"}
 
225
  >
226
  <Gamepad2 className="w-3 h-3 mr-0.5" /> control
227
  </Button>
228
+ <Button
229
+ variant="outline"
230
+ size="sm"
231
+ onClick={() => onRecord(robot)}
232
+ className="font-mono text-xs uppercase px-2"
233
+ >
234
+ <Disc className="w-3 h-3 mr-0.5" /> record
235
+ </Button>
236
  <Button
237
  variant="destructive"
238
  size="sm"
examples/cyberpunk-standalone/src/components/docs-section.tsx CHANGED
@@ -592,6 +592,113 @@ await releaseMotors(robot, [1, 2, 3]);`}
592
  </ul>
593
  </div>
594
  </div>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
595
  </div>
596
  </div>
597
  </TabsContent>
 
592
  </ul>
593
  </div>
594
  </div>
595
+
596
+ {/* record */}
597
+ <div>
598
+ <h4 className="font-bold text-primary">record(config)</h4>
599
+ <p className="text-sm mt-1">
600
+ Records robot teleoperator data and camera streams in the LeRobot dataset format.
601
+ </p>
602
+ <CodeBlock>
603
+ {`import { teleoperate, record } from "@lerobot/web";
604
+
605
+ // Start teleoperation first
606
+ const teleoperationProcess = await teleoperate({
607
+ robot: connectedRobot,
608
+ teleop: { type: "keyboard" },
609
+ calibrationData: calibrationData,
610
+ });
611
+
612
+ // Create recording with teleoperator
613
+ const recordProcess = await record({
614
+ teleoperator: teleoperationProcess.teleoperator,
615
+ videoStreams: {
616
+ main: mainCameraStream,
617
+ wrist: wristCameraStream,
618
+ },
619
+ robotType: "so100",
620
+ options: {
621
+ fps: 30,
622
+ taskDescription: "Pick and place task",
623
+ onStateUpdate: (state) => {
624
+ console.log(\`Recording: \${state.frameCount} frames\`);
625
+ },
626
+ },
627
+ });
628
+
629
+ // Start both recording and teleoperation
630
+ teleoperationProcess.start();
631
+ recordProcess.start();
632
+
633
+ // Manage recording during operation
634
+ recordProcess.nextEpisode(); // Start new episode if needed
635
+
636
+ // Stop recording when finished
637
+ const robotData = await recordProcess.stop();
638
+ await recordProcess.exportForLeRobot("zip-download");`}
639
+ </CodeBlock>
640
+ <div className="mt-3">
641
+ <h5 className="font-bold text-sm tracking-wider">
642
+ Options
643
+ </h5>
644
+ <ul className="mt-1 ml-4 space-y-1 text-sm">
645
+ <li>
646
+ • <code>teleoperator: WebTeleoperator</code> - The teleoperator to record from
647
+ </li>
648
+ <li>
649
+ • <code>videoStreams?: {`{ [name: string]: MediaStream }`}</code> - Optional camera streams
650
+ </li>
651
+ <li>
652
+ • <code>robotType?: string</code> - Robot metadata (e.g., "so100")
653
+ </li>
654
+ <li>
655
+ • <code>options?: RecordOptions</code> - Optional configuration:
656
+ <ul className="mt-1 ml-4 space-y-1">
657
+ <li>
658
+ • <code>fps?: number</code> - Target frames per second (default: 30)
659
+ </li>
660
+ <li>
661
+ • <code>taskDescription?: string</code> - Task description
662
+ </li>
663
+ <li>
664
+ • <code>onStateUpdate?: (state: RecordingState) =&gt; void</code> - State changes
665
+ </li>
666
+ </ul>
667
+ </li>
668
+ </ul>
669
+ </div>
670
+ <div className="mt-3">
671
+ <h5 className="font-bold text-sm tracking-wider">
672
+ Returns: RecordProcess
673
+ </h5>
674
+ <ul className="mt-1 ml-4 space-y-1 text-sm">
675
+ <li>
676
+ • <code>start(): void</code> - Start recording
677
+ </li>
678
+ <li>
679
+ • <code>stop(): Promise&lt;RobotRecordingData&gt;</code> - Stop recording
680
+ </li>
681
+ <li>
682
+ • <code>getState(): RecordingState</code> - Current recording state
683
+ </li>
684
+ <li>
685
+ • <code>getEpisodeCount(): number</code> - Get number of episodes
686
+ </li>
687
+ <li>
688
+ • <code>nextEpisode(): Promise&lt;number&gt;</code> - Start new episode
689
+ </li>
690
+ <li>
691
+ • <code>clearEpisodes(): void</code> - Delete all episodes
692
+ </li>
693
+ <li>
694
+ • <code>addCamera(name: string, stream: MediaStream): void</code> - Add camera dynamically
695
+ </li>
696
+ <li>
697
+ • <code>exportForLeRobot(format?: "blobs" | "zip" | "zip-download"): Promise&lt;any&gt;</code> - Export dataset
698
+ </li>
699
+ </ul>
700
+ </div>
701
+ </div>
702
  </div>
703
  </div>
704
  </TabsContent>
examples/cyberpunk-standalone/src/components/recorder.tsx CHANGED
@@ -1,17 +1,7 @@
1
  "use client";
2
 
3
- import { useState, useEffect, useRef, useCallback, useMemo } from "react";
4
  import { Button } from "@/components/ui/button";
5
- import { Card } from "@/components/ui/card";
6
- import {
7
- Table,
8
- TableBody,
9
- TableCaption,
10
- TableCell,
11
- TableHead,
12
- TableHeader,
13
- TableRow,
14
- } from "@/components/ui/table";
15
  import { Input } from "@/components/ui/input";
16
  import { Badge } from "@/components/ui/badge";
17
  import {
@@ -36,20 +26,38 @@ import {
36
  Edit2,
37
  Check,
38
  } from "lucide-react";
39
- import { LeRobotDatasetRecorder, LeRobotDatasetRow, NonIndexedLeRobotDatasetRow, LeRobotEpisode } from "@lerobot/web";
 
 
 
 
 
 
 
 
 
 
 
40
  import { TeleoperatorEpisodesView } from "./teleoperator-episodes-view";
 
41
 
42
  interface RecorderProps {
43
  teleoperators: any[];
44
  robot: any; // eslint-disable-line @typescript-eslint/no-explicit-any
45
  onNeedsTeleoperation: () => Promise<boolean>;
 
 
 
 
 
 
46
  videoStreams?: { [key: string]: MediaStream };
47
  }
48
 
49
-
50
-
51
  interface RecorderSettings {
52
  huggingfaceApiKey: string;
 
 
53
  cameraConfigs: {
54
  [cameraName: string]: {
55
  deviceId: string;
@@ -72,6 +80,8 @@ function getRecorderSettings(): RecorderSettings {
72
  }
73
  return {
74
  huggingfaceApiKey: "",
 
 
75
  cameraConfigs: {},
76
  };
77
  }
@@ -88,9 +98,15 @@ export function Recorder({
88
  teleoperators,
89
  robot,
90
  onNeedsTeleoperation,
 
 
91
  }: RecorderProps) {
92
  const [isRecording, setIsRecording] = useState(false);
93
  const [currentEpisode, setCurrentEpisode] = useState(0);
 
 
 
 
94
  // Use huggingfaceApiKey from recorderSettings instead of separate state
95
  const [cameraName, setCameraName] = useState("");
96
  const [additionalCameras, setAdditionalCameras] = useState<{
@@ -105,8 +121,7 @@ export function Recorder({
105
  const [cameraPermissionState, setCameraPermissionState] = useState<
106
  "unknown" | "granted" | "denied"
107
  >("unknown");
108
- const [showCameraConfig, setShowCameraConfig] = useState(false);
109
- const [showConfigure, setShowConfigure] = useState(false);
110
  const [recorderSettings, setRecorderSettings] = useState<RecorderSettings>(
111
  () => getRecorderSettings()
112
  );
@@ -115,68 +130,106 @@ export function Recorder({
115
  null
116
  );
117
  const [editingCameraNewName, setEditingCameraNewName] = useState("");
118
- const [huggingfaceApiKey, setHuggingfaceApiKey] = useState("")
119
- const recorderRef = useRef<LeRobotDatasetRecorder | null>(null);
 
 
 
 
 
 
120
  const videoRef = useRef<HTMLVideoElement | null>(null);
121
  const { toast } = useToast();
122
 
123
  // Initialize the recorder when teleoperators are available
124
  useEffect(() => {
125
- if (teleoperators.length > 0) {
126
- recorderRef.current = new LeRobotDatasetRecorder(
127
- teleoperators,
128
- additionalCameras,
129
- 30, // fps
130
- "Robot teleoperation recording"
131
- );
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
132
  }
133
- }, [teleoperators, additionalCameras]);
134
 
135
- const handleStartRecording = async () => {
136
- // If teleoperators aren't available, initialize teleoperation first
137
- if (teleoperators.length === 0) {
138
- toast({
139
- title: "Initializing...",
140
- description: `Setting up robot control for ${robot.robotId || "robot"}`,
141
- });
142
 
143
- const success = await onNeedsTeleoperation();
144
- if (!success) {
145
- toast({
146
- title: "Recording Error",
147
- description: "Failed to initialize robot control",
148
- variant: "destructive",
149
- });
150
- return;
151
  }
 
 
152
 
153
- // Wait a moment for the recorder to initialize with new teleoperators
154
- await new Promise((resolve) => setTimeout(resolve, 100));
 
 
 
 
 
 
155
  }
 
156
 
157
- if (!recorderRef.current) {
 
 
158
  toast({
159
  title: "Recording Error",
160
- description: "Recorder not ready yet. Please try again.",
161
  variant: "destructive",
162
  });
163
  return;
164
  }
165
 
166
  try {
167
- // Set the episode index
168
- recorderRef.current.setEpisodeIndex(currentEpisode);
169
- recorderRef.current.setTaskIndex(0); // Default task index
170
-
171
  // Start recording
172
- recorderRef.current.startRecording();
173
  setIsRecording(true);
174
  setHasRecordedFrames(true);
175
 
176
- toast({
177
- title: "Recording Started",
178
- description: `Episode ${currentEpisode} is now recording`,
179
- });
180
  } catch (error) {
181
  const errorMessage =
182
  error instanceof Error ? error.message : "Failed to start recording";
@@ -189,18 +242,18 @@ export function Recorder({
189
  };
190
 
191
  const handleStopRecording = async () => {
192
- if (!recorderRef.current || !isRecording) {
193
  return;
194
  }
195
 
196
  try {
197
- const result = await recorderRef.current.stopRecording();
198
  setIsRecording(false);
199
 
200
- toast({
201
- title: "Recording Stopped",
202
- description: `Episode ${currentEpisode} completed with ${result.teleoperatorData.length} frames`,
203
- });
204
  } catch (error) {
205
  const errorMessage =
206
  error instanceof Error ? error.message : "Failed to stop recording";
@@ -212,37 +265,47 @@ export function Recorder({
212
  }
213
  };
214
 
215
- const handleNextEpisode = () => {
216
- // Make sure we're not recording
217
- if (isRecording) {
218
- handleStopRecording();
219
  }
220
 
221
- // Increment episode counter
222
- setCurrentEpisode((prev) => prev + 1);
 
 
223
 
224
- toast({
225
- title: "New Episode",
226
- description: `Ready to record episode ${currentEpisode + 1}`,
227
- });
 
228
  };
229
 
230
- // Reset frames by clearing the recorder data
231
- const handleResetFrames = useCallback(() => {
232
- if (isRecording) {
233
- handleStopRecording();
234
  }
235
 
236
- if (recorderRef.current) {
237
- recorderRef.current.clearRecording();
238
- setHasRecordedFrames(false);
239
-
240
- toast({
241
- title: "Frames Reset",
242
- description: "All recorded frames have been cleared",
243
  });
244
- }
245
- }, [isRecording, toast]);
 
 
 
 
 
 
 
 
 
 
246
 
247
  // Load available cameras
248
  const loadAvailableCameras = useCallback(
@@ -375,6 +438,79 @@ export function Recorder({
375
  [previewStream, toast]
376
  );
377
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
378
  // Add a new camera to the recorder
379
  const handleAddCamera = useCallback(async () => {
380
  if (!cameraName.trim()) {
@@ -622,31 +758,26 @@ export function Recorder({
622
  }, [availableCameras, cameraPermissionState, restoreSavedCameras]);
623
 
624
  const handleDownloadZip = async () => {
625
- if (!recorderRef.current) {
 
 
 
 
 
 
626
  toast({
627
- title: "Download Error",
628
- description: "Recorder not initialized",
 
629
  variant: "destructive",
630
  });
631
  return;
632
  }
633
-
634
- await recorderRef.current.exportForLeRobot("zip-download");
635
- toast({
636
- title: "Download Started",
637
- description: "Your dataset is being downloaded as a ZIP file",
638
- });
639
  };
640
 
641
  const handleUploadToHuggingFace = async () => {
642
- if (!recorderRef.current) {
643
- toast({
644
- title: "Upload Error",
645
- description: "Recorder not initialized",
646
- variant: "destructive",
647
- });
648
- return;
649
- }
650
 
651
  if (!recorderSettings.huggingfaceApiKey) {
652
  toast({
@@ -658,25 +789,36 @@ export function Recorder({
658
  }
659
 
660
  try {
661
- toast({
662
- title: "Upload Started",
663
- description: "Uploading dataset to Hugging Face...",
664
- });
665
-
666
- // Generate a unique repository name
667
- const repoName = `lerobot-recording-${Date.now()}`;
 
 
668
 
669
- const uploader = await recorderRef.current.exportForLeRobot(
670
- "huggingface",
671
- {
672
- repoName,
673
- accessToken: recorderSettings.huggingfaceApiKey,
674
- }
675
  );
676
 
677
- uploader.addEventListener("progress", (event: Event) => {
678
- console.log(event);
679
- });
 
 
 
 
 
 
 
 
 
680
  } catch (error) {
681
  const errorMessage =
682
  error instanceof Error
@@ -690,444 +832,212 @@ export function Recorder({
690
  }
691
  };
692
 
693
- // Helper function to format duration
694
- const formatDuration = (seconds: number): string => {
695
- const mins = Math.floor(seconds / 60);
696
- const secs = Math.floor(seconds % 60);
697
- return `${mins.toString().padStart(2, "0")}:${secs
698
- .toString()
699
- .padStart(2, "0")}`;
700
- };
701
-
702
  return (
703
- <Card className="border-0 rounded-none mt-6">
704
- <div className="p-4 border-b border-white/10">
705
- <div className="flex items-center justify-between">
706
- <div className="flex items-center gap-4">
707
- <div className="w-1 h-8 bg-primary"></div>
708
- <div>
709
- <h3 className="text-xl font-bold text-foreground font-mono tracking-wider uppercase">
710
- robot movement recorder
711
- </h3>
712
- <p className="text-sm text-muted-foreground font-mono">
713
- dataset <span className="text-muted-foreground">recording</span>{" "}
714
- interface
715
- </p>
716
- </div>
717
- </div>
718
- <div className="flex items-center gap-6">
719
- <div className="border-l border-white/10 pl-6 flex items-center gap-4">
720
- <Button
721
- variant="outline"
722
- size="lg"
723
- className="gap-2"
724
- onClick={() => setShowConfigure(!showConfigure)}
725
- >
726
- <Settings className="w-5 h-5" />
727
- Configure
728
- </Button>
729
-
730
- <Button
731
- variant={isRecording ? "destructive" : "default"}
732
- size="lg"
733
- className="gap-2"
734
- onClick={
735
- isRecording ? handleStopRecording : handleStartRecording
736
- }
737
- >
738
- {isRecording ? (
739
- <>
740
- <Square className="w-5 h-5" />
741
- Stop Recording
742
- </>
743
- ) : (
744
- <>
745
- <Record className="w-5 h-5" />
746
- Start Recording
747
- </>
748
- )}
749
- </Button>
750
- </div>
751
- </div>
752
- </div>
753
- </div>
754
-
755
- <div className="p-6 space-y-6">
756
- {/* Recorder Settings - Toggleable Inline */}
757
- {showConfigure && (
758
- <div className="space-y-6">
759
- {/* Hugging Face Settings */}
760
- <div className="space-y-3">
761
- <h3 className="text-lg font-semibold text-foreground">
762
- Settings
763
- </h3>
764
- <div className="grid grid-cols-1 md:grid-cols-2 gap-4">
765
- <div className="space-y-2">
766
- <label className="text-sm text-muted-foreground">
767
- Hugging Face API Key
768
- </label>
769
- <Input
770
- placeholder="Enter your Hugging Face API key"
771
- value={recorderSettings.huggingfaceApiKey}
772
  onChange={(e) => {
773
  const newSettings = {
774
  ...recorderSettings,
775
- huggingfaceApiKey: e.target.value,
776
  };
777
  setRecorderSettings(newSettings);
778
  saveRecorderSettings(newSettings);
779
  }}
780
- type="password"
781
- className="bg-black/20 border-white/10"
782
  />
783
- <p className="text-xs text-white/50">
784
- Required to upload datasets to Hugging Face Hub
785
- </p>
786
- </div>
787
  </div>
788
  </div>
 
789
 
790
- {/* Camera Configuration */}
791
- <div className="space-y-4">
792
- <h3 className="text-lg font-semibold text-foreground">
793
- Camera Setup
794
- </h3>
795
- <div className="bg-black/40 border border-white/20 p-6">
796
- <div className="grid grid-cols-1 lg:grid-cols-2 gap-6">
797
- {/* Left Column: Camera Selection & Adding */}
798
- <div className="space-y-4">
799
- {/* Camera Selection and Refresh */}
800
-
801
- {/* Camera Access Request */}
802
- {cameraPermissionState === "unknown" && (
803
- <div className="space-y-3">
804
- <div className="bg-black/60 border border-white/20 rounded-lg p-4 text-center">
805
- <Camera className="w-8 h-8 mx-auto mb-2 opacity-50" />
806
- <p className="text-sm text-white/70 mb-3">
807
- Camera access needed to configure cameras
808
- </p>
809
- <Button
810
- onClick={() => loadAvailableCameras(false)}
811
- variant="outline"
812
- className="gap-2"
813
- disabled={isLoadingCameras}
814
- >
815
- <Camera className="w-4 h-4" />
816
- {isLoadingCameras
817
- ? "Loading..."
818
- : "Request Camera Access"}
819
- </Button>
820
- </div>
821
  </div>
822
- )}
823
-
824
- {/* Camera Access Denied */}
825
- {cameraPermissionState === "denied" && (
826
- <div className="space-y-3">
827
- <div className="bg-red-900/20 border border-red-500/20 rounded-lg p-4 text-center">
828
- <Camera className="w-8 h-8 mx-auto mb-2 opacity-50 text-red-400" />
829
- <p className="text-sm text-red-300 mb-1">
830
- Camera access denied
831
- </p>
832
- <p className="text-xs text-red-400">
833
- Please allow camera access in your browser settings
834
- and refresh
835
- </p>
836
- </div>
837
  </div>
838
- )}
839
-
840
- {/* Camera List with Refresh Button */}
841
- {cameraPermissionState === "granted" &&
842
- availableCameras.length > 0 && (
843
- <div className="space-y-2">
844
- <label className="text-sm text-white/70">
845
- Select Camera:
846
- </label>
847
- <div className="flex items-center gap-2">
848
- <Select
849
- value={selectedCameraId}
850
- onValueChange={switchCameraPreview}
851
- disabled={hasRecordedFrames}
852
- >
853
- <SelectTrigger className="flex-1 bg-black/20 border-white/10">
854
- <SelectValue placeholder="Choose a camera" />
855
- </SelectTrigger>
856
- <SelectContent>
857
- {availableCameras.map((camera) => (
858
- <SelectItem
859
- key={camera.deviceId}
860
- value={camera.deviceId}
861
- >
862
- {camera.label ||
863
- `Camera ${camera.deviceId.slice(
864
- 0,
865
- 8
866
- )}...`}
867
- </SelectItem>
868
- ))}
869
- </SelectContent>
870
- </Select>
871
- <Button
872
- onClick={() => loadAvailableCameras(false)}
873
- variant="ghost"
874
- size="sm"
875
- className="gap-2 text-white/70 hover:text-white"
876
- disabled={isLoadingCameras}
877
- >
878
- <RefreshCw
879
- className={`w-4 h-4 ${
880
- isLoadingCameras ? "animate-spin" : ""
881
- }`}
882
- />
883
- Refresh
884
- </Button>
885
- </div>
886
- </div>
887
- )}
888
 
889
- {/* Camera Name Input */}
890
- {selectedCameraId && (
 
891
  <div className="space-y-2">
892
  <label className="text-sm text-white/70">
893
- Camera Name:
894
  </label>
895
- <Input
896
- placeholder="e.g., 'Overhead View', 'Side Angle', 'Close-up'"
897
- value={cameraName}
898
- onChange={(e) => setCameraName(e.target.value)}
899
- className="bg-black/20 border-white/10"
900
- disabled={hasRecordedFrames}
901
- />
902
- <p className="text-xs text-white/50">
903
- Give this camera a descriptive name for your recording
904
- setup
905
- </p>
906
- </div>
907
- )}
908
-
909
- {/* Add Camera Button */}
910
- {selectedCameraId && (
911
- <div className="flex justify-end">
912
- <Button
913
- onClick={handleAddCamera}
914
- className="gap-2"
915
- disabled={
916
- hasRecordedFrames ||
917
- !cameraName.trim() ||
918
- !selectedCameraId ||
919
- !previewStream
920
- }
921
- >
922
- <PlusCircle className="w-4 h-4" />
923
- Add Camera to Recorder
924
- </Button>
 
 
 
 
 
 
925
  </div>
926
  )}
927
- </div>
928
 
929
- {/* Right Column: Camera Preview */}
930
- <div className="space-y-4">
931
- <div className="aspect-video bg-black/60 border border-white/20 rounded-lg overflow-hidden">
932
- {previewStream ? (
933
- <video
934
- ref={videoRef}
935
- autoPlay
936
- muted
937
- playsInline
938
- className="w-full h-full object-cover"
939
- />
940
- ) : (
941
- <div className="w-full h-full flex items-center justify-center text-white/60">
942
- <div className="text-center">
943
- <Camera className="w-12 h-12 mx-auto mb-2 opacity-50" />
944
- {cameraPermissionState === "unknown" ? (
945
- <p className="text-sm">
946
- Request camera access to preview
947
- </p>
948
- ) : cameraPermissionState === "denied" ? (
949
- <p className="text-sm">Camera access denied</p>
950
- ) : availableCameras.length === 0 ? (
951
- <p className="text-sm">No cameras available</p>
952
- ) : !selectedCameraId ? (
953
- <p className="text-sm">
954
- Select a camera to preview
955
- </p>
956
- ) : (
957
- <p className="text-sm">Loading preview...</p>
958
- )}
959
- </div>
960
- </div>
961
- )}
962
  </div>
963
- </div>
964
- </div>
965
- </div>
966
- </div>
967
- </div>
968
- )}
969
-
970
- {/* Added Camera Previews */}
971
- {Object.keys(additionalCameras).length > 0 && (
972
- <div className="space-y-4">
973
- <h3 className="text-lg font-semibold text-foreground">
974
- Active Cameras
975
- </h3>
976
- <div className="grid grid-cols-2 md:grid-cols-3 lg:grid-cols-4 gap-4">
977
- {Object.entries(additionalCameras).map(([cameraName, stream]) => (
978
- <div
979
- key={cameraName}
980
- className="bg-black/40 border border-white/20 rounded-lg p-3 space-y-2"
981
- >
982
- <div className="aspect-video bg-black/60 border border-white/10 rounded overflow-hidden">
983
- <video
984
- autoPlay
985
- muted
986
- playsInline
987
- className="w-full h-full object-cover"
988
- ref={(video) => {
989
- if (video && stream) {
990
- video.srcObject = stream;
991
  }
992
- }}
993
- />
994
- </div>
995
- <div className="flex items-center justify-between">
996
- {editingCameraName === cameraName ? (
997
- <div className="flex items-center gap-1 flex-1">
998
- <Input
999
- value={editingCameraNewName}
1000
- onChange={(e) =>
1001
- setEditingCameraNewName(e.target.value)
1002
- }
1003
- className="text-xs h-6 bg-black/20 border-white/10"
1004
- onKeyDown={(e) => {
1005
- if (e.key === "Enter") {
1006
- handleConfirmCameraNameEdit(cameraName);
1007
- } else if (e.key === "Escape") {
1008
- handleCancelCameraNameEdit();
1009
- }
1010
- }}
1011
- autoFocus
1012
- />
1013
- <button
1014
- onClick={() =>
1015
- handleConfirmCameraNameEdit(cameraName)
1016
- }
1017
- className="text-green-400 hover:text-green-300 p-1"
1018
- >
1019
- <Check className="w-3 h-3" />
1020
- </button>
1021
- <button
1022
- onClick={handleCancelCameraNameEdit}
1023
- className="text-red-400 hover:text-red-300 p-1"
1024
- >
1025
- <X className="w-3 h-3" />
1026
- </button>
1027
- </div>
1028
- ) : (
1029
- <button
1030
- onClick={() => handleStartEditingCameraName(cameraName)}
1031
- className="text-sm font-medium text-white/90 truncate hover:text-white cursor-pointer flex items-center gap-1 flex-1"
1032
- disabled={hasRecordedFrames}
1033
  >
1034
- {cameraName}
1035
- <Edit2 className="w-3 h-3 opacity-50" />
1036
- </button>
1037
- )}
1038
- <button
1039
- onClick={() => handleRemoveCamera(cameraName)}
1040
- className="text-red-400 hover:text-red-300 p-1 ml-2"
1041
- disabled={hasRecordedFrames}
1042
- title="Remove camera"
1043
- >
1044
- <X className="w-4 h-4" />
1045
- </button>
1046
- </div>
1047
  </div>
1048
- ))}
1049
- </div>
1050
- </div>
1051
- )}
1052
-
1053
- {/* Episode Management & Dataset Actions */}
1054
- <div className="flex justify-between items-center">
1055
- <div className="flex items-center gap-2">
1056
- <Button
1057
- variant="outline"
1058
- className="gap-2"
1059
- onClick={handleResetFrames}
1060
- disabled={isRecording || !hasRecordedFrames}
1061
- >
1062
- <Trash2 className="w-4 h-4" />
1063
- Reset Frames
1064
- </Button>
1065
- <Button
1066
- variant="outline"
1067
- className="gap-2"
1068
- onClick={handleNextEpisode}
1069
- disabled={isRecording}
1070
- >
1071
- <PlusCircle className="w-4 h-4" />
1072
- Next Episode
1073
- </Button>
1074
- </div>
1075
-
1076
- <div className="flex items-center gap-2">
1077
- <Button
1078
- variant="outline"
1079
- className="gap-2"
1080
- onClick={handleDownloadZip}
1081
- disabled={recorderRef.current?.teleoperatorData.length === 0 || isRecording}
1082
- >
1083
- <Download className="w-4 h-4" />
1084
- Download as ZIP
1085
- </Button>
1086
- <Button
1087
- variant="outline"
1088
- className="gap-2"
1089
- onClick={handleUploadToHuggingFace}
1090
- disabled={
1091
- recorderRef.current?.teleoperatorData.length === 0 ||
1092
- isRecording ||
1093
- !recorderSettings.huggingfaceApiKey
1094
- }
1095
- >
1096
- <Upload className="w-4 h-4" />
1097
- Upload to Hugging Face
1098
- </Button>
1099
- </div>
1100
- </div>
1101
-
1102
- <div className="border border-white/10 rounded-md overflow-hidden">
1103
- <TeleoperatorEpisodesView teleoperatorData={recorderRef.current?.teleoperatorData} />
1104
- </div>
1105
 
1106
- {/* Camera Configuration */}
1107
- <div className="border-t border-white/10 pt-4 mt-4">
1108
- <div className="flex items-center justify-between mb-4">
1109
- <h3 className="text-lg font-semibold">Camera Setup</h3>
1110
- <Button
1111
- onClick={() => setShowCameraConfig(!showCameraConfig)}
1112
- variant="outline"
1113
- size="sm"
1114
- className="gap-2"
1115
- >
1116
- <Camera className="w-4 h-4" />
1117
- {showCameraConfig ? "Hide Camera Config" : "Configure Cameras"}
1118
- </Button>
1119
- </div>
1120
-
1121
- {showCameraConfig && (
1122
- <div className="bg-black/40 border border-white/20 rounded-lg p-6 mb-4">
1123
- <div className="grid grid-cols-1 lg:grid-cols-2 gap-6">
1124
- {/* Left Column: Camera Preview & Selection */}
1125
  <div className="space-y-4">
1126
- <h4 className="text-md font-semibold text-white/90">
1127
- Camera Preview
1128
- </h4>
1129
-
1130
- {/* Camera Preview */}
1131
  <div className="aspect-video bg-black/60 border border-white/20 rounded-lg overflow-hidden">
1132
  {previewStream ? (
1133
  <video
@@ -1141,178 +1051,270 @@ export function Recorder({
1141
  <div className="w-full h-full flex items-center justify-center text-white/60">
1142
  <div className="text-center">
1143
  <Camera className="w-12 h-12 mx-auto mb-2 opacity-50" />
1144
- {isLoadingCameras ? (
1145
- <p>Loading cameras...</p>
 
 
1146
  ) : cameraPermissionState === "denied" ? (
1147
- <div>
1148
- <p>Camera access denied</p>
1149
- <p className="text-xs mt-1">
1150
- Please allow camera access and refresh
1151
- </p>
1152
- </div>
1153
  ) : availableCameras.length === 0 ? (
1154
- <p>Click "Load Cameras" to start</p>
 
 
 
 
1155
  ) : (
1156
- <p>Select a camera to preview</p>
1157
  )}
1158
  </div>
1159
  </div>
1160
  )}
1161
  </div>
1162
-
1163
- {/* Camera Controls */}
1164
- <div className="space-y-3">
1165
- <Button
1166
- onClick={() => loadAvailableCameras(false)}
1167
- variant="outline"
1168
- className="w-full gap-2"
1169
- disabled={hasRecordedFrames || isLoadingCameras}
1170
- >
1171
- <Camera className="w-4 h-4" />
1172
- {isLoadingCameras
1173
- ? "Loading..."
1174
- : availableCameras.length > 0
1175
- ? "Refresh Cameras"
1176
- : "Load Cameras"}
1177
- </Button>
1178
-
1179
- {availableCameras.length > 0 && (
1180
- <div className="space-y-2">
1181
- <label className="text-sm text-white/70">
1182
- Select Camera:
1183
- </label>
1184
- <Select
1185
- value={selectedCameraId}
1186
- onValueChange={switchCameraPreview}
1187
- disabled={hasRecordedFrames}
1188
- >
1189
- <SelectTrigger className="w-full bg-black/20 border-white/10">
1190
- <SelectValue placeholder="Choose a camera" />
1191
- </SelectTrigger>
1192
- <SelectContent>
1193
- {availableCameras.map((camera) => (
1194
- <SelectItem
1195
- key={camera.deviceId}
1196
- value={camera.deviceId}
1197
- >
1198
- {camera.label ||
1199
- `Camera ${camera.deviceId.slice(0, 8)}...`}
1200
- </SelectItem>
1201
- ))}
1202
- </SelectContent>
1203
- </Select>
1204
- </div>
1205
- )}
1206
- </div>
1207
  </div>
1208
-
1209
- {/* Right Column: Camera Naming & Adding */}
1210
- <div className="space-y-4">
1211
- <h4 className="text-md font-semibold text-white/90">
1212
- Add to Recorder
1213
- </h4>
1214
-
1215
- <div className="space-y-4">
1216
- <div className="space-y-2">
1217
- <label className="text-sm text-white/70">
1218
- Camera Name:
1219
- </label>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1220
  <Input
1221
- placeholder="e.g., 'Overhead View', 'Side Angle', 'Close-up'"
1222
- value={cameraName}
1223
- onChange={(e) => setCameraName(e.target.value)}
1224
- className="bg-black/20 border-white/10"
1225
- disabled={hasRecordedFrames}
 
 
 
 
 
 
 
 
1226
  />
1227
- <p className="text-xs text-white/50">
1228
- Give this camera a descriptive name for your recording
1229
- setup
1230
- </p>
 
 
 
 
 
 
 
 
1231
  </div>
1232
-
1233
- <Button
1234
- onClick={handleAddCamera}
1235
- className="w-full gap-2"
1236
- disabled={
1237
- hasRecordedFrames ||
1238
- !cameraName.trim() ||
1239
- !selectedCameraId ||
1240
- !previewStream
1241
- }
1242
  >
1243
- <PlusCircle className="w-4 h-4" />
1244
- Add Camera to Recorder
1245
- </Button>
1246
- </div>
1247
- </div>
1248
- </div>
1249
- </div>
1250
- )}
1251
-
1252
- {/* Display added cameras */}
1253
- {Object.keys(additionalCameras).length > 0 && (
1254
- <div className="mb-4 space-y-2">
1255
- <p className="text-sm text-muted-foreground">Added Cameras:</p>
1256
- <div className="flex flex-wrap gap-2">
1257
- {Object.keys(additionalCameras).map((name) => (
1258
- <Badge
1259
- key={name}
1260
- variant="secondary"
1261
- className="flex items-center gap-1 py-1 px-2"
1262
- >
1263
- {name}
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1264
  <button
1265
- onClick={() => handleRemoveCamera(name)}
1266
- className="ml-1 text-muted-foreground hover:text-destructive"
1267
  disabled={hasRecordedFrames}
 
1268
  >
1269
- ×
1270
  </button>
1271
- </Badge>
1272
- ))}
1273
  </div>
1274
- </div>
1275
- )}
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1276
  </div>
1277
 
1278
- <div className="flex items-center gap-4">
1279
- <div className="flex items-center gap-2">
1280
- <Button
1281
- variant="outline"
1282
- className="gap-2"
1283
- onClick={handleDownloadZip}
1284
- disabled={recorderRef.current?.teleoperatorData.length === 0 || isRecording}
1285
- >
1286
- <Download className="w-4 h-4" />
1287
- Download as ZIP
1288
- </Button>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1289
 
1290
- {/* Reset Frames button moved to top bar */}
1291
- </div>
 
 
 
 
 
 
 
 
1292
 
1293
- <div className="flex items-center gap-2 flex-1">
1294
- <Button
1295
- variant="outline"
1296
- className="gap-2"
1297
- onClick={handleUploadToHuggingFace}
1298
- disabled={
1299
- recorderRef.current?.teleoperatorData.length === 0 || isRecording || !huggingfaceApiKey
1300
- }
 
 
 
 
 
 
 
 
 
 
1301
  >
1302
- <Upload className="w-4 h-4" />
1303
- Upload to HuggingFace
1304
- </Button>
1305
-
1306
- <Input
1307
- placeholder="HuggingFace API Key"
1308
- value={huggingfaceApiKey}
1309
- onChange={(e) => setHuggingfaceApiKey(e.target.value)}
1310
- className="flex-1 bg-black/20 border-white/10"
1311
- type="password"
1312
- />
1313
- </div>
1314
- </div>
1315
- </div>
1316
- </Card>
1317
  );
1318
  }
 
 
 
1
  "use client";
2
 
3
+ import { useState, useEffect, useRef, useCallback, memo } from "react";
4
  import { Button } from "@/components/ui/button";
 
 
 
 
 
 
 
 
 
 
5
  import { Input } from "@/components/ui/input";
6
  import { Badge } from "@/components/ui/badge";
7
  import {
 
26
  Edit2,
27
  Check,
28
  } from "lucide-react";
29
+ import {
30
+ AlertDialog,
31
+ AlertDialogAction,
32
+ AlertDialogCancel,
33
+ AlertDialogContent,
34
+ AlertDialogDescription,
35
+ AlertDialogFooter,
36
+ AlertDialogHeader,
37
+ AlertDialogTitle,
38
+ } from "@/components/ui/alert-dialog";
39
+ import { record, LeRobotEpisode } from "@lerobot/web";
40
+ import type { RecordProcess } from "@lerobot/web";
41
  import { TeleoperatorEpisodesView } from "./teleoperator-episodes-view";
42
+ import { uploadToHuggingFace } from "@/utils/dataset-uploader";
43
 
44
  interface RecorderProps {
45
  teleoperators: any[];
46
  robot: any; // eslint-disable-line @typescript-eslint/no-explicit-any
47
  onNeedsTeleoperation: () => Promise<boolean>;
48
+ showConfigure: boolean;
49
+ onRecorderReady?: (callbacks: {
50
+ startRecording: () => Promise<void>;
51
+ stopRecording: () => Promise<void>;
52
+ isRecording: boolean;
53
+ }) => void;
54
  videoStreams?: { [key: string]: MediaStream };
55
  }
56
 
 
 
57
  interface RecorderSettings {
58
  huggingfaceApiKey: string;
59
+ huggingfaceRepoName?: string;
60
+ huggingfacePrivate?: boolean;
61
  cameraConfigs: {
62
  [cameraName: string]: {
63
  deviceId: string;
 
80
  }
81
  return {
82
  huggingfaceApiKey: "",
83
+ huggingfaceRepoName: "",
84
+ huggingfacePrivate: true,
85
  cameraConfigs: {},
86
  };
87
  }
 
98
  teleoperators,
99
  robot,
100
  onNeedsTeleoperation,
101
+ showConfigure,
102
+ onRecorderReady,
103
  }: RecorderProps) {
104
  const [isRecording, setIsRecording] = useState(false);
105
  const [currentEpisode, setCurrentEpisode] = useState(0);
106
+ const [persistedEpisodes, setPersistedEpisodes] = useState<any[]>([]);
107
+ const [uiTick, setUiTick] = useState(0);
108
+ const [showDeleteEpisodesDialog, setShowDeleteEpisodesDialog] =
109
+ useState(false);
110
  // Use huggingfaceApiKey from recorderSettings instead of separate state
111
  const [cameraName, setCameraName] = useState("");
112
  const [additionalCameras, setAdditionalCameras] = useState<{
 
121
  const [cameraPermissionState, setCameraPermissionState] = useState<
122
  "unknown" | "granted" | "denied"
123
  >("unknown");
124
+
 
125
  const [recorderSettings, setRecorderSettings] = useState<RecorderSettings>(
126
  () => getRecorderSettings()
127
  );
 
130
  null
131
  );
132
  const [editingCameraNewName, setEditingCameraNewName] = useState("");
133
+ const [sourceSelectorOpenFor, setSourceSelectorOpenFor] = useState<
134
+ string | null
135
+ >(null);
136
+ const [uploadState, setUploadState] = useState<
137
+ { status: "idle" } | { status: "uploading" } | { status: "done" }
138
+ >({ status: "idle" });
139
+
140
+ const recordProcessRef = useRef<RecordProcess | null>(null);
141
  const videoRef = useRef<HTMLVideoElement | null>(null);
142
  const { toast } = useToast();
143
 
144
  // Initialize the recorder when teleoperators are available
145
  useEffect(() => {
146
+ if (teleoperators.length > 0 && !recordProcessRef.current) {
147
+ (async () => {
148
+ // Get robot type/family for metadata
149
+ let robotType = "unknown";
150
+ try {
151
+ const type = (robot?.robotType || "").toString();
152
+ robotType = type.split("_")[0] || type || "unknown";
153
+ } catch {}
154
+
155
+ // Create record process with upfront config
156
+ const recordProcess = await record({
157
+ teleoperator: teleoperators[0],
158
+ videoStreams: additionalCameras,
159
+ robotType,
160
+ options: {
161
+ fps: 30,
162
+ taskDescription: "Robot teleoperation recording",
163
+ },
164
+ });
165
+ recordProcessRef.current = recordProcess;
166
+
167
+ // Restore episodes if any were persisted
168
+ if (persistedEpisodes.length > 0) {
169
+ recordProcess.restoreEpisodes(persistedEpisodes);
170
+ setCurrentEpisode(persistedEpisodes.length - 1);
171
+ }
172
+
173
+ // Attach any cameras that were configured before control was enabled
174
+ for (const [key, stream] of Object.entries(additionalCameras)) {
175
+ try {
176
+ recordProcess.addCamera(key, stream as MediaStream);
177
+ } catch (e) {
178
+ console.warn("[Recorder] init: addCamera failed", key, e);
179
+ }
180
+ }
181
+ })();
182
  }
183
+ }, [teleoperators, persistedEpisodes, additionalCameras]);
184
 
185
+ // Sync additional cameras into recorder without re-creating it
186
+ useEffect(() => {
187
+ if (!recordProcessRef.current) return;
188
+ if (isRecording) return; // don't change streams during recording
189
+ const recordProcess = recordProcessRef.current;
 
 
190
 
191
+ // Add any new cameras
192
+ for (const [key, stream] of Object.entries(additionalCameras)) {
193
+ try {
194
+ recordProcess.addCamera(key, stream);
195
+ } catch (e) {
196
+ console.warn("Failed to add camera", key, e);
 
 
197
  }
198
+ }
199
+ }, [additionalCameras, isRecording]);
200
 
201
+ // Notify parent of recorder state changes
202
+ useEffect(() => {
203
+ if (onRecorderReady) {
204
+ onRecorderReady({
205
+ startRecording: handleStartRecordingClick,
206
+ stopRecording: handleStopRecording,
207
+ isRecording: isRecording,
208
+ });
209
  }
210
+ }, [isRecording, onRecorderReady]);
211
 
212
+ // Simple recording start - just like the original working version
213
+ const handleStartRecordingClick = async () => {
214
+ if (!recordProcessRef.current) {
215
  toast({
216
  title: "Recording Error",
217
+ description: "Recorder not ready yet. Please enable control first.",
218
  variant: "destructive",
219
  });
220
  return;
221
  }
222
 
223
  try {
 
 
 
 
224
  // Start recording
225
+ recordProcessRef.current.start();
226
  setIsRecording(true);
227
  setHasRecordedFrames(true);
228
 
229
+ // Sync current episode to recorder's latest
230
+ setCurrentEpisode(
231
+ Math.max(0, (recordProcessRef.current.getEpisodeCount() || 1) - 1)
232
+ );
233
  } catch (error) {
234
  const errorMessage =
235
  error instanceof Error ? error.message : "Failed to start recording";
 
242
  };
243
 
244
  const handleStopRecording = async () => {
245
+ if (!recordProcessRef.current || !isRecording) {
246
  return;
247
  }
248
 
249
  try {
250
+ const result = await recordProcessRef.current.stop();
251
  setIsRecording(false);
252
 
253
+ // No need to manually persist - episodes are managed by record process
254
+
255
+ // Force a small delay to ensure videoBlobs populated before export
256
+ await new Promise((r) => setTimeout(r, 50));
257
  } catch (error) {
258
  const errorMessage =
259
  error instanceof Error ? error.message : "Failed to stop recording";
 
265
  }
266
  };
267
 
268
+ const handleDeleteEpisodes = async () => {
269
+ if (recordProcessRef.current) {
270
+ recordProcessRef.current.clearEpisodes();
 
271
  }
272
 
273
+ // Clear persisted episodes only if not recording
274
+ if (!isRecording) {
275
+ setPersistedEpisodes([]);
276
+ }
277
 
278
+ setCurrentEpisode(0);
279
+ setHasRecordedFrames(isRecording); // Keep true if recording, false if not
280
+ setShowDeleteEpisodesDialog(false);
281
+
282
+ // No toast needed; dialog confirmation provides sufficient feedback
283
  };
284
 
285
+ const handleNextEpisode = () => {
286
+ if (!isRecording || !recordProcessRef.current) {
287
+ return;
 
288
  }
289
 
290
+ // Finalize current video segment and start a new one
291
+ recordProcessRef.current
292
+ .nextEpisode()
293
+ .then((newIndex: number) => setCurrentEpisode(newIndex))
294
+ .catch(() => {
295
+ /* noop */
 
296
  });
297
+ };
298
+
299
+ // Force lightweight UI refresh while recording so episode table updates in near real-time
300
+ useEffect(() => {
301
+ if (!isRecording) return;
302
+ const id = setInterval(() => {
303
+ setUiTick((t) => (t + 1) % 1_000_000);
304
+ }, 500);
305
+ return () => clearInterval(id);
306
+ }, [isRecording]);
307
+
308
+ // (Removed) Reset frames button in favor of Delete Episodes with confirmation
309
 
310
  // Load available cameras
311
  const loadAvailableCameras = useCallback(
 
438
  [previewStream, toast]
439
  );
440
 
441
+ // Change camera source for an already-added camera card
442
+ const handleChangeCameraSourceForCard = useCallback(
443
+ async (cameraName: string, deviceId: string) => {
444
+ if (hasRecordedFrames) {
445
+ toast({
446
+ title: "Camera Error",
447
+ description:
448
+ "Cannot change camera source after recording has started",
449
+ variant: "destructive",
450
+ });
451
+ return;
452
+ }
453
+
454
+ try {
455
+ // Create new stream for selected device
456
+ const newStream = await navigator.mediaDevices.getUserMedia({
457
+ video: {
458
+ deviceId: { exact: deviceId },
459
+ width: { ideal: 1280 },
460
+ height: { ideal: 720 },
461
+ },
462
+ });
463
+
464
+ // Replace existing stream in additionalCameras
465
+ setAdditionalCameras((prev) => {
466
+ const next = { ...prev };
467
+ const old = next[cameraName];
468
+ if (old) {
469
+ old.getTracks().forEach((t) => t.stop());
470
+ }
471
+ next[cameraName] = newStream;
472
+ return next;
473
+ });
474
+
475
+ // Update persistent config
476
+ const selected = availableCameras.find((c) => c.deviceId === deviceId);
477
+ const newSettings = {
478
+ ...recorderSettings,
479
+ cameraConfigs: { ...recorderSettings.cameraConfigs },
480
+ };
481
+ if (!newSettings.cameraConfigs[cameraName]) {
482
+ newSettings.cameraConfigs[cameraName] = {
483
+ deviceId,
484
+ deviceLabel: selected?.label || `Camera ${deviceId.slice(0, 8)}...`,
485
+ };
486
+ } else {
487
+ newSettings.cameraConfigs[cameraName].deviceId = deviceId;
488
+ newSettings.cameraConfigs[cameraName].deviceLabel =
489
+ selected?.label || `Camera ${deviceId.slice(0, 8)}...`;
490
+ }
491
+ setRecorderSettings(newSettings);
492
+ saveRecorderSettings(newSettings);
493
+
494
+ setSourceSelectorOpenFor(null);
495
+ toast({
496
+ title: "Camera Source Updated",
497
+ description: `\"${cameraName}\" now uses \"${
498
+ selected?.label || deviceId
499
+ }\"`,
500
+ });
501
+ } catch (error) {
502
+ toast({
503
+ title: "Camera Error",
504
+ description: `Failed to switch camera: ${
505
+ error instanceof Error ? error.message : String(error)
506
+ }`,
507
+ variant: "destructive",
508
+ });
509
+ }
510
+ },
511
+ [hasRecordedFrames, availableCameras, recorderSettings, toast]
512
+ );
513
+
514
  // Add a new camera to the recorder
515
  const handleAddCamera = useCallback(async () => {
516
  if (!cameraName.trim()) {
 
758
  }, [availableCameras, cameraPermissionState, restoreSavedCameras]);
759
 
760
  const handleDownloadZip = async () => {
761
+ if (!recordProcessRef.current) return;
762
+ try {
763
+ if (isRecording) {
764
+ await handleStopRecording();
765
+ }
766
+ await recordProcessRef.current.exportForLeRobot("zip-download");
767
+ } catch (e) {
768
  toast({
769
+ title: "Export Error",
770
+ description:
771
+ e instanceof Error ? e.message : "Failed to export the dataset",
772
  variant: "destructive",
773
  });
774
  return;
775
  }
776
+ // No toast; the browser download prompt is sufficient feedback
 
 
 
 
 
777
  };
778
 
779
  const handleUploadToHuggingFace = async () => {
780
+ if (!recordProcessRef.current) return;
 
 
 
 
 
 
 
781
 
782
  if (!recorderSettings.huggingfaceApiKey) {
783
  toast({
 
789
  }
790
 
791
  try {
792
+ // Use provided repo name or generate one
793
+ const repoName =
794
+ (recorderSettings.huggingfaceRepoName || "").trim() ||
795
+ `lerobot-recording-${Date.now()}`;
796
+
797
+ // Get blobs from recorder
798
+ const blobArray = await recordProcessRef.current.exportForLeRobot(
799
+ "blobs"
800
+ );
801
 
802
+ // Upload using demo utility
803
+ const uploader = await uploadToHuggingFace(
804
+ blobArray,
805
+ recorderSettings.huggingfaceApiKey,
806
+ repoName,
807
+ !!recorderSettings.huggingfacePrivate
808
  );
809
 
810
+ // Progress UI next to button
811
+ setUploadState({ status: "uploading" });
812
+ const onProgress = (_event: Event) => {
813
+ // Spinner-only UI; keep uploading state
814
+ setUploadState({ status: "uploading" });
815
+ };
816
+ const onFinished = () => setUploadState({ status: "done" });
817
+ const onError = () => setUploadState({ status: "idle" });
818
+ uploader.addEventListener("progress", onProgress);
819
+ uploader.addEventListener("finished", onFinished);
820
+ uploader.addEventListener("error", onError);
821
+ // No fake progress; spinner indicates activity
822
  } catch (error) {
823
  const errorMessage =
824
  error instanceof Error
 
832
  }
833
  };
834
 
 
 
 
 
 
 
 
 
 
835
  return (
836
+ <div className="space-y-6">
837
+ {/* Recorder Settings - Toggleable Inline */}
838
+ {showConfigure && (
839
+ <div className="space-y-6">
840
+ {/* Hugging Face Settings */}
841
+ <div className="space-y-3">
842
+ <h3 className="text-lg font-semibold text-foreground">Settings</h3>
843
+ <div className="grid grid-cols-1 md:grid-cols-2 gap-4">
844
+ <div className="space-y-2">
845
+ <label className="text-sm text-muted-foreground">
846
+ Hugging Face API Key
847
+ </label>
848
+ <Input
849
+ placeholder="Enter your Hugging Face API key"
850
+ value={recorderSettings.huggingfaceApiKey}
851
+ onChange={(e) => {
852
+ const newSettings = {
853
+ ...recorderSettings,
854
+ huggingfaceApiKey: e.target.value,
855
+ };
856
+ setRecorderSettings(newSettings);
857
+ saveRecorderSettings(newSettings);
858
+ }}
859
+ type="password"
860
+ className="bg-black/20 border-white/10"
861
+ />
862
+ <p className="text-xs text-white/50">
863
+ Required to upload datasets to Hugging Face Hub
864
+ </p>
865
+ </div>
866
+ <div className="space-y-2">
867
+ <label className="text-sm text-muted-foreground">
868
+ Hugging Face Repo Name
869
+ </label>
870
+ <Input
871
+ placeholder="e.g. my-dataset-name"
872
+ value={recorderSettings.huggingfaceRepoName || ""}
873
+ onChange={(e) => {
874
+ const newSettings = {
875
+ ...recorderSettings,
876
+ huggingfaceRepoName: e.target.value,
877
+ };
878
+ setRecorderSettings(newSettings);
879
+ saveRecorderSettings(newSettings);
880
+ }}
881
+ className="bg-black/20 border-white/10"
882
+ />
883
+ <label className="flex items-center gap-2 text-sm text-muted-foreground">
884
+ <input
885
+ type="checkbox"
886
+ className="accent-white"
887
+ checked={!!recorderSettings.huggingfacePrivate}
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
888
  onChange={(e) => {
889
  const newSettings = {
890
  ...recorderSettings,
891
+ huggingfacePrivate: e.target.checked,
892
  };
893
  setRecorderSettings(newSettings);
894
  saveRecorderSettings(newSettings);
895
  }}
 
 
896
  />
897
+ Private repository
898
+ </label>
 
 
899
  </div>
900
  </div>
901
+ </div>
902
 
903
+ {/* Camera Configuration */}
904
+ <div className="space-y-4">
905
+ <h3 className="text-lg font-semibold text-foreground">
906
+ Camera Setup
907
+ </h3>
908
+ <div className="bg-black/40 border border-white/20 p-6">
909
+ <div className="grid grid-cols-1 lg:grid-cols-2 gap-6">
910
+ {/* Left Column: Camera Selection & Adding */}
911
+ <div className="space-y-4">
912
+ {/* Camera Selection and Refresh */}
913
+
914
+ {/* Camera Access Request */}
915
+ {cameraPermissionState === "unknown" && (
916
+ <div className="space-y-3">
917
+ <div className="bg-black/60 border border-white/20 rounded-lg p-4 text-center">
918
+ <Camera className="w-8 h-8 mx-auto mb-2 opacity-50" />
919
+ <p className="text-sm text-white/70 mb-3">
920
+ Camera access needed to configure cameras
921
+ </p>
922
+ <Button
923
+ onClick={() => loadAvailableCameras(false)}
924
+ variant="outline"
925
+ className="gap-2"
926
+ disabled={isLoadingCameras}
927
+ >
928
+ <Camera className="w-4 h-4" />
929
+ {isLoadingCameras
930
+ ? "Loading..."
931
+ : "Request Camera Access"}
932
+ </Button>
 
933
  </div>
934
+ </div>
935
+ )}
936
+
937
+ {/* Camera Access Denied */}
938
+ {cameraPermissionState === "denied" && (
939
+ <div className="space-y-3">
940
+ <div className="bg-red-900/20 border border-red-500/20 rounded-lg p-4 text-center">
941
+ <Camera className="w-8 h-8 mx-auto mb-2 opacity-50 text-red-400" />
942
+ <p className="text-sm text-red-300 mb-1">
943
+ Camera access denied
944
+ </p>
945
+ <p className="text-xs text-red-400">
946
+ Please allow camera access in your browser settings
947
+ and refresh
948
+ </p>
949
  </div>
950
+ </div>
951
+ )}
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
952
 
953
+ {/* Camera List with Refresh Button */}
954
+ {cameraPermissionState === "granted" &&
955
+ availableCameras.length > 0 && (
956
  <div className="space-y-2">
957
  <label className="text-sm text-white/70">
958
+ Select Camera:
959
  </label>
960
+ <div className="flex items-center gap-2">
961
+ <Select
962
+ value={selectedCameraId}
963
+ onValueChange={switchCameraPreview}
964
+ disabled={hasRecordedFrames}
965
+ >
966
+ <SelectTrigger className="flex-1 bg-black/20 border-white/10">
967
+ <SelectValue placeholder="Choose a camera" />
968
+ </SelectTrigger>
969
+ <SelectContent>
970
+ {availableCameras.map((camera) => (
971
+ <SelectItem
972
+ key={camera.deviceId}
973
+ value={camera.deviceId}
974
+ >
975
+ {camera.label ||
976
+ `Camera ${camera.deviceId.slice(0, 8)}...`}
977
+ </SelectItem>
978
+ ))}
979
+ </SelectContent>
980
+ </Select>
981
+ <Button
982
+ onClick={() => loadAvailableCameras(false)}
983
+ variant="ghost"
984
+ size="sm"
985
+ className="gap-2 text-white/70 hover:text-white"
986
+ disabled={isLoadingCameras}
987
+ >
988
+ <RefreshCw
989
+ className={`w-4 h-4 ${
990
+ isLoadingCameras ? "animate-spin" : ""
991
+ }`}
992
+ />
993
+ Refresh
994
+ </Button>
995
+ </div>
996
  </div>
997
  )}
 
998
 
999
+ {/* Camera Name Input */}
1000
+ {selectedCameraId && (
1001
+ <div className="space-y-2">
1002
+ <label className="text-sm text-white/70">
1003
+ Camera Name:
1004
+ </label>
1005
+ <Input
1006
+ placeholder="e.g., 'Overhead View', 'Side Angle', 'Close-up'"
1007
+ value={cameraName}
1008
+ onChange={(e) => setCameraName(e.target.value)}
1009
+ className="bg-black/20 border-white/10"
1010
+ disabled={hasRecordedFrames}
1011
+ />
1012
+ <p className="text-xs text-white/50">
1013
+ Give this camera a descriptive name for your recording
1014
+ setup
1015
+ </p>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1016
  </div>
1017
+ )}
1018
+
1019
+ {/* Add Camera Button */}
1020
+ {selectedCameraId && (
1021
+ <div className="flex justify-end">
1022
+ <Button
1023
+ onClick={handleAddCamera}
1024
+ className="gap-2"
1025
+ disabled={
1026
+ hasRecordedFrames ||
1027
+ !cameraName.trim() ||
1028
+ !selectedCameraId ||
1029
+ !previewStream
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1030
  }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1031
  >
1032
+ <PlusCircle className="w-4 h-4" />
1033
+ Add Camera to Recorder
1034
+ </Button>
1035
+ </div>
1036
+ )}
 
 
 
 
 
 
 
 
1037
  </div>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1038
 
1039
+ {/* Right Column: Camera Preview */}
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1040
  <div className="space-y-4">
 
 
 
 
 
1041
  <div className="aspect-video bg-black/60 border border-white/20 rounded-lg overflow-hidden">
1042
  {previewStream ? (
1043
  <video
 
1051
  <div className="w-full h-full flex items-center justify-center text-white/60">
1052
  <div className="text-center">
1053
  <Camera className="w-12 h-12 mx-auto mb-2 opacity-50" />
1054
+ {cameraPermissionState === "unknown" ? (
1055
+ <p className="text-sm">
1056
+ Request camera access to preview
1057
+ </p>
1058
  ) : cameraPermissionState === "denied" ? (
1059
+ <p className="text-sm">Camera access denied</p>
 
 
 
 
 
1060
  ) : availableCameras.length === 0 ? (
1061
+ <p className="text-sm">No cameras available</p>
1062
+ ) : !selectedCameraId ? (
1063
+ <p className="text-sm">
1064
+ Select a camera to preview
1065
+ </p>
1066
  ) : (
1067
+ <p className="text-sm">Loading preview...</p>
1068
  )}
1069
  </div>
1070
  </div>
1071
  )}
1072
  </div>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1073
  </div>
1074
+ </div>
1075
+ </div>
1076
+ </div>
1077
+ </div>
1078
+ )}
1079
+
1080
+ {/* Added Camera Previews */}
1081
+ {Object.keys(additionalCameras).length > 0 && (
1082
+ <div className="space-y-4">
1083
+ <h3 className="text-lg font-semibold text-foreground">
1084
+ Active Cameras
1085
+ </h3>
1086
+ <div className="grid grid-cols-2 md:grid-cols-3 lg:grid-cols-4 gap-4">
1087
+ {Object.entries(additionalCameras).map(([cameraName, stream]) => (
1088
+ <div
1089
+ key={cameraName}
1090
+ className="bg-black/40 border border-white/20 rounded-lg p-3 space-y-2"
1091
+ >
1092
+ <div className="aspect-video bg-black/60 border border-white/10 rounded overflow-hidden">
1093
+ <video
1094
+ autoPlay
1095
+ muted
1096
+ playsInline
1097
+ className="w-full h-full object-cover"
1098
+ ref={(video) => {
1099
+ if (video && stream) {
1100
+ const current = video.srcObject as MediaStream | null;
1101
+ if (current !== stream) {
1102
+ video.srcObject = stream;
1103
+ }
1104
+ }
1105
+ }}
1106
+ />
1107
+ </div>
1108
+ <div className="flex items-center justify-between">
1109
+ {editingCameraName === cameraName ? (
1110
+ <div className="flex items-center gap-1 flex-1">
1111
  <Input
1112
+ value={editingCameraNewName}
1113
+ onChange={(e) =>
1114
+ setEditingCameraNewName(e.target.value)
1115
+ }
1116
+ className="text-xs h-6 bg-black/20 border-white/10"
1117
+ onKeyDown={(e) => {
1118
+ if (e.key === "Enter") {
1119
+ handleConfirmCameraNameEdit(cameraName);
1120
+ } else if (e.key === "Escape") {
1121
+ handleCancelCameraNameEdit();
1122
+ }
1123
+ }}
1124
+ autoFocus
1125
  />
1126
+ <button
1127
+ onClick={() => handleConfirmCameraNameEdit(cameraName)}
1128
+ className="text-green-400 hover:text-green-300 p-1"
1129
+ >
1130
+ <Check className="w-3 h-3" />
1131
+ </button>
1132
+ <button
1133
+ onClick={handleCancelCameraNameEdit}
1134
+ className="text-red-400 hover:text-red-300 p-1"
1135
+ >
1136
+ <X className="w-3 h-3" />
1137
+ </button>
1138
  </div>
1139
+ ) : (
1140
+ <button
1141
+ onClick={() => handleStartEditingCameraName(cameraName)}
1142
+ className="text-sm font-medium text-white/90 truncate hover:text-white cursor-pointer flex items-center gap-1 flex-1"
1143
+ disabled={hasRecordedFrames}
 
 
 
 
 
1144
  >
1145
+ {cameraName}
1146
+ <Edit2 className="w-3 h-3 opacity-50" />
1147
+ </button>
1148
+ )}
1149
+ <div className="flex items-center gap-2 ml-2">
1150
+ {/* Inline camera source selector trigger */}
1151
+ <div className="relative">
1152
+ <button
1153
+ onClick={() =>
1154
+ setSourceSelectorOpenFor((prev) =>
1155
+ prev === cameraName ? null : cameraName
1156
+ )
1157
+ }
1158
+ className="text-white/80 hover:text-white p-1"
1159
+ disabled={hasRecordedFrames}
1160
+ title="Change camera source"
1161
+ >
1162
+ <Camera className="w-4 h-4" />
1163
+ </button>
1164
+ {sourceSelectorOpenFor === cameraName && (
1165
+ <div className="absolute right-0 mt-1 z-20 bg-black/90 border border-white/20 rounded p-2 w-64">
1166
+ <label className="text-xs text-white/70">
1167
+ Camera Source
1168
+ </label>
1169
+ <Select
1170
+ value={
1171
+ recorderSettings.cameraConfigs[cameraName]
1172
+ ?.deviceId || ""
1173
+ }
1174
+ onValueChange={(deviceId) =>
1175
+ handleChangeCameraSourceForCard(
1176
+ cameraName,
1177
+ deviceId
1178
+ )
1179
+ }
1180
+ disabled={hasRecordedFrames}
1181
+ >
1182
+ <SelectTrigger className="w-full h-8 bg-black/20 border-white/10">
1183
+ <SelectValue placeholder="Choose a camera" />
1184
+ </SelectTrigger>
1185
+ <SelectContent>
1186
+ {availableCameras.map((cam) => (
1187
+ <SelectItem
1188
+ key={cam.deviceId}
1189
+ value={cam.deviceId}
1190
+ >
1191
+ {cam.label ||
1192
+ `Camera ${cam.deviceId.slice(0, 8)}...`}
1193
+ </SelectItem>
1194
+ ))}
1195
+ </SelectContent>
1196
+ </Select>
1197
+ </div>
1198
+ )}
1199
+ </div>
1200
  <button
1201
+ onClick={() => handleRemoveCamera(cameraName)}
1202
+ className="text-red-400 hover:text-red-300 p-1"
1203
  disabled={hasRecordedFrames}
1204
+ title="Remove camera"
1205
  >
1206
+ <X className="w-4 h-4" />
1207
  </button>
1208
+ </div>
1209
+ </div>
1210
  </div>
1211
+ ))}
1212
+ </div>
1213
+ </div>
1214
+ )}
1215
+
1216
+ {/* Episode Management & Dataset Actions */}
1217
+ <div className="flex justify-between items-center">
1218
+ <div className="flex items-center gap-2">
1219
+ <Button
1220
+ variant="outline"
1221
+ className="gap-2"
1222
+ onClick={handleNextEpisode}
1223
+ disabled={!isRecording}
1224
+ >
1225
+ <PlusCircle className="w-4 h-4" />
1226
+ Next Episode
1227
+ </Button>
1228
+ <Button
1229
+ variant="outline"
1230
+ className="gap-2"
1231
+ onClick={() => setShowDeleteEpisodesDialog(true)}
1232
+ disabled={
1233
+ persistedEpisodes.length === 0 &&
1234
+ (recordProcessRef.current?.getEpisodeCount() || 0) === 0
1235
+ }
1236
+ >
1237
+ <Trash2 className="w-4 h-4" />
1238
+ Delete Episodes
1239
+ </Button>
1240
  </div>
1241
 
1242
+ <div className="flex items-center gap-2">
1243
+ <Button
1244
+ variant="outline"
1245
+ className="gap-2"
1246
+ onClick={handleDownloadZip}
1247
+ disabled={
1248
+ (recordProcessRef.current?.getEpisodeCount() || 0) === 0 ||
1249
+ isRecording
1250
+ }
1251
+ >
1252
+ <Download className="w-4 h-4" />
1253
+ Download as ZIP
1254
+ </Button>
1255
+ <Button
1256
+ variant="outline"
1257
+ className="gap-2 relative"
1258
+ onClick={handleUploadToHuggingFace}
1259
+ disabled={
1260
+ (recordProcessRef.current?.getEpisodeCount() || 0) === 0 ||
1261
+ isRecording ||
1262
+ !recorderSettings.huggingfaceApiKey ||
1263
+ uploadState.status === "uploading"
1264
+ }
1265
+ >
1266
+ <Upload className="w-4 h-4" />
1267
+ {uploadState.status === "uploading" ? (
1268
+ <span className="inline-flex items-center gap-2">
1269
+ Uploading…
1270
+ <span className="inline-block w-4 h-4 border-2 border-white/50 border-t-white rounded-full animate-spin" />
1271
+ </span>
1272
+ ) : uploadState.status === "done" ? (
1273
+ <span>Uploaded ✓</span>
1274
+ ) : (
1275
+ <span>Upload to Hugging Face</span>
1276
+ )}
1277
+ </Button>
1278
+ </div>
1279
+ </div>
1280
 
1281
+ <div className="border border-white/10 rounded-md overflow-hidden">
1282
+ <TeleoperatorEpisodesView
1283
+ teleoperatorData={
1284
+ (uiTick,
1285
+ recordProcessRef.current?.getEpisodes() || persistedEpisodes)
1286
+ }
1287
+ isRecording={isRecording}
1288
+ refreshTick={uiTick}
1289
+ />
1290
+ </div>
1291
 
1292
+ {/* Delete Episodes Confirmation Dialog */}
1293
+ <AlertDialog
1294
+ open={showDeleteEpisodesDialog}
1295
+ onOpenChange={setShowDeleteEpisodesDialog}
1296
+ >
1297
+ <AlertDialogContent>
1298
+ <AlertDialogHeader>
1299
+ <AlertDialogTitle>Delete All Episodes</AlertDialogTitle>
1300
+ <AlertDialogDescription>
1301
+ Are you sure you want to delete all recorded episodes? This action
1302
+ cannot be undone.
1303
+ </AlertDialogDescription>
1304
+ </AlertDialogHeader>
1305
+ <AlertDialogFooter>
1306
+ <AlertDialogCancel>Cancel</AlertDialogCancel>
1307
+ <AlertDialogAction
1308
+ onClick={handleDeleteEpisodes}
1309
+ className="bg-destructive text-destructive-foreground hover:bg-destructive/90"
1310
  >
1311
+ Delete Episodes
1312
+ </AlertDialogAction>
1313
+ </AlertDialogFooter>
1314
+ </AlertDialogContent>
1315
+ </AlertDialog>
1316
+ </div>
 
 
 
 
 
 
 
 
 
1317
  );
1318
  }
1319
+
1320
+ export const MemoRecorder = memo(Recorder);
examples/cyberpunk-standalone/src/components/recording-view.tsx ADDED
@@ -0,0 +1,598 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ "use client";
2
+
3
+ import {
4
+ useState,
5
+ useEffect,
6
+ useMemo,
7
+ useRef,
8
+ useCallback,
9
+ startTransition,
10
+ } from "react";
11
+ import { Card } from "@/components/ui/card";
12
+ import { Badge } from "@/components/ui/badge";
13
+ import { Button } from "@/components/ui/button";
14
+ import { cn } from "@/lib/utils";
15
+ import {
16
+ Gamepad2,
17
+ Keyboard,
18
+ Settings,
19
+ Square,
20
+ Disc as Record,
21
+ Check,
22
+ } from "lucide-react";
23
+ import {
24
+ Select,
25
+ SelectContent,
26
+ SelectItem,
27
+ SelectTrigger,
28
+ SelectValue,
29
+ } from "@/components/ui/select";
30
+ import {
31
+ teleoperate,
32
+ type TeleoperationProcess,
33
+ type TeleoperationState,
34
+ type TeleoperateConfig,
35
+ type RobotConnection,
36
+ } from "@lerobot/web";
37
+ import { getUnifiedRobotData } from "@/lib/unified-storage";
38
+ import { MemoRecorder as Recorder } from "@/components/recorder";
39
+
40
+ const UI_UPDATE_INTERVAL_MS = 100;
41
+
42
+ interface RecordingViewProps {
43
+ robot: RobotConnection;
44
+ }
45
+
46
+ export function RecordingView({ robot }: RecordingViewProps) {
47
+ const [teleopState, setTeleopState] = useState<TeleoperationState>({
48
+ isActive: false,
49
+ motorConfigs: [],
50
+ lastUpdate: 0,
51
+ keyStates: {},
52
+ });
53
+
54
+ const [isInitialized, setIsInitialized] = useState(false);
55
+ const [controlEnabled, setControlEnabled] = useState(false);
56
+ const [selectedTeleoperatorType, setSelectedTeleoperatorType] = useState<
57
+ "direct" | "keyboard"
58
+ >("keyboard");
59
+ const [showConfigure, setShowConfigure] = useState(false);
60
+ const [recorderCallbacks, setRecorderCallbacks] = useState<{
61
+ startRecording: () => Promise<void>;
62
+ stopRecording: () => Promise<void>;
63
+ isRecording: boolean;
64
+ } | null>(null);
65
+
66
+ // Current step tracking
67
+ const currentStep = useMemo(() => {
68
+ if (!robot.isConnected) return 0; // No steps active if robot offline
69
+ if (!controlEnabled) return 1; // Step 1: Enable Control
70
+ if (!recorderCallbacks?.isRecording) return 2; // Step 2: Start Recording
71
+ return 3; // Step 3: Move Robot (recording active)
72
+ }, [robot.isConnected, controlEnabled, recorderCallbacks?.isRecording]);
73
+ const keyboardProcessRef = useRef<TeleoperationProcess | null>(null);
74
+ const directProcessRef = useRef<TeleoperationProcess | null>(null);
75
+ const lastUiUpdateRef = useRef(0);
76
+
77
+ // Load calibration data from unified storage
78
+ const calibrationData = useMemo(() => {
79
+ if (!robot.serialNumber) return undefined;
80
+
81
+ const data = getUnifiedRobotData(robot.serialNumber);
82
+ if (data?.calibration) {
83
+ return data.calibration;
84
+ }
85
+
86
+ // Return undefined if no calibration data - let library handle defaults
87
+ return undefined;
88
+ }, [robot.serialNumber]);
89
+
90
+ // Initialize teleoperation for recording
91
+ const initializeTeleoperation = useCallback(async () => {
92
+ if (!robot || !robot.robotType) {
93
+ return false;
94
+ }
95
+
96
+ try {
97
+ // Create keyboard teleoperation process
98
+ const keyboardConfig: TeleoperateConfig = {
99
+ robot: robot,
100
+ teleop: {
101
+ type: "keyboard",
102
+ },
103
+ calibrationData,
104
+ onStateUpdate: (state: TeleoperationState) => {
105
+ const now = performance.now();
106
+ if (now - lastUiUpdateRef.current >= UI_UPDATE_INTERVAL_MS) {
107
+ lastUiUpdateRef.current = now;
108
+ startTransition(() => setTeleopState(state));
109
+ }
110
+ },
111
+ };
112
+ const keyboardProcess = await teleoperate(keyboardConfig);
113
+
114
+ // Create direct teleoperation process
115
+ const directConfig: TeleoperateConfig = {
116
+ robot: robot,
117
+ teleop: {
118
+ type: "direct",
119
+ },
120
+ calibrationData,
121
+ onStateUpdate: (state: TeleoperationState) => {
122
+ const now = performance.now();
123
+ if (now - lastUiUpdateRef.current >= UI_UPDATE_INTERVAL_MS) {
124
+ lastUiUpdateRef.current = now;
125
+ startTransition(() => setTeleopState(state));
126
+ }
127
+ },
128
+ };
129
+ const directProcess = await teleoperate(directConfig);
130
+
131
+ keyboardProcessRef.current = keyboardProcess;
132
+ directProcessRef.current = directProcess;
133
+ setTeleopState(directProcess.getState());
134
+
135
+ setIsInitialized(true);
136
+
137
+ return true;
138
+ } catch (error) {
139
+ const errorMessage =
140
+ error instanceof Error
141
+ ? error.message
142
+ : "Failed to initialize teleoperation for recording";
143
+ toast({
144
+ title: "Teleoperation Error",
145
+ description: errorMessage,
146
+ variant: "destructive",
147
+ });
148
+ return false;
149
+ }
150
+ }, [robot, robot.robotType, calibrationData]);
151
+
152
+ // Enable robot control for recording
153
+ const enableControl = useCallback(async () => {
154
+ if (!robot.isConnected) {
155
+ return false;
156
+ }
157
+
158
+ const success = await initializeTeleoperation();
159
+ if (success) {
160
+ // Start the appropriate teleoperator based on selection
161
+ if (
162
+ selectedTeleoperatorType === "keyboard" &&
163
+ keyboardProcessRef.current
164
+ ) {
165
+ keyboardProcessRef.current.start();
166
+ } else if (
167
+ selectedTeleoperatorType === "direct" &&
168
+ directProcessRef.current
169
+ ) {
170
+ directProcessRef.current.start();
171
+ }
172
+
173
+ setControlEnabled(true);
174
+ return true;
175
+ }
176
+ return false;
177
+ }, [robot.isConnected, initializeTeleoperation, selectedTeleoperatorType]);
178
+
179
+ // Disable robot control
180
+ const disableControl = useCallback(async () => {
181
+ if (keyboardProcessRef.current) {
182
+ keyboardProcessRef.current.stop();
183
+ }
184
+ if (directProcessRef.current) {
185
+ directProcessRef.current.stop();
186
+ }
187
+ setControlEnabled(false);
188
+ }, [selectedTeleoperatorType]);
189
+
190
+ // Keyboard event handlers (guarded to not interfere with inputs/shortcuts)
191
+ const handleKeyDown = useCallback(
192
+ (event: KeyboardEvent) => {
193
+ if (!teleopState.isActive || !keyboardProcessRef.current) return;
194
+
195
+ // Ignore when user is typing in inputs/textareas or contenteditable elements
196
+ const target = event.target as HTMLElement | null;
197
+ const isEditableTarget = !!(
198
+ target &&
199
+ (target.tagName === "INPUT" ||
200
+ target.tagName === "TEXTAREA" ||
201
+ target.tagName === "SELECT" ||
202
+ target.isContentEditable ||
203
+ target.closest(
204
+ '[role="textbox"], [contenteditable="true"], input, textarea, select'
205
+ ))
206
+ );
207
+ if (isEditableTarget) return;
208
+
209
+ // Allow browser/system shortcuts (e.g. Ctrl/Cmd+R, Ctrl/Cmd+L, etc.)
210
+ if (event.metaKey || event.ctrlKey || event.altKey) return;
211
+
212
+ // Only handle specific teleop keys
213
+ const rawKey = event.key;
214
+ const normalizedKey = rawKey.length === 1 ? rawKey.toLowerCase() : rawKey;
215
+ const allowedKeys = new Set([
216
+ "ArrowUp",
217
+ "ArrowDown",
218
+ "ArrowLeft",
219
+ "ArrowRight",
220
+ "w",
221
+ "a",
222
+ "s",
223
+ "d",
224
+ "q",
225
+ "e",
226
+ "o",
227
+ "c",
228
+ "Escape",
229
+ ]);
230
+ if (!allowedKeys.has(normalizedKey)) return;
231
+
232
+ event.preventDefault();
233
+ const keyboardTeleoperator = keyboardProcessRef.current.teleoperator;
234
+ if (keyboardTeleoperator && "updateKeyState" in keyboardTeleoperator) {
235
+ (
236
+ keyboardTeleoperator as {
237
+ updateKeyState: (key: string, pressed: boolean) => void;
238
+ }
239
+ ).updateKeyState(normalizedKey, true);
240
+ }
241
+ },
242
+ [teleopState.isActive]
243
+ );
244
+
245
+ const handleKeyUp = useCallback(
246
+ (event: KeyboardEvent) => {
247
+ if (!teleopState.isActive || !keyboardProcessRef.current) return;
248
+
249
+ // Ignore when user is typing in inputs/textareas or contenteditable elements
250
+ const target = event.target as HTMLElement | null;
251
+ const isEditableTarget = !!(
252
+ target &&
253
+ (target.tagName === "INPUT" ||
254
+ target.tagName === "TEXTAREA" ||
255
+ target.tagName === "SELECT" ||
256
+ target.isContentEditable ||
257
+ target.closest(
258
+ '[role="textbox"], [contenteditable="true"], input, textarea, select'
259
+ ))
260
+ );
261
+ if (isEditableTarget) return;
262
+
263
+ // Allow browser/system shortcuts (e.g. Ctrl/Cmd+R, Ctrl/Cmd+L, etc.)
264
+ if (event.metaKey || event.ctrlKey || event.altKey) return;
265
+
266
+ // Only handle specific teleop keys
267
+ const rawKey = event.key;
268
+ const normalizedKey = rawKey.length === 1 ? rawKey.toLowerCase() : rawKey;
269
+ const allowedKeys = new Set([
270
+ "ArrowUp",
271
+ "ArrowDown",
272
+ "ArrowLeft",
273
+ "ArrowRight",
274
+ "w",
275
+ "a",
276
+ "s",
277
+ "d",
278
+ "q",
279
+ "e",
280
+ "o",
281
+ "c",
282
+ "Escape",
283
+ ]);
284
+ if (!allowedKeys.has(normalizedKey)) return;
285
+
286
+ event.preventDefault();
287
+ const keyboardTeleoperator = keyboardProcessRef.current.teleoperator;
288
+ if (keyboardTeleoperator && "updateKeyState" in keyboardTeleoperator) {
289
+ (
290
+ keyboardTeleoperator as {
291
+ updateKeyState: (key: string, pressed: boolean) => void;
292
+ }
293
+ ).updateKeyState(normalizedKey, false);
294
+ }
295
+ },
296
+ [teleopState.isActive]
297
+ );
298
+
299
+ // Register keyboard events
300
+ useEffect(() => {
301
+ if (teleopState.isActive && selectedTeleoperatorType === "keyboard") {
302
+ window.addEventListener("keydown", handleKeyDown);
303
+ window.addEventListener("keyup", handleKeyUp);
304
+
305
+ return () => {
306
+ window.removeEventListener("keydown", handleKeyDown);
307
+ window.removeEventListener("keyup", handleKeyUp);
308
+ };
309
+ }
310
+ }, [
311
+ teleopState.isActive,
312
+ selectedTeleoperatorType,
313
+ handleKeyDown,
314
+ handleKeyUp,
315
+ ]);
316
+
317
+ // Cleanup on unmount
318
+ useEffect(() => {
319
+ return () => {
320
+ const cleanup = async () => {
321
+ try {
322
+ if (keyboardProcessRef.current) {
323
+ await keyboardProcessRef.current.disconnect();
324
+ keyboardProcessRef.current = null;
325
+ }
326
+ if (directProcessRef.current) {
327
+ await directProcessRef.current.disconnect();
328
+ directProcessRef.current = null;
329
+ }
330
+ } catch (error) {
331
+ console.warn("Error during teleoperation cleanup:", error);
332
+ }
333
+ };
334
+ cleanup();
335
+ };
336
+ }, []);
337
+
338
+ // Memoize teleoperators array for the Recorder component
339
+ const memoizedTeleoperators = useMemo(() => {
340
+ if (!controlEnabled) return [];
341
+
342
+ // Return the active teleoperator based on selection
343
+ const activeTeleoperator =
344
+ selectedTeleoperatorType === "keyboard"
345
+ ? keyboardProcessRef.current?.teleoperator
346
+ : directProcessRef.current?.teleoperator;
347
+
348
+ return activeTeleoperator ? [activeTeleoperator] : [];
349
+ }, [controlEnabled, selectedTeleoperatorType]);
350
+
351
+ return (
352
+ <div className="space-y-6">
353
+ {/* Robot Movement Recorder Header */}
354
+ <Card className="border-0 rounded-none">
355
+ <div className="p-4 border-b border-white/10">
356
+ <div className="flex items-center justify-between">
357
+ <div className="flex items-center gap-4">
358
+ <div className="w-1 h-8 bg-primary"></div>
359
+ <div>
360
+ <h3 className="text-xl font-bold text-foreground font-mono tracking-wider uppercase">
361
+ robot movement recorder
362
+ </h3>
363
+ <p className="text-sm text-muted-foreground font-mono">
364
+ dataset{" "}
365
+ <span className="text-muted-foreground">recording</span>{" "}
366
+ interface
367
+ </p>
368
+ </div>
369
+ </div>
370
+ <div className="flex items-center gap-6">
371
+ <div className="border-l border-white/10 pl-6 flex items-center gap-4">
372
+ <Button
373
+ variant="outline"
374
+ size="lg"
375
+ className="gap-2"
376
+ onClick={() => setShowConfigure(!showConfigure)}
377
+ >
378
+ <Settings className="w-5 h-5" />
379
+ Configure
380
+ </Button>
381
+
382
+ <Button
383
+ variant={
384
+ recorderCallbacks?.isRecording ? "destructive" : "default"
385
+ }
386
+ size="lg"
387
+ className="gap-2"
388
+ disabled={!robot.isConnected || !recorderCallbacks}
389
+ onClick={
390
+ recorderCallbacks?.isRecording
391
+ ? recorderCallbacks.stopRecording
392
+ : recorderCallbacks?.startRecording
393
+ }
394
+ >
395
+ {recorderCallbacks?.isRecording ? (
396
+ <>
397
+ <Square className="w-5 h-5" />
398
+ Stop Recording
399
+ </>
400
+ ) : (
401
+ <>
402
+ <Record className="w-5 h-5" />
403
+ Start Recording
404
+ </>
405
+ )}
406
+ </Button>
407
+ </div>
408
+ <div className="flex items-center gap-2">
409
+ <span className="text-sm font-mono text-muted-foreground uppercase">
410
+ robot:
411
+ </span>
412
+ <Badge
413
+ variant="outline"
414
+ className={cn(
415
+ "border-primary/50 bg-primary/20 text-primary font-mono text-xs",
416
+ robot.isConnected
417
+ ? "border-green-500/50 bg-green-500/20 text-green-400"
418
+ : "border-red-500/50 bg-red-500/20 text-red-400"
419
+ )}
420
+ >
421
+ {robot.isConnected ? "ONLINE" : "OFFLINE"}
422
+ </Badge>
423
+ </div>
424
+ </div>
425
+ </div>
426
+ </div>
427
+ <div className="p-6 space-y-6">
428
+ {/* Step-by-Step Recording Guide */}
429
+ <div className="bg-black/40 border border-white/20 rounded-lg p-6">
430
+ <h4 className="text-lg font-semibold text-foreground mb-4">
431
+ How to Record Robot Data
432
+ </h4>
433
+ <div className="space-y-4">
434
+ {/* Step 1 */}
435
+ <div className="flex items-start gap-4">
436
+ <div
437
+ className={cn(
438
+ "flex-shrink-0 w-8 h-8 border rounded-full flex items-center justify-center text-sm font-mono",
439
+ currentStep > 1
440
+ ? "bg-green-500/20 border-green-500/50 text-green-400" // Completed
441
+ : currentStep === 1
442
+ ? "bg-yellow-500/20 border-yellow-500/50 text-yellow-400" // Active
443
+ : "bg-muted/20 border-muted/50 text-muted-foreground" // Inactive
444
+ )}
445
+ >
446
+ {currentStep > 1 ? <Check className="w-4 h-4" /> : "1"}
447
+ </div>
448
+ <div className="flex-1">
449
+ <h5 className="font-semibold text-foreground">
450
+ Enable Robot Control
451
+ </h5>
452
+ <p className="text-sm text-muted-foreground mb-3">
453
+ Choose how you want to control the robot during recording.
454
+ </p>
455
+
456
+ {!controlEnabled && robot.isConnected && (
457
+ <div className="space-y-3">
458
+ <div className="flex items-center gap-3">
459
+ <label className="text-sm text-muted-foreground min-w-0">
460
+ Control Type:
461
+ </label>
462
+ <Select
463
+ value={selectedTeleoperatorType}
464
+ onValueChange={(value: "direct" | "keyboard") =>
465
+ setSelectedTeleoperatorType(value)
466
+ }
467
+ >
468
+ <SelectTrigger className="w-48 bg-black/20 border-white/10">
469
+ <SelectValue />
470
+ </SelectTrigger>
471
+ <SelectContent>
472
+ <SelectItem value="direct">
473
+ <div className="flex items-center gap-2">
474
+ <Gamepad2 className="w-4 h-4" />
475
+ Direct Control
476
+ </div>
477
+ </SelectItem>
478
+ <SelectItem value="keyboard">
479
+ <div className="flex items-center gap-2">
480
+ <Keyboard className="w-4 h-4" />
481
+ Keyboard Control
482
+ </div>
483
+ </SelectItem>
484
+ </SelectContent>
485
+ </Select>
486
+ </div>
487
+ <p className="text-xs text-muted-foreground">
488
+ {selectedTeleoperatorType === "direct"
489
+ ? "Programmatic control (use teleoperation page for manual sliders)"
490
+ : "Move robot using keyboard keys (WASD, arrows, Q/E, O/C)"}
491
+ </p>
492
+ </div>
493
+ )}
494
+
495
+ <div className="mt-3">
496
+ <Button
497
+ variant={controlEnabled ? "destructive" : "default"}
498
+ onClick={controlEnabled ? disableControl : enableControl}
499
+ disabled={!robot.isConnected}
500
+ className="gap-2"
501
+ size="sm"
502
+ >
503
+ {selectedTeleoperatorType === "keyboard" ? (
504
+ <Keyboard className="w-4 h-4" />
505
+ ) : (
506
+ <Gamepad2 className="w-4 h-4" />
507
+ )}
508
+ {controlEnabled
509
+ ? `${selectedTeleoperatorType} Control Active`
510
+ : `Enable ${selectedTeleoperatorType} Control`}
511
+ </Button>
512
+ </div>
513
+ </div>
514
+ </div>
515
+
516
+ {/* Step 2 */}
517
+ <div className="flex items-start gap-4">
518
+ <div
519
+ className={cn(
520
+ "flex-shrink-0 w-8 h-8 border rounded-full flex items-center justify-center text-sm font-mono",
521
+ currentStep > 2
522
+ ? "bg-green-500/20 border-green-500/50 text-green-400" // Completed
523
+ : currentStep === 2
524
+ ? "bg-yellow-500/20 border-yellow-500/50 text-yellow-400" // Active
525
+ : "bg-muted/20 border-muted/50 text-muted-foreground" // Inactive
526
+ )}
527
+ >
528
+ {currentStep > 2 ? <Check className="w-4 h-4" /> : "2"}
529
+ </div>
530
+ <div>
531
+ <h5
532
+ className={cn(
533
+ "font-semibold",
534
+ controlEnabled
535
+ ? "text-foreground"
536
+ : "text-muted-foreground"
537
+ )}
538
+ >
539
+ Start Recording
540
+ </h5>
541
+ <p className="text-sm text-muted-foreground">
542
+ Click "Start Recording" to begin capturing robot movements.
543
+ </p>
544
+ {null}
545
+ </div>
546
+ </div>
547
+
548
+ {/* Step 3 */}
549
+ <div className="flex items-start gap-4">
550
+ <div
551
+ className={cn(
552
+ "flex-shrink-0 w-8 h-8 border rounded-full flex items-center justify-center text-sm font-mono",
553
+ currentStep === 3
554
+ ? "bg-yellow-500/20 border-yellow-500/50 text-yellow-400 animate-pulse" // Active with pulse
555
+ : "bg-muted/20 border-muted/50 text-muted-foreground" // Inactive
556
+ )}
557
+ >
558
+ 3
559
+ </div>
560
+ <div>
561
+ <h5
562
+ className={cn(
563
+ "font-semibold",
564
+ currentStep === 3
565
+ ? "text-foreground"
566
+ : "text-muted-foreground"
567
+ )}
568
+ >
569
+ Move the Robot
570
+ </h5>
571
+ <p className="text-sm text-muted-foreground">
572
+ {currentStep === 3 ? (
573
+ <span className="text-yellow-400">
574
+ 🎯 Recording active! Move the robot to demonstrate your
575
+ task.
576
+ </span>
577
+ ) : (
578
+ "Move the robot manually to demonstrate the task you want to teach. Recording will capture your movements."
579
+ )}
580
+ </p>
581
+ </div>
582
+ </div>
583
+ </div>
584
+ </div>
585
+
586
+ {/* Robot Movement Recorder */}
587
+ <Recorder
588
+ teleoperators={memoizedTeleoperators}
589
+ robot={robot}
590
+ onNeedsTeleoperation={enableControl}
591
+ showConfigure={showConfigure}
592
+ onRecorderReady={setRecorderCallbacks}
593
+ />
594
+ </div>
595
+ </Card>
596
+ </div>
597
+ );
598
+ }
examples/cyberpunk-standalone/src/components/roadmap-section.tsx CHANGED
@@ -36,14 +36,14 @@ const roadmapItems: RoadmapItem[] = [
36
  description: "Node.js CLI tools for robot control and automation scripts",
37
  status: "completed",
38
  },
39
- {
40
- title: "SO-100 leader arm",
41
- description: "Leader arm teleoperation support for intuitive robot control",
42
- status: "in_progress",
43
- },
44
  {
45
  title: "record",
46
  description: "Record robot trajectories and sensor data to create datasets",
 
 
 
 
 
47
  status: "in_progress",
48
  },
49
  {
 
36
  description: "Node.js CLI tools for robot control and automation scripts",
37
  status: "completed",
38
  },
 
 
 
 
 
39
  {
40
  title: "record",
41
  description: "Record robot trajectories and sensor data to create datasets",
42
+ status: "completed",
43
+ },
44
+ {
45
+ title: "SO-100 leader arm",
46
+ description: "Leader arm teleoperation support for intuitive robot control",
47
  status: "in_progress",
48
  },
49
  {
examples/cyberpunk-standalone/src/components/teleoperation-view.tsx CHANGED
@@ -22,7 +22,6 @@ import {
22
  } from "@lerobot/web";
23
  import { getUnifiedRobotData } from "@/lib/unified-storage";
24
  import VirtualKey from "@/components/VirtualKey";
25
- import { Recorder } from "@/components/recorder";
26
  import { Canvas } from "@react-three/fiber";
27
  import { Physics } from "@react-three/cannon";
28
  import * as THREE from "three";
@@ -196,12 +195,48 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
196
  };
197
  }, []);
198
 
199
- // Keyboard event handlers
200
  const handleKeyDown = useCallback(
201
  (event: KeyboardEvent) => {
202
  if (!teleopState.isActive || !keyboardProcessRef.current) return;
203
 
204
- const key = event.key;
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
205
  event.preventDefault();
206
 
207
  const keyboardTeleoperator = keyboardProcessRef.current.teleoperator;
@@ -210,7 +245,7 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
210
  keyboardTeleoperator as {
211
  updateKeyState: (key: string, pressed: boolean) => void;
212
  }
213
- ).updateKeyState(key, true);
214
  }
215
  },
216
  [teleopState.isActive]
@@ -220,7 +255,43 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
220
  (event: KeyboardEvent) => {
221
  if (!teleopState.isActive || !keyboardProcessRef.current) return;
222
 
223
- const key = event.key;
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
224
  event.preventDefault();
225
 
226
  const keyboardTeleoperator = keyboardProcessRef.current.teleoperator;
@@ -229,7 +300,7 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
229
  keyboardTeleoperator as {
230
  updateKeyState: (key: string, pressed: boolean) => void;
231
  }
232
- ).updateKeyState(key, false);
233
  }
234
  },
235
  [teleopState.isActive]
@@ -414,15 +485,6 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
414
  const keyStates = teleopState?.keyStates || {};
415
  const controls = SO100_KEYBOARD_CONTROLS;
416
 
417
- // Memoize teleoperators array to prevent unnecessary re-renders of the Recorder component
418
- const memoizedTeleoperators = useMemo(() => {
419
- if (!directProcessRef.current) return [];
420
-
421
- return [
422
- directProcessRef.current?.teleoperator,
423
- ].filter(Boolean);
424
- }, [directProcessRef.current]);
425
-
426
  return (
427
  <>
428
  <Card className="border-0 rounded-none">
@@ -456,19 +518,37 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
456
  <Power className="w-5 h-5 mr-2" /> Control Robot
457
  </Button>
458
  )}
459
- <div className="flex items-center gap-2">
460
- <span className="text-sm font-mono text-muted-foreground uppercase">
461
- status:
462
- </span>
463
- <Badge
464
- variant="outline"
465
- className={cn(
466
- "border-primary/50 bg-primary/20 text-primary font-mono text-xs",
467
- teleopState?.isActive && "animate-pulse-slow"
468
- )}
469
- >
470
- {teleopState?.isActive ? "ACTIVE" : "STOPPED"}
471
- </Badge>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
472
  </div>
473
  </div>
474
  </div>
@@ -759,13 +839,6 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
759
  </div>
760
  </div>
761
  </Card>
762
-
763
- {/* Robot Movement Recorder - Always show UI */}
764
- <Recorder
765
- teleoperators={memoizedTeleoperators}
766
- robot={robot}
767
- onNeedsTeleoperation={initializeTeleoperation}
768
- />
769
  </>
770
  );
771
  }
 
22
  } from "@lerobot/web";
23
  import { getUnifiedRobotData } from "@/lib/unified-storage";
24
  import VirtualKey from "@/components/VirtualKey";
 
25
  import { Canvas } from "@react-three/fiber";
26
  import { Physics } from "@react-three/cannon";
27
  import * as THREE from "three";
 
195
  };
196
  }, []);
197
 
198
+ // Keyboard event handlers (guarded to not interfere with inputs/shortcuts)
199
  const handleKeyDown = useCallback(
200
  (event: KeyboardEvent) => {
201
  if (!teleopState.isActive || !keyboardProcessRef.current) return;
202
 
203
+ // Ignore when user is typing in inputs/textareas or contenteditable elements
204
+ const target = event.target as HTMLElement | null;
205
+ const isEditableTarget = !!(
206
+ target &&
207
+ (target.tagName === "INPUT" ||
208
+ target.tagName === "TEXTAREA" ||
209
+ target.tagName === "SELECT" ||
210
+ target.isContentEditable ||
211
+ target.closest(
212
+ '[role="textbox"], [contenteditable="true"], input, textarea, select'
213
+ ))
214
+ );
215
+ if (isEditableTarget) return;
216
+
217
+ // Allow browser/system shortcuts (e.g. Ctrl/Cmd+R, Ctrl/Cmd+L, etc.)
218
+ if (event.metaKey || event.ctrlKey || event.altKey) return;
219
+
220
+ // Only handle specific teleop keys
221
+ const rawKey = event.key;
222
+ const normalizedKey = rawKey.length === 1 ? rawKey.toLowerCase() : rawKey;
223
+ const allowedKeys = new Set([
224
+ "ArrowUp",
225
+ "ArrowDown",
226
+ "ArrowLeft",
227
+ "ArrowRight",
228
+ "w",
229
+ "a",
230
+ "s",
231
+ "d",
232
+ "q",
233
+ "e",
234
+ "o",
235
+ "c",
236
+ "Escape",
237
+ ]);
238
+ if (!allowedKeys.has(normalizedKey)) return;
239
+
240
  event.preventDefault();
241
 
242
  const keyboardTeleoperator = keyboardProcessRef.current.teleoperator;
 
245
  keyboardTeleoperator as {
246
  updateKeyState: (key: string, pressed: boolean) => void;
247
  }
248
+ ).updateKeyState(normalizedKey, true);
249
  }
250
  },
251
  [teleopState.isActive]
 
255
  (event: KeyboardEvent) => {
256
  if (!teleopState.isActive || !keyboardProcessRef.current) return;
257
 
258
+ // Ignore when user is typing in inputs/textareas or contenteditable elements
259
+ const target = event.target as HTMLElement | null;
260
+ const isEditableTarget = !!(
261
+ target &&
262
+ (target.tagName === "INPUT" ||
263
+ target.tagName === "TEXTAREA" ||
264
+ target.tagName === "SELECT" ||
265
+ target.isContentEditable ||
266
+ target.closest(
267
+ '[role="textbox"], [contenteditable="true"], input, textarea, select'
268
+ ))
269
+ );
270
+ if (isEditableTarget) return;
271
+
272
+ // Allow browser/system shortcuts (e.g. Ctrl/Cmd+R, Ctrl/Cmd+L, etc.)
273
+ if (event.metaKey || event.ctrlKey || event.altKey) return;
274
+
275
+ // Only handle specific teleop keys
276
+ const rawKey = event.key;
277
+ const normalizedKey = rawKey.length === 1 ? rawKey.toLowerCase() : rawKey;
278
+ const allowedKeys = new Set([
279
+ "ArrowUp",
280
+ "ArrowDown",
281
+ "ArrowLeft",
282
+ "ArrowRight",
283
+ "w",
284
+ "a",
285
+ "s",
286
+ "d",
287
+ "q",
288
+ "e",
289
+ "o",
290
+ "c",
291
+ "Escape",
292
+ ]);
293
+ if (!allowedKeys.has(normalizedKey)) return;
294
+
295
  event.preventDefault();
296
 
297
  const keyboardTeleoperator = keyboardProcessRef.current.teleoperator;
 
300
  keyboardTeleoperator as {
301
  updateKeyState: (key: string, pressed: boolean) => void;
302
  }
303
+ ).updateKeyState(normalizedKey, false);
304
  }
305
  },
306
  [teleopState.isActive]
 
485
  const keyStates = teleopState?.keyStates || {};
486
  const controls = SO100_KEYBOARD_CONTROLS;
487
 
 
 
 
 
 
 
 
 
 
488
  return (
489
  <>
490
  <Card className="border-0 rounded-none">
 
518
  <Power className="w-5 h-5 mr-2" /> Control Robot
519
  </Button>
520
  )}
521
+ <div className="flex items-center gap-4">
522
+ <div className="flex items-center gap-2">
523
+ <span className="text-sm font-mono text-muted-foreground uppercase">
524
+ robot:
525
+ </span>
526
+ <Badge
527
+ variant="outline"
528
+ className={cn(
529
+ "border-primary/50 bg-primary/20 text-primary font-mono text-xs",
530
+ robot.isConnected
531
+ ? "border-green-500/50 bg-green-500/20 text-green-400"
532
+ : "border-red-500/50 bg-red-500/20 text-red-400"
533
+ )}
534
+ >
535
+ {robot.isConnected ? "ONLINE" : "OFFLINE"}
536
+ </Badge>
537
+ </div>
538
+ <div className="flex items-center gap-2">
539
+ <span className="text-sm font-mono text-muted-foreground uppercase">
540
+ control:
541
+ </span>
542
+ <Badge
543
+ variant="outline"
544
+ className={cn(
545
+ "border-primary/50 bg-primary/20 text-primary font-mono text-xs",
546
+ teleopState?.isActive && "animate-pulse-slow"
547
+ )}
548
+ >
549
+ {teleopState?.isActive ? "ACTIVE" : "STOPPED"}
550
+ </Badge>
551
+ </div>
552
  </div>
553
  </div>
554
  </div>
 
839
  </div>
840
  </div>
841
  </Card>
 
 
 
 
 
 
 
842
  </>
843
  );
844
  }
examples/cyberpunk-standalone/src/components/teleoperator-episodes-view.tsx CHANGED
@@ -7,23 +7,38 @@ import { Button } from "./ui/button";
7
 
8
  interface TeleoperatorEpisodesViewProps {
9
  teleoperatorData?: LeRobotEpisode[];
 
 
10
  }
11
 
12
- export function TeleoperatorEpisodesView({ teleoperatorData }: TeleoperatorEpisodesViewProps) {
 
 
 
 
13
  // State to track which episodes are expanded
14
- const [expandedEpisodes, setExpandedEpisodes] = useState<Record<number, boolean>>({});
15
-
 
 
 
 
 
 
 
16
  // Toggle expanded state for an episode
17
  const toggleEpisode = (index: number) => {
18
- setExpandedEpisodes(prev => ({
19
  ...prev,
20
- [index]: !prev[index]
21
  }));
22
  };
23
-
24
  return (
25
  <div className="flex flex-col gap-2">
26
- <div className="text-sm text-center text-muted-foreground mb-2">List of recorded episodes</div>
 
 
27
  <div className="flex flex-col gap-1">
28
  {/* Header */}
29
  <div className="flex flex-row font-medium text-sm">
@@ -31,7 +46,7 @@ export function TeleoperatorEpisodesView({ teleoperatorData }: TeleoperatorEpiso
31
  <div className="flex-1 px-4 py-2">time length</div>
32
  <div className="flex-1 px-4 py-2">frames</div>
33
  </div>
34
-
35
  {/* Body */}
36
  {teleoperatorData && teleoperatorData.length > 0 ? (
37
  teleoperatorData.map((episode: LeRobotEpisode, i: number) => (
@@ -39,26 +54,37 @@ export function TeleoperatorEpisodesView({ teleoperatorData }: TeleoperatorEpiso
39
  {/* Episode row */}
40
  <div className="flex flex-row">
41
  <div className="flex-1 px-4 py-2 font-mono">{i}</div>
42
- <div className="flex-1 px-4 py-2 font-mono">{episode.timespan}</div>
43
- <div className="flex-1 px-4 py-2 font-mono">{episode.length}</div>
44
- <div className="px-4 py-2 cursor-pointer" onClick={() => toggleEpisode(i)}>
45
- {expandedEpisodes[i] ?
46
- <Button>hide frames</Button> :
47
- <Button>show frames</Button>}
 
 
 
 
 
 
 
 
 
48
  </div>
49
  </div>
50
-
51
  {/* Frames (collapsible) */}
52
  {expandedEpisodes[i] && (
53
- <TeleoperatorFramesView frames={episode.frames} />
 
 
 
 
54
  )}
55
  </div>
56
  ))
57
  ) : (
58
  <div className="flex flex-row border-t border-gray-700">
59
- <div
60
- className="flex-1 px-4 py-4 text-center text-muted-foreground"
61
- >
62
  No episodes recorded yet. Click "Start Recording" to begin.
63
  </div>
64
  </div>
 
7
 
8
  interface TeleoperatorEpisodesViewProps {
9
  teleoperatorData?: LeRobotEpisode[];
10
+ isRecording?: boolean;
11
+ refreshTick?: number;
12
  }
13
 
14
+ export function TeleoperatorEpisodesView({
15
+ teleoperatorData,
16
+ isRecording,
17
+ refreshTick,
18
+ }: TeleoperatorEpisodesViewProps) {
19
  // State to track which episodes are expanded
20
+ const [expandedEpisodes, setExpandedEpisodes] = useState<
21
+ Record<number, boolean>
22
+ >({});
23
+
24
+ const formatSeconds = (s: number): string => {
25
+ if (!Number.isFinite(s)) return String(s);
26
+ return s.toFixed(3);
27
+ };
28
+
29
  // Toggle expanded state for an episode
30
  const toggleEpisode = (index: number) => {
31
+ setExpandedEpisodes((prev) => ({
32
  ...prev,
33
+ [index]: !prev[index],
34
  }));
35
  };
36
+
37
  return (
38
  <div className="flex flex-col gap-2">
39
+ <div className="text-sm text-center text-muted-foreground mb-2">
40
+ List of recorded episodes
41
+ </div>
42
  <div className="flex flex-col gap-1">
43
  {/* Header */}
44
  <div className="flex flex-row font-medium text-sm">
 
46
  <div className="flex-1 px-4 py-2">time length</div>
47
  <div className="flex-1 px-4 py-2">frames</div>
48
  </div>
49
+
50
  {/* Body */}
51
  {teleoperatorData && teleoperatorData.length > 0 ? (
52
  teleoperatorData.map((episode: LeRobotEpisode, i: number) => (
 
54
  {/* Episode row */}
55
  <div className="flex flex-row">
56
  <div className="flex-1 px-4 py-2 font-mono">{i}</div>
57
+ <div className="flex-1 px-4 py-2 font-mono">
58
+ {formatSeconds(episode.timespan)}
59
+ </div>
60
+ <div className="flex-1 px-4 py-2 font-mono">
61
+ {episode.length}
62
+ </div>
63
+ <div
64
+ className="px-4 py-2 cursor-pointer"
65
+ onClick={() => toggleEpisode(i)}
66
+ >
67
+ {expandedEpisodes[i] ? (
68
+ <Button>hide frames</Button>
69
+ ) : (
70
+ <Button>show frames</Button>
71
+ )}
72
  </div>
73
  </div>
74
+
75
  {/* Frames (collapsible) */}
76
  {expandedEpisodes[i] && (
77
+ <TeleoperatorFramesView
78
+ frames={episode.frames}
79
+ isRecording={isRecording}
80
+ refreshTick={refreshTick}
81
+ />
82
  )}
83
  </div>
84
  ))
85
  ) : (
86
  <div className="flex flex-row border-t border-gray-700">
87
+ <div className="flex-1 px-4 py-4 text-center text-muted-foreground">
 
 
88
  No episodes recorded yet. Click "Start Recording" to begin.
89
  </div>
90
  </div>
examples/cyberpunk-standalone/src/components/teleoperator-frames-view.tsx CHANGED
@@ -1,21 +1,28 @@
1
  "use client";
2
 
 
3
  import { NonIndexedLeRobotDatasetRow } from "@lerobot/web";
4
  import { TeleoperatorJointGraph } from "./teleoperator-joint-graph";
5
 
6
  interface TeleoperatorFramesViewProps {
7
  frames: NonIndexedLeRobotDatasetRow[];
 
 
8
  }
9
 
10
- export function TeleoperatorFramesView({ frames }: TeleoperatorFramesViewProps) {
 
 
 
 
11
  // Joint names in the order they appear in the arrays
12
  const jointNames = [
13
- "shoulder_pan",
14
  "shoulder_lift",
15
  "elbow_flex",
16
  "wrist_flex",
17
  "wrist_roll",
18
- "gripper"
19
  ];
20
 
21
  // Helper function to format an object as a column of key-value pairs with joint names
@@ -24,55 +31,77 @@ export function TeleoperatorFramesView({ frames }: TeleoperatorFramesViewProps)
24
  .map(([key, value]) => {
25
  // Convert numeric key to joint name if possible
26
  const index = parseInt(key);
27
- const jointName = !isNaN(index) && index < jointNames.length ? jointNames[index] : key;
28
- return `${jointName}: ${value}`;
 
 
 
 
29
  })
30
- .join('\n');
31
  };
32
-
 
 
 
 
 
33
  return (
34
  <div className="ml-8 mr-4 mb-2">
35
  {/* Joint visualization graph */}
36
- <TeleoperatorJointGraph frames={frames} />
37
-
 
 
 
 
38
  {/* Frames container with horizontal scroll */}
39
  <div className="bg-gray-800/50 rounded-md overflow-hidden">
40
- <div className="overflow-x-auto">
41
- {/* Frames header */}
42
- <table className="w-full min-w-max table-fixed">
43
- <thead>
44
- <tr className="text-xs font-medium bg-gray-800/80 text-gray-300">
45
- <th className="w-16 px-2 py-1 text-left">Frame</th>
46
- <th className="w-64 px-2 py-1 text-left">Timestamp</th>
47
- <th className="w-[300px] px-2 py-1 text-left">Action</th>
48
- <th className="w-[500px] px-2 py-1 text-left">State</th>
49
- </tr>
50
- </thead>
51
-
52
- {/* Frame rows */}
53
- <tbody className="max-h-60 overflow-y-auto">
54
- {frames.map((frame: NonIndexedLeRobotDatasetRow, frameIndex: number) => (
55
- <tr key={frameIndex} className="text-xs border-t border-gray-700/50">
56
- <td className="w-16 px-2 py-1 font-mono whitespace-nowrap">{frameIndex}</td>
57
- <td className="w-64 px-2 py-1 font-mono whitespace-nowrap">
58
- {frame.timestamp}
59
- </td>
60
- <td className="w-[200px] px-2 py-1 font-mono whitespace-pre-wrap align-top">
61
- {Object.keys(frame.action).length > 0 ?
62
- formatArrayAsColumn(frame.action) :
63
- '-'}
64
- </td>
65
- <td className="w-[300px] px-2 py-1 font-mono whitespace-pre-wrap align-top">
66
- {Object.keys(frame["observation.state"]).length > 0 ?
67
- formatArrayAsColumn(frame["observation.state"]) :
68
- '-'}
69
- </td>
70
  </tr>
71
- ))}
72
- </tbody>
73
- </table>
74
- </div>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
75
  </div>
76
  </div>
77
  );
78
- }
 
1
  "use client";
2
 
3
+ import React, { memo, useMemo } from "react";
4
  import { NonIndexedLeRobotDatasetRow } from "@lerobot/web";
5
  import { TeleoperatorJointGraph } from "./teleoperator-joint-graph";
6
 
7
  interface TeleoperatorFramesViewProps {
8
  frames: NonIndexedLeRobotDatasetRow[];
9
+ isRecording?: boolean;
10
+ refreshTick?: number;
11
  }
12
 
13
+ export const TeleoperatorFramesView = memo(function TeleoperatorFramesView({
14
+ frames,
15
+ isRecording,
16
+ refreshTick,
17
+ }: TeleoperatorFramesViewProps) {
18
  // Joint names in the order they appear in the arrays
19
  const jointNames = [
20
+ "shoulder_pan",
21
  "shoulder_lift",
22
  "elbow_flex",
23
  "wrist_flex",
24
  "wrist_roll",
25
+ "gripper",
26
  ];
27
 
28
  // Helper function to format an object as a column of key-value pairs with joint names
 
31
  .map(([key, value]) => {
32
  // Convert numeric key to joint name if possible
33
  const index = parseInt(key);
34
+ const jointName =
35
+ !isNaN(index) && index < jointNames.length ? jointNames[index] : key;
36
+ const formatted = Number.isFinite(value)
37
+ ? value.toFixed(2)
38
+ : String(value);
39
+ return `${jointName}: ${formatted}`;
40
  })
41
+ .join("\n");
42
  };
43
+
44
+ const visibleFrames = useMemo(
45
+ () => frames || ([] as NonIndexedLeRobotDatasetRow[]),
46
+ [frames, refreshTick]
47
+ );
48
+
49
  return (
50
  <div className="ml-8 mr-4 mb-2">
51
  {/* Joint visualization graph */}
52
+ <TeleoperatorJointGraph
53
+ frames={visibleFrames}
54
+ refreshTick={refreshTick}
55
+ isRecording={isRecording}
56
+ />
57
+
58
  {/* Frames container with horizontal scroll */}
59
  <div className="bg-gray-800/50 rounded-md overflow-hidden">
60
+ <div className="overflow-x-auto">
61
+ {/* Frames header */}
62
+ <table className="w-full min-w-max table-fixed">
63
+ <thead>
64
+ <tr className="text-xs font-medium bg-gray-800/80 text-gray-300">
65
+ <th className="w-16 px-2 py-1 text-left">Frame</th>
66
+ <th className="w-64 px-2 py-1 text-left">Timestamp</th>
67
+ <th className="w-[300px] px-2 py-1 text-left">Action</th>
68
+ <th className="w-[500px] px-2 py-1 text-left">State</th>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
69
  </tr>
70
+ </thead>
71
+
72
+ {/* Frame rows */}
73
+ <tbody className="max-h-60 overflow-y-auto">
74
+ {visibleFrames.map(
75
+ (frame: NonIndexedLeRobotDatasetRow, frameIndex: number) => (
76
+ <tr
77
+ key={frameIndex}
78
+ className="text-xs border-t border-gray-700/50"
79
+ >
80
+ <td className="w-16 px-2 py-1 font-mono whitespace-nowrap">
81
+ {frameIndex}
82
+ </td>
83
+ <td className="w-64 px-2 py-1 font-mono whitespace-nowrap">
84
+ {Number.isFinite(frame.timestamp)
85
+ ? frame.timestamp.toFixed(3)
86
+ : String(frame.timestamp)}
87
+ </td>
88
+ <td className="w-[200px] px-2 py-1 font-mono whitespace-pre-wrap align-top">
89
+ {Object.keys(frame.action).length > 0
90
+ ? formatArrayAsColumn(frame.action)
91
+ : "-"}
92
+ </td>
93
+ <td className="w-[300px] px-2 py-1 font-mono whitespace-pre-wrap align-top">
94
+ {Object.keys(frame["observation.state"]).length > 0
95
+ ? formatArrayAsColumn(frame["observation.state"])
96
+ : "-"}
97
+ </td>
98
+ </tr>
99
+ )
100
+ )}
101
+ </tbody>
102
+ </table>
103
+ </div>
104
  </div>
105
  </div>
106
  );
107
+ });
examples/cyberpunk-standalone/src/components/teleoperator-joint-graph.tsx CHANGED
@@ -1,4 +1,4 @@
1
- import React from "react";
2
  import {
3
  LineChart,
4
  Line,
@@ -7,15 +7,21 @@ import {
7
  CartesianGrid,
8
  Tooltip,
9
  Legend,
10
- ResponsiveContainer
11
  } from "recharts";
12
  import { NonIndexedLeRobotDatasetRow } from "@lerobot/web";
13
 
14
  interface TeleoperatorJointGraphProps {
15
  frames: NonIndexedLeRobotDatasetRow[];
 
16
  }
17
 
18
- export function TeleoperatorJointGraph({ frames }: TeleoperatorJointGraphProps) {
 
 
 
 
 
19
  // Skip rendering if no frames
20
  if (!frames || frames.length === 0) {
21
  return null;
@@ -23,100 +29,129 @@ export function TeleoperatorJointGraph({ frames }: TeleoperatorJointGraphProps)
23
 
24
  // Use hardcoded joint names that match the LeRobot dataset format
25
  const jointNames = [
26
- "shoulder_pan",
27
  "shoulder_lift",
28
  "elbow_flex",
29
  "wrist_flex",
30
  "wrist_roll",
31
- "gripper"
32
  ];
33
-
34
  // Generate a color palette for the joints
35
  const colors = [
36
- "#8884d8", "#82ca9d", "#ffc658", "#ff8042", "#0088fe", "#00C49F",
37
- "#FFBB28", "#FF8042", "#a4de6c", "#d0ed57"
 
 
 
 
 
 
 
 
38
  ];
39
-
40
  // Prepare data for the chart - handling arrays
41
- const chartData = frames.map((frame, index) => {
42
- // Create base data point with index
43
- const dataPoint: any = {
44
- name: index,
45
- timestamp: frame.timestamp
46
- };
47
-
48
- // Add action values (assuming action is an array)
49
- if (Array.isArray(frame.action)) {
50
- // Map each array index to the corresponding joint name
51
- jointNames.forEach((jointName, i) => {
52
- if (i < frame.action.length) {
53
- dataPoint[`action_${jointName}`] = frame.action[i];
 
 
 
 
54
  }
55
- });
56
- }
57
-
58
- // Add observation state values (assuming observation.state is an array)
59
- if (Array.isArray(frame["observation.state"])) {
60
- // Map each array index to the corresponding joint name
61
- jointNames.forEach((jointName, i) => {
62
- if (i < frame["observation.state"].length) {
63
- dataPoint[`state_${jointName}`] = frame["observation.state"][i];
64
  }
65
- });
66
- }
67
-
68
- return dataPoint;
69
- });
70
-
 
 
 
 
 
 
 
71
  // Create lines for each joint
72
- const linesToRender = jointNames.flatMap(jointName => [
73
  {
74
  key: `action_${jointName}`,
75
  dataKey: `action_${jointName}`,
76
  name: `Action: ${jointName}`,
77
- isDotted: true
78
  },
79
  {
80
  key: `state_${jointName}`,
81
  dataKey: `state_${jointName}`,
82
  name: `State: ${jointName}`,
83
- isDotted: false
84
- }
85
  ]);
86
 
87
  return (
88
  <div className="w-full bg-gray-800/50 rounded-md p-4 mb-4">
89
- <h3 className="text-sm font-medium text-gray-300 mb-2">Joint Positions Over Time</h3>
 
 
90
  <ResponsiveContainer width="100%" height={300}>
91
  <LineChart
92
- data={chartData}
93
  margin={{
94
  top: 5,
95
  right: 30,
96
  left: 20,
97
- bottom: 5
98
  }}
99
  >
100
  <CartesianGrid strokeDasharray="3 3" stroke="#444" />
101
- <XAxis
102
- dataKey="name"
103
- label={{ value: 'Frame Index', position: 'insideBottomRight', offset: -10 }}
104
- stroke="#aaa"
 
 
 
 
105
  />
106
  <YAxis stroke="#aaa" />
107
- <Tooltip
108
- contentStyle={{ backgroundColor: '#333', borderColor: '#555' }}
109
- labelStyle={{ color: '#eee' }}
110
- itemStyle={{ color: '#eee' }}
111
  />
112
  <Legend />
113
-
114
  {/* Render all lines */}
115
  {linesToRender.map((lineConfig, index) => {
116
- const jointName = lineConfig.dataKey.replace(/^(action|state)_/, '');
 
 
 
117
  const jointIndex = jointNames.indexOf(jointName);
118
- const colorIndex = jointIndex >= 0 ? jointIndex : index % colors.length;
119
-
 
120
  return (
121
  <Line
122
  key={lineConfig.key}
@@ -134,4 +169,4 @@ export function TeleoperatorJointGraph({ frames }: TeleoperatorJointGraphProps)
134
  </ResponsiveContainer>
135
  </div>
136
  );
137
- }
 
1
+ import React, { memo, useMemo } from "react";
2
  import {
3
  LineChart,
4
  Line,
 
7
  CartesianGrid,
8
  Tooltip,
9
  Legend,
10
+ ResponsiveContainer,
11
  } from "recharts";
12
  import { NonIndexedLeRobotDatasetRow } from "@lerobot/web";
13
 
14
  interface TeleoperatorJointGraphProps {
15
  frames: NonIndexedLeRobotDatasetRow[];
16
+ refreshTick?: number;
17
  }
18
 
19
+ const MAX_POINTS = 5000;
20
+
21
+ export const TeleoperatorJointGraph = memo(function TeleoperatorJointGraph({
22
+ frames,
23
+ refreshTick,
24
+ }: TeleoperatorJointGraphProps) {
25
  // Skip rendering if no frames
26
  if (!frames || frames.length === 0) {
27
  return null;
 
29
 
30
  // Use hardcoded joint names that match the LeRobot dataset format
31
  const jointNames = [
32
+ "shoulder_pan",
33
  "shoulder_lift",
34
  "elbow_flex",
35
  "wrist_flex",
36
  "wrist_roll",
37
+ "gripper",
38
  ];
39
+
40
  // Generate a color palette for the joints
41
  const colors = [
42
+ "#8884d8",
43
+ "#82ca9d",
44
+ "#ffc658",
45
+ "#ff8042",
46
+ "#0088fe",
47
+ "#00C49F",
48
+ "#FFBB28",
49
+ "#FF8042",
50
+ "#a4de6c",
51
+ "#d0ed57",
52
  ];
53
+
54
  // Prepare data for the chart - handling arrays
55
+ const chartData = useMemo(
56
+ () =>
57
+ frames.map((frame, index) => {
58
+ // Create base data point with index
59
+ const dataPoint: any = {
60
+ name: index,
61
+ timestamp: frame.timestamp,
62
+ };
63
+
64
+ // Add action values (assuming action is an array)
65
+ if (Array.isArray(frame.action)) {
66
+ // Map each array index to the corresponding joint name
67
+ jointNames.forEach((jointName, i) => {
68
+ if (i < frame.action.length) {
69
+ dataPoint[`action_${jointName}`] = frame.action[i];
70
+ }
71
+ });
72
  }
73
+
74
+ // Add observation state values (assuming observation.state is an array)
75
+ if (Array.isArray(frame["observation.state"])) {
76
+ // Map each array index to the corresponding joint name
77
+ jointNames.forEach((jointName, i) => {
78
+ if (i < frame["observation.state"].length) {
79
+ dataPoint[`state_${jointName}`] = frame["observation.state"][i];
80
+ }
81
+ });
82
  }
83
+
84
+ return dataPoint;
85
+ }),
86
+ [frames, refreshTick]
87
+ );
88
+
89
+ const limitedData = useMemo(() => {
90
+ if (!chartData) return [] as typeof chartData;
91
+ const len = chartData.length;
92
+ if (len <= MAX_POINTS) return chartData;
93
+ return chartData.slice(len - MAX_POINTS, len);
94
+ }, [chartData]);
95
+
96
  // Create lines for each joint
97
+ const linesToRender = jointNames.flatMap((jointName) => [
98
  {
99
  key: `action_${jointName}`,
100
  dataKey: `action_${jointName}`,
101
  name: `Action: ${jointName}`,
102
+ isDotted: true,
103
  },
104
  {
105
  key: `state_${jointName}`,
106
  dataKey: `state_${jointName}`,
107
  name: `State: ${jointName}`,
108
+ isDotted: false,
109
+ },
110
  ]);
111
 
112
  return (
113
  <div className="w-full bg-gray-800/50 rounded-md p-4 mb-4">
114
+ <h3 className="text-sm font-medium text-gray-300 mb-2">
115
+ Joint Positions Over Time
116
+ </h3>
117
  <ResponsiveContainer width="100%" height={300}>
118
  <LineChart
119
+ data={limitedData}
120
  margin={{
121
  top: 5,
122
  right: 30,
123
  left: 20,
124
+ bottom: 5,
125
  }}
126
  >
127
  <CartesianGrid strokeDasharray="3 3" stroke="#444" />
128
+ <XAxis
129
+ dataKey="name"
130
+ label={{
131
+ value: "Frame Index",
132
+ position: "insideBottomRight",
133
+ offset: -10,
134
+ }}
135
+ stroke="#aaa"
136
  />
137
  <YAxis stroke="#aaa" />
138
+ <Tooltip
139
+ contentStyle={{ backgroundColor: "#333", borderColor: "#555" }}
140
+ labelStyle={{ color: "#eee" }}
141
+ itemStyle={{ color: "#eee" }}
142
  />
143
  <Legend />
144
+
145
  {/* Render all lines */}
146
  {linesToRender.map((lineConfig, index) => {
147
+ const jointName = lineConfig.dataKey.replace(
148
+ /^(action|state)_/,
149
+ ""
150
+ );
151
  const jointIndex = jointNames.indexOf(jointName);
152
+ const colorIndex =
153
+ jointIndex >= 0 ? jointIndex : index % colors.length;
154
+
155
  return (
156
  <Line
157
  key={lineConfig.key}
 
169
  </ResponsiveContainer>
170
  </div>
171
  );
172
+ });
examples/cyberpunk-standalone/src/utils/dataset-uploader.ts ADDED
@@ -0,0 +1,243 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import * as hub from "@huggingface/hub";
2
+ import { S3Client, HeadBucketCommand } from "@aws-sdk/client-s3";
3
+ import { Upload } from "@aws-sdk/lib-storage";
4
+
5
+ type FileArray = Array<{ path: string; content: Blob | Uint8Array }>;
6
+
7
+ /**
8
+ * Uploads a leRobot dataset to Hugging Face
9
+ *
10
+ * @param files Array of files to upload
11
+ * @param accessToken Hugging Face access token
12
+ * @param repoName Repository name (will be created if it doesn't exist)
13
+ * @param privateRepo Whether the repo should be private (default: false)
14
+ * @returns EventTarget that emits 'repoCreated', 'progress', 'finished', and 'error' events
15
+ */
16
+ export async function uploadToHuggingFace(
17
+ files: FileArray,
18
+ accessToken: string,
19
+ repoName: string,
20
+ privateRepo: boolean = false
21
+ ): Promise<EventTarget> {
22
+ const eventTarget = new EventTarget();
23
+
24
+ // Run upload asynchronously so UI can subscribe to events immediately
25
+ (async () => {
26
+ try {
27
+ // Get username from token
28
+ const { name: username } = await hub.whoAmI({ accessToken });
29
+
30
+ const repoDesignation = {
31
+ name: `${username}/${repoName}`,
32
+ type: "dataset" as const,
33
+ };
34
+
35
+ // Try to create repo; if it already exists (409), continue and upload
36
+ try {
37
+ await hub.createRepo({
38
+ repo: repoDesignation,
39
+ accessToken,
40
+ license: "mit",
41
+ private: privateRepo,
42
+ });
43
+ eventTarget.dispatchEvent(
44
+ new CustomEvent("repoCreated", { detail: repoDesignation })
45
+ );
46
+ } catch (error: any) {
47
+ const message = (error && (error.message || `${error}`)) as string;
48
+ const isConflict =
49
+ message?.includes("409") ||
50
+ message?.toLowerCase()?.includes("already created") ||
51
+ message?.toLowerCase()?.includes("already exists");
52
+ if (!isConflict) {
53
+ eventTarget.dispatchEvent(new CustomEvent("error", { detail: error }));
54
+ throw error;
55
+ }
56
+ // Repo exists: proceed as created
57
+ eventTarget.dispatchEvent(
58
+ new CustomEvent("repoCreated", { detail: repoDesignation })
59
+ );
60
+ }
61
+
62
+ // Upload files to v2.1 branch, fallback to main if branch doesn't exist
63
+ let uploadedBranch = "v2.1";
64
+ try {
65
+ await uploadFilesWithProgress(
66
+ files,
67
+ accessToken,
68
+ repoDesignation,
69
+ uploadedBranch,
70
+ eventTarget
71
+ );
72
+ } catch (error: any) {
73
+ const message = (error && (error.message || `${error}`)) as string;
74
+ const invalidRev = message?.toLowerCase()?.includes("invalid rev id");
75
+ if (invalidRev) {
76
+ console.warn(
77
+ "v2.1 branch not available. Falling back to main branch."
78
+ );
79
+ uploadedBranch = "main";
80
+ await uploadFilesWithProgress(
81
+ files,
82
+ accessToken,
83
+ repoDesignation,
84
+ uploadedBranch,
85
+ eventTarget
86
+ );
87
+ } else {
88
+ throw error;
89
+ }
90
+ }
91
+
92
+ console.log(
93
+ `Successfully uploaded dataset to ${username}/${repoName} (${uploadedBranch})`
94
+ );
95
+ eventTarget.dispatchEvent(
96
+ new CustomEvent("finished", { detail: { branch: uploadedBranch } })
97
+ );
98
+ } catch (error) {
99
+ console.error("Error uploading to Hugging Face:", error);
100
+ eventTarget.dispatchEvent(new CustomEvent("error", { detail: error }));
101
+ }
102
+ })();
103
+
104
+ return eventTarget;
105
+ }
106
+
107
+ /**
108
+ * Uploads a leRobot dataset to Amazon S3
109
+ *
110
+ * @param files Array of files to upload
111
+ * @param bucketName S3 bucket name
112
+ * @param accessKeyId AWS access key ID
113
+ * @param secretAccessKey AWS secret access key
114
+ * @param region AWS region (default: us-east-1)
115
+ * @param prefix Optional prefix/folder for uploaded files
116
+ * @returns EventTarget that emits 'bucketVerified', 'progress', 'finished', and 'error' events
117
+ */
118
+ export async function uploadToS3(
119
+ files: FileArray,
120
+ bucketName: string,
121
+ accessKeyId: string,
122
+ secretAccessKey: string,
123
+ region: string = "us-east-1",
124
+ prefix: string = ""
125
+ ): Promise<EventTarget> {
126
+ const eventTarget = new EventTarget();
127
+
128
+ // Run upload asynchronously
129
+ (async () => {
130
+ try {
131
+ const s3Client = new S3Client({
132
+ region,
133
+ credentials: {
134
+ accessKeyId,
135
+ secretAccessKey,
136
+ },
137
+ });
138
+
139
+ // Verify bucket exists
140
+ try {
141
+ await s3Client.send(
142
+ new HeadBucketCommand({ Bucket: bucketName })
143
+ );
144
+ eventTarget.dispatchEvent(
145
+ new CustomEvent("bucketVerified", {
146
+ detail: { bucketName, region },
147
+ })
148
+ );
149
+ } catch (error: any) {
150
+ const message = error?.message || `${error}`;
151
+ if (message.includes("404") || message.includes("NotFound")) {
152
+ throw new Error(
153
+ `S3 bucket "${bucketName}" not found in region "${region}"`
154
+ );
155
+ }
156
+ throw error;
157
+ }
158
+
159
+ // Upload files
160
+ for (const file of files) {
161
+ const key = prefix ? `${prefix}/${file.path}` : file.path;
162
+
163
+ const upload = new Upload({
164
+ client: s3Client,
165
+ params: {
166
+ Bucket: bucketName,
167
+ Key: key,
168
+ Body:
169
+ file.content instanceof Blob
170
+ ? Buffer.from(await file.content.arrayBuffer())
171
+ : Buffer.from(file.content),
172
+ },
173
+ });
174
+
175
+ upload.on("httpUploadProgress", (progress) => {
176
+ eventTarget.dispatchEvent(
177
+ new CustomEvent("progress", { detail: { file: file.path, progress } })
178
+ );
179
+ });
180
+
181
+ await upload.done();
182
+ console.log(`Uploaded ${key}`);
183
+ }
184
+
185
+ console.log(
186
+ `Successfully uploaded dataset to S3 bucket: ${bucketName}${prefix ? `/${prefix}` : ""}`
187
+ );
188
+ eventTarget.dispatchEvent(
189
+ new CustomEvent("finished", {
190
+ detail: { bucketName, prefix, filesCount: files.length },
191
+ })
192
+ );
193
+ } catch (error) {
194
+ console.error("Error uploading to S3:", error);
195
+ eventTarget.dispatchEvent(new CustomEvent("error", { detail: error }));
196
+ }
197
+ })();
198
+
199
+ return eventTarget;
200
+ }
201
+
202
+ /**
203
+ * Helper function to upload files to Hugging Face with progress tracking
204
+ */
205
+ async function uploadFilesWithProgress(
206
+ files: FileArray,
207
+ accessToken: string,
208
+ repoDesignation: { name: string; type: "dataset" },
209
+ branch: string,
210
+ eventTarget: EventTarget
211
+ ): Promise<void> {
212
+ const referenceId = `lerobot-upload-${Date.now()}`;
213
+
214
+ // Upload each file
215
+ for (const file of files) {
216
+ let blob: Blob;
217
+
218
+ if (file.content instanceof Blob) {
219
+ blob = file.content;
220
+ } else {
221
+ blob = new Blob([file.content]);
222
+ }
223
+
224
+ await hub.uploadFile({
225
+ repo: repoDesignation,
226
+ credentials: { accessToken },
227
+ file: {
228
+ content: blob,
229
+ path: file.path,
230
+ },
231
+ revision: branch,
232
+ });
233
+
234
+ eventTarget.dispatchEvent(
235
+ new CustomEvent("progress", {
236
+ detail: { file: file.path, referenceId },
237
+ })
238
+ );
239
+
240
+ console.log(`Uploaded ${file.path}`);
241
+ }
242
+ }
243
+
package.json CHANGED
@@ -1,7 +1,7 @@
1
  {
2
  "name": "lerobot",
3
  "version": "0.0.0",
4
- "description": "State-of-the-art AI for real-world robotics in JS",
5
  "type": "module",
6
  "workspaces": [
7
  "packages/*"
 
1
  {
2
  "name": "lerobot",
3
  "version": "0.0.0",
4
+ "description": "Control your robots with JS, inspired by LeRobot (python)",
5
  "type": "module",
6
  "workspaces": [
7
  "packages/*"
packages/web/README.md CHANGED
@@ -20,7 +20,13 @@ yarn add @lerobot/web
20
  ## Quick Start
21
 
22
  ```typescript
23
- import { findPort, releaseMotors, calibrate, teleoperate } from "@lerobot/web";
 
 
 
 
 
 
24
 
25
  // 1. find available robot ports (shows browser port dialog)
26
  console.log("🔍 finding available robot ports...");
@@ -387,72 +393,68 @@ await releaseMotors(robot, [1, 2, 3]);
387
 
388
  ---
389
 
390
- ## Dataset Recording and Export
391
 
392
- The LeRobot.js library provides functionality to record teleoperator data and export it in the LeRobot dataset format, compatible with machine learning models.
393
-
394
- ### `LeRobotDatasetRecorder`
395
-
396
- Records teleoperator movements and camera streams, then exports them in the LeRobot dataset format.
397
 
398
  ```typescript
399
- import { LeRobotDatasetRecorder } from "@lerobot/web";
400
-
401
- // Create a recorder with teleoperator and video streams
402
- const recorder = new LeRobotDatasetRecorder(
403
- [teleoperator], // Array of teleoperators to record, currently only supports 1 teleoperator
404
- { "main": videoStream }, // Video streams by camera key
405
- 30, // Target FPS
406
- "Pick and place task" // Task description
407
- );
408
-
409
- // Start recording
410
- await recorder.startRecording();
411
 
412
- // ... robot performs task ...
413
-
414
- // Stop recording and get the data
415
- const recordingData = await recorder.stopRecording();
416
-
417
- // Export the dataset in various formats
418
- // 1. As a downloadable zip file
419
- await recorder.exportForLeRobot('zip-download');
420
-
421
- // 2. Upload to Hugging Face
422
- const hfUploader = await recorder.exportForLeRobot('huggingface', {
423
- repoName: 'my-robot-dataset',
424
- accessToken: 'hf_...',
425
  });
426
 
427
- // 3. Upload to S3
428
- const s3Uploader = await recorder.exportForLeRobot('s3', {
429
- bucketName: 'my-bucket',
430
- accessKeyId: 'AKIA...',
431
- secretAccessKey: '...',
432
- region: 'us-east-1',
 
 
 
 
 
433
  });
434
- ```
435
 
436
- #### Key Features
 
 
437
 
438
- - **Multi-source Recording**: Records teleoperator movements and synchronized video
439
- - **Regular Interpolation**: Generates frames at consistent intervals with `episodes` getter
440
- - **Multiple Export Formats**: Supports local download, Hugging Face, and S3 upload
441
- - **LeRobot Dataset Format**: Follows the standard format for compatibility with ML models
442
 
443
- > **Note:** The dataset statistical data currently generated is incorrect and needs to be updated in a future release.
 
 
 
444
 
445
- #### Dataset Format
446
 
447
- The exported dataset follows the LeRobot format with this structure:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
448
 
449
- ```
450
- /data/chunk-000/file-000.parquet # Teleoperator data
451
- /videos/observation.images.{camera-key}/chunk-000/file-000.mp4 # Video data
452
- /metadata.json # Dataset metadata
453
- /statistics.json # Dataset statistics (currently incorrect)
454
- /README.md # Dataset documentation
455
- ```
456
 
457
  ## Browser Requirements
458
 
 
20
  ## Quick Start
21
 
22
  ```typescript
23
+ import {
24
+ findPort,
25
+ releaseMotors,
26
+ calibrate,
27
+ teleoperate,
28
+ record,
29
+ } from "@lerobot/web";
30
 
31
  // 1. find available robot ports (shows browser port dialog)
32
  console.log("🔍 finding available robot ports...");
 
393
 
394
  ---
395
 
396
+ ### `record(config): Promise<RecordProcess>`
397
 
398
+ Records robot motor positions and teleoperation data using a clean function API that matches the patterns established by `calibrate()` and `teleoperate()`.
 
 
 
 
399
 
400
  ```typescript
401
+ import { teleoperate, record } from "@lerobot/web";
 
 
 
 
 
 
 
 
 
 
 
402
 
403
+ // 1. Create teleoperation first
404
+ const teleoperationProcess = await teleoperate({
405
+ robot: connectedRobot,
406
+ teleop: { type: "keyboard" },
407
+ calibrationData: calibrationData,
 
 
 
 
 
 
 
 
408
  });
409
 
410
+ // 2. Create recording with teleoperator
411
+ const recordProcess = await record({
412
+ teleoperator: teleoperationProcess.teleoperator,
413
+ videoStreams: {
414
+ main: mainCameraStream,
415
+ },
416
+ robotType: "so100",
417
+ options: {
418
+ fps: 30,
419
+ taskDescription: "Pick and place task",
420
+ },
421
  });
 
422
 
423
+ // 3. Start both processes
424
+ teleoperationProcess.start();
425
+ recordProcess.start();
426
 
427
+ // 4. Manage recording during operation
428
+ recordProcess.nextEpisode(); // Start new episode if needed
 
 
429
 
430
+ // 5. Stop recording and export
431
+ const robotData = await recordProcess.stop();
432
+ await recordProcess.exportForLeRobot("zip-download");
433
+ ```
434
 
435
+ #### Options
436
 
437
+ - `config: RecordConfig`
438
+ - `teleoperator: WebTeleoperator` - The teleoperator to record from
439
+ - `videoStreams?: { [name: string]: MediaStream }` - Optional camera streams (e.g., `{ main: stream1, wrist: stream2 }`)
440
+ - `robotType?: string` - Robot metadata (e.g., "so100")
441
+ - `options?: RecordOptions` - Optional recording configuration:
442
+ - `fps?: number` - Target frames per second (default: 30)
443
+ - `taskDescription?: string` - Task description
444
+ - `onStateUpdate?: (state: RecordingState) => void` - Recording state changes
445
+
446
+ #### Returns: `RecordProcess`
447
+
448
+ - `start(): void` - Start recording
449
+ - `stop(): Promise<RobotRecordingData>` - Stop recording and get data
450
+ - `getState(): RecordingState` - Current recording state
451
+ - `getEpisodeCount(): number` - Get total episodes
452
+ - `nextEpisode(): Promise<number>` - Start new episode
453
+ - `clearEpisodes(): void` - Delete all episodes
454
+ - `addCamera(name: string, stream: MediaStream): void` - Add camera dynamically
455
+ - `exportForLeRobot(format?: "blobs" | "zip" | "zip-download"): Promise<any>` - Export dataset
456
 
457
+ ---
 
 
 
 
 
 
458
 
459
  ## Browser Requirements
460
 
packages/web/package.json CHANGED
@@ -68,9 +68,6 @@
68
  "access": "public"
69
  },
70
  "dependencies": {
71
- "@aws-sdk/client-s3": "^3.856.0",
72
- "@aws-sdk/lib-storage": "^3.856.0",
73
- "@huggingface/hub": "^2.4.0",
74
  "apache-arrow": "^21.0.0",
75
  "jszip": "^3.10.1",
76
  "parquet-wasm": "^0.6.1"
 
68
  "access": "public"
69
  },
70
  "dependencies": {
 
 
 
71
  "apache-arrow": "^21.0.0",
72
  "jszip": "^3.10.1",
73
  "parquet-wasm": "^0.6.1"
packages/web/src/hf_uploader.ts DELETED
@@ -1,86 +0,0 @@
1
- import * as hub from "@huggingface/hub";
2
- import type { RepoDesignation } from "@huggingface/hub";
3
- import type { ContentSource } from "@huggingface/hub";
4
-
5
- type FileArray = Array<URL | File | { path: string; content: ContentSource }>;
6
- /**
7
- * Uploads a leRobot dataset to huggingface
8
- */
9
-
10
- export class LeRobotHFUploader extends EventTarget {
11
- private _repoDesignation: RepoDesignation;
12
- private _uploaded : boolean;
13
- private _created_repo : boolean;
14
-
15
- constructor(username: string, repoName: string) {
16
- super();
17
- this._repoDesignation = {
18
- name : `${username}/${repoName}`,
19
- type : "dataset"
20
- };
21
-
22
- this._uploaded = false;
23
- this._created_repo = false;
24
- }
25
-
26
- /**
27
- * Returns whether the repository has been successfully created
28
- */
29
- get createdRepo(): boolean {
30
- return this._created_repo;
31
- }
32
-
33
- get uploaded() : boolean {
34
- return this._uploaded;
35
- }
36
-
37
-
38
- /**
39
- * Uploads the dataset to huggingface
40
- *
41
- * A referenceId is used to be able to track progress of the upload,
42
- * this provides a progressEvent from huggingface.js (see : https://github.com/huggingface/huggingface.js/blob/main/packages/hub/README.md#usage)
43
- *
44
- * both this and huggingface.js have pretty bad documentation, some exploration will be required (sorry!, I have university, and work and I already feel guilty procastinating those to to write this)
45
- *
46
- * @param dataset The dataset to upload
47
- * @param accessToken The access token for huggingface
48
- * @param referenceId The reference id for the upload, to track it (optional)
49
- */
50
- async createRepoAndUploadFiles(files : FileArray, accessToken : string, referenceId : string = "") {
51
- await hub.createRepo({
52
- repo: this._repoDesignation,
53
- accessToken: accessToken,
54
- license: "mit"
55
- });
56
-
57
- this._created_repo = true;
58
- this.dispatchEvent(new CustomEvent("repoCreated", { detail: this._repoDesignation }));
59
-
60
- const uploadPromises : Promise<void>[] = [];
61
- uploadPromises.push(this.uploadFilesWithProgress(files, accessToken, referenceId));
62
-
63
- await Promise.all(uploadPromises);
64
- }
65
-
66
- /**
67
- * Uploads files to huggingface with progress events
68
- *
69
- * @param files The files to upload
70
- * @param accessToken The access token for huggingface
71
- * @param referenceId The reference id for the upload, to track it (optional)
72
- */
73
- async uploadFilesWithProgress(files : FileArray, accessToken : string, referenceId : string = "") {
74
- for await (const progressEvent of hub.uploadFilesWithProgress({
75
- repo: this._repoDesignation,
76
- accessToken: accessToken,
77
- files: files,
78
- })) {
79
- this.dispatchEvent(new CustomEvent("progress", { detail: {
80
- progressEvent,
81
- repoDesignation: this._repoDesignation,
82
- referenceId
83
- } }));
84
- }
85
- }
86
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
packages/web/src/index.ts CHANGED
@@ -9,6 +9,7 @@ export { calibrate } from "./calibrate.js";
9
  export { teleoperate } from "./teleoperate.js";
10
  export { findPort } from "./find_port.js";
11
  export { releaseMotors } from "./release_motors.js";
 
12
 
13
  // Browser support utilities
14
  export {
@@ -51,6 +52,14 @@ export type {
51
  KeyboardControl,
52
  } from "./types/robot-config.js";
53
 
 
 
 
 
 
 
 
 
54
  // Utilities (advanced users)
55
  export { WebSerialPortWrapper } from "./utils/serial-port-wrapper.js";
56
  export {
@@ -64,5 +73,4 @@ export {
64
  export { KEYBOARD_TELEOPERATOR_DEFAULTS } from "./teleoperators/index.js";
65
 
66
  // Record
67
- export { LeRobotDatasetRecorder } from "./record.js";
68
- export { LeRobotHFUploader } from "./hf_uploader.js";
 
9
  export { teleoperate } from "./teleoperate.js";
10
  export { findPort } from "./find_port.js";
11
  export { releaseMotors } from "./release_motors.js";
12
+ export { record } from "./record.js";
13
 
14
  // Browser support utilities
15
  export {
 
52
  KeyboardControl,
53
  } from "./types/robot-config.js";
54
 
55
+ export type {
56
+ RecordConfig,
57
+ RecordProcess,
58
+ RecordingState,
59
+ RecordingData,
60
+ RobotRecordingData,
61
+ } from "./types/recording.js";
62
+
63
  // Utilities (advanced users)
64
  export { WebSerialPortWrapper } from "./utils/serial-port-wrapper.js";
65
  export {
 
73
  export { KEYBOARD_TELEOPERATOR_DEFAULTS } from "./teleoperators/index.js";
74
 
75
  // Record
76
+ export { LeRobotDatasetRecorder, LeRobotEpisode } from "./record.js";
 
packages/web/src/record.ts CHANGED
@@ -3,12 +3,7 @@ import { MotorConfig } from "./types/teleoperation";
3
  import * as parquet from "parquet-wasm";
4
  import * as arrow from "apache-arrow";
5
  import JSZip from "jszip";
6
- import getMetadataInfo from "./utils/record/metadataInfo";
7
- import type { VideoInfo } from "./utils/record/metadataInfo";
8
- import getStats from "./utils/record/stats";
9
  import generateREADME from "./utils/record/generateREADME";
10
- import { LeRobotHFUploader } from "./hf_uploader";
11
- import { LeRobotS3Uploader } from "./s3_uploader";
12
 
13
  // declare a type leRobot action that's basically an array of numbers
14
  interface LeRobotAction {
@@ -338,12 +333,18 @@ export class LeRobotDatasetRecorder {
338
  mediaRecorders: { [key: string]: MediaRecorder };
339
  videoChunks: { [key: string]: Blob[] };
340
  videoBlobs: { [key: string]: Blob };
 
 
 
 
341
  teleoperatorData: LeRobotEpisode[];
342
  private _isRecording: boolean;
343
  private episodeIndex: number = 0;
344
  private taskIndex: number = 0;
 
345
  fps: number;
346
  taskDescription: string;
 
347
 
348
  /**
349
  * Ensures BlobPart compatibility across environments by converting Uint8Array
@@ -382,17 +383,45 @@ export class LeRobotDatasetRecorder {
382
  this.mediaRecorders = {};
383
  this.videoChunks = {};
384
  this.videoBlobs = {};
 
385
  this.videoStreams = {};
 
386
  this.teleoperatorData = [];
387
  this._isRecording = false;
388
  this.fps = fps;
389
  this.taskDescription = taskDescription;
 
390
 
391
  for (const [key, stream] of Object.entries(videoStreams)) {
392
  this.addVideoStream(key, stream);
393
  }
394
  }
395
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
396
  get isRecording(): boolean {
397
  return this._isRecording;
398
  }
@@ -407,21 +436,18 @@ export class LeRobotDatasetRecorder {
407
  * @param stream The media stream to record from
408
  */
409
  addVideoStream(key: string, stream: MediaStream) {
410
- console.log("Adding video stream", key);
411
  if (this._isRecording) {
412
  throw new Error("Cannot add video streams while recording");
413
  }
414
 
415
  // Add to video streams dictionary
416
  this.videoStreams[key] = stream;
417
-
418
- // Initialize MediaRecorder for this stream
419
- this.mediaRecorders[key] = new MediaRecorder(stream, {
420
- mimeType: "video/mp4",
421
- });
422
-
423
- // add a video chunk array for this stream
424
  this.videoChunks[key] = [];
 
 
 
 
425
  }
426
 
427
  /**
@@ -466,7 +492,6 @@ export class LeRobotDatasetRecorder {
466
  * Starts recording for all teleoperators and video streams
467
  */
468
  startRecording() {
469
- console.log("Starting recording");
470
  if (this._isRecording) {
471
  console.warn("Recording already in progress");
472
  return;
@@ -474,19 +499,26 @@ export class LeRobotDatasetRecorder {
474
 
475
  this._isRecording = true;
476
 
477
- // add a new episode
 
478
  this.teleoperatorData.push(new LeRobotEpisode());
 
479
 
480
  // Start recording video streams
481
  Object.entries(this.videoStreams).forEach(([key, stream]) => {
482
- // Create a media recorder for this stream
483
- const mediaRecorder = new MediaRecorder(stream, {
484
- mimeType: "video/mp4",
485
- });
 
 
 
 
 
 
486
 
487
  // Handle data available events
488
  mediaRecorder.ondataavailable = (event) => {
489
- console.log("data available for", key);
490
  if (event.data && event.data.size > 0) {
491
  this.videoChunks[key].push(event.data);
492
  }
@@ -495,8 +527,6 @@ export class LeRobotDatasetRecorder {
495
  // Save the recorder and start recording
496
  this.mediaRecorders[key] = mediaRecorder;
497
  mediaRecorder.start(1000); // Capture in 1-second chunks
498
-
499
- console.log(`Started recording video stream: ${key}`);
500
  });
501
  }
502
 
@@ -560,8 +590,17 @@ export class LeRobotDatasetRecorder {
560
  recorder.onstop = () => {
561
  // Combine all chunks into a single blob
562
  const chunks = this.videoChunks[key] || [];
563
- const blob = new Blob(chunks, { type: "video/mp4" });
 
564
  this.videoBlobs[key] = blob;
 
 
 
 
 
 
 
 
565
  resolve();
566
  };
567
 
@@ -579,12 +618,95 @@ export class LeRobotDatasetRecorder {
579
  };
580
  }
581
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
582
  /**
583
  * Clears the teleoperator data and video blobs
584
  */
585
  clearRecording() {
586
  this.teleoperatorData = [];
587
  this.videoBlobs = {};
 
 
 
 
 
 
 
588
  }
589
 
590
  /**
@@ -939,68 +1061,17 @@ export class LeRobotDatasetRecorder {
939
  * Generates metadata for the dataset
940
  * @returns Metadata object for the LeRobot dataset
941
  */
942
- async generateMetadata(data: any[]): Promise<any> {
943
- // Calculate total episodes, frames, and tasks
944
- let total_episodes = 0;
945
- const total_frames = data.length;
946
- let total_tasks = 0;
947
-
948
- for (const row of data) {
949
- total_episodes = Math.max(total_episodes, row.episode_index);
950
- total_tasks = Math.max(total_tasks, row.task_index);
951
- }
952
-
953
- // Create video info objects for each video stream
954
- const videos_info: VideoInfo[] = Object.keys(this.videoBlobs).map((key) => {
955
- // Default values - in a production environment, you would extract
956
- // these from the actual video metadata using the key to identify the video source
957
- console.log(`Generating metadata for video stream: ${key}`);
958
- return {
959
- height: 480,
960
- width: 640,
961
- channels: 3,
962
- codec: "h264",
963
- pix_fmt: "yuv420p",
964
- is_depth_map: false,
965
- has_audio: false,
966
- };
967
- });
968
-
969
- // Calculate approximate file sizes in MB
970
- const data_files_size_in_mb = Math.round(data.length * 0.001); // Estimate
971
-
972
- // Calculate video size by summing the sizes of all video blobs and converting to MB
973
- let video_files_size_in_mb = 0;
974
- for (const blob of Object.values(this.videoBlobs)) {
975
- video_files_size_in_mb += blob.size / (1024 * 1024);
976
- }
977
- video_files_size_in_mb = Math.round(video_files_size_in_mb);
978
-
979
- // Generate and return the metadata
980
- return getMetadataInfo({
981
- total_episodes,
982
- total_frames,
983
- total_tasks,
984
- chunks_size: 1000, // Default chunk size
985
- fps: this.fps,
986
- splits: { train: `0:${total_episodes}` }, // All episodes in train split
987
- features: {}, // Additional features can be added here
988
- videos_info,
989
- data_files_size_in_mb,
990
- video_files_size_in_mb,
991
- });
992
  }
993
 
994
  /**
995
  * Generates statistics for the dataset
996
  * @returns Statistics object for the LeRobot dataset
997
  */
998
- async getStatistics(data: any[]): Promise<any> {
999
- // Get camera keys from the video blobs
1000
- const cameraKeys = Object.keys(this.videoBlobs);
1001
-
1002
- // Generate stats using the data and camera keys
1003
- return getStats(data, cameraKeys);
1004
  }
1005
 
1006
  /**
@@ -1057,299 +1128,8 @@ export class LeRobotDatasetRecorder {
1057
  * Creates the episodes statistics parquet file
1058
  * @returns A Uint8Array blob containing the parquet data
1059
  */
1060
- async getEpisodeStatistics(data: any[]): Promise<Uint8Array> {
1061
- const { vectorFromArray } = arrow;
1062
- const statistics = await this.getStatistics(data);
1063
-
1064
- // Calculate total episodes and frames
1065
- let total_episodes = 0;
1066
-
1067
- for (let row of data) {
1068
- total_episodes = Math.max(total_episodes, row.episode_index);
1069
- }
1070
-
1071
- total_episodes += 1; // +1 since episodes start from 0
1072
-
1073
- const episodes: any[] = [];
1074
-
1075
- // we'll create one row per episode
1076
- for (
1077
- let episode_index = 0;
1078
- episode_index < total_episodes;
1079
- episode_index++
1080
- ) {
1081
- // Get data for this episode only
1082
- const episodeData = data.filter(
1083
- (row) => row.episode_index === episode_index
1084
- );
1085
-
1086
- // Extract timestamps for this episode
1087
- const timestamps = episodeData.map((row) => row.timestamp);
1088
- let min_timestamp = Infinity;
1089
- let max_timestamp = -Infinity;
1090
-
1091
- for (let timestamp of timestamps) {
1092
- min_timestamp = Math.min(min_timestamp, timestamp);
1093
- max_timestamp = Math.max(max_timestamp, timestamp);
1094
- }
1095
-
1096
- // Camera keys from video blobs
1097
- const cameraKeys = Object.keys(this.videoBlobs);
1098
-
1099
- // Create entry for this episode
1100
- const episodeEntry: any = {
1101
- // Basic episode information
1102
- episode_index: episode_index,
1103
- "data/chunk_index": 0,
1104
- "data/file_index": 0,
1105
- dataset_from_index: 0,
1106
- dataset_to_index: episodeData.length - 1,
1107
- length: episodeData.length,
1108
- tasks: [0], // Task index 0, could be extended for multiple tasks
1109
-
1110
- // Meta information
1111
- "meta/episodes/chunk_index": 0,
1112
- "meta/episodes/file_index": 0,
1113
- };
1114
-
1115
- // Add video information for each camera
1116
- cameraKeys.forEach((key) => {
1117
- episodeEntry[`videos/observation.images.${key}/chunk_index`] = 0;
1118
- episodeEntry[`videos/observation.images.${key}/file_index`] = 0;
1119
- episodeEntry[`videos/observation.images.${key}/from_timestamp`] =
1120
- min_timestamp;
1121
- episodeEntry[`videos/observation.images.${key}/to_timestamp`] =
1122
- max_timestamp;
1123
- });
1124
-
1125
- // Add statistics for each field
1126
- // This is a simplified approach - in a real implementation, you'd calculate
1127
- // these values for each episode individually
1128
-
1129
- // Add timestamp statistics
1130
- episodeEntry["stats/timestamp/min"] = [statistics.timestamp.min];
1131
- episodeEntry["stats/timestamp/max"] = [statistics.timestamp.max];
1132
- episodeEntry["stats/timestamp/mean"] = [statistics.timestamp.mean];
1133
- episodeEntry["stats/timestamp/std"] = [statistics.timestamp.std];
1134
- episodeEntry["stats/timestamp/count"] = [statistics.timestamp.count];
1135
-
1136
- // Add frame_index statistics
1137
- episodeEntry["stats/frame_index/min"] = [statistics.frame_index.min];
1138
- episodeEntry["stats/frame_index/max"] = [statistics.frame_index.max];
1139
- episodeEntry["stats/frame_index/mean"] = [statistics.frame_index.mean];
1140
- episodeEntry["stats/frame_index/std"] = [statistics.frame_index.std];
1141
- episodeEntry["stats/frame_index/count"] = [statistics.frame_index.count];
1142
-
1143
- // Add episode_index statistics
1144
- episodeEntry["stats/episode_index/min"] = [statistics.episode_index.min];
1145
- episodeEntry["stats/episode_index/max"] = [statistics.episode_index.max];
1146
- episodeEntry["stats/episode_index/mean"] = [
1147
- statistics.episode_index.mean,
1148
- ];
1149
- episodeEntry["stats/episode_index/std"] = [statistics.episode_index.std];
1150
- episodeEntry["stats/episode_index/count"] = [
1151
- statistics.episode_index.count,
1152
- ];
1153
-
1154
- // Add task_index statistics
1155
- episodeEntry["stats/task_index/min"] = [statistics.task_index.min];
1156
- episodeEntry["stats/task_index/max"] = [statistics.task_index.max];
1157
- episodeEntry["stats/task_index/mean"] = [statistics.task_index.mean];
1158
- episodeEntry["stats/task_index/std"] = [statistics.task_index.std];
1159
- episodeEntry["stats/task_index/count"] = [statistics.task_index.count];
1160
-
1161
- // Add index statistics
1162
- episodeEntry["stats/index/min"] = [0];
1163
- episodeEntry["stats/index/max"] = [episodeData.length - 1];
1164
- episodeEntry["stats/index/mean"] = [episodeData.length / 2];
1165
- episodeEntry["stats/index/std"] = [episodeData.length / 4]; // Approximate std
1166
- episodeEntry["stats/index/count"] = [episodeData.length];
1167
-
1168
- // Add action statistics (placeholder)
1169
- episodeEntry["stats/action/min"] = [0.0];
1170
- episodeEntry["stats/action/max"] = [1.0];
1171
- episodeEntry["stats/action/mean"] = [0.5];
1172
- episodeEntry["stats/action/std"] = [0.2];
1173
- episodeEntry["stats/action/count"] = [episodeData.length];
1174
-
1175
- // Add observation.state statistics (placeholder)
1176
- episodeEntry["stats/observation.state/min"] = [0.0];
1177
- episodeEntry["stats/observation.state/max"] = [1.0];
1178
- episodeEntry["stats/observation.state/mean"] = [0.5];
1179
- episodeEntry["stats/observation.state/std"] = [0.2];
1180
- episodeEntry["stats/observation.state/count"] = [episodeData.length];
1181
-
1182
- // Add observation.images statistics for each camera
1183
- cameraKeys.forEach((key) => {
1184
- // Get the image statistics from the overall statistics
1185
- const imageStats = statistics[`observation.images.${key}`] || {
1186
- min: [[[0.0]], [[0.0]], [[0.0]]],
1187
- max: [[[255.0]], [[255.0]], [[255.0]]],
1188
- mean: [[[127.5]], [[127.5]], [[127.5]]],
1189
- std: [[[50.0]], [[50.0]], [[50.0]]],
1190
- count: [[[episodeData.length * 3]]],
1191
- };
1192
-
1193
- episodeEntry[`stats/observation.images.${key}/min`] = imageStats.min;
1194
- episodeEntry[`stats/observation.images.${key}/max`] = imageStats.max;
1195
- episodeEntry[`stats/observation.images.${key}/mean`] = imageStats.mean;
1196
- episodeEntry[`stats/observation.images.${key}/std`] = imageStats.std;
1197
- episodeEntry[`stats/observation.images.${key}/count`] =
1198
- imageStats.count;
1199
- });
1200
-
1201
- episodes.push(episodeEntry);
1202
- }
1203
-
1204
- // Create vector arrays for each column
1205
- const columns: any = {};
1206
-
1207
- // Define column names and default types
1208
- const columnNames = [
1209
- "episode_index",
1210
- "data/chunk_index",
1211
- "data/file_index",
1212
- "dataset_from_index",
1213
- "dataset_to_index",
1214
- "length",
1215
- "meta/episodes/chunk_index",
1216
- "meta/episodes/file_index",
1217
- "tasks",
1218
- ];
1219
-
1220
- // Add camera-specific columns
1221
- const cameraKeys = Object.keys(this.videoBlobs);
1222
- cameraKeys.forEach((key) => {
1223
- columnNames.push(
1224
- `videos/observation.images.${key}/chunk_index`,
1225
- `videos/observation.images.${key}/file_index`,
1226
- `videos/observation.images.${key}/from_timestamp`,
1227
- `videos/observation.images.${key}/to_timestamp`
1228
- );
1229
- });
1230
-
1231
- // Add statistic columns for each field
1232
- const statFields = [
1233
- "timestamp",
1234
- "frame_index",
1235
- "episode_index",
1236
- "task_index",
1237
- "index",
1238
- "action",
1239
- "observation.state",
1240
- ];
1241
- statFields.forEach((field) => {
1242
- columnNames.push(
1243
- `stats/${field}/min`,
1244
- `stats/${field}/max`,
1245
- `stats/${field}/mean`,
1246
- `stats/${field}/std`,
1247
- `stats/${field}/count`
1248
- );
1249
- });
1250
-
1251
- // Add image statistic columns for each camera
1252
- cameraKeys.forEach((key) => {
1253
- columnNames.push(
1254
- `stats/observation.images.${key}/min`,
1255
- `stats/observation.images.${key}/max`,
1256
- `stats/observation.images.${key}/mean`,
1257
- `stats/observation.images.${key}/std`,
1258
- `stats/observation.images.${key}/count`
1259
- );
1260
- });
1261
-
1262
- // Create vector arrays for each column
1263
- columnNames.forEach((columnName) => {
1264
- const values = episodes.map((ep) => ep[columnName] || 0);
1265
-
1266
- // Check if the column is an array type and needs special handling
1267
- if (columnName.includes("stats/") || columnName === "tasks") {
1268
- // Handle different types of array columns based on their naming pattern
1269
- if (columnName.includes("/count")) {
1270
- // Bigint arrays for count fields
1271
- // @ts-ignore
1272
- columns[columnName] = vectorFromArray(
1273
- values.map((v) => Number(v)),
1274
- new arrow.List(new arrow.Field("item", new arrow.Int64()))
1275
- );
1276
- } else if (
1277
- columnName.includes("/min") ||
1278
- columnName.includes("/max") ||
1279
- columnName.includes("/mean") ||
1280
- columnName.includes("/std")
1281
- ) {
1282
- // Double arrays for min, max, mean, std fields
1283
- if (
1284
- columnName.includes("observation.images") &&
1285
- (columnName.includes("/min") ||
1286
- columnName.includes("/max") ||
1287
- columnName.includes("/mean") ||
1288
- columnName.includes("/std"))
1289
- ) {
1290
- // These are 3D arrays [[[value]]]
1291
- // For 3D arrays, we need nested Lists
1292
- // @ts-ignore
1293
- columns[columnName] = vectorFromArray(
1294
- values,
1295
- new arrow.List(
1296
- new arrow.Field(
1297
- "item",
1298
- new arrow.List(
1299
- new arrow.Field(
1300
- "subitem",
1301
- new arrow.List(
1302
- new arrow.Field("value", new arrow.Float64())
1303
- )
1304
- )
1305
- )
1306
- )
1307
- )
1308
- );
1309
- } else {
1310
- // These are normal arrays [value]
1311
- // @ts-ignore
1312
- columns[columnName] = vectorFromArray(
1313
- values,
1314
- new arrow.List(new arrow.Field("item", new arrow.Float64()))
1315
- );
1316
- }
1317
- } else {
1318
- // Default to Float64 List for other array types
1319
- // @ts-ignore
1320
- columns[columnName] = vectorFromArray(
1321
- values,
1322
- new arrow.List(new arrow.Field("item", new arrow.Float64()))
1323
- );
1324
- }
1325
- } else {
1326
- // For non-array columns, use regular vectorFromArray
1327
- // @ts-ignore
1328
- columns[columnName] = vectorFromArray(values);
1329
- }
1330
- });
1331
-
1332
- // Create the table with all columns
1333
- const table = arrow.tableFromArrays(columns);
1334
-
1335
- // Initialize the WASM module
1336
- const wasmUrl =
1337
- "https://cdn.jsdelivr.net/npm/parquet-wasm@0.6.1/esm/parquet_wasm_bg.wasm";
1338
- const initWasm = parquet.default;
1339
- await initWasm(wasmUrl);
1340
-
1341
- // Convert Arrow table to Parquet WASM table
1342
- const wasmTable = parquet.Table.fromIPCStream(
1343
- arrow.tableToIPC(table, "stream")
1344
- );
1345
-
1346
- // Set compression properties
1347
- const writerProperties = new parquet.WriterPropertiesBuilder()
1348
- .setCompression(parquet.Compression.UNCOMPRESSED)
1349
- .build();
1350
-
1351
- // Write the Parquet file
1352
- return parquet.writeParquet(wasmTable, writerProperties);
1353
  }
1354
 
1355
  generateREADME(metaInfo: string) {
@@ -1363,59 +1143,236 @@ export class LeRobotDatasetRecorder {
1363
  * @private
1364
  */
1365
  async _exportForLeRobotBlobs() {
1366
- const teleoperatorDataJson = (await this.exportEpisodes("json")) as any[];
 
 
1367
  const parquetEpisodeDataFiles = await this._exportEpisodesToBlob(
1368
- teleoperatorDataJson
1369
  );
1370
- const videoBlobs = await this.exportMediaData();
1371
- const metadata = await this.generateMetadata(teleoperatorDataJson);
1372
- const statistics = await this.getStatistics(teleoperatorDataJson);
1373
- const tasksParquet = await this.createTasksParquet();
1374
- const episodesParquet = await this.getEpisodeStatistics(
1375
- teleoperatorDataJson
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1376
  );
1377
- const readme = this.generateREADME(JSON.stringify(metadata));
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1378
 
1379
- // Create the blob array with proper paths
1380
- const blobArray = [
1381
- ...parquetEpisodeDataFiles,
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1382
  {
1383
  path: "meta/info.json",
1384
- content: new Blob([JSON.stringify(metadata, null, 2)], {
1385
  type: "application/json",
1386
  }),
1387
  },
1388
  {
1389
- path: "meta/stats.json",
1390
- content: new Blob([JSON.stringify(statistics, null, 2)], {
1391
- type: "application/json",
1392
  }),
1393
  },
1394
  {
1395
- path: "meta/tasks.parquet",
1396
- content: new Blob([
1397
- LeRobotDatasetRecorder.toArrayBuffer(tasksParquet as Uint8Array),
1398
- ]),
1399
  },
1400
  {
1401
- path: "meta/episodes/chunk-000/file-000.parquet",
1402
- content: new Blob([
1403
- LeRobotDatasetRecorder.toArrayBuffer(episodesParquet as Uint8Array),
1404
- ]),
1405
  },
1406
  {
1407
  path: "README.md",
1408
  content: new Blob([readme], { type: "text/markdown" }),
1409
- },
1410
- ];
1411
 
1412
- // Add video blobs with proper paths
1413
- for (const [key, blob] of Object.entries(videoBlobs)) {
1414
- blobArray.push({
1415
- path: `videos/chunk-000/observation.images.${key}/episode_000000.mp4`,
1416
- content: blob,
1417
- });
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1418
  }
 
 
 
 
 
 
1419
 
1420
  return blobArray;
1421
  }
@@ -1452,130 +1409,15 @@ export class LeRobotDatasetRecorder {
1452
  return await zip.generateAsync({ type: "blob" });
1453
  }
1454
 
1455
- /**
1456
- * Uploads the LeRobot dataset to Hugging Face
1457
- *
1458
- * @param username Hugging Face username
1459
- * @param repoName Repository name for the dataset
1460
- * @param accessToken Hugging Face access token
1461
- * @returns The LeRobotHFUploader instance used for upload
1462
- */
1463
- async _exportForLeRobotHuggingface(
1464
- username: string,
1465
- repoName: string,
1466
- accessToken: string
1467
- ) {
1468
- // Create the blobs array for upload
1469
- const blobArray = await this._exportForLeRobotBlobs();
1470
-
1471
- // Create the uploader
1472
- const uploader = new LeRobotHFUploader(username, repoName);
1473
-
1474
- // Convert blobs to File objects for HF uploader
1475
- const files = blobArray.map((item) => {
1476
- return {
1477
- path: item.path,
1478
- content: item.content,
1479
- };
1480
- });
1481
-
1482
- // Generate a unique reference ID for tracking the upload
1483
- const referenceId = `lerobot-upload-${Date.now()}`;
1484
-
1485
- try {
1486
- // Start the upload process
1487
- uploader.createRepoAndUploadFiles(files, accessToken, referenceId);
1488
- console.log(`Successfully uploaded dataset to ${username}/${repoName}`);
1489
- return uploader;
1490
- } catch (error) {
1491
- console.error("Error uploading to Hugging Face:", error);
1492
- throw error;
1493
- }
1494
- }
1495
-
1496
- /**
1497
- * Uploads the LeRobot dataset to Amazon S3
1498
- *
1499
- * @param bucketName S3 bucket name
1500
- * @param accessKeyId AWS access key ID
1501
- * @param secretAccessKey AWS secret access key
1502
- * @param region AWS region (default: us-east-1)
1503
- * @param prefix Optional prefix (folder) to upload files to within the bucket
1504
- * @returns The LeRobotS3Uploader instance used for upload
1505
- */
1506
- async _exportForLeRobotS3(
1507
- bucketName: string,
1508
- accessKeyId: string,
1509
- secretAccessKey: string,
1510
- region: string = "us-east-1",
1511
- prefix: string = ""
1512
- ) {
1513
- // Create the blobs array for upload
1514
- const blobArray = await this._exportForLeRobotBlobs();
1515
-
1516
- // Create the uploader
1517
- const uploader = new LeRobotS3Uploader(bucketName, region);
1518
-
1519
- // Convert blobs to File objects for S3 uploader
1520
- const files = blobArray.map((item) => {
1521
- return {
1522
- path: item.path,
1523
- content: item.content,
1524
- };
1525
- });
1526
-
1527
- // Generate a unique reference ID for tracking the upload
1528
- const referenceId = `lerobot-s3-upload-${Date.now()}`;
1529
-
1530
- try {
1531
- // Start the upload process
1532
- uploader.checkBucketAndUploadFiles(
1533
- files,
1534
- accessKeyId,
1535
- secretAccessKey,
1536
- prefix,
1537
- referenceId
1538
- );
1539
- console.log(`Successfully uploaded dataset to S3 bucket: ${bucketName}`);
1540
- return uploader;
1541
- } catch (error) {
1542
- console.error("Error uploading to S3:", error);
1543
- throw error;
1544
- }
1545
- }
1546
-
1547
  /**
1548
  * Exports the LeRobot dataset in various formats
1549
  *
1550
- * @param format The export format - 'blobs', 'zip', 'zip-download', 'huggingface', or 's3'
1551
- * @param options Additional options for specific formats
1552
- * @param options.username Hugging Face username (if not provided for "huggingface" format, it will use the default username)
1553
- * @param options.repoName Hugging Face repository name (required for 'huggingface' format)
1554
- * @param options.accessToken Hugging Face access token (required for 'huggingface' format)
1555
- * @param options.bucketName S3 bucket name (required for 's3' format)
1556
- * @param options.accessKeyId AWS access key ID (required for 's3' format)
1557
- * @param options.secretAccessKey AWS secret access key (required for 's3' format)
1558
- * @param options.region AWS region (optional for 's3' format, default: us-east-1)
1559
- * @param options.prefix S3 prefix/folder (optional for 's3' format)
1560
- * @returns The exported data in the requested format or the uploader instance for 'huggingface'/'s3' formats
1561
  */
1562
  async exportForLeRobot(
1563
- format:
1564
- | "blobs"
1565
- | "zip"
1566
- | "zip-download"
1567
- | "huggingface"
1568
- | "s3" = "zip-download",
1569
- options?: {
1570
- username?: string;
1571
- repoName?: string;
1572
- accessToken?: string;
1573
- bucketName?: string;
1574
- accessKeyId?: string;
1575
- secretAccessKey?: string;
1576
- region?: string;
1577
- prefix?: string;
1578
- }
1579
  ) {
1580
  switch (format) {
1581
  case "blobs":
@@ -1584,49 +1426,6 @@ export class LeRobotDatasetRecorder {
1584
  case "zip":
1585
  return this._exportForLeRobotZip();
1586
 
1587
- case "huggingface":
1588
- // Validate required options for Hugging Face upload
1589
- if (!options || !options.repoName || !options.accessToken) {
1590
- throw new Error(
1591
- "Hugging Face upload requires repoName, and accessToken options"
1592
- );
1593
- }
1594
-
1595
- if (!options.username) {
1596
- const hub = await import("@huggingface/hub");
1597
- const { name: username } = await hub.whoAmI({
1598
- accessToken: options.accessToken,
1599
- });
1600
- options.username = username;
1601
- }
1602
-
1603
- return this._exportForLeRobotHuggingface(
1604
- options.username,
1605
- options.repoName,
1606
- options.accessToken
1607
- );
1608
-
1609
- case "s3":
1610
- // Validate required options for S3 upload
1611
- if (
1612
- !options ||
1613
- !options.bucketName ||
1614
- !options.accessKeyId ||
1615
- !options.secretAccessKey
1616
- ) {
1617
- throw new Error(
1618
- "S3 upload requires bucketName, accessKeyId, and secretAccessKey options"
1619
- );
1620
- }
1621
-
1622
- return this._exportForLeRobotS3(
1623
- options.bucketName,
1624
- options.accessKeyId,
1625
- options.secretAccessKey,
1626
- options.region,
1627
- options.prefix
1628
- );
1629
-
1630
  case "zip-download":
1631
  default:
1632
  // Get the zip blob
@@ -1654,3 +1453,185 @@ export class LeRobotDatasetRecorder {
1654
  }
1655
  }
1656
  }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  import * as parquet from "parquet-wasm";
4
  import * as arrow from "apache-arrow";
5
  import JSZip from "jszip";
 
 
 
6
  import generateREADME from "./utils/record/generateREADME";
 
 
7
 
8
  // declare a type leRobot action that's basically an array of numbers
9
  interface LeRobotAction {
 
333
  mediaRecorders: { [key: string]: MediaRecorder };
334
  videoChunks: { [key: string]: Blob[] };
335
  videoBlobs: { [key: string]: Blob };
336
+ private videoBlobsByEpisode: {
337
+ [episodeIndex: number]: { [key: string]: Blob };
338
+ };
339
+ private videoMimeByKey: { [key: string]: { mime: string; ext: string } };
340
  teleoperatorData: LeRobotEpisode[];
341
  private _isRecording: boolean;
342
  private episodeIndex: number = 0;
343
  private taskIndex: number = 0;
344
+ private currentVideoSegmentEpisodeIndex: number | null = null;
345
  fps: number;
346
  taskDescription: string;
347
+ private robotLabel?: string;
348
 
349
  /**
350
  * Ensures BlobPart compatibility across environments by converting Uint8Array
 
383
  this.mediaRecorders = {};
384
  this.videoChunks = {};
385
  this.videoBlobs = {};
386
+ this.videoBlobsByEpisode = {};
387
  this.videoStreams = {};
388
+ this.videoMimeByKey = {};
389
  this.teleoperatorData = [];
390
  this._isRecording = false;
391
  this.fps = fps;
392
  this.taskDescription = taskDescription;
393
+ this.robotLabel = undefined;
394
 
395
  for (const [key, stream] of Object.entries(videoStreams)) {
396
  this.addVideoStream(key, stream);
397
  }
398
  }
399
 
400
+ setRobotLabel(label: string) {
401
+ this.robotLabel = label;
402
+ }
403
+
404
+ private static getSupportedRecorderType(): { mime: string; ext: string } {
405
+ // Prefer H.264 MP4 for viewer compatibility; fall back to WebM
406
+ const candidates: { mime: string; ext: string }[] = [
407
+ { mime: "video/mp4;codecs=h264", ext: "mp4" },
408
+ { mime: "video/mp4", ext: "mp4" },
409
+ { mime: "video/webm;codecs=vp9", ext: "webm" },
410
+ { mime: "video/webm;codecs=vp8", ext: "webm" },
411
+ { mime: "video/webm", ext: "webm" },
412
+ ];
413
+ for (const c of candidates) {
414
+ if (
415
+ (window as any).MediaRecorder &&
416
+ MediaRecorder.isTypeSupported &&
417
+ MediaRecorder.isTypeSupported(c.mime)
418
+ ) {
419
+ return c;
420
+ }
421
+ }
422
+ return { mime: "video/webm", ext: "webm" };
423
+ }
424
+
425
  get isRecording(): boolean {
426
  return this._isRecording;
427
  }
 
436
  * @param stream The media stream to record from
437
  */
438
  addVideoStream(key: string, stream: MediaStream) {
 
439
  if (this._isRecording) {
440
  throw new Error("Cannot add video streams while recording");
441
  }
442
 
443
  // Add to video streams dictionary
444
  this.videoStreams[key] = stream;
445
+ // Initialize chunks storage
 
 
 
 
 
 
446
  this.videoChunks[key] = [];
447
+
448
+ // Pre-warm container selection for consistent extension even if added before start
449
+ const { mime, ext } = LeRobotDatasetRecorder.getSupportedRecorderType();
450
+ this.videoMimeByKey[key] = { mime, ext };
451
  }
452
 
453
  /**
 
492
  * Starts recording for all teleoperators and video streams
493
  */
494
  startRecording() {
 
495
  if (this._isRecording) {
496
  console.warn("Recording already in progress");
497
  return;
 
499
 
500
  this._isRecording = true;
501
 
502
+ // Always start a brand new episode using the next available index
503
+ this.episodeIndex = this.teleoperatorData.length;
504
  this.teleoperatorData.push(new LeRobotEpisode());
505
+ this.currentVideoSegmentEpisodeIndex = this.episodeIndex;
506
 
507
  // Start recording video streams
508
  Object.entries(this.videoStreams).forEach(([key, stream]) => {
509
+ // Pick a supported mime/container pair per browser
510
+ const supported =
511
+ this.videoMimeByKey[key] ||
512
+ LeRobotDatasetRecorder.getSupportedRecorderType();
513
+ const { mime, ext } = supported;
514
+ this.videoMimeByKey[key] = { mime, ext };
515
+
516
+ // Reset chunks for a clean segment start
517
+ this.videoChunks[key] = [];
518
+ const mediaRecorder = new MediaRecorder(stream, { mimeType: mime });
519
 
520
  // Handle data available events
521
  mediaRecorder.ondataavailable = (event) => {
 
522
  if (event.data && event.data.size > 0) {
523
  this.videoChunks[key].push(event.data);
524
  }
 
527
  // Save the recorder and start recording
528
  this.mediaRecorders[key] = mediaRecorder;
529
  mediaRecorder.start(1000); // Capture in 1-second chunks
 
 
530
  });
531
  }
532
 
 
590
  recorder.onstop = () => {
591
  // Combine all chunks into a single blob
592
  const chunks = this.videoChunks[key] || [];
593
+ const mime = this.videoMimeByKey[key]?.mime || "video/webm";
594
+ const blob = new Blob(chunks, { type: mime });
595
  this.videoBlobs[key] = blob;
596
+ const segmentEpisodeIndex =
597
+ this.currentVideoSegmentEpisodeIndex ?? this.episodeIndex;
598
+ if (!this.videoBlobsByEpisode[segmentEpisodeIndex]) {
599
+ this.videoBlobsByEpisode[segmentEpisodeIndex] = {} as any;
600
+ }
601
+ this.videoBlobsByEpisode[segmentEpisodeIndex][key] = blob;
602
+ // Prepare for any subsequent recording
603
+ this.videoChunks[key] = [];
604
  resolve();
605
  };
606
 
 
618
  };
619
  }
620
 
621
+ /**
622
+ * Finalizes the current video segment and immediately starts a new one
623
+ * while continuing the recording session. Also advances to the next
624
+ * episode and begins collecting frames under the new episode index.
625
+ *
626
+ * @returns The new episode index
627
+ */
628
+ async nextEpisodeSegment(): Promise<number> {
629
+ if (!this._isRecording) {
630
+ console.warn("nextEpisodeSegment() called while not recording");
631
+ // Ensure episode index points to last episode if any
632
+ this.episodeIndex = Math.max(0, this.teleoperatorData.length - 1);
633
+ return this.episodeIndex;
634
+ }
635
+
636
+ const oldSegmentEpisodeIndex =
637
+ this.currentVideoSegmentEpisodeIndex ?? this.episodeIndex;
638
+
639
+ // Stop current media recorders and persist the segment blobs under the old episode index
640
+ const stopPromises = Object.entries(this.mediaRecorders).map(
641
+ ([key, recorder]) => {
642
+ return new Promise<void>((resolve) => {
643
+ if (recorder.state === "inactive") {
644
+ resolve();
645
+ return;
646
+ }
647
+ recorder.onstop = () => {
648
+ const chunks = this.videoChunks[key] || [];
649
+ const mime = this.videoMimeByKey[key]?.mime || "video/webm";
650
+ const blob = new Blob(chunks, { type: mime });
651
+ if (!this.videoBlobsByEpisode[oldSegmentEpisodeIndex]) {
652
+ this.videoBlobsByEpisode[oldSegmentEpisodeIndex] = {} as any;
653
+ }
654
+ this.videoBlobsByEpisode[oldSegmentEpisodeIndex][key] = blob;
655
+ // Reset chunks for the next segment
656
+ this.videoChunks[key] = [];
657
+ resolve();
658
+ };
659
+ recorder.stop();
660
+ });
661
+ }
662
+ );
663
+
664
+ await Promise.all(stopPromises);
665
+
666
+ // Advance to the next episode
667
+ const newEpisodeIndex = this.teleoperatorData.length;
668
+ this.episodeIndex = newEpisodeIndex;
669
+ this.teleoperatorData.push(new LeRobotEpisode());
670
+
671
+ // Start new media recorders for the next segment
672
+ Object.entries(this.videoStreams).forEach(([key, stream]) => {
673
+ const supported =
674
+ this.videoMimeByKey[key] ||
675
+ LeRobotDatasetRecorder.getSupportedRecorderType();
676
+ const { mime, ext } = supported;
677
+ this.videoMimeByKey[key] = { mime, ext };
678
+
679
+ // Ensure a fresh chunk buffer
680
+ this.videoChunks[key] = [];
681
+ const mediaRecorder = new MediaRecorder(stream, { mimeType: mime });
682
+ mediaRecorder.ondataavailable = (event) => {
683
+ if (event.data && event.data.size > 0) {
684
+ this.videoChunks[key].push(event.data);
685
+ }
686
+ };
687
+ this.mediaRecorders[key] = mediaRecorder;
688
+ mediaRecorder.start(1000);
689
+ });
690
+
691
+ // Point subsequent stop() to the new episode index
692
+ this.currentVideoSegmentEpisodeIndex = newEpisodeIndex;
693
+
694
+ return newEpisodeIndex;
695
+ }
696
+
697
  /**
698
  * Clears the teleoperator data and video blobs
699
  */
700
  clearRecording() {
701
  this.teleoperatorData = [];
702
  this.videoBlobs = {};
703
+ this.videoBlobsByEpisode = {} as any;
704
+ // Reset video chunk buffers so future segments don't include cleared data
705
+ for (const key of Object.keys(this.videoChunks)) {
706
+ this.videoChunks[key] = [];
707
+ }
708
+ this.episodeIndex = 0;
709
+ this.currentVideoSegmentEpisodeIndex = null;
710
  }
711
 
712
  /**
 
1061
  * Generates metadata for the dataset
1062
  * @returns Metadata object for the LeRobot dataset
1063
  */
1064
+ // Deprecated for v2.1 exporter; left for backwards-compat APIs
1065
+ async generateMetadata(_data: any[]): Promise<any> {
1066
+ return {};
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1067
  }
1068
 
1069
  /**
1070
  * Generates statistics for the dataset
1071
  * @returns Statistics object for the LeRobot dataset
1072
  */
1073
+ async getStatistics(_data: any[]): Promise<any> {
1074
+ return {};
 
 
 
 
1075
  }
1076
 
1077
  /**
 
1128
  * Creates the episodes statistics parquet file
1129
  * @returns A Uint8Array blob containing the parquet data
1130
  */
1131
+ async getEpisodeStatistics(_data: any[]): Promise<Uint8Array> {
1132
+ return new Uint8Array();
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1133
  }
1134
 
1135
  generateREADME(metaInfo: string) {
 
1143
  * @private
1144
  */
1145
  async _exportForLeRobotBlobs() {
1146
+ const regularizedEpisodes = (await this.exportEpisodes("json")) as any[];
1147
+
1148
+ // Build episodes parquet files under data/chunk-000/episode_<id>.parquet
1149
  const parquetEpisodeDataFiles = await this._exportEpisodesToBlob(
1150
+ regularizedEpisodes
1151
  );
1152
+
1153
+ // Rewrite parquet file paths to v2.1 layout with chunk folder
1154
+ const rewrittenParquet = parquetEpisodeDataFiles.map((file, idx) => {
1155
+ return {
1156
+ path: `data/chunk-000/episode_${idx
1157
+ .toString()
1158
+ .padStart(6, "0")}.parquet`,
1159
+ content: file.content,
1160
+ };
1161
+ });
1162
+
1163
+ // Videos: videos/chunk-000/observation.images.<camera>/episode_<id>.<ext>
1164
+ const blobArray: { path: string; content: Blob }[] = [...rewrittenParquet];
1165
+
1166
+ const allEpisodeIndices = regularizedEpisodes.map((_: any, i: number) => i);
1167
+ const cameraKeySet = new Set<string>();
1168
+ const episodesVideoMap: { [ep: number]: { [cam: string]: string } } = {};
1169
+
1170
+ for (const epIdx of allEpisodeIndices) {
1171
+ const byCam = this.videoBlobsByEpisode[epIdx] || {};
1172
+ episodesVideoMap[epIdx] = {};
1173
+ for (const [key, blob] of Object.entries(byCam)) {
1174
+ const ext = this.videoMimeByKey[key]?.ext || "mp4";
1175
+ const episodeId = epIdx.toString().padStart(6, "0");
1176
+ const path = `videos/chunk-000/observation.images.${key}/episode_${episodeId}.${ext}`;
1177
+ cameraKeySet.add(key);
1178
+ episodesVideoMap[epIdx][key] = path;
1179
+ blobArray.push({ path, content: blob });
1180
+ }
1181
+ }
1182
+
1183
+ // info.json (v2.1)
1184
+ const cameras = Array.from(cameraKeySet);
1185
+ const numEpisodes = regularizedEpisodes.length;
1186
+ const totalFrames = regularizedEpisodes.reduce(
1187
+ (sum: number, ep: any) => sum + ep.frames.length,
1188
+ 0
1189
  );
1190
+ // Determine a default video extension for video_path pattern
1191
+ let defaultVideoExt = "mp4";
1192
+ if (cameras.length > 0) {
1193
+ const firstKey = cameras[0];
1194
+ const ext = this.videoMimeByKey[firstKey]?.ext;
1195
+ if (ext) defaultVideoExt = ext;
1196
+ }
1197
+
1198
+ // Build features object
1199
+ const features: Record<string, any> = {
1200
+ action: {
1201
+ dtype: "float32",
1202
+ shape: [6],
1203
+ names: [
1204
+ "main_shoulder_pan",
1205
+ "main_shoulder_lift",
1206
+ "main_elbow_flex",
1207
+ "main_wrist_flex",
1208
+ "main_wrist_roll",
1209
+ "main_gripper",
1210
+ ],
1211
+ },
1212
+ "observation.state": {
1213
+ dtype: "float32",
1214
+ shape: [6],
1215
+ names: [
1216
+ "main_shoulder_pan",
1217
+ "main_shoulder_lift",
1218
+ "main_elbow_flex",
1219
+ "main_wrist_flex",
1220
+ "main_wrist_roll",
1221
+ "main_gripper",
1222
+ ],
1223
+ },
1224
+ timestamp: { dtype: "float32", shape: [1], names: null },
1225
+ frame_index: { dtype: "int64", shape: [1], names: null },
1226
+ episode_index: { dtype: "int64", shape: [1], names: null },
1227
+ index: { dtype: "int64", shape: [1], names: null },
1228
+ task_index: { dtype: "int64", shape: [1], names: null },
1229
+ };
1230
+ for (const cam of cameras) {
1231
+ // Map mime to codec
1232
+ const mime = this.videoMimeByKey[cam]?.mime || "video/mp4";
1233
+ let codec = "avc1";
1234
+ if (mime.includes("vp9")) codec = "vp9";
1235
+ else if (mime.includes("vp8")) codec = "vp8";
1236
+ features[`observation.images.${cam}`] = {
1237
+ dtype: "video",
1238
+ shape: [480, 640, 3],
1239
+ names: ["height", "width", "channels"],
1240
+ info: {
1241
+ "video.fps": this.fps,
1242
+ "video.height": 480,
1243
+ "video.width": 640,
1244
+ "video.channels": 3,
1245
+ "video.codec": codec,
1246
+ "video.pix_fmt": "yuv420p",
1247
+ "video.is_depth_map": false,
1248
+ has_audio: false,
1249
+ },
1250
+ };
1251
+ }
1252
+
1253
+ const infoJson = {
1254
+ version: "2.1",
1255
+ // Compatibility fields consumed by visualizers
1256
+ total_episodes: numEpisodes,
1257
+ total_frames: totalFrames,
1258
+ fps: this.fps,
1259
+ data_path:
1260
+ "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
1261
+ video_path: `videos/chunk-{episode_chunk:03d}/{video_key}/episode_{episode_index:06d}.${defaultVideoExt}`,
1262
+ features,
1263
+ // Extra descriptive fields
1264
+ name: `lerobot_dataset_${new Date().toISOString().slice(0, 10)}`,
1265
+ robot: this.robotLabel || "unknown",
1266
+ cameras,
1267
+ action_space: "joint_position",
1268
+ frame_rate: this.fps,
1269
+ num_episodes: numEpisodes,
1270
+ created_by: "lerobot.js",
1271
+ created_at: new Date().toISOString(),
1272
+ } as any;
1273
+
1274
+ // episodes.jsonl
1275
+ const episodesJsonlLines: string[] = [];
1276
+ for (let epIdx = 0; epIdx < numEpisodes; epIdx++) {
1277
+ const episodeId = epIdx.toString().padStart(6, "0");
1278
+ const length = regularizedEpisodes[epIdx]?.frames.length || 0;
1279
+ const videos: any = {};
1280
+ Object.entries(episodesVideoMap[epIdx] || {}).forEach(([cam, path]) => {
1281
+ videos[cam] = path;
1282
+ });
1283
+ const row = {
1284
+ episode_id: episodeId,
1285
+ task: this.taskDescription || "default",
1286
+ length,
1287
+ videos,
1288
+ };
1289
+ episodesJsonlLines.push(JSON.stringify(row));
1290
+ }
1291
 
1292
+ // tasks.jsonl (single task)
1293
+ const tasksJsonl = JSON.stringify({
1294
+ task: this.taskDescription || "default",
1295
+ description: this.taskDescription || "default",
1296
+ });
1297
+
1298
+ // stats.json (minimal)
1299
+ const lengths = regularizedEpisodes.map((ep: any) => ep.frames.length);
1300
+ const epMin = lengths.length ? Math.min(...lengths) : 0;
1301
+ const epMax = lengths.length ? Math.max(...lengths) : 0;
1302
+ const epMean = lengths.length
1303
+ ? lengths.reduce((a, b) => a + b, 0) / lengths.length
1304
+ : 0;
1305
+ const statsJson = {
1306
+ total_frames: totalFrames,
1307
+ episode_lengths: { min: epMin, max: epMax, mean: Math.round(epMean) },
1308
+ } as any;
1309
+
1310
+ const readme = this.generateREADME(JSON.stringify(infoJson));
1311
+
1312
+ blobArray.push(
1313
  {
1314
  path: "meta/info.json",
1315
+ content: new Blob([JSON.stringify(infoJson, null, 2)], {
1316
  type: "application/json",
1317
  }),
1318
  },
1319
  {
1320
+ path: "meta/episodes.jsonl",
1321
+ content: new Blob([episodesJsonlLines.join("\n") + "\n"], {
1322
+ type: "application/jsonlines",
1323
  }),
1324
  },
1325
  {
1326
+ path: "meta/tasks.jsonl",
1327
+ content: new Blob([tasksJsonl + "\n"], {
1328
+ type: "application/jsonlines",
1329
+ }),
1330
  },
1331
  {
1332
+ path: "meta/stats.json",
1333
+ content: new Blob([JSON.stringify(statsJson, null, 2)], {
1334
+ type: "application/json",
1335
+ }),
1336
  },
1337
  {
1338
  path: "README.md",
1339
  content: new Blob([readme], { type: "text/markdown" }),
1340
+ }
1341
+ );
1342
 
1343
+ // episodes_stats.jsonl (v2.1)
1344
+ const episodesStatsLines: string[] = [];
1345
+ for (let epIdx = 0; epIdx < numEpisodes; epIdx++) {
1346
+ const episodeId = epIdx.toString().padStart(6, "0");
1347
+ const length = regularizedEpisodes[epIdx]?.frames.length || 0;
1348
+ const timestamps = (regularizedEpisodes[epIdx]?.frames || []).map(
1349
+ (f: any) => f.timestamp
1350
+ );
1351
+ const fromTs = timestamps.length ? Math.min(...timestamps) : 0;
1352
+ const toTs = timestamps.length ? Math.max(...timestamps) : 0;
1353
+
1354
+ const row: any = {
1355
+ episode_id: episodeId,
1356
+ "data/chunk_index": 0,
1357
+ "data/file": `data/chunk-000/episode_${episodeId}.parquet`,
1358
+ length,
1359
+ };
1360
+ // add per-camera video fields
1361
+ const map = episodesVideoMap[epIdx] || {};
1362
+ for (const [cam, path] of Object.entries(map)) {
1363
+ row[`videos/observation.images.${cam}/chunk_index`] = 0;
1364
+ row[`videos/observation.images.${cam}/file`] = path;
1365
+ row[`videos/observation.images.${cam}/from_timestamp`] = fromTs;
1366
+ row[`videos/observation.images.${cam}/to_timestamp`] = toTs;
1367
+ }
1368
+ episodesStatsLines.push(JSON.stringify(row));
1369
  }
1370
+ blobArray.push({
1371
+ path: "meta/episodes_stats.jsonl",
1372
+ content: new Blob([episodesStatsLines.join("\n") + "\n"], {
1373
+ type: "application/jsonlines",
1374
+ }),
1375
+ });
1376
 
1377
  return blobArray;
1378
  }
 
1409
  return await zip.generateAsync({ type: "blob" });
1410
  }
1411
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1412
  /**
1413
  * Exports the LeRobot dataset in various formats
1414
  *
1415
+ * @param format The export format - 'blobs', 'zip', or 'zip-download'
1416
+ * @param options Additional options (currently unused)
1417
+ * @returns The exported data in the requested format
 
 
 
 
 
 
 
 
1418
  */
1419
  async exportForLeRobot(
1420
+ format: "blobs" | "zip" | "zip-download" = "zip-download"
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1421
  ) {
1422
  switch (format) {
1423
  case "blobs":
 
1426
  case "zip":
1427
  return this._exportForLeRobotZip();
1428
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1429
  case "zip-download":
1430
  default:
1431
  // Get the zip blob
 
1453
  }
1454
  }
1455
  }
1456
+
1457
+ // Simple record() function API - wraps LeRobotDatasetRecorder
1458
+ import type {
1459
+ RecordConfig,
1460
+ RecordProcess,
1461
+ RecordingState,
1462
+ RecordingData,
1463
+ RobotRecordingData,
1464
+ } from "./types/recording.js";
1465
+
1466
+ /**
1467
+ * Simple recording function that follows LeRobot.js conventions
1468
+ *
1469
+ * Records robot motor positions and teleoperation data using a clean function API
1470
+ * that matches the patterns established by calibrate() and teleoperate().
1471
+ *
1472
+ * @param config Recording configuration with explicit teleoperator dependency
1473
+ * @returns RecordProcess with start(), stop(), getState(), and result
1474
+ *
1475
+ * @example
1476
+ * ```typescript
1477
+ * // 1. Create teleoperation
1478
+ * const teleoperationProcess = await teleoperate({
1479
+ * robot: connectedRobot,
1480
+ * teleop: { type: "keyboard" },
1481
+ * calibrationData: calibrationData,
1482
+ * });
1483
+ *
1484
+ * // 2. Create recording with explicit teleoperator dependency
1485
+ * const recordProcess = await record({
1486
+ * teleoperator: teleoperationProcess.teleoperator,
1487
+ * options: {
1488
+ * fps: 30,
1489
+ * taskDescription: "Pick and place task",
1490
+ * onDataUpdate: (data) => console.log(`Recorded ${data.frameCount} frames`),
1491
+ * }
1492
+ * });
1493
+ *
1494
+ * // 3. Start both processes
1495
+ * teleoperationProcess.start();
1496
+ * recordProcess.start();
1497
+ *
1498
+ * // 4. Stop recording
1499
+ * const robotData = await recordProcess.stop();
1500
+ * ```
1501
+ */
1502
+ export async function record(config: RecordConfig): Promise<RecordProcess> {
1503
+ // Use the provided teleoperator (explicit dependency - good architecture!)
1504
+ const recorder = new LeRobotDatasetRecorder(
1505
+ [config.teleoperator],
1506
+ config.videoStreams || {},
1507
+ config.options?.fps || 30,
1508
+ config.options?.taskDescription || "Robot recording"
1509
+ );
1510
+
1511
+ // Set robot metadata if provided
1512
+ if (config.robotType) {
1513
+ (recorder as any).setRobotLabel?.(config.robotType);
1514
+ }
1515
+
1516
+ let startTime = 0;
1517
+ let resultPromise: Promise<RobotRecordingData> | null = null;
1518
+ let stateUpdateInterval: NodeJS.Timeout | null = null;
1519
+
1520
+ const recordProcess: RecordProcess = {
1521
+ start(): void {
1522
+ startTime = Date.now();
1523
+ recorder.startRecording();
1524
+
1525
+ // Set up state update polling for callbacks
1526
+ if (config.options?.onStateUpdate || config.options?.onDataUpdate) {
1527
+ stateUpdateInterval = setInterval(() => {
1528
+ if (recorder.isRecording) {
1529
+ const state = recordProcess.getState();
1530
+
1531
+ if (config.options?.onStateUpdate) {
1532
+ config.options.onStateUpdate(state);
1533
+ }
1534
+
1535
+ if (config.options?.onDataUpdate) {
1536
+ config.options.onDataUpdate({
1537
+ frameCount: state.frameCount,
1538
+ currentEpisode: state.episodeCount,
1539
+ recentFrames: [],
1540
+ });
1541
+ }
1542
+ }
1543
+ }, 100);
1544
+ }
1545
+ },
1546
+
1547
+ async stop(): Promise<RobotRecordingData> {
1548
+ if (stateUpdateInterval) {
1549
+ clearInterval(stateUpdateInterval);
1550
+ stateUpdateInterval = null;
1551
+ }
1552
+
1553
+ const result = await recorder.stopRecording();
1554
+
1555
+ const robotData: RobotRecordingData = {
1556
+ episodes: recorder.episodes.map((episode) => episode.frames),
1557
+ metadata: {
1558
+ fps: config.options?.fps || 30,
1559
+ robotType: config.robotType || "unknown",
1560
+ startTime: startTime,
1561
+ endTime: Date.now(),
1562
+ totalFrames: recorder.teleoperatorData.reduce(
1563
+ (sum, ep) => sum + ep.length,
1564
+ 0
1565
+ ),
1566
+ totalEpisodes: recorder.teleoperatorData.length,
1567
+ },
1568
+ };
1569
+
1570
+ return robotData;
1571
+ },
1572
+
1573
+ getState(): RecordingState {
1574
+ return {
1575
+ isActive: recorder.isRecording,
1576
+ frameCount: recorder.teleoperatorData.reduce(
1577
+ (sum, ep) => sum + ep.length,
1578
+ 0
1579
+ ),
1580
+ episodeCount: recorder.teleoperatorData.length,
1581
+ duration: recorder.isRecording ? Date.now() - startTime : 0,
1582
+ lastUpdate: Date.now(),
1583
+ };
1584
+ },
1585
+
1586
+ get result(): Promise<RobotRecordingData> {
1587
+ if (!resultPromise) {
1588
+ resultPromise = new Promise((resolve) => {
1589
+ const originalStop = recordProcess.stop;
1590
+ recordProcess.stop = async () => {
1591
+ const data = await originalStop();
1592
+ resolve(data);
1593
+ return data;
1594
+ };
1595
+ });
1596
+ }
1597
+ return resultPromise;
1598
+ },
1599
+
1600
+ getEpisodeCount(): number {
1601
+ return recorder.teleoperatorData.length;
1602
+ },
1603
+
1604
+ getEpisodes(): any[] {
1605
+ return recorder.teleoperatorData;
1606
+ },
1607
+
1608
+ clearEpisodes(): void {
1609
+ (recorder as any).clearRecording();
1610
+ },
1611
+
1612
+ async nextEpisode(): Promise<number> {
1613
+ return (recorder as any).nextEpisodeSegment();
1614
+ },
1615
+
1616
+ restoreEpisodes(episodes: any[]): void {
1617
+ (recorder as any).teleoperatorData = [...episodes];
1618
+ },
1619
+
1620
+ addCamera(name: string, stream: MediaStream): void {
1621
+ (recorder as any).addVideoStream(name, stream);
1622
+ },
1623
+
1624
+ removeCamera(name: string): void {
1625
+ const videoStreams = (recorder as any).videoStreams || {};
1626
+ delete videoStreams[name];
1627
+ },
1628
+
1629
+ async exportForLeRobot(
1630
+ format: "blobs" | "zip" | "zip-download" = "zip-download"
1631
+ ): Promise<any> {
1632
+ return recorder.exportForLeRobot(format);
1633
+ },
1634
+ };
1635
+
1636
+ return recordProcess;
1637
+ }
packages/web/src/s3_uploader.ts DELETED
@@ -1,172 +0,0 @@
1
- // Note: To use this module, you'll need to install the AWS SDK:
2
- // npm install @aws-sdk/client-s3 @aws-sdk/lib-storage
3
-
4
- import { S3Client, HeadBucketCommand } from "@aws-sdk/client-s3";
5
- import { Upload } from "@aws-sdk/lib-storage";
6
-
7
- // Define ContentSource type locally to avoid HuggingFace dependency
8
- type ContentSource = Blob | ArrayBuffer | Uint8Array | string;
9
- type FileArray = Array<URL | File | { path: string; content: ContentSource }>;
10
-
11
- /**
12
- * Uploads a leRobot dataset to Amazon S3
13
- */
14
- export class LeRobotS3Uploader extends EventTarget {
15
- private _bucketName: string;
16
- private _region: string;
17
- private _uploaded: boolean;
18
- private _bucket_exists: boolean;
19
- private _s3Client: S3Client | null;
20
-
21
- constructor(bucketName: string, region: string = "us-east-1") {
22
- super();
23
- this._bucketName = bucketName;
24
- this._region = region;
25
- this._uploaded = false;
26
- this._bucket_exists = false;
27
- this._s3Client = null;
28
- }
29
-
30
- /**
31
- * Returns whether the bucket has been successfully checked/created
32
- */
33
- get bucketExists(): boolean {
34
- return this._bucket_exists;
35
- }
36
-
37
- get uploaded(): boolean {
38
- return this._uploaded;
39
- }
40
-
41
- /**
42
- * Initialize the S3 client with credentials
43
- *
44
- * @param accessKeyId AWS access key ID
45
- * @param secretAccessKey AWS secret access key
46
- */
47
- initializeClient(accessKeyId: string, secretAccessKey: string): void {
48
- this._s3Client = new S3Client({
49
- region: this._region,
50
- credentials: {
51
- accessKeyId,
52
- secretAccessKey
53
- }
54
- });
55
- }
56
-
57
- /**
58
- * Checks if the bucket exists and uploads files to it
59
- *
60
- * @param files The files to upload
61
- * @param accessKeyId AWS access key ID
62
- * @param secretAccessKey AWS secret access key
63
- * @param prefix Optional prefix (folder) to upload files to within the bucket
64
- * @param referenceId The reference id for the upload, to track it (optional)
65
- */
66
- async checkBucketAndUploadFiles(
67
- files: FileArray,
68
- accessKeyId: string,
69
- secretAccessKey: string,
70
- prefix: string = "",
71
- referenceId: string = ""
72
- ): Promise<void> {
73
- // Initialize the client if not already done
74
- if (!this._s3Client) {
75
- this.initializeClient(accessKeyId, secretAccessKey);
76
- }
77
-
78
- // Check if bucket exists
79
- try {
80
- await this._s3Client!.send(new HeadBucketCommand({ Bucket: this._bucketName }));
81
- this._bucket_exists = true;
82
- this.dispatchEvent(new CustomEvent("bucketExists", {
83
- detail: { bucketName: this._bucketName }
84
- }));
85
- } catch (error) {
86
- throw new Error(`Bucket ${this._bucketName} does not exist or you don't have permission to access it`);
87
- }
88
-
89
- // Upload files
90
- const uploadPromises: Promise<void>[] = [];
91
- for (const file of files) {
92
- uploadPromises.push(this.uploadFileWithProgress([file], prefix, referenceId));
93
- }
94
-
95
- await Promise.all(uploadPromises);
96
- this._uploaded = true;
97
- }
98
-
99
- /**
100
- * Uploads files to S3 with progress events
101
- *
102
- * @param files The files to upload
103
- * @param prefix Optional prefix (folder) to upload files to within the bucket
104
- * @param referenceId The reference id for the upload, to track it (optional)
105
- */
106
- async uploadFileWithProgress(
107
- files: FileArray,
108
- prefix: string = "",
109
- referenceId: string = ""
110
- ): Promise<void> {
111
- if (!this._s3Client) {
112
- throw new Error("S3 client not initialized. Call initializeClient first.");
113
- }
114
-
115
- for (const file of files) {
116
- let key: string;
117
- let body: any;
118
-
119
- if (file instanceof URL) {
120
- const response = await fetch(file);
121
- body = await response.blob();
122
- key = `${prefix}${prefix ? '/' : ''}${file.pathname.split('/').pop()}`;
123
- } else if (file instanceof File) {
124
- body = file;
125
- key = `${prefix}${prefix ? '/' : ''}${file.name}`;
126
- } else {
127
- body = file.content;
128
- key = `${prefix}${prefix ? '/' : ''}${file.path}`;
129
- }
130
-
131
- const upload = new Upload({
132
- client: this._s3Client,
133
- params: {
134
- Bucket: this._bucketName,
135
- Key: key,
136
- Body: body,
137
- },
138
- });
139
-
140
- // Set up progress tracking
141
- upload.on('httpUploadProgress', (progress) => {
142
- this.dispatchEvent(new CustomEvent("progress", {
143
- detail: {
144
- progressEvent: progress,
145
- bucketName: this._bucketName,
146
- key,
147
- referenceId
148
- }
149
- }));
150
- });
151
-
152
- await upload.done();
153
- }
154
- }
155
-
156
- /**
157
- * Generates a pre-signed URL for downloading a file from S3
158
- *
159
- * @param _key The key (path) of the file in the S3 bucket
160
- * @param _expiresIn The number of seconds until the URL expires (default: 3600 = 1 hour)
161
- * @returns A pre-signed URL for downloading the file
162
- */
163
- async generatePresignedUrl(_key: string, _expiresIn: number = 3600): Promise<string> {
164
- if (!this._s3Client) {
165
- throw new Error("S3 client not initialized. Call initializeClient first.");
166
- }
167
-
168
- // Note: This requires the @aws-sdk/s3-request-presigner package
169
- // Implementation would go here
170
- throw new Error("generatePresignedUrl not implemented. Requires @aws-sdk/s3-request-presigner package.");
171
- }
172
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
packages/web/src/types/recording.ts ADDED
@@ -0,0 +1,122 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ /**
2
+ * Types for the simple record() function API
3
+ */
4
+
5
+ import type { WebTeleoperator } from "../teleoperators/base-teleoperator.js";
6
+ import type { LeRobotDatasetRecorder } from "../record.js";
7
+
8
+ /**
9
+ * Configuration for the record() function
10
+ * Supports both upfront configuration and runtime management
11
+ */
12
+ export interface RecordConfig {
13
+ /** The teleoperator to record from (explicit dependency) */
14
+ teleoperator: WebTeleoperator;
15
+
16
+ /** Optional: video streams to record by camera name (e.g. { main: videoStream, wrist: videoStream }) */
17
+ videoStreams?: { [cameraName: string]: MediaStream };
18
+
19
+ /** Optional: robot type/model name for metadata (e.g. "so100") */
20
+ robotType?: string;
21
+
22
+ /** Optional recording configuration */
23
+ options?: {
24
+ /** Target frames per second (default: 30) */
25
+ fps?: number;
26
+ /** Task description for the recording */
27
+ taskDescription?: string;
28
+ /** Callback for real-time recording data updates */
29
+ onDataUpdate?: (data: RecordingData) => void;
30
+ /** Callback for recording state changes */
31
+ onStateUpdate?: (state: RecordingState) => void;
32
+ };
33
+ }
34
+
35
+ /**
36
+ * Process interface returned by record() function
37
+ * Supports flexible upfront config and runtime management
38
+ */
39
+ export interface RecordProcess {
40
+ // Recording control
41
+ /** Start recording */
42
+ start(): void;
43
+ /** Stop recording and return the result */
44
+ stop(): Promise<RobotRecordingData>;
45
+ /** Get current recording state */
46
+ getState(): RecordingState;
47
+ /** Promise that resolves when recording is stopped with the data */
48
+ result: Promise<RobotRecordingData>;
49
+
50
+ // Episode management (runtime)
51
+ /** Get total number of episodes recorded */
52
+ getEpisodeCount(): number;
53
+ /** Get raw episode data for viewing/analysis */
54
+ getEpisodes(): any[];
55
+ /** Delete all recorded episodes */
56
+ clearEpisodes(): void;
57
+ /** Start a new episode segment and get the new episode index */
58
+ nextEpisode(): Promise<number>;
59
+ /** Restore previously recorded episodes */
60
+ restoreEpisodes(episodes: any[]): void;
61
+
62
+ // Camera management (runtime)
63
+ /** Add a camera stream for recording */
64
+ addCamera(name: string, stream: MediaStream): void;
65
+ /** Remove a camera from recording */
66
+ removeCamera(name: string): void;
67
+
68
+ // Export
69
+ /** Export the recorded dataset in various formats */
70
+ exportForLeRobot(format?: "blobs" | "zip" | "zip-download"): Promise<any>;
71
+ }
72
+
73
+ /**
74
+ * Current state of the recording process
75
+ */
76
+ export interface RecordingState {
77
+ /** Whether recording is currently active */
78
+ isActive: boolean;
79
+ /** Total number of frames recorded */
80
+ frameCount: number;
81
+ /** Number of episodes recorded */
82
+ episodeCount: number;
83
+ /** Duration of current recording in milliseconds */
84
+ duration: number;
85
+ /** Timestamp of last update */
86
+ lastUpdate: number;
87
+ }
88
+
89
+ /**
90
+ * Real-time recording data for UI feedback
91
+ */
92
+ export interface RecordingData {
93
+ /** Total frames recorded */
94
+ frameCount: number;
95
+ /** Current episode number */
96
+ currentEpisode: number;
97
+ /** Recent frames for preview (last few frames) */
98
+ recentFrames: any[];
99
+ }
100
+
101
+ /**
102
+ * Final robot recording data (hardware only, no video)
103
+ */
104
+ export interface RobotRecordingData {
105
+ /** Recorded episodes with motor position data */
106
+ episodes: any[];
107
+ /** Recording metadata */
108
+ metadata: {
109
+ /** Frames per second */
110
+ fps: number;
111
+ /** Robot type if available */
112
+ robotType: string;
113
+ /** Recording start time */
114
+ startTime: number;
115
+ /** Recording end time */
116
+ endTime: number;
117
+ /** Total frames recorded */
118
+ totalFrames: number;
119
+ /** Total episodes recorded */
120
+ totalEpisodes: number;
121
+ };
122
+ }
packages/web/src/utils/record/metadataInfo.ts DELETED
@@ -1,159 +0,0 @@
1
- export interface VideoInfo {
2
- height: number;
3
- width: number;
4
- channels: number;
5
- codec: string;
6
- pix_fmt: string;
7
- is_depth_map: boolean;
8
- has_audio: boolean;
9
- }
10
-
11
- /**
12
- * Metadata parameters interface
13
- */
14
- interface MetadataParams {
15
- total_episodes: number;
16
- total_frames: number;
17
- total_tasks: number;
18
- chunks_size: number;
19
- fps: number;
20
- splits: { [key: string]: string };
21
- features: { [key: string]: any };
22
- videos_info: VideoInfo[];
23
- data_files_size_in_mb: number;
24
- video_files_size_in_mb: number;
25
- }
26
-
27
- /**
28
- * Generates and returns a metadata information dictionary
29
- * Needs some named parameters passed as parameters
30
- */
31
- function getMetadataInfo(params: MetadataParams) {
32
- return {
33
- "codebase_version": "v2.1",
34
- "robot_type": "so100",
35
- "total_episodes": params.total_episodes,
36
- "total_frames": params.total_frames,
37
- "total_tasks": params.total_tasks,
38
- "total_videos": params.videos_info.length,
39
- "total_chunks": 1,
40
- "chunks_size": params.chunks_size,
41
- "fps": params.fps,
42
- "splits": {
43
- "train": `0:${params.total_episodes}`
44
- },
45
- "data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
46
- "video_path": "videos/chunk-{episode_chunk:03d}/{video_key}/episode_{episode_index:06d}.mp4",
47
- "features": {
48
- "action": {
49
- "dtype": "float32",
50
- "shape": [
51
- 6
52
- ],
53
- "names": [
54
- "main_shoulder_pan",
55
- "main_shoulder_lift",
56
- "main_elbow_flex",
57
- "main_wrist_flex",
58
- "main_wrist_roll",
59
- "main_gripper"
60
- ],
61
- "fps": params.fps
62
- },
63
- "observation.state": {
64
- "dtype": "float32",
65
- "shape": [
66
- 6
67
- ],
68
- "names": [
69
- "main_shoulder_pan",
70
- "main_shoulder_lift",
71
- "main_elbow_flex",
72
- "main_wrist_flex",
73
- "main_wrist_roll",
74
- "main_gripper"
75
- ],
76
- "fps": params.fps
77
- },
78
- "observation.images.front": {
79
- "dtype": "video",
80
- "shape": [
81
- 480,
82
- 640,
83
- 3
84
- ],
85
- "names": [
86
- "height",
87
- "width",
88
- "channels"
89
- ],
90
- "info": {
91
- "video.fps": params.fps,
92
- "video.height": 480,
93
- "video.width": 640,
94
- "video.channels": 3,
95
- "video.codec": "av1",
96
- "video.pix_fmt": "yuv420p",
97
- "video.is_depth_map": false,
98
- "has_audio": false
99
- }
100
- },
101
- "timestamp": {
102
- "dtype": "float32",
103
- "shape": [
104
- 1
105
- ],
106
- "names": null,
107
- "fps": params.fps
108
- },
109
- "frame_index": {
110
- "dtype": "int64",
111
- "shape": [
112
- 1
113
- ],
114
- "names": null,
115
- "fps": params.fps
116
- },
117
- "episode_index": {
118
- "dtype": "int64",
119
- "shape": [
120
- 1
121
- ],
122
- "names": null,
123
- "fps": params.fps
124
- },
125
- "index": {
126
- "dtype": "int64",
127
- "shape": [
128
- 1
129
- ],
130
- "names": null,
131
- "fps": params.fps
132
- },
133
- "task_index": {
134
- "dtype": "int64",
135
- "shape": [
136
- 1
137
- ],
138
- "names": null,
139
- "fps": params.fps
140
- }
141
- },
142
- "data_files_size_in_mb": 100,
143
- "video_files_size_in_mb": 500
144
- }
145
- }
146
-
147
- export function getVideoInfo(width: number, height: number): VideoInfo {
148
- return {
149
- height,
150
- width,
151
- channels: 3,
152
- codec: "h264",
153
- pix_fmt: "yuv420p",
154
- is_depth_map: false,
155
- has_audio: false
156
- };
157
- }
158
-
159
- export default getMetadataInfo;
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
packages/web/src/utils/record/stats.ts DELETED
@@ -1,79 +0,0 @@
1
- /**
2
- * Calculates basic statistics (min, max, mean, std, count) for a numeric array
3
- */
4
- function calculateStats(values: number[]): { min: number[], max: number[], mean: number[], std: number[], count: number[] } {
5
- const count = values.length;
6
- if (count === 0) {
7
- return {
8
- min: [0],
9
- max: [0],
10
- mean: [0],
11
- std: [0],
12
- count: [0]
13
- };
14
- }
15
-
16
- let min = Infinity;
17
- let max = -Infinity;
18
-
19
- for(let value of values){
20
- min = Math.min(min, value);
21
- max = Math.max(max, value);
22
- }
23
-
24
- const sum = values.reduce((acc, val) => acc + val, 0);
25
- const mean = sum / count;
26
-
27
- // Calculate standard deviation
28
- const squareDiffs = values.map(value => Math.pow(value - mean, 2));
29
- const avgSquareDiff = squareDiffs.reduce((acc, val) => acc + val, 0) / count;
30
- const std = Math.sqrt(avgSquareDiff);
31
-
32
- return {
33
- min: [min],
34
- max: [max],
35
- mean: [mean],
36
- std: [std],
37
- count: [count]
38
- };
39
- }
40
-
41
- /**
42
- * Generates statistics for a dataset
43
- * @param data The dataset to analyze
44
- * @param cameraKeys Array of camera keys for dynamic observation.images entries
45
- */
46
- export function getStats(data: any[], cameraKeys: string[] = []): any {
47
- // Extract timestamp and episode_index values
48
- const timestamps = data.map(item => item.timestamp);
49
- const episodeIndices = data.map(item => item.episode_index);
50
-
51
- // Extract other common fields if they exist
52
- const frameIndices = data.map(item => item.frame_index || 0);
53
- const taskIndices = data.map(item => item.task_index || 0);
54
-
55
- const stats: any = {
56
- // Standard fields
57
- "timestamp": calculateStats(timestamps),
58
- "episode_index": calculateStats(episodeIndices),
59
- "frame_index": calculateStats(frameIndices),
60
- "task_index": calculateStats(taskIndices),
61
- };
62
-
63
- // Add observation.images entries for each camera key
64
- cameraKeys.forEach(key => {
65
- // In a real implementation, you would calculate actual stats from video data
66
- // Since we don't have actual video frame data to analyze, we'll use placeholder values
67
- stats[`observation.images.${key}`] = {
68
- "min": [[[0.0]], [[0.0]], [[0.0]]], // R,G,B channels min
69
- "max": [[[255.0]], [[255.0]], [[255.0]]], // R,G,B channels max
70
- "mean": [[[127.5]], [[127.5]], [[127.5]]], // R,G,B channels mean
71
- "std": [[[50.0]], [[50.0]], [[50.0]]], // R,G,B channels std
72
- "count": [[[data.length * 3]]] // Number of pixels × 3 channels
73
- };
74
- });
75
-
76
- return stats;
77
- }
78
-
79
- export default getStats;