Spaces:
Configuration error
feat: record (#14)
Browse files* feat: moved record-demo to own view
* feat: simplified recording; interactive how to
* feat: add visual progress indicators to recording how-to; fix keyboard teleop not to hijack inputs or browser shortcuts; handle only teleop keys with normalized casing; allow editing camera names while teleop is active; add per-camera source selector on card via camera icon with fixed-width inline select; switch streams and persist camera config; disable source changes during active recording
* feat: improved performance of webcam while teleop is active
* feat: select supported video format
* fix: attach camera to recorder
* docs: best pratcise when recording video in the web
* fix: make video work
* fix: each episode has its own video
* feat(record): making sure that we use version 2.1 of the spec
* refactor: move upload logic from library to demo, unify on record() api
remove cloud storage uploaders (LeRobotHFUploader, LeRobotS3Uploader) from
@lerobot/web library, reducing dependencies by ~700kb (@huggingface/hub,
@aws-sdk/client-s3, @aws-sdk/lib-storage).
add dataset-uploader.ts utility to demo for handling HuggingFace and S3 uploads,
allowing applications to implement custom upload strategies.
update RecordConfig to support videoStreams parameter for multi-camera recording.
enhance RecordProcess to expose underlying recorder instance for advanced use cases
(episode persistence, dynamic camera management, custom metadata).
update cyberpunk demo to use record() function with recorder access for all
advanced features, eliminating direct LeRobotDatasetRecorder instantiation.
update package.json, README, index.ts exports, and type definitions accordingly.
BREAKING CHANGE: exportForLeRobot('huggingface') and exportForLeRobot('s3')
removed from library (users should export 'blobs' and handle upload in their apps)
* feat(record): add flexible recordprocess api with runtime management
expand RecordConfig to support robotType metadata for upfront configuration.
enhance RecordProcess with runtime management methods:
- getEpisodeCount() / getEpisodes() - episode introspection
- clearEpisodes() - delete all recorded episodes
- nextEpisode() - create new episode segment
- restoreEpisodes() - restore persisted episodes
- addCamera() / removeCamera() - dynamic camera management
remove direct recorder access requirement from demo. update cyberpunk example
to use clean RecordProcess API instead of accessing internal recorder instance.
support both upfront configuration (videos, robot type, options) and runtime
operations (add cameras, manage episodes, export), providing maximum flexibility
for both simple and complex recording workflows.
update recorder.tsx to use new RecordProcess methods:
- record() call with robotType metadata
- episode management via getEpisodeCount() instead of direct access
- camera management via addCamera() instead of direct recorder access
- episode display via getEpisodes() for clean data access
this eliminates need for 'as any' casts and direct recorder access in demo,
maintaining clean API surface while supporting all advanced recording features.
* docs(record): simplify readme to show unified record() api only
remove references to LeRobotDatasetRecorder class and advanced use cases.
show single, clean record() function with all features available through
unified RecordProcess API.
simplify example to show:
- record() creation with config (teleoperator, videoStreams, robotType, options)
- recording control (start, stop, getState)
- episode management (getEpisodeCount, nextEpisode, clearEpisodes)
- dynamic camera management (addCamera)
- export options (zip-download, blobs)
users dont need to know about internal classes - record() handles everything.
* docs(readme): remove duplicate record() section
the 'dataset recording and export' section was a duplicate of the
record() documentation already in the core api section. remove it
to avoid confusion - one section, one api.
* docs(demo): add record() api documentation to docs section
add complete record() api reference to the demo's docs section including:
- usage example with teleoperation and video streams
- complete options and return types
- episode management methods (getEpisodeCount, nextEpisode, clearEpisodes)
- dynamic camera management (addCamera)
- export formats (blobs, zip, zip-download)
keeps documentation in sync with the library api.
* docs(demo): simplify record() example, remove setTimeout
remove unnecessary setTimeout from docs example and show cleaner,
more straightforward usage pattern. add nextEpisode() call to demonstrate
runtime episode management feature.
* docs(readme): fix record() example and api reference
remove setTimeout and show clean, straightforward usage.
add videoStreams, robotType to example and options documentation.
update returns to document all RecordProcess methods:
- episode management (getEpisodeCount, nextEpisode, clearEpisodes)
- dynamic cameras (addCamera)
- export (exportForLeRobot)
* chore: add changeset for @lerobot/web minor release
record api enhancements including flexible runtime management,
video stream support, and episode/camera management methods.
* chore(demo): mark record() as completed on roadmap
move record from in_progress to completed status.
reorder roadmap so record appears before SO-100 leader arm.
keep train and eval as planned features.
- .changeset/odd-mice-eat.md +24 -0
- docs/conventions.md +20 -0
- docs/dataset/v2.1.md +209 -0
- docs/planning/{007_record.md → 007_refactor_record.md} +273 -166
- examples/cyberpunk-standalone/src/App.tsx +66 -18
- examples/cyberpunk-standalone/src/components/calibration-view.tsx +42 -22
- examples/cyberpunk-standalone/src/components/device-dashboard.tsx +14 -1
- examples/cyberpunk-standalone/src/components/docs-section.tsx +107 -0
- examples/cyberpunk-standalone/src/components/recorder.tsx +679 -677
- examples/cyberpunk-standalone/src/components/recording-view.tsx +598 -0
- examples/cyberpunk-standalone/src/components/roadmap-section.tsx +5 -5
- examples/cyberpunk-standalone/src/components/teleoperation-view.tsx +108 -35
- examples/cyberpunk-standalone/src/components/teleoperator-episodes-view.tsx +45 -19
- examples/cyberpunk-standalone/src/components/teleoperator-frames-view.tsx +73 -44
- examples/cyberpunk-standalone/src/components/teleoperator-joint-graph.tsx +92 -57
- examples/cyberpunk-standalone/src/utils/dataset-uploader.ts +243 -0
- package.json +1 -1
- packages/web/README.md +56 -54
- packages/web/package.json +0 -3
- packages/web/src/hf_uploader.ts +0 -86
- packages/web/src/index.ts +10 -2
- packages/web/src/record.ts +547 -566
- packages/web/src/s3_uploader.ts +0 -172
- packages/web/src/types/recording.ts +122 -0
- packages/web/src/utils/record/metadataInfo.ts +0 -159
- packages/web/src/utils/record/stats.ts +0 -79
|
@@ -0,0 +1,24 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
"@lerobot/web": minor
|
| 3 |
+
---
|
| 4 |
+
|
| 5 |
+
enhance record() api with flexible runtime management and video stream support.
|
| 6 |
+
|
| 7 |
+
add videoStreams and robotType to RecordConfig for upfront configuration.
|
| 8 |
+
expose episode management methods on RecordProcess:
|
| 9 |
+
- getEpisodeCount() / getEpisodes() - introspect recorded episodes
|
| 10 |
+
- clearEpisodes() - delete all episodes
|
| 11 |
+
- nextEpisode() - create new episode segment
|
| 12 |
+
- restoreEpisodes() - restore persisted episodes
|
| 13 |
+
|
| 14 |
+
expose dynamic camera management methods:
|
| 15 |
+
- addCamera(name, stream) - add camera during recording
|
| 16 |
+
- removeCamera(name) - remove camera
|
| 17 |
+
|
| 18 |
+
all functionality previously requiring direct LeRobotDatasetRecorder access
|
| 19 |
+
now available through clean, type-safe RecordProcess interface. supports both
|
| 20 |
+
upfront configuration and runtime operations for maximum flexibility.
|
| 21 |
+
|
| 22 |
+
update demo to use unified record() api exclusively, removing direct recorder
|
| 23 |
+
access and supporting all advanced features (episode management, dynamic cameras,
|
| 24 |
+
custom metadata) through consistent api surface.
|
|
@@ -441,6 +441,26 @@ const response = await new Promise((resolve, reject) => {
|
|
| 441 |
}
|
| 442 |
```
|
| 443 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 444 |
#### Web Implementation Blockers Solved
|
| 445 |
|
| 446 |
**These blockers were identified during SO-100 web calibration development:**
|
|
|
|
| 441 |
}
|
| 442 |
```
|
| 443 |
|
| 444 |
+
##### 6. Media Recording Workflow (Web Cameras)
|
| 445 |
+
|
| 446 |
+
**Required order of operations for video export to work correctly:**
|
| 447 |
+
|
| 448 |
+
- Attach camera streams before recording: call `recorder.addVideoStream(key, stream)` on the same `LeRobotDatasetRecorder` instance before `startRecording()`.
|
| 449 |
+
- Do not add/replace streams while recording: changing streams during an active recording is unsupported and will throw.
|
| 450 |
+
- Stop before export: always stop recording before calling any export method so `MediaRecorder` can finalize the video blob.
|
| 451 |
+
|
| 452 |
+
**Implementation notes:**
|
| 453 |
+
|
| 454 |
+
- Use a separate preview stream and clone for recording (e.g., `const recordingStream = previewStream.clone()`) to avoid preview interruptions.
|
| 455 |
+
- If your UI allows configuring cameras before enabling control, attach those streams to the recorder immediately after the recorder is created.
|
| 456 |
+
- The recorder selects a supported container/mime per browser; exported files are placed under `videos/chunk-000/observation.images.<cameraName>/episode_000000.<ext>`.
|
| 457 |
+
|
| 458 |
+
**Anti-patterns (avoid):**
|
| 459 |
+
|
| 460 |
+
- Creating the recorder, starting recording, then adding cameras.
|
| 461 |
+
- Mutating `videoStreams` during recording.
|
| 462 |
+
- Exporting while recording is still active.
|
| 463 |
+
|
| 464 |
#### Web Implementation Blockers Solved
|
| 465 |
|
| 466 |
**These blockers were identified during SO-100 web calibration development:**
|
|
@@ -0,0 +1,209 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# LeRobot Dataset Format v2.1 — Complete Spec
|
| 2 |
+
|
| 3 |
+
This document describes the **exact structure and syntax** of a dataset repository following the **LeRobot 2.1 standard**.
|
| 4 |
+
It is intended as an implementation guide — follow it exactly to ensure 100% compatibility with the dataset viewer and `LeRobotDataset` loader.
|
| 5 |
+
|
| 6 |
+
---
|
| 7 |
+
|
| 8 |
+
## 1) Repository Layout (typical)
|
| 9 |
+
|
| 10 |
+
```
|
| 11 |
+
<dataset-root>/
|
| 12 |
+
├─ data/
|
| 13 |
+
│ ├─ chunk-000/
|
| 14 |
+
│ │ ├─ episode_000000.parquet
|
| 15 |
+
│ │ └─ episode_000001.parquet
|
| 16 |
+
│ └─ chunk-001/
|
| 17 |
+
│ └─ ...
|
| 18 |
+
├─ videos/
|
| 19 |
+
│ ├─ chunk-000/
|
| 20 |
+
│ │ ├─ observation.images.front/
|
| 21 |
+
│ │ │ ├─ episode_000000.mp4
|
| 22 |
+
│ │ │ └─ episode_000001.mp4
|
| 23 |
+
│ │ └─ observation.images.wrist/
|
| 24 |
+
│ │ ├─ episode_000000.mp4
|
| 25 |
+
│ │ └─ episode_000001.mp4
|
| 26 |
+
│ └─ chunk-001/
|
| 27 |
+
│ └─ ...
|
| 28 |
+
├─ meta/
|
| 29 |
+
│ ├─ info.json
|
| 30 |
+
│ ├─ episodes.jsonl
|
| 31 |
+
│ ├─ tasks.jsonl
|
| 32 |
+
│ └─ episodes_stats.jsonl # (v2.1); older v2.0 used stats.json
|
| 33 |
+
└─ README.md
|
| 34 |
+
```
|
| 35 |
+
|
| 36 |
+
---
|
| 37 |
+
|
| 38 |
+
## 2) `data/`
|
| 39 |
+
|
| 40 |
+
- Contains **one Parquet file per episode**.
|
| 41 |
+
- Path pattern (from `info.json.data_path`):
|
| 42 |
+
```
|
| 43 |
+
data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet
|
| 44 |
+
```
|
| 45 |
+
|
| 46 |
+
Each Parquet file stores synchronized timesteps with:
|
| 47 |
+
|
| 48 |
+
- `timestamp`
|
| 49 |
+
- `observations` (sensor values, images, poses, etc.)
|
| 50 |
+
- `actions` (target joint commands, end-effector states, etc.)
|
| 51 |
+
|
| 52 |
+
---
|
| 53 |
+
|
| 54 |
+
## 3) `videos/`
|
| 55 |
+
|
| 56 |
+
- Contains **one MP4 video per episode per camera key**.
|
| 57 |
+
- Path pattern (from `info.json.video_path`):
|
| 58 |
+
```
|
| 59 |
+
videos/chunk-{episode_chunk:03d}/{video_key}/episode_{episode_index:06d}.mp4
|
| 60 |
+
```
|
| 61 |
+
- `{video_key}` MUST exactly match a feature key in `info.json.features` under `observation.images.*`.
|
| 62 |
+
|
| 63 |
+
**Requirements:**
|
| 64 |
+
|
| 65 |
+
- Codec: H.264 (`avc1`)
|
| 66 |
+
- FPS matches `info.json.fps`
|
| 67 |
+
- Frame count == number of timesteps in corresponding Parquet
|
| 68 |
+
|
| 69 |
+
---
|
| 70 |
+
|
| 71 |
+
## 4) `meta/`
|
| 72 |
+
|
| 73 |
+
### `info.json`
|
| 74 |
+
|
| 75 |
+
Top-level dataset metadata and templates.
|
| 76 |
+
|
| 77 |
+
```json
|
| 78 |
+
{
|
| 79 |
+
"version": "2.1",
|
| 80 |
+
"fps": 30,
|
| 81 |
+
"chunks_size": 1000,
|
| 82 |
+
"data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
|
| 83 |
+
"video_path": "videos/chunk-{episode_chunk:03d}/{video_key}/episode_{episode_index:06d}.mp4",
|
| 84 |
+
"features": {
|
| 85 |
+
"observation.images.front": {
|
| 86 |
+
"dtype": "video",
|
| 87 |
+
"shape": [240, 320, 3],
|
| 88 |
+
"names": ["height", "width", "channel"],
|
| 89 |
+
"video_info": { "video.fps": 30.0, "video.codec": "avc1" }
|
| 90 |
+
}
|
| 91 |
+
}
|
| 92 |
+
}
|
| 93 |
+
```
|
| 94 |
+
|
| 95 |
+
---
|
| 96 |
+
|
| 97 |
+
### `episodes.jsonl`
|
| 98 |
+
|
| 99 |
+
Line-delimited JSON, one entry per episode.
|
| 100 |
+
|
| 101 |
+
```json
|
| 102 |
+
{"episode_id":"000000","task":"pick_and_place","length":243,"chunk":0}
|
| 103 |
+
{"episode_id":"000001","task":"pick_and_place","length":198,"chunk":0}
|
| 104 |
+
```
|
| 105 |
+
|
| 106 |
+
Fields:
|
| 107 |
+
|
| 108 |
+
- `episode_id` → must match filenames (`episode_000000`)
|
| 109 |
+
- `task` → must exist in `tasks.jsonl`
|
| 110 |
+
- `length` → number of timesteps (and frames)
|
| 111 |
+
- `chunk` → numeric index (0,1,2…) matching folder
|
| 112 |
+
|
| 113 |
+
---
|
| 114 |
+
|
| 115 |
+
### `tasks.jsonl`
|
| 116 |
+
|
| 117 |
+
Defines available tasks.
|
| 118 |
+
|
| 119 |
+
```json
|
| 120 |
+
{"task":"pick_and_place","description":"Pick an object and place it"}
|
| 121 |
+
{"task":"push_button","description":"Push the button until LED lights up"}
|
| 122 |
+
```
|
| 123 |
+
|
| 124 |
+
---
|
| 125 |
+
|
| 126 |
+
### `episodes_stats.jsonl`
|
| 127 |
+
|
| 128 |
+
Dataset statistics for analysis & normalization.
|
| 129 |
+
|
| 130 |
+
```json
|
| 131 |
+
{
|
| 132 |
+
"total_frames": 25873,
|
| 133 |
+
"episode_lengths": { "min": 120, "max": 450, "mean": 215 },
|
| 134 |
+
"action_stats": {
|
| 135 |
+
"x": { "min": -0.12, "max": 0.12, "mean": 0.0, "std": 0.05 },
|
| 136 |
+
"y": { "min": -0.1, "max": 0.1, "mean": 0.0, "std": 0.04 },
|
| 137 |
+
"z": { "min": 0.05, "max": 0.35, "mean": 0.2, "std": 0.06 }
|
| 138 |
+
}
|
| 139 |
+
}
|
| 140 |
+
```
|
| 141 |
+
|
| 142 |
+
---
|
| 143 |
+
|
| 144 |
+
## 5) Chunking explained
|
| 145 |
+
|
| 146 |
+
- **Purpose:** purely structural; keeps folders manageable and enables fast lookups.
|
| 147 |
+
- **Decided by dataset builder:** choose `chunks_size` (often 1000).
|
| 148 |
+
- **Rule:**
|
| 149 |
+
```
|
| 150 |
+
episode_chunk = episode_index // chunks_size
|
| 151 |
+
```
|
| 152 |
+
|
| 153 |
+
### Examples
|
| 154 |
+
|
| 155 |
+
- Small dataset (N ≤ 1000, chunks_size=1000) → all in `chunk-000`
|
| 156 |
+
- Large dataset (N=2000, chunks_size=1000):
|
| 157 |
+
- episodes 0–999 → `chunk-000`
|
| 158 |
+
- episodes 1000–1999 → `chunk-001`
|
| 159 |
+
|
| 160 |
+
### Naming
|
| 161 |
+
|
| 162 |
+
- Chunk folder must be `chunk-XXX` with zero padding.
|
| 163 |
+
|
| 164 |
+
---
|
| 165 |
+
|
| 166 |
+
## 6) Synchronization rules
|
| 167 |
+
|
| 168 |
+
- Parquet rows = MP4 frames = `episodes.jsonl.length`
|
| 169 |
+
- Episode IDs consistent across parquet, mp4, and JSONL
|
| 170 |
+
- Video folder name = exact feature key (`observation.images.front`)
|
| 171 |
+
- Codecs = H.264 (`avc1`)
|
| 172 |
+
- Forward slashes in all paths
|
| 173 |
+
|
| 174 |
+
---
|
| 175 |
+
|
| 176 |
+
## 7) Common pitfalls
|
| 177 |
+
|
| 178 |
+
1. Wrong folder naming (`front/` instead of `observation.images.front/`)
|
| 179 |
+
2. Skipping `chunk-000` entirely (must exist, even for small datasets)
|
| 180 |
+
3. Filename mismatch (`000001.mp4` vs `episode_000001.mp4`)
|
| 181 |
+
4. Nested JSON instead of JSONL
|
| 182 |
+
5. Using unsupported codec (HEVC/AV1)
|
| 183 |
+
|
| 184 |
+
---
|
| 185 |
+
|
| 186 |
+
## 8) Quick implementation checklist
|
| 187 |
+
|
| 188 |
+
- [ ] `info.json.version == "2.1"`
|
| 189 |
+
- [ ] `chunks_size` defined (e.g., 1000)
|
| 190 |
+
- [ ] `data_path` and `video_path` templates correct
|
| 191 |
+
- [ ] All `video_key` entries match `features` keys under `observation.images.*`
|
| 192 |
+
- [ ] Episode IDs zero‑padded: `episode_{:06d}`
|
| 193 |
+
- [ ] `episodes.jsonl` one JSON per line
|
| 194 |
+
- [ ] Parquet rows == MP4 frames == `length`
|
| 195 |
+
- [ ] Codec = H.264 (`avc1`), FPS correct
|
| 196 |
+
- [ ] All paths use `/` slashes
|
| 197 |
+
|
| 198 |
+
---
|
| 199 |
+
|
| 200 |
+
## 9) Minimal valid dataset (≤1000 episodes)
|
| 201 |
+
|
| 202 |
+
```
|
| 203 |
+
my_dataset/
|
| 204 |
+
├─ data/chunk-000/episode_000000.parquet
|
| 205 |
+
├─ data/chunk-000/episode_000001.parquet
|
| 206 |
+
├─ videos/chunk-000/observation.images.front/episode_000000.mp4
|
| 207 |
+
├─ videos/chunk-000/observation.images.front/episode_000001.mp4
|
| 208 |
+
└─ meta/{info.json, episodes.jsonl, tasks.jsonl, episodes_stats.jsonl}
|
| 209 |
+
```
|
|
@@ -4,76 +4,126 @@
|
|
| 4 |
|
| 5 |
**As a** robotics developer building teleoperation recording systems
|
| 6 |
**I want** to record robot motor positions and control data using a clean `record()` function API
|
| 7 |
-
**So that** I can capture teleoperation sessions for training AI models, analysis, and replay
|
| 8 |
|
| 9 |
## Background
|
| 10 |
|
| 11 |
-
A community contributor has
|
| 12 |
|
| 13 |
-
### Current Implementation
|
| 14 |
|
| 15 |
-
The existing
|
| 16 |
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
|
| 26 |
### Convention Alignment Needed
|
| 27 |
|
| 28 |
Our established patterns from `calibrate()`, `teleoperate()`, and `findPort()` follow these principles:
|
| 29 |
|
| 30 |
-
- **Simple Function API**: `const process = await record(config)`
|
| 31 |
-
- **Clean Process Objects**: Consistent `start()`, `stop()`, `getState()`, `result` interface
|
| 32 |
-
- **Hardware-Only Library**: Standard library handles only robotics hardware
|
| 33 |
- **Demo Handles UI**: Examples handle video, export formats, browser storage, file downloads
|
| 34 |
-
- **Immediate Callbacks**: Real-time updates via callbacks, not polling or unused events
|
| 35 |
- **Direct Usage**: End users call library functions directly without complex setup
|
| 36 |
|
| 37 |
## Acceptance Criteria
|
| 38 |
|
| 39 |
### Core Functionality
|
| 40 |
|
| 41 |
-
- [ ] **Standard Library API**: Clean `record(config)` function matching our established patterns
|
| 42 |
-
- [ ] **Process Object Interface**: Consistent `RecordProcess` with `start()`, `stop()`, `getState()`, `result` methods
|
| 43 |
-
- [ ] **Hardware-Only Recording**: Library captures only robot motor positions and teleoperation data
|
| 44 |
-
- [ ] **
|
| 45 |
-
- [ ] **
|
| 46 |
-
- [ ] **Clean Teleoperator Integration**: Recording subscribes to teleoperation changes without embedding in teleoperator classes
|
| 47 |
|
| 48 |
### User Experience
|
| 49 |
|
| 50 |
- [ ] **Simple Integration**: Easy to add recording to existing teleoperation workflows
|
| 51 |
- [ ] **Consistent API**: Same patterns as `calibrate()` and `teleoperate()` for familiar developer experience
|
| 52 |
-
- [ ] **
|
| 53 |
- [ ] **Error Handling**: Clear error messages for recording failures or invalid configurations
|
| 54 |
- [ ] **Resource Management**: Proper cleanup of recording resources on stop/disconnect
|
| 55 |
|
| 56 |
### Technical Requirements
|
| 57 |
|
| 58 |
-
- [ ] **Library/Demo Separation**: Move video
|
| 59 |
-
- [ ] **
|
| 60 |
-
- [ ] **
|
| 61 |
- [ ] **TypeScript**: Fully typed with proper interfaces for recording configuration and data
|
| 62 |
-
- [ ] **No
|
| 63 |
-
- [ ] **Performance**: Immediate callbacks when data changes, no unnecessary polling
|
| 64 |
|
| 65 |
## Expected User Flow
|
| 66 |
|
| 67 |
-
### Basic Robot Recording
|
| 68 |
|
| 69 |
```typescript
|
| 70 |
-
import { record } from "@lerobot/web";
|
| 71 |
|
| 72 |
-
//
|
| 73 |
-
const
|
| 74 |
robot: connectedRobot,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 75 |
options: {
|
| 76 |
fps: 30,
|
|
|
|
| 77 |
onDataUpdate: (data) => {
|
| 78 |
// Real-time recording data for UI feedback
|
| 79 |
console.log(`Recorded ${data.frameCount} frames`);
|
|
@@ -87,26 +137,57 @@ const recordProcess = await record({
|
|
| 87 |
},
|
| 88 |
});
|
| 89 |
|
| 90 |
-
//
|
|
|
|
| 91 |
recordProcess.start();
|
| 92 |
|
| 93 |
-
// Recording
|
| 94 |
setTimeout(() => {
|
| 95 |
recordProcess.stop();
|
| 96 |
}, 30000);
|
| 97 |
|
| 98 |
-
// Get pure robot recording data
|
| 99 |
const robotData = await recordProcess.result;
|
| 100 |
console.log("Episodes:", robotData.episodes);
|
| 101 |
console.log("Metadata:", robotData.metadata);
|
| 102 |
```
|
| 103 |
|
| 104 |
-
###
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 105 |
|
| 106 |
```typescript
|
| 107 |
import { teleoperate, record } from "@lerobot/web";
|
| 108 |
|
| 109 |
-
// Start teleoperation
|
| 110 |
const teleoperationProcess = await teleoperate({
|
| 111 |
robot: connectedRobot,
|
| 112 |
teleop: { type: "keyboard" },
|
|
@@ -116,21 +197,23 @@ const teleoperationProcess = await teleoperate({
|
|
| 116 |
},
|
| 117 |
});
|
| 118 |
|
| 119 |
-
// Add recording
|
| 120 |
const recordProcess = await record({
|
| 121 |
-
|
| 122 |
options: {
|
|
|
|
|
|
|
| 123 |
onDataUpdate: (data) => {
|
| 124 |
console.log(`Recording frame ${data.frameCount}`);
|
| 125 |
},
|
| 126 |
},
|
| 127 |
});
|
| 128 |
|
| 129 |
-
// Both run independently
|
| 130 |
teleoperationProcess.start();
|
| 131 |
recordProcess.start();
|
| 132 |
|
| 133 |
-
// Control independently
|
| 134 |
setTimeout(() => {
|
| 135 |
recordProcess.stop(); // Stop recording, keep teleoperation
|
| 136 |
}, 60000);
|
|
@@ -140,12 +223,19 @@ setTimeout(() => {
|
|
| 140 |
}, 120000);
|
| 141 |
```
|
| 142 |
|
| 143 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 144 |
|
| 145 |
```typescript
|
| 146 |
// In examples/demo - NOT in standard library
|
| 147 |
import { record } from "@lerobot/web";
|
| 148 |
-
import { DatasetExporter } from "./dataset-exporter"; //
|
| 149 |
|
| 150 |
const recordProcess = await record({ robot, options });
|
| 151 |
recordProcess.start();
|
|
@@ -153,21 +243,35 @@ recordProcess.start();
|
|
| 153 |
// ... recording session ...
|
| 154 |
|
| 155 |
recordProcess.stop();
|
| 156 |
-
const robotData = await recordProcess.result;
|
| 157 |
|
| 158 |
-
//
|
| 159 |
const exporter = new DatasetExporter({
|
| 160 |
robotData,
|
| 161 |
videoStreams: cameraStreams, // Demo manages video
|
| 162 |
taskDescription: "Pick and place task",
|
| 163 |
});
|
| 164 |
|
| 165 |
-
//
|
| 166 |
await exporter.downloadZip();
|
| 167 |
await exporter.uploadToHuggingFace({ apiKey, repoName });
|
| 168 |
await exporter.uploadToS3({ credentials });
|
| 169 |
```
|
| 170 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 171 |
### Component Integration
|
| 172 |
|
| 173 |
```typescript
|
|
@@ -203,45 +307,61 @@ const handleStopRecording = async () => {
|
|
| 203 |
|
| 204 |
## Implementation Details
|
| 205 |
|
| 206 |
-
### File Structure
|
| 207 |
|
| 208 |
```
|
| 209 |
packages/web/src/
|
| 210 |
-
├── record.ts #
|
|
|
|
| 211 |
├── types/
|
| 212 |
-
│ └── recording.ts # NEW: Recording-specific types
|
| 213 |
-
├── utils/
|
| 214 |
-
│ └── recording-manager.ts # NEW: Internal recording logic
|
| 215 |
├── teleoperators/
|
| 216 |
-
│ └── base-teleoperator.ts #
|
| 217 |
-
└── [
|
| 218 |
-
├──
|
| 219 |
-
├──
|
| 220 |
-
└──
|
| 221 |
```
|
| 222 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 223 |
### Key Dependencies
|
| 224 |
|
| 225 |
-
####
|
| 226 |
|
| 227 |
-
- **Existing**:
|
| 228 |
-
- **
|
| 229 |
|
| 230 |
-
#### Demo Dependencies (Moved)
|
| 231 |
|
| 232 |
-
- **Video/Export**: `parquet-wasm`, `apache-arrow`, `jszip` -
|
| 233 |
-
- **Upload**: `@huggingface/hub`, AWS SDK -
|
|
|
|
| 234 |
|
| 235 |
### Core Functions to Implement
|
| 236 |
|
| 237 |
-
####
|
| 238 |
|
| 239 |
```typescript
|
| 240 |
-
// record.ts -
|
| 241 |
interface RecordConfig {
|
| 242 |
-
|
| 243 |
options?: {
|
| 244 |
fps?: number; // Default: 30
|
|
|
|
| 245 |
onDataUpdate?: (data: RecordingData) => void;
|
| 246 |
onStateUpdate?: (state: RecordingState) => void;
|
| 247 |
};
|
|
@@ -265,11 +385,11 @@ interface RecordingState {
|
|
| 265 |
interface RecordingData {
|
| 266 |
frameCount: number;
|
| 267 |
currentEpisode: number;
|
| 268 |
-
recentFrames:
|
| 269 |
}
|
| 270 |
|
| 271 |
interface RobotRecordingData {
|
| 272 |
-
episodes:
|
| 273 |
metadata: {
|
| 274 |
fps: number;
|
| 275 |
robotType: string;
|
|
@@ -280,106 +400,80 @@ interface RobotRecordingData {
|
|
| 280 |
};
|
| 281 |
}
|
| 282 |
|
| 283 |
-
//
|
|
|
|
| 284 |
export async function record(config: RecordConfig): Promise<RecordProcess>;
|
| 285 |
```
|
| 286 |
|
| 287 |
-
####
|
| 288 |
-
|
| 289 |
-
```typescript
|
| 290 |
-
// utils/recording-manager.ts - Internal implementation
|
| 291 |
-
class RecordingManager {
|
| 292 |
-
private robot: RobotConnection;
|
| 293 |
-
private isActive: boolean = false;
|
| 294 |
-
private episodes: MotorPositionFrame[][] = [];
|
| 295 |
-
private currentEpisode: MotorPositionFrame[] = [];
|
| 296 |
-
private startTime: number = 0;
|
| 297 |
-
private frameCount: number = 0;
|
| 298 |
-
|
| 299 |
-
constructor(
|
| 300 |
-
robot: RobotConnection,
|
| 301 |
-
private options: RecordOptions,
|
| 302 |
-
private onDataUpdate?: (data: RecordingData) => void,
|
| 303 |
-
private onStateUpdate?: (state: RecordingState) => void
|
| 304 |
-
) {
|
| 305 |
-
this.robot = robot;
|
| 306 |
-
}
|
| 307 |
-
|
| 308 |
-
start(): void {
|
| 309 |
-
if (this.isActive) return;
|
| 310 |
|
| 311 |
-
|
| 312 |
-
this.startTime = Date.now();
|
| 313 |
|
| 314 |
-
|
| 315 |
-
|
| 316 |
-
|
| 317 |
-
|
| 318 |
-
|
| 319 |
-
|
| 320 |
-
|
| 321 |
-
|
| 322 |
-
|
| 323 |
-
|
| 324 |
-
|
| 325 |
-
|
| 326 |
-
|
| 327 |
-
this.notifyStateUpdate();
|
| 328 |
-
}
|
| 329 |
-
|
| 330 |
-
private subscribeToRobotChanges(): void {
|
| 331 |
-
// Listen to existing teleoperation callbacks - no new events needed
|
| 332 |
-
// This integrates with the existing onStateUpdate mechanism
|
| 333 |
-
}
|
| 334 |
-
|
| 335 |
-
private recordFrame(motorConfigs: MotorConfig[]): void {
|
| 336 |
-
const frame: MotorPositionFrame = {
|
| 337 |
-
timestamp: Date.now() - this.startTime,
|
| 338 |
-
motorPositions: motorConfigs.map((config) => ({
|
| 339 |
-
id: config.id,
|
| 340 |
-
name: config.name,
|
| 341 |
-
position: config.currentPosition,
|
| 342 |
-
})),
|
| 343 |
-
frameIndex: this.frameCount++,
|
| 344 |
-
};
|
| 345 |
-
|
| 346 |
-
this.currentEpisode.push(frame);
|
| 347 |
-
|
| 348 |
-
if (this.onDataUpdate) {
|
| 349 |
-
this.onDataUpdate({
|
| 350 |
-
frameCount: this.frameCount,
|
| 351 |
-
currentEpisode: this.episodes.length,
|
| 352 |
-
recentFrames: this.currentEpisode.slice(-10), // Last 10 frames
|
| 353 |
-
});
|
| 354 |
-
}
|
| 355 |
-
}
|
| 356 |
|
| 357 |
-
|
| 358 |
-
|
| 359 |
-
|
| 360 |
-
|
| 361 |
-
|
| 362 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 363 |
lastUpdate: Date.now(),
|
| 364 |
-
}
|
| 365 |
-
|
| 366 |
-
|
| 367 |
-
async getResult(): Promise<RobotRecordingData> {
|
| 368 |
-
return {
|
| 369 |
-
episodes: [...this.episodes],
|
| 370 |
metadata: {
|
| 371 |
-
fps:
|
| 372 |
-
robotType:
|
| 373 |
-
startTime:
|
| 374 |
endTime: Date.now(),
|
| 375 |
-
totalFrames:
|
| 376 |
-
totalEpisodes:
|
| 377 |
},
|
| 378 |
-
}
|
| 379 |
-
}
|
| 380 |
}
|
| 381 |
```
|
| 382 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 383 |
#### Updated Teleoperate Integration
|
| 384 |
|
| 385 |
```typescript
|
|
@@ -468,16 +562,29 @@ class AdvancedDatasetExporter extends DatasetExporter {
|
|
| 468 |
|
| 469 |
## Definition of Done
|
| 470 |
|
| 471 |
-
|
|
|
|
|
|
|
| 472 |
- [ ] **Process Interface**: `RecordProcess` with consistent `start()`, `stop()`, `getState()`, `result` methods
|
| 473 |
-
- [ ] **Hardware-Only
|
| 474 |
-
- [ ] **
|
| 475 |
-
- [ ] **
|
| 476 |
-
|
| 477 |
-
|
| 478 |
-
|
| 479 |
-
- [ ] **
|
| 480 |
-
- [ ] **
|
| 481 |
-
- [ ] **
|
| 482 |
-
|
| 483 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4 |
|
| 5 |
**As a** robotics developer building teleoperation recording systems
|
| 6 |
**I want** to record robot motor positions and control data using a clean `record()` function API
|
| 7 |
+
**So that** I can capture teleoperation sessions for training AI models, analysis, and replay with the same simple patterns as other LeRobot.js functions
|
| 8 |
|
| 9 |
## Background
|
| 10 |
|
| 11 |
+
A community contributor has successfully implemented comprehensive recording functionality, including a `LeRobotDatasetRecorder` class with video recording, data export, and LeRobot dataset format support. The implementation is functional and well-integrated, but doesn't follow our established simple function API patterns from `calibrate()`, `teleoperate()`, and `findPort()`.
|
| 12 |
|
| 13 |
+
### Current Recording Implementation (From README)
|
| 14 |
|
| 15 |
+
The existing system works as documented in the web package README:
|
| 16 |
|
| 17 |
+
```typescript
|
| 18 |
+
import { LeRobotDatasetRecorder } from "@lerobot/web";
|
| 19 |
+
|
| 20 |
+
// Create a recorder with teleoperator and video streams
|
| 21 |
+
const recorder = new LeRobotDatasetRecorder(
|
| 22 |
+
[teleoperator], // Array of teleoperators to record
|
| 23 |
+
{ main: videoStream }, // Video streams by camera key
|
| 24 |
+
30, // Target FPS
|
| 25 |
+
"Pick and place task" // Task description
|
| 26 |
+
);
|
| 27 |
+
|
| 28 |
+
// Start recording
|
| 29 |
+
await recorder.startRecording();
|
| 30 |
+
// ... robot performs task ...
|
| 31 |
+
const recordingData = await recorder.stopRecording();
|
| 32 |
+
|
| 33 |
+
// Export the dataset in various formats
|
| 34 |
+
await recorder.exportForLeRobot("zip-download");
|
| 35 |
+
await recorder.exportForLeRobot("huggingface", { repoName, accessToken });
|
| 36 |
+
await recorder.exportForLeRobot("s3", { bucketName, credentials });
|
| 37 |
+
```
|
| 38 |
+
|
| 39 |
+
This implementation has **excellent architecture** - the explicit teleoperator dependency makes it clear, testable, and flexible.
|
| 40 |
+
|
| 41 |
+
### Current Implementation Status
|
| 42 |
+
|
| 43 |
+
The existing recording system is **fully functional** with these components:
|
| 44 |
+
|
| 45 |
+
✅ **Already Working Well:**
|
| 46 |
+
|
| 47 |
+
- `LeRobotDatasetRecorder` class with complete functionality
|
| 48 |
+
- Proper callback-based integration with teleoperators (no polling issues)
|
| 49 |
+
- Full LeRobot dataset format support with Parquet export
|
| 50 |
+
- Video recording and synchronization capabilities
|
| 51 |
+
- Complete cyberpunk example integration with camera management
|
| 52 |
+
- Clean separation between recording logic and teleoperator classes
|
| 53 |
+
- Export to ZIP, Hugging Face, and S3
|
| 54 |
+
|
| 55 |
+
### What Works Well (Keep This)
|
| 56 |
+
|
| 57 |
+
The current implementation has **excellent architectural decisions**:
|
| 58 |
+
|
| 59 |
+
- **Explicit Teleoperator Dependency**: `LeRobotDatasetRecorder([teleoperator], ...)` makes dependencies clear and predictable
|
| 60 |
+
- **Clean Separation**: Recording subscribes to teleoperator callbacks without tight coupling
|
| 61 |
+
- **Flexible Architecture**: Can record from any teleoperator, multiple teleoperators, or no teleoperator at all
|
| 62 |
+
|
| 63 |
+
### Areas for Improvement
|
| 64 |
+
|
| 65 |
+
The only issues are **API consistency** and **UI organization**:
|
| 66 |
+
|
| 67 |
+
- **Missing Simple Function API**: Users must instantiate `LeRobotDatasetRecorder` class directly instead of calling `record()` like other functions
|
| 68 |
+
- **UI Integration Pattern**: Recording is embedded within teleoperation view instead of being its own separate component/page
|
| 69 |
+
- **Library vs Demo Boundary**: Complex export functionality (video processing, cloud uploads) should be in demo layer, not standard library
|
| 70 |
|
| 71 |
### Convention Alignment Needed
|
| 72 |
|
| 73 |
Our established patterns from `calibrate()`, `teleoperate()`, and `findPort()` follow these principles:
|
| 74 |
|
| 75 |
+
- **Simple Function API**: `const process = await record(config)` (currently requires class instantiation)
|
| 76 |
+
- **Clean Process Objects**: Consistent `start()`, `stop()`, `getState()`, `result` interface (class has `startRecording()`, `stopRecording()`)
|
| 77 |
+
- **Hardware-Only Library**: Standard library handles only robotics hardware (currently includes video/export)
|
| 78 |
- **Demo Handles UI**: Examples handle video, export formats, browser storage, file downloads
|
|
|
|
| 79 |
- **Direct Usage**: End users call library functions directly without complex setup
|
| 80 |
|
| 81 |
## Acceptance Criteria
|
| 82 |
|
| 83 |
### Core Functionality
|
| 84 |
|
| 85 |
+
- [ ] **Standard Library API**: Clean `record(config)` function matching our established patterns (wrap existing `LeRobotDatasetRecorder`)
|
| 86 |
+
- [ ] **Process Object Interface**: Consistent `RecordProcess` with `start()`, `stop()`, `getState()`, `result` methods (adapt existing methods)
|
| 87 |
+
- [ ] **Hardware-Only Recording**: Library captures only robot motor positions and teleoperation data (move video/export to demo)
|
| 88 |
+
- [ ] **Clean Teleoperator Integration**: Recording uses existing callback system in `BaseWebTeleoperator`
|
| 89 |
+
- [ ] **Preserve Advanced Features**: Keep `LeRobotDatasetRecorder` class for users who need full control
|
|
|
|
| 90 |
|
| 91 |
### User Experience
|
| 92 |
|
| 93 |
- [ ] **Simple Integration**: Easy to add recording to existing teleoperation workflows
|
| 94 |
- [ ] **Consistent API**: Same patterns as `calibrate()` and `teleoperate()` for familiar developer experience
|
| 95 |
+
- [ ] **Separate UI Component**: Move recording from teleoperation view to its own dedicated page/section in cyberpunk example
|
| 96 |
- [ ] **Error Handling**: Clear error messages for recording failures or invalid configurations
|
| 97 |
- [ ] **Resource Management**: Proper cleanup of recording resources on stop/disconnect
|
| 98 |
|
| 99 |
### Technical Requirements
|
| 100 |
|
| 101 |
+
- [ ] **Library/Demo Separation**: Move video recording, complex export logic (HF, S3) to examples/demo layer
|
| 102 |
+
- [ ] **Wrapper Function**: Create simple `record()` function that wraps existing `LeRobotDatasetRecorder`
|
| 103 |
+
- [ ] **Preserve Existing Integration**: Keep current callback system in `BaseWebTeleoperator` (it works well)
|
| 104 |
- [ ] **TypeScript**: Fully typed with proper interfaces for recording configuration and data
|
| 105 |
+
- [ ] **No Breaking Changes**: Existing `LeRobotDatasetRecorder` class should remain available for advanced users
|
|
|
|
| 106 |
|
| 107 |
## Expected User Flow
|
| 108 |
|
| 109 |
+
### Basic Robot Recording (Proposed Simple API)
|
| 110 |
|
| 111 |
```typescript
|
| 112 |
+
import { teleoperate, record } from "@lerobot/web";
|
| 113 |
|
| 114 |
+
// 1. Create teleoperation first (existing pattern)
|
| 115 |
+
const teleoperationProcess = await teleoperate({
|
| 116 |
robot: connectedRobot,
|
| 117 |
+
teleop: { type: "keyboard" },
|
| 118 |
+
calibrationData: calibrationData,
|
| 119 |
+
});
|
| 120 |
+
|
| 121 |
+
// 2. NEW: Clean API with explicit teleoperator dependency
|
| 122 |
+
const recordProcess = await record({
|
| 123 |
+
teleoperator: teleoperationProcess.teleoperator, // ← Explicit dependency
|
| 124 |
options: {
|
| 125 |
fps: 30,
|
| 126 |
+
taskDescription: "Pick and place task",
|
| 127 |
onDataUpdate: (data) => {
|
| 128 |
// Real-time recording data for UI feedback
|
| 129 |
console.log(`Recorded ${data.frameCount} frames`);
|
|
|
|
| 137 |
},
|
| 138 |
});
|
| 139 |
|
| 140 |
+
// 3. Start both processes
|
| 141 |
+
teleoperationProcess.start();
|
| 142 |
recordProcess.start();
|
| 143 |
|
| 144 |
+
// 4. Recording captures teleoperation automatically via callbacks
|
| 145 |
setTimeout(() => {
|
| 146 |
recordProcess.stop();
|
| 147 |
}, 30000);
|
| 148 |
|
| 149 |
+
// 5. Get pure robot recording data (no video/export complexity)
|
| 150 |
const robotData = await recordProcess.result;
|
| 151 |
console.log("Episodes:", robotData.episodes);
|
| 152 |
console.log("Metadata:", robotData.metadata);
|
| 153 |
```
|
| 154 |
|
| 155 |
+
### Current Implementation (Works Well, Just Different API Style)
|
| 156 |
+
|
| 157 |
+
```typescript
|
| 158 |
+
import { LeRobotDatasetRecorder } from "@lerobot/web";
|
| 159 |
+
|
| 160 |
+
// CURRENT: Class-based API with explicit dependencies (good architecture!)
|
| 161 |
+
const recorder = new LeRobotDatasetRecorder(
|
| 162 |
+
[teleoperator], // ← GOOD: Explicit teleoperator dependency
|
| 163 |
+
{ main: videoStream }, // Video complexity in library (to be moved)
|
| 164 |
+
30, // fps
|
| 165 |
+
"Pick and place task" // Task description
|
| 166 |
+
);
|
| 167 |
+
|
| 168 |
+
// Different method names than our conventions (but functional)
|
| 169 |
+
await recorder.startRecording();
|
| 170 |
+
// ... robot performs task ...
|
| 171 |
+
const result = await recorder.stopRecording();
|
| 172 |
+
|
| 173 |
+
// Complex export in standard library (should be demo-only)
|
| 174 |
+
await recorder.exportForLeRobot("zip-download");
|
| 175 |
+
await recorder.exportForLeRobot("huggingface", { repoName, accessToken });
|
| 176 |
+
```
|
| 177 |
+
|
| 178 |
+
**What's Good About Current Implementation:**
|
| 179 |
+
|
| 180 |
+
- ✅ **Explicit Dependencies**: Clear what the recorder needs to work
|
| 181 |
+
- ✅ **Clean Architecture**: Recording subscribes to teleoperator via callbacks
|
| 182 |
+
- ✅ **Full Functionality**: Complete LeRobot dataset format support
|
| 183 |
+
- ✅ **Flexible**: Can record from any teleoperator instance
|
| 184 |
+
|
| 185 |
+
### Recording with Teleoperation (Proposed Simple API)
|
| 186 |
|
| 187 |
```typescript
|
| 188 |
import { teleoperate, record } from "@lerobot/web";
|
| 189 |
|
| 190 |
+
// 1. Start teleoperation (existing pattern)
|
| 191 |
const teleoperationProcess = await teleoperate({
|
| 192 |
robot: connectedRobot,
|
| 193 |
teleop: { type: "keyboard" },
|
|
|
|
| 197 |
},
|
| 198 |
});
|
| 199 |
|
| 200 |
+
// 2. NEW: Add recording with explicit teleoperator dependency
|
| 201 |
const recordProcess = await record({
|
| 202 |
+
teleoperator: teleoperationProcess.teleoperator, // ← Explicit dependency (good!)
|
| 203 |
options: {
|
| 204 |
+
fps: 30,
|
| 205 |
+
taskDescription: "Pick and place task",
|
| 206 |
onDataUpdate: (data) => {
|
| 207 |
console.log(`Recording frame ${data.frameCount}`);
|
| 208 |
},
|
| 209 |
},
|
| 210 |
});
|
| 211 |
|
| 212 |
+
// 3. Both run independently
|
| 213 |
teleoperationProcess.start();
|
| 214 |
recordProcess.start();
|
| 215 |
|
| 216 |
+
// 4. Control independently
|
| 217 |
setTimeout(() => {
|
| 218 |
recordProcess.stop(); // Stop recording, keep teleoperation
|
| 219 |
}, 60000);
|
|
|
|
| 223 |
}, 120000);
|
| 224 |
```
|
| 225 |
|
| 226 |
+
**Why Explicit Teleoperator Dependency is Good:**
|
| 227 |
+
|
| 228 |
+
- 🎯 **Clear**: You know exactly what gets recorded
|
| 229 |
+
- 🔧 **Flexible**: Can record from any teleoperator
|
| 230 |
+
- 🧪 **Testable**: Easy to mock teleoperator for testing
|
| 231 |
+
- 📦 **Reusable**: Same teleoperator can serve multiple recorders
|
| 232 |
+
|
| 233 |
+
### Demo-Layer Dataset Export (Proposed Architecture)
|
| 234 |
|
| 235 |
```typescript
|
| 236 |
// In examples/demo - NOT in standard library
|
| 237 |
import { record } from "@lerobot/web";
|
| 238 |
+
import { DatasetExporter } from "./dataset-exporter"; // MOVE complex logic here
|
| 239 |
|
| 240 |
const recordProcess = await record({ robot, options });
|
| 241 |
recordProcess.start();
|
|
|
|
| 243 |
// ... recording session ...
|
| 244 |
|
| 245 |
recordProcess.stop();
|
| 246 |
+
const robotData = await recordProcess.result; // Pure motor data only
|
| 247 |
|
| 248 |
+
// MOVE TO DEMO: Complex export logic with video/cloud features
|
| 249 |
const exporter = new DatasetExporter({
|
| 250 |
robotData,
|
| 251 |
videoStreams: cameraStreams, // Demo manages video
|
| 252 |
taskDescription: "Pick and place task",
|
| 253 |
});
|
| 254 |
|
| 255 |
+
// MOVE TO DEMO: Export options
|
| 256 |
await exporter.downloadZip();
|
| 257 |
await exporter.uploadToHuggingFace({ apiKey, repoName });
|
| 258 |
await exporter.uploadToS3({ credentials });
|
| 259 |
```
|
| 260 |
|
| 261 |
+
### Current Cyberpunk Example Integration
|
| 262 |
+
|
| 263 |
+
```typescript
|
| 264 |
+
// CURRENT: Recording embedded in teleoperation view
|
| 265 |
+
// examples/cyberpunk-standalone/src/components/teleoperation-view.tsx
|
| 266 |
+
<TeleoperationView robot={robot} />
|
| 267 |
+
// ^ Contains embedded <Recorder /> component
|
| 268 |
+
|
| 269 |
+
// PROPOSED: Separate recording page/component
|
| 270 |
+
// examples/cyberpunk-standalone/src/components/recording-view.tsx
|
| 271 |
+
<RecordingView robot={robot} />
|
| 272 |
+
// ^ Dedicated component with full recording interface
|
| 273 |
+
```
|
| 274 |
+
|
| 275 |
### Component Integration
|
| 276 |
|
| 277 |
```typescript
|
|
|
|
| 307 |
|
| 308 |
## Implementation Details
|
| 309 |
|
| 310 |
+
### File Structure Changes
|
| 311 |
|
| 312 |
```
|
| 313 |
packages/web/src/
|
| 314 |
+
├── record.ts # UPDATE: Add simple record() function wrapper
|
| 315 |
+
├── record-class.ts # RENAME: Move LeRobotDatasetRecorder here
|
| 316 |
├── types/
|
| 317 |
+
│ └── recording.ts # NEW: Recording-specific types for simple API
|
|
|
|
|
|
|
| 318 |
├── teleoperators/
|
| 319 |
+
│ └── base-teleoperator.ts # KEEP: Current callback system works well
|
| 320 |
+
└── [MOVE TO EXAMPLES]
|
| 321 |
+
├── dataset-exporter.ts # Video recording + export functionality
|
| 322 |
+
├── hf_uploader.ts # HuggingFace upload logic
|
| 323 |
+
└── s3_uploader.ts # S3 upload logic
|
| 324 |
```
|
| 325 |
|
| 326 |
+
### Current vs Proposed Architecture
|
| 327 |
+
|
| 328 |
+
**Current (Working):**
|
| 329 |
+
|
| 330 |
+
- `LeRobotDatasetRecorder` class with full functionality
|
| 331 |
+
- Integrated in cyberpunk example within teleoperation view
|
| 332 |
+
- Video, HF, S3 export in standard library
|
| 333 |
+
|
| 334 |
+
**Proposed (Convention-Aligned):**
|
| 335 |
+
|
| 336 |
+
- Simple `record()` function wrapping existing class
|
| 337 |
+
- Separate recording component/page in cyberpunk example
|
| 338 |
+
- Video, HF, S3 export moved to demo layer
|
| 339 |
+
- Keep existing class available for advanced users
|
| 340 |
+
|
| 341 |
### Key Dependencies
|
| 342 |
|
| 343 |
+
#### Standard Library (Minimal Changes)
|
| 344 |
|
| 345 |
+
- **Keep Existing**: All current dependencies for core recording functionality
|
| 346 |
+
- **Wrapper Only**: Simple `record()` function is just a wrapper, no new dependencies
|
| 347 |
|
| 348 |
+
#### Demo Dependencies (To Be Moved)
|
| 349 |
|
| 350 |
+
- **Video/Export**: `parquet-wasm`, `apache-arrow`, `jszip` - move to examples
|
| 351 |
+
- **Upload**: `@huggingface/hub`, AWS SDK - move to examples
|
| 352 |
+
- **Keep Available**: Advanced users can still import `LeRobotDatasetRecorder` for full features
|
| 353 |
|
| 354 |
### Core Functions to Implement
|
| 355 |
|
| 356 |
+
#### Simple Record API (Wrapper)
|
| 357 |
|
| 358 |
```typescript
|
| 359 |
+
// record.ts - Simple wrapper around existing LeRobotDatasetRecorder
|
| 360 |
interface RecordConfig {
|
| 361 |
+
teleoperator: WebTeleoperator; // ← Explicit dependency (keep this!)
|
| 362 |
options?: {
|
| 363 |
fps?: number; // Default: 30
|
| 364 |
+
taskDescription?: string;
|
| 365 |
onDataUpdate?: (data: RecordingData) => void;
|
| 366 |
onStateUpdate?: (state: RecordingState) => void;
|
| 367 |
};
|
|
|
|
| 385 |
interface RecordingData {
|
| 386 |
frameCount: number;
|
| 387 |
currentEpisode: number;
|
| 388 |
+
recentFrames: any[]; // Simplified for basic API
|
| 389 |
}
|
| 390 |
|
| 391 |
interface RobotRecordingData {
|
| 392 |
+
episodes: any[]; // Pure motor data only (no video)
|
| 393 |
metadata: {
|
| 394 |
fps: number;
|
| 395 |
robotType: string;
|
|
|
|
| 400 |
};
|
| 401 |
}
|
| 402 |
|
| 403 |
+
// Simple wrapper function - internally uses LeRobotDatasetRecorder
|
| 404 |
+
// Preserves the excellent explicit dependency architecture
|
| 405 |
export async function record(config: RecordConfig): Promise<RecordProcess>;
|
| 406 |
```
|
| 407 |
|
| 408 |
+
#### Implementation Strategy
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 409 |
|
| 410 |
+
**Phase 1: Simple Wrapper (Preserves Current Architecture)**
|
|
|
|
| 411 |
|
| 412 |
+
```typescript
|
| 413 |
+
// record.ts - Simple wrapper implementation
|
| 414 |
+
import { LeRobotDatasetRecorder } from "./record-class.js";
|
| 415 |
+
|
| 416 |
+
export async function record(config: RecordConfig): Promise<RecordProcess> {
|
| 417 |
+
// Use the provided teleoperator (explicit dependency - good!)
|
| 418 |
+
const recorder = new LeRobotDatasetRecorder(
|
| 419 |
+
[config.teleoperator], // ← Use explicit teleoperator dependency
|
| 420 |
+
{}, // No video streams in simple API (move to demo)
|
| 421 |
+
config.options?.fps || 30,
|
| 422 |
+
config.options?.taskDescription || "Robot recording"
|
| 423 |
+
);
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 424 |
|
| 425 |
+
return {
|
| 426 |
+
start: () => {
|
| 427 |
+
recorder.startRecording();
|
| 428 |
+
if (config.options?.onStateUpdate) {
|
| 429 |
+
// Set up state update polling for simple API
|
| 430 |
+
const updateLoop = () => {
|
| 431 |
+
if (recorder.isRecording) {
|
| 432 |
+
config.options.onStateUpdate!({
|
| 433 |
+
isActive: recorder.isRecording,
|
| 434 |
+
frameCount: recorder.teleoperatorData.length,
|
| 435 |
+
episodeCount: recorder.teleoperatorData.length,
|
| 436 |
+
duration: Date.now() - (recorder as any).startTime,
|
| 437 |
+
lastUpdate: Date.now(),
|
| 438 |
+
});
|
| 439 |
+
setTimeout(updateLoop, 100);
|
| 440 |
+
}
|
| 441 |
+
};
|
| 442 |
+
updateLoop();
|
| 443 |
+
}
|
| 444 |
+
},
|
| 445 |
+
stop: () => {
|
| 446 |
+
return recorder.stopRecording();
|
| 447 |
+
},
|
| 448 |
+
getState: () => ({
|
| 449 |
+
isActive: recorder.isRecording,
|
| 450 |
+
frameCount: recorder.teleoperatorData.length,
|
| 451 |
+
episodeCount: recorder.teleoperatorData.length,
|
| 452 |
+
duration: 0, // Calculate from recorder
|
| 453 |
lastUpdate: Date.now(),
|
| 454 |
+
}),
|
| 455 |
+
result: recorder.stopRecording().then(() => ({
|
| 456 |
+
episodes: recorder.episodes, // Pure motor data
|
|
|
|
|
|
|
|
|
|
| 457 |
metadata: {
|
| 458 |
+
fps: config.options?.fps || 30,
|
| 459 |
+
robotType: "unknown", // Get from teleoperator if possible
|
| 460 |
+
startTime: Date.now(),
|
| 461 |
endTime: Date.now(),
|
| 462 |
+
totalFrames: recorder.teleoperatorData.length,
|
| 463 |
+
totalEpisodes: recorder.teleoperatorData.length,
|
| 464 |
},
|
| 465 |
+
})),
|
| 466 |
+
};
|
| 467 |
}
|
| 468 |
```
|
| 469 |
|
| 470 |
+
**Key Benefits of This Approach:**
|
| 471 |
+
|
| 472 |
+
- ✅ **Preserves Explicit Dependencies**: Keeps the excellent `teleoperator` parameter
|
| 473 |
+
- ✅ **Minimal Changes**: Just wraps existing `LeRobotDatasetRecorder`
|
| 474 |
+
- ✅ **No Breaking Changes**: Current class remains available
|
| 475 |
+
- ✅ **Consistent API**: Follows `start()`, `stop()`, `getState()`, `result` pattern
|
| 476 |
+
|
| 477 |
#### Updated Teleoperate Integration
|
| 478 |
|
| 479 |
```typescript
|
|
|
|
| 562 |
|
| 563 |
## Definition of Done
|
| 564 |
|
| 565 |
+
### Phase 1: Simple Function API (Priority)
|
| 566 |
+
|
| 567 |
+
- [ ] **Clean Record API**: `record(config)` function implemented as wrapper around existing `LeRobotDatasetRecorder`
|
| 568 |
- [ ] **Process Interface**: `RecordProcess` with consistent `start()`, `stop()`, `getState()`, `result` methods
|
| 569 |
+
- [ ] **Hardware-Only Simple API**: Simple `record()` function captures only robot motor data (no video)
|
| 570 |
+
- [ ] **Preserve Advanced Features**: Keep existing `LeRobotDatasetRecorder` class available for full functionality
|
| 571 |
+
- [ ] **TypeScript Coverage**: Full type safety with proper interfaces for simple recording API
|
| 572 |
+
|
| 573 |
+
### Phase 2: UI Separation (Secondary)
|
| 574 |
+
|
| 575 |
+
- [ ] **Separate Recording Component**: Move recording from teleoperation view to dedicated component/page
|
| 576 |
+
- [ ] **Clean Navigation**: Add recording as separate section in cyberpunk example navigation
|
| 577 |
+
- [ ] **No Breaking Changes**: Existing functionality continues to work during transition
|
| 578 |
+
|
| 579 |
+
### Phase 3: Library/Demo Boundary (Future)
|
| 580 |
+
|
| 581 |
+
- [ ] **Demo Separation**: Video recording, export formats moved to examples layer (optional enhancement)
|
| 582 |
+
- [ ] **Advanced Export Demo**: Create demo showing complex export features using `LeRobotDatasetRecorder`
|
| 583 |
+
- [ ] **Documentation**: Clear examples showing simple API vs advanced class usage
|
| 584 |
+
|
| 585 |
+
### Success Criteria
|
| 586 |
+
|
| 587 |
+
- [ ] **API Consistency**: `record()` function follows same patterns as `calibrate()` and `teleoperate()`
|
| 588 |
+
- [ ] **No Regression**: All existing recording functionality preserved and working
|
| 589 |
+
- [ ] **Easy Migration**: Users can easily switch between simple API and advanced class
|
| 590 |
+
- [ ] **Clean Example**: Recording has its own dedicated UI section in cyberpunk demo
|
|
@@ -13,6 +13,7 @@ import { EditRobotDialog } from "@/components/edit-robot-dialog";
|
|
| 13 |
import { DeviceDashboard } from "@/components/device-dashboard";
|
| 14 |
import { CalibrationView } from "@/components/calibration-view";
|
| 15 |
import { TeleoperationView } from "@/components/teleoperation-view";
|
|
|
|
| 16 |
import { SetupCards } from "@/components/setup-cards";
|
| 17 |
import { DocsSection } from "@/components/docs-section";
|
| 18 |
import { RoadmapSection } from "@/components/roadmap-section";
|
|
@@ -225,31 +226,17 @@ function App() {
|
|
| 225 |
};
|
| 226 |
|
| 227 |
const handleCalibrate = (robot: RobotConnection) => {
|
| 228 |
-
if (!robot.isConnected) {
|
| 229 |
-
toast({
|
| 230 |
-
title: "Robot Not Connected",
|
| 231 |
-
description: "Please connect the robot before calibrating",
|
| 232 |
-
variant: "destructive",
|
| 233 |
-
});
|
| 234 |
-
return;
|
| 235 |
-
}
|
| 236 |
-
|
| 237 |
navigate(`/device/${robot.serialNumber}/calibrate`);
|
| 238 |
};
|
| 239 |
|
| 240 |
const handleTeleoperate = (robot: RobotConnection) => {
|
| 241 |
-
if (!robot.isConnected) {
|
| 242 |
-
toast({
|
| 243 |
-
title: "Robot Not Connected",
|
| 244 |
-
description: "Please connect the robot before teleoperating",
|
| 245 |
-
variant: "destructive",
|
| 246 |
-
});
|
| 247 |
-
return;
|
| 248 |
-
}
|
| 249 |
-
|
| 250 |
navigate(`/device/${robot.serialNumber}/control`);
|
| 251 |
};
|
| 252 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 253 |
const handleBackToDashboard = () => {
|
| 254 |
navigate("/");
|
| 255 |
};
|
|
@@ -270,6 +257,7 @@ function App() {
|
|
| 270 |
robots={robots}
|
| 271 |
onCalibrate={handleCalibrate}
|
| 272 |
onTeleoperate={handleTeleoperate}
|
|
|
|
| 273 |
onRemove={handleRemoveRobot}
|
| 274 |
onEdit={setEditingRobot}
|
| 275 |
onFindNew={handleFindNewRobots}
|
|
@@ -297,6 +285,15 @@ function App() {
|
|
| 297 |
/>
|
| 298 |
}
|
| 299 |
/>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 300 |
</Routes>
|
| 301 |
<EditRobotDialog
|
| 302 |
robot={editingRobot}
|
|
@@ -316,6 +313,7 @@ function DashboardPage({
|
|
| 316 |
robots,
|
| 317 |
onCalibrate,
|
| 318 |
onTeleoperate,
|
|
|
|
| 319 |
onRemove,
|
| 320 |
onEdit,
|
| 321 |
onFindNew,
|
|
@@ -326,6 +324,7 @@ function DashboardPage({
|
|
| 326 |
robots: RobotConnection[];
|
| 327 |
onCalibrate: (robot: RobotConnection) => void;
|
| 328 |
onTeleoperate: (robot: RobotConnection) => void;
|
|
|
|
| 329 |
onRemove: (robot: RobotConnection) => void;
|
| 330 |
onEdit: (robot: RobotConnection | null) => void;
|
| 331 |
onFindNew: () => void;
|
|
@@ -341,6 +340,7 @@ function DashboardPage({
|
|
| 341 |
robots={robots}
|
| 342 |
onCalibrate={onCalibrate}
|
| 343 |
onTeleoperate={onTeleoperate}
|
|
|
|
| 344 |
onRemove={onRemove}
|
| 345 |
onEdit={onEdit}
|
| 346 |
onFindNew={onFindNew}
|
|
@@ -442,6 +442,43 @@ function ControlPage({
|
|
| 442 |
);
|
| 443 |
}
|
| 444 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 445 |
// Page Header Component
|
| 446 |
function PageHeader({
|
| 447 |
onBackToDashboard,
|
|
@@ -454,6 +491,7 @@ function PageHeader({
|
|
| 454 |
const isDashboard = location.pathname === "/";
|
| 455 |
const isCalibrating = location.pathname.includes("/calibrate");
|
| 456 |
const isTeleoperating = location.pathname.includes("/control");
|
|
|
|
| 457 |
|
| 458 |
return (
|
| 459 |
<div className="flex items-center justify-between mb-12">
|
|
@@ -483,6 +521,16 @@ function PageHeader({
|
|
| 483 |
{selectedRobot.robotId?.toUpperCase()}
|
| 484 |
</span>
|
| 485 |
</h1>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 486 |
) : (
|
| 487 |
<h1
|
| 488 |
className="font-mono text-4xl font-bold text-primary tracking-wider text-glitch uppercase"
|
|
|
|
| 13 |
import { DeviceDashboard } from "@/components/device-dashboard";
|
| 14 |
import { CalibrationView } from "@/components/calibration-view";
|
| 15 |
import { TeleoperationView } from "@/components/teleoperation-view";
|
| 16 |
+
import { RecordingView } from "@/components/recording-view";
|
| 17 |
import { SetupCards } from "@/components/setup-cards";
|
| 18 |
import { DocsSection } from "@/components/docs-section";
|
| 19 |
import { RoadmapSection } from "@/components/roadmap-section";
|
|
|
|
| 226 |
};
|
| 227 |
|
| 228 |
const handleCalibrate = (robot: RobotConnection) => {
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 229 |
navigate(`/device/${robot.serialNumber}/calibrate`);
|
| 230 |
};
|
| 231 |
|
| 232 |
const handleTeleoperate = (robot: RobotConnection) => {
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 233 |
navigate(`/device/${robot.serialNumber}/control`);
|
| 234 |
};
|
| 235 |
|
| 236 |
+
const handleRecord = (robot: RobotConnection) => {
|
| 237 |
+
navigate(`/device/${robot.serialNumber}/record`);
|
| 238 |
+
};
|
| 239 |
+
|
| 240 |
const handleBackToDashboard = () => {
|
| 241 |
navigate("/");
|
| 242 |
};
|
|
|
|
| 257 |
robots={robots}
|
| 258 |
onCalibrate={handleCalibrate}
|
| 259 |
onTeleoperate={handleTeleoperate}
|
| 260 |
+
onRecord={handleRecord}
|
| 261 |
onRemove={handleRemoveRobot}
|
| 262 |
onEdit={setEditingRobot}
|
| 263 |
onFindNew={handleFindNewRobots}
|
|
|
|
| 285 |
/>
|
| 286 |
}
|
| 287 |
/>
|
| 288 |
+
<Route
|
| 289 |
+
path="/device/:serialNumber/record"
|
| 290 |
+
element={
|
| 291 |
+
<RecordPage
|
| 292 |
+
robots={robots}
|
| 293 |
+
onBackToDashboard={handleBackToDashboard}
|
| 294 |
+
/>
|
| 295 |
+
}
|
| 296 |
+
/>
|
| 297 |
</Routes>
|
| 298 |
<EditRobotDialog
|
| 299 |
robot={editingRobot}
|
|
|
|
| 313 |
robots,
|
| 314 |
onCalibrate,
|
| 315 |
onTeleoperate,
|
| 316 |
+
onRecord,
|
| 317 |
onRemove,
|
| 318 |
onEdit,
|
| 319 |
onFindNew,
|
|
|
|
| 324 |
robots: RobotConnection[];
|
| 325 |
onCalibrate: (robot: RobotConnection) => void;
|
| 326 |
onTeleoperate: (robot: RobotConnection) => void;
|
| 327 |
+
onRecord: (robot: RobotConnection) => void;
|
| 328 |
onRemove: (robot: RobotConnection) => void;
|
| 329 |
onEdit: (robot: RobotConnection | null) => void;
|
| 330 |
onFindNew: () => void;
|
|
|
|
| 340 |
robots={robots}
|
| 341 |
onCalibrate={onCalibrate}
|
| 342 |
onTeleoperate={onTeleoperate}
|
| 343 |
+
onRecord={onRecord}
|
| 344 |
onRemove={onRemove}
|
| 345 |
onEdit={onEdit}
|
| 346 |
onFindNew={onFindNew}
|
|
|
|
| 442 |
);
|
| 443 |
}
|
| 444 |
|
| 445 |
+
// Record Page Component
|
| 446 |
+
function RecordPage({
|
| 447 |
+
robots,
|
| 448 |
+
onBackToDashboard,
|
| 449 |
+
}: {
|
| 450 |
+
robots: RobotConnection[];
|
| 451 |
+
onBackToDashboard: () => void;
|
| 452 |
+
}) {
|
| 453 |
+
const { serialNumber } = useParams<{ serialNumber: string }>();
|
| 454 |
+
const selectedRobot = robots.find(
|
| 455 |
+
(robot) => robot.serialNumber === serialNumber
|
| 456 |
+
);
|
| 457 |
+
|
| 458 |
+
if (!selectedRobot) {
|
| 459 |
+
return (
|
| 460 |
+
<div>
|
| 461 |
+
<PageHeader onBackToDashboard={onBackToDashboard} />
|
| 462 |
+
<div className="text-center py-20">
|
| 463 |
+
<p className="text-muted-foreground">
|
| 464 |
+
Device not found or not connected.
|
| 465 |
+
</p>
|
| 466 |
+
</div>
|
| 467 |
+
</div>
|
| 468 |
+
);
|
| 469 |
+
}
|
| 470 |
+
|
| 471 |
+
return (
|
| 472 |
+
<div>
|
| 473 |
+
<PageHeader
|
| 474 |
+
onBackToDashboard={onBackToDashboard}
|
| 475 |
+
selectedRobot={selectedRobot}
|
| 476 |
+
/>
|
| 477 |
+
<RecordingView robot={selectedRobot} />
|
| 478 |
+
</div>
|
| 479 |
+
);
|
| 480 |
+
}
|
| 481 |
+
|
| 482 |
// Page Header Component
|
| 483 |
function PageHeader({
|
| 484 |
onBackToDashboard,
|
|
|
|
| 491 |
const isDashboard = location.pathname === "/";
|
| 492 |
const isCalibrating = location.pathname.includes("/calibrate");
|
| 493 |
const isTeleoperating = location.pathname.includes("/control");
|
| 494 |
+
const isRecording = location.pathname.includes("/record");
|
| 495 |
|
| 496 |
return (
|
| 497 |
<div className="flex items-center justify-between mb-12">
|
|
|
|
| 521 |
{selectedRobot.robotId?.toUpperCase()}
|
| 522 |
</span>
|
| 523 |
</h1>
|
| 524 |
+
) : isRecording && selectedRobot ? (
|
| 525 |
+
<h1 className="font-mono text-4xl font-bold tracking-wider">
|
| 526 |
+
<span className="text-muted-foreground uppercase">record:</span>{" "}
|
| 527 |
+
<span
|
| 528 |
+
className="text-primary text-glitch uppercase"
|
| 529 |
+
data-text={selectedRobot.robotId}
|
| 530 |
+
>
|
| 531 |
+
{selectedRobot.robotId?.toUpperCase()}
|
| 532 |
+
</span>
|
| 533 |
+
</h1>
|
| 534 |
) : (
|
| 535 |
<h1
|
| 536 |
className="font-mono text-4xl font-bold text-primary tracking-wider text-glitch uppercase"
|
|
@@ -3,6 +3,7 @@ import { useState, useMemo, useEffect, useCallback } from "react";
|
|
| 3 |
import { Download } from "lucide-react";
|
| 4 |
import { Button } from "@/components/ui/button";
|
| 5 |
import { Card } from "@/components/ui/card";
|
|
|
|
| 6 |
import {
|
| 7 |
AlertDialog,
|
| 8 |
AlertDialogAction,
|
|
@@ -28,6 +29,7 @@ import {
|
|
| 28 |
type CalibrationMetadata,
|
| 29 |
} from "@/lib/unified-storage";
|
| 30 |
import { MotorCalibrationVisual } from "@/components/motor-calibration-visual";
|
|
|
|
| 31 |
|
| 32 |
interface CalibrationViewProps {
|
| 33 |
robot: RobotConnection;
|
|
@@ -281,32 +283,50 @@ export function CalibrationView({ robot }: CalibrationViewProps) {
|
|
| 281 |
</p>
|
| 282 |
</div>
|
| 283 |
</div>
|
| 284 |
-
<div className="flex gap-4">
|
| 285 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 286 |
<Button
|
| 287 |
-
onClick={
|
|
|
|
| 288 |
size="lg"
|
| 289 |
-
disabled={
|
| 290 |
>
|
| 291 |
-
|
| 292 |
-
? "Preparing..."
|
| 293 |
-
: calibrationResults
|
| 294 |
-
? "Re-calibrate"
|
| 295 |
-
: "Start Calibration"}
|
| 296 |
</Button>
|
| 297 |
-
|
| 298 |
-
<Button onClick={handleFinish} variant="destructive" size="lg">
|
| 299 |
-
Finish Recording
|
| 300 |
-
</Button>
|
| 301 |
-
)}
|
| 302 |
-
<Button
|
| 303 |
-
onClick={downloadJson}
|
| 304 |
-
variant="outline"
|
| 305 |
-
size="lg"
|
| 306 |
-
disabled={!calibrationResults}
|
| 307 |
-
>
|
| 308 |
-
<Download className="w-4 h-4 mr-2" /> Download JSON
|
| 309 |
-
</Button>
|
| 310 |
</div>
|
| 311 |
</div>
|
| 312 |
<div className="pt-6 p-6">
|
|
|
|
| 3 |
import { Download } from "lucide-react";
|
| 4 |
import { Button } from "@/components/ui/button";
|
| 5 |
import { Card } from "@/components/ui/card";
|
| 6 |
+
import { Badge } from "@/components/ui/badge";
|
| 7 |
import {
|
| 8 |
AlertDialog,
|
| 9 |
AlertDialogAction,
|
|
|
|
| 29 |
type CalibrationMetadata,
|
| 30 |
} from "@/lib/unified-storage";
|
| 31 |
import { MotorCalibrationVisual } from "@/components/motor-calibration-visual";
|
| 32 |
+
import { cn } from "@/lib/utils";
|
| 33 |
|
| 34 |
interface CalibrationViewProps {
|
| 35 |
robot: RobotConnection;
|
|
|
|
| 283 |
</p>
|
| 284 |
</div>
|
| 285 |
</div>
|
| 286 |
+
<div className="flex items-center gap-4">
|
| 287 |
+
<div className="flex items-center gap-2">
|
| 288 |
+
<span className="text-sm font-mono text-muted-foreground uppercase">
|
| 289 |
+
robot status:
|
| 290 |
+
</span>
|
| 291 |
+
<Badge
|
| 292 |
+
variant="outline"
|
| 293 |
+
className={cn(
|
| 294 |
+
"border-primary/50 bg-primary/20 text-primary font-mono text-xs",
|
| 295 |
+
robot.isConnected
|
| 296 |
+
? "border-green-500/50 bg-green-500/20 text-green-400"
|
| 297 |
+
: "border-red-500/50 bg-red-500/20 text-red-400"
|
| 298 |
+
)}
|
| 299 |
+
>
|
| 300 |
+
{robot.isConnected ? "ONLINE" : "OFFLINE"}
|
| 301 |
+
</Badge>
|
| 302 |
+
</div>
|
| 303 |
+
<div className="flex gap-4">
|
| 304 |
+
{!isCalibrating ? (
|
| 305 |
+
<Button
|
| 306 |
+
onClick={handleStart}
|
| 307 |
+
size="lg"
|
| 308 |
+
disabled={isPreparing || !robot.isConnected}
|
| 309 |
+
>
|
| 310 |
+
{isPreparing
|
| 311 |
+
? "Preparing..."
|
| 312 |
+
: calibrationResults
|
| 313 |
+
? "Re-calibrate"
|
| 314 |
+
: "Start Calibration"}
|
| 315 |
+
</Button>
|
| 316 |
+
) : (
|
| 317 |
+
<Button onClick={handleFinish} variant="destructive" size="lg">
|
| 318 |
+
Finish Recording
|
| 319 |
+
</Button>
|
| 320 |
+
)}
|
| 321 |
<Button
|
| 322 |
+
onClick={downloadJson}
|
| 323 |
+
variant="outline"
|
| 324 |
size="lg"
|
| 325 |
+
disabled={!calibrationResults}
|
| 326 |
>
|
| 327 |
+
<Download className="w-4 h-4 mr-2" /> Download JSON
|
|
|
|
|
|
|
|
|
|
|
|
|
| 328 |
</Button>
|
| 329 |
+
</div>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 330 |
</div>
|
| 331 |
</div>
|
| 332 |
<div className="pt-6 p-6">
|
|
@@ -8,6 +8,7 @@ import {
|
|
| 8 |
Pencil,
|
| 9 |
Plus,
|
| 10 |
ExternalLink,
|
|
|
|
| 11 |
} from "lucide-react";
|
| 12 |
import { Button } from "@/components/ui/button";
|
| 13 |
import { Card, CardFooter } from "@/components/ui/card";
|
|
@@ -31,6 +32,7 @@ interface DeviceDashboardProps {
|
|
| 31 |
robots: RobotConnection[];
|
| 32 |
onCalibrate: (robot: RobotConnection) => void;
|
| 33 |
onTeleoperate: (robot: RobotConnection) => void;
|
|
|
|
| 34 |
onRemove: (robotId: string) => void;
|
| 35 |
onEdit: (robot: RobotConnection) => void;
|
| 36 |
onFindNew: () => void;
|
|
@@ -42,6 +44,7 @@ export function DeviceDashboard({
|
|
| 42 |
robots,
|
| 43 |
onCalibrate,
|
| 44 |
onTeleoperate,
|
|
|
|
| 45 |
onRemove,
|
| 46 |
onEdit,
|
| 47 |
onFindNew,
|
|
@@ -162,7 +165,9 @@ export function DeviceDashboard({
|
|
| 162 |
variant="outline"
|
| 163 |
className={cn(
|
| 164 |
"border-primary/50 bg-primary/20 text-primary font-mono text-xs",
|
| 165 |
-
robot.isConnected
|
|
|
|
|
|
|
| 166 |
)}
|
| 167 |
>
|
| 168 |
{robot.isConnected ? "ONLINE" : "OFFLINE"}
|
|
@@ -220,6 +225,14 @@ export function DeviceDashboard({
|
|
| 220 |
>
|
| 221 |
<Gamepad2 className="w-3 h-3 mr-0.5" /> control
|
| 222 |
</Button>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 223 |
<Button
|
| 224 |
variant="destructive"
|
| 225 |
size="sm"
|
|
|
|
| 8 |
Pencil,
|
| 9 |
Plus,
|
| 10 |
ExternalLink,
|
| 11 |
+
Disc,
|
| 12 |
} from "lucide-react";
|
| 13 |
import { Button } from "@/components/ui/button";
|
| 14 |
import { Card, CardFooter } from "@/components/ui/card";
|
|
|
|
| 32 |
robots: RobotConnection[];
|
| 33 |
onCalibrate: (robot: RobotConnection) => void;
|
| 34 |
onTeleoperate: (robot: RobotConnection) => void;
|
| 35 |
+
onRecord: (robot: RobotConnection) => void;
|
| 36 |
onRemove: (robotId: string) => void;
|
| 37 |
onEdit: (robot: RobotConnection) => void;
|
| 38 |
onFindNew: () => void;
|
|
|
|
| 44 |
robots,
|
| 45 |
onCalibrate,
|
| 46 |
onTeleoperate,
|
| 47 |
+
onRecord,
|
| 48 |
onRemove,
|
| 49 |
onEdit,
|
| 50 |
onFindNew,
|
|
|
|
| 165 |
variant="outline"
|
| 166 |
className={cn(
|
| 167 |
"border-primary/50 bg-primary/20 text-primary font-mono text-xs",
|
| 168 |
+
robot.isConnected
|
| 169 |
+
? "border-green-500/50 bg-green-500/20 text-green-400 animate-pulse-slow"
|
| 170 |
+
: "border-red-500/50 bg-red-500/20 text-red-400"
|
| 171 |
)}
|
| 172 |
>
|
| 173 |
{robot.isConnected ? "ONLINE" : "OFFLINE"}
|
|
|
|
| 225 |
>
|
| 226 |
<Gamepad2 className="w-3 h-3 mr-0.5" /> control
|
| 227 |
</Button>
|
| 228 |
+
<Button
|
| 229 |
+
variant="outline"
|
| 230 |
+
size="sm"
|
| 231 |
+
onClick={() => onRecord(robot)}
|
| 232 |
+
className="font-mono text-xs uppercase px-2"
|
| 233 |
+
>
|
| 234 |
+
<Disc className="w-3 h-3 mr-0.5" /> record
|
| 235 |
+
</Button>
|
| 236 |
<Button
|
| 237 |
variant="destructive"
|
| 238 |
size="sm"
|
|
@@ -592,6 +592,113 @@ await releaseMotors(robot, [1, 2, 3]);`}
|
|
| 592 |
</ul>
|
| 593 |
</div>
|
| 594 |
</div>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 595 |
</div>
|
| 596 |
</div>
|
| 597 |
</TabsContent>
|
|
|
|
| 592 |
</ul>
|
| 593 |
</div>
|
| 594 |
</div>
|
| 595 |
+
|
| 596 |
+
{/* record */}
|
| 597 |
+
<div>
|
| 598 |
+
<h4 className="font-bold text-primary">record(config)</h4>
|
| 599 |
+
<p className="text-sm mt-1">
|
| 600 |
+
Records robot teleoperator data and camera streams in the LeRobot dataset format.
|
| 601 |
+
</p>
|
| 602 |
+
<CodeBlock>
|
| 603 |
+
{`import { teleoperate, record } from "@lerobot/web";
|
| 604 |
+
|
| 605 |
+
// Start teleoperation first
|
| 606 |
+
const teleoperationProcess = await teleoperate({
|
| 607 |
+
robot: connectedRobot,
|
| 608 |
+
teleop: { type: "keyboard" },
|
| 609 |
+
calibrationData: calibrationData,
|
| 610 |
+
});
|
| 611 |
+
|
| 612 |
+
// Create recording with teleoperator
|
| 613 |
+
const recordProcess = await record({
|
| 614 |
+
teleoperator: teleoperationProcess.teleoperator,
|
| 615 |
+
videoStreams: {
|
| 616 |
+
main: mainCameraStream,
|
| 617 |
+
wrist: wristCameraStream,
|
| 618 |
+
},
|
| 619 |
+
robotType: "so100",
|
| 620 |
+
options: {
|
| 621 |
+
fps: 30,
|
| 622 |
+
taskDescription: "Pick and place task",
|
| 623 |
+
onStateUpdate: (state) => {
|
| 624 |
+
console.log(\`Recording: \${state.frameCount} frames\`);
|
| 625 |
+
},
|
| 626 |
+
},
|
| 627 |
+
});
|
| 628 |
+
|
| 629 |
+
// Start both recording and teleoperation
|
| 630 |
+
teleoperationProcess.start();
|
| 631 |
+
recordProcess.start();
|
| 632 |
+
|
| 633 |
+
// Manage recording during operation
|
| 634 |
+
recordProcess.nextEpisode(); // Start new episode if needed
|
| 635 |
+
|
| 636 |
+
// Stop recording when finished
|
| 637 |
+
const robotData = await recordProcess.stop();
|
| 638 |
+
await recordProcess.exportForLeRobot("zip-download");`}
|
| 639 |
+
</CodeBlock>
|
| 640 |
+
<div className="mt-3">
|
| 641 |
+
<h5 className="font-bold text-sm tracking-wider">
|
| 642 |
+
Options
|
| 643 |
+
</h5>
|
| 644 |
+
<ul className="mt-1 ml-4 space-y-1 text-sm">
|
| 645 |
+
<li>
|
| 646 |
+
• <code>teleoperator: WebTeleoperator</code> - The teleoperator to record from
|
| 647 |
+
</li>
|
| 648 |
+
<li>
|
| 649 |
+
• <code>videoStreams?: {`{ [name: string]: MediaStream }`}</code> - Optional camera streams
|
| 650 |
+
</li>
|
| 651 |
+
<li>
|
| 652 |
+
• <code>robotType?: string</code> - Robot metadata (e.g., "so100")
|
| 653 |
+
</li>
|
| 654 |
+
<li>
|
| 655 |
+
• <code>options?: RecordOptions</code> - Optional configuration:
|
| 656 |
+
<ul className="mt-1 ml-4 space-y-1">
|
| 657 |
+
<li>
|
| 658 |
+
• <code>fps?: number</code> - Target frames per second (default: 30)
|
| 659 |
+
</li>
|
| 660 |
+
<li>
|
| 661 |
+
• <code>taskDescription?: string</code> - Task description
|
| 662 |
+
</li>
|
| 663 |
+
<li>
|
| 664 |
+
• <code>onStateUpdate?: (state: RecordingState) => void</code> - State changes
|
| 665 |
+
</li>
|
| 666 |
+
</ul>
|
| 667 |
+
</li>
|
| 668 |
+
</ul>
|
| 669 |
+
</div>
|
| 670 |
+
<div className="mt-3">
|
| 671 |
+
<h5 className="font-bold text-sm tracking-wider">
|
| 672 |
+
Returns: RecordProcess
|
| 673 |
+
</h5>
|
| 674 |
+
<ul className="mt-1 ml-4 space-y-1 text-sm">
|
| 675 |
+
<li>
|
| 676 |
+
• <code>start(): void</code> - Start recording
|
| 677 |
+
</li>
|
| 678 |
+
<li>
|
| 679 |
+
• <code>stop(): Promise<RobotRecordingData></code> - Stop recording
|
| 680 |
+
</li>
|
| 681 |
+
<li>
|
| 682 |
+
• <code>getState(): RecordingState</code> - Current recording state
|
| 683 |
+
</li>
|
| 684 |
+
<li>
|
| 685 |
+
• <code>getEpisodeCount(): number</code> - Get number of episodes
|
| 686 |
+
</li>
|
| 687 |
+
<li>
|
| 688 |
+
• <code>nextEpisode(): Promise<number></code> - Start new episode
|
| 689 |
+
</li>
|
| 690 |
+
<li>
|
| 691 |
+
• <code>clearEpisodes(): void</code> - Delete all episodes
|
| 692 |
+
</li>
|
| 693 |
+
<li>
|
| 694 |
+
• <code>addCamera(name: string, stream: MediaStream): void</code> - Add camera dynamically
|
| 695 |
+
</li>
|
| 696 |
+
<li>
|
| 697 |
+
• <code>exportForLeRobot(format?: "blobs" | "zip" | "zip-download"): Promise<any></code> - Export dataset
|
| 698 |
+
</li>
|
| 699 |
+
</ul>
|
| 700 |
+
</div>
|
| 701 |
+
</div>
|
| 702 |
</div>
|
| 703 |
</div>
|
| 704 |
</TabsContent>
|
|
@@ -1,17 +1,7 @@
|
|
| 1 |
"use client";
|
| 2 |
|
| 3 |
-
import { useState, useEffect, useRef, useCallback,
|
| 4 |
import { Button } from "@/components/ui/button";
|
| 5 |
-
import { Card } from "@/components/ui/card";
|
| 6 |
-
import {
|
| 7 |
-
Table,
|
| 8 |
-
TableBody,
|
| 9 |
-
TableCaption,
|
| 10 |
-
TableCell,
|
| 11 |
-
TableHead,
|
| 12 |
-
TableHeader,
|
| 13 |
-
TableRow,
|
| 14 |
-
} from "@/components/ui/table";
|
| 15 |
import { Input } from "@/components/ui/input";
|
| 16 |
import { Badge } from "@/components/ui/badge";
|
| 17 |
import {
|
|
@@ -36,20 +26,38 @@ import {
|
|
| 36 |
Edit2,
|
| 37 |
Check,
|
| 38 |
} from "lucide-react";
|
| 39 |
-
import {
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
import { TeleoperatorEpisodesView } from "./teleoperator-episodes-view";
|
|
|
|
| 41 |
|
| 42 |
interface RecorderProps {
|
| 43 |
teleoperators: any[];
|
| 44 |
robot: any; // eslint-disable-line @typescript-eslint/no-explicit-any
|
| 45 |
onNeedsTeleoperation: () => Promise<boolean>;
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 46 |
videoStreams?: { [key: string]: MediaStream };
|
| 47 |
}
|
| 48 |
|
| 49 |
-
|
| 50 |
-
|
| 51 |
interface RecorderSettings {
|
| 52 |
huggingfaceApiKey: string;
|
|
|
|
|
|
|
| 53 |
cameraConfigs: {
|
| 54 |
[cameraName: string]: {
|
| 55 |
deviceId: string;
|
|
@@ -72,6 +80,8 @@ function getRecorderSettings(): RecorderSettings {
|
|
| 72 |
}
|
| 73 |
return {
|
| 74 |
huggingfaceApiKey: "",
|
|
|
|
|
|
|
| 75 |
cameraConfigs: {},
|
| 76 |
};
|
| 77 |
}
|
|
@@ -88,9 +98,15 @@ export function Recorder({
|
|
| 88 |
teleoperators,
|
| 89 |
robot,
|
| 90 |
onNeedsTeleoperation,
|
|
|
|
|
|
|
| 91 |
}: RecorderProps) {
|
| 92 |
const [isRecording, setIsRecording] = useState(false);
|
| 93 |
const [currentEpisode, setCurrentEpisode] = useState(0);
|
|
|
|
|
|
|
|
|
|
|
|
|
| 94 |
// Use huggingfaceApiKey from recorderSettings instead of separate state
|
| 95 |
const [cameraName, setCameraName] = useState("");
|
| 96 |
const [additionalCameras, setAdditionalCameras] = useState<{
|
|
@@ -105,8 +121,7 @@ export function Recorder({
|
|
| 105 |
const [cameraPermissionState, setCameraPermissionState] = useState<
|
| 106 |
"unknown" | "granted" | "denied"
|
| 107 |
>("unknown");
|
| 108 |
-
|
| 109 |
-
const [showConfigure, setShowConfigure] = useState(false);
|
| 110 |
const [recorderSettings, setRecorderSettings] = useState<RecorderSettings>(
|
| 111 |
() => getRecorderSettings()
|
| 112 |
);
|
|
@@ -115,68 +130,106 @@ export function Recorder({
|
|
| 115 |
null
|
| 116 |
);
|
| 117 |
const [editingCameraNewName, setEditingCameraNewName] = useState("");
|
| 118 |
-
const [
|
| 119 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 120 |
const videoRef = useRef<HTMLVideoElement | null>(null);
|
| 121 |
const { toast } = useToast();
|
| 122 |
|
| 123 |
// Initialize the recorder when teleoperators are available
|
| 124 |
useEffect(() => {
|
| 125 |
-
if (teleoperators.length > 0) {
|
| 126 |
-
|
| 127 |
-
|
| 128 |
-
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 132 |
}
|
| 133 |
-
}, [teleoperators, additionalCameras]);
|
| 134 |
|
| 135 |
-
|
| 136 |
-
|
| 137 |
-
if (
|
| 138 |
-
|
| 139 |
-
|
| 140 |
-
description: `Setting up robot control for ${robot.robotId || "robot"}`,
|
| 141 |
-
});
|
| 142 |
|
| 143 |
-
|
| 144 |
-
|
| 145 |
-
|
| 146 |
-
|
| 147 |
-
|
| 148 |
-
|
| 149 |
-
});
|
| 150 |
-
return;
|
| 151 |
}
|
|
|
|
|
|
|
| 152 |
|
| 153 |
-
|
| 154 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 155 |
}
|
|
|
|
| 156 |
|
| 157 |
-
|
|
|
|
|
|
|
| 158 |
toast({
|
| 159 |
title: "Recording Error",
|
| 160 |
-
description: "Recorder not ready yet. Please
|
| 161 |
variant: "destructive",
|
| 162 |
});
|
| 163 |
return;
|
| 164 |
}
|
| 165 |
|
| 166 |
try {
|
| 167 |
-
// Set the episode index
|
| 168 |
-
recorderRef.current.setEpisodeIndex(currentEpisode);
|
| 169 |
-
recorderRef.current.setTaskIndex(0); // Default task index
|
| 170 |
-
|
| 171 |
// Start recording
|
| 172 |
-
|
| 173 |
setIsRecording(true);
|
| 174 |
setHasRecordedFrames(true);
|
| 175 |
|
| 176 |
-
|
| 177 |
-
|
| 178 |
-
|
| 179 |
-
|
| 180 |
} catch (error) {
|
| 181 |
const errorMessage =
|
| 182 |
error instanceof Error ? error.message : "Failed to start recording";
|
|
@@ -189,18 +242,18 @@ export function Recorder({
|
|
| 189 |
};
|
| 190 |
|
| 191 |
const handleStopRecording = async () => {
|
| 192 |
-
if (!
|
| 193 |
return;
|
| 194 |
}
|
| 195 |
|
| 196 |
try {
|
| 197 |
-
const result = await
|
| 198 |
setIsRecording(false);
|
| 199 |
|
| 200 |
-
|
| 201 |
-
|
| 202 |
-
|
| 203 |
-
|
| 204 |
} catch (error) {
|
| 205 |
const errorMessage =
|
| 206 |
error instanceof Error ? error.message : "Failed to stop recording";
|
|
@@ -212,37 +265,47 @@ export function Recorder({
|
|
| 212 |
}
|
| 213 |
};
|
| 214 |
|
| 215 |
-
const
|
| 216 |
-
|
| 217 |
-
|
| 218 |
-
handleStopRecording();
|
| 219 |
}
|
| 220 |
|
| 221 |
-
//
|
| 222 |
-
|
|
|
|
|
|
|
| 223 |
|
| 224 |
-
|
| 225 |
-
|
| 226 |
-
|
| 227 |
-
|
|
|
|
| 228 |
};
|
| 229 |
|
| 230 |
-
|
| 231 |
-
|
| 232 |
-
|
| 233 |
-
handleStopRecording();
|
| 234 |
}
|
| 235 |
|
| 236 |
-
|
| 237 |
-
|
| 238 |
-
|
| 239 |
-
|
| 240 |
-
|
| 241 |
-
|
| 242 |
-
description: "All recorded frames have been cleared",
|
| 243 |
});
|
| 244 |
-
|
| 245 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 246 |
|
| 247 |
// Load available cameras
|
| 248 |
const loadAvailableCameras = useCallback(
|
|
@@ -375,6 +438,79 @@ export function Recorder({
|
|
| 375 |
[previewStream, toast]
|
| 376 |
);
|
| 377 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 378 |
// Add a new camera to the recorder
|
| 379 |
const handleAddCamera = useCallback(async () => {
|
| 380 |
if (!cameraName.trim()) {
|
|
@@ -622,31 +758,26 @@ export function Recorder({
|
|
| 622 |
}, [availableCameras, cameraPermissionState, restoreSavedCameras]);
|
| 623 |
|
| 624 |
const handleDownloadZip = async () => {
|
| 625 |
-
if (!
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 626 |
toast({
|
| 627 |
-
title: "
|
| 628 |
-
description:
|
|
|
|
| 629 |
variant: "destructive",
|
| 630 |
});
|
| 631 |
return;
|
| 632 |
}
|
| 633 |
-
|
| 634 |
-
await recorderRef.current.exportForLeRobot("zip-download");
|
| 635 |
-
toast({
|
| 636 |
-
title: "Download Started",
|
| 637 |
-
description: "Your dataset is being downloaded as a ZIP file",
|
| 638 |
-
});
|
| 639 |
};
|
| 640 |
|
| 641 |
const handleUploadToHuggingFace = async () => {
|
| 642 |
-
if (!
|
| 643 |
-
toast({
|
| 644 |
-
title: "Upload Error",
|
| 645 |
-
description: "Recorder not initialized",
|
| 646 |
-
variant: "destructive",
|
| 647 |
-
});
|
| 648 |
-
return;
|
| 649 |
-
}
|
| 650 |
|
| 651 |
if (!recorderSettings.huggingfaceApiKey) {
|
| 652 |
toast({
|
|
@@ -658,25 +789,36 @@ export function Recorder({
|
|
| 658 |
}
|
| 659 |
|
| 660 |
try {
|
| 661 |
-
|
| 662 |
-
|
| 663 |
-
|
| 664 |
-
|
| 665 |
-
|
| 666 |
-
//
|
| 667 |
-
const
|
|
|
|
|
|
|
| 668 |
|
| 669 |
-
|
| 670 |
-
|
| 671 |
-
|
| 672 |
-
|
| 673 |
-
|
| 674 |
-
|
| 675 |
);
|
| 676 |
|
| 677 |
-
|
| 678 |
-
|
| 679 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 680 |
} catch (error) {
|
| 681 |
const errorMessage =
|
| 682 |
error instanceof Error
|
|
@@ -690,444 +832,212 @@ export function Recorder({
|
|
| 690 |
}
|
| 691 |
};
|
| 692 |
|
| 693 |
-
// Helper function to format duration
|
| 694 |
-
const formatDuration = (seconds: number): string => {
|
| 695 |
-
const mins = Math.floor(seconds / 60);
|
| 696 |
-
const secs = Math.floor(seconds % 60);
|
| 697 |
-
return `${mins.toString().padStart(2, "0")}:${secs
|
| 698 |
-
.toString()
|
| 699 |
-
.padStart(2, "0")}`;
|
| 700 |
-
};
|
| 701 |
-
|
| 702 |
return (
|
| 703 |
-
<
|
| 704 |
-
|
| 705 |
-
|
| 706 |
-
|
| 707 |
-
|
| 708 |
-
|
| 709 |
-
|
| 710 |
-
|
| 711 |
-
<
|
| 712 |
-
|
| 713 |
-
|
| 714 |
-
|
| 715 |
-
|
| 716 |
-
|
| 717 |
-
|
| 718 |
-
|
| 719 |
-
|
| 720 |
-
|
| 721 |
-
|
| 722 |
-
|
| 723 |
-
|
| 724 |
-
|
| 725 |
-
|
| 726 |
-
|
| 727 |
-
|
| 728 |
-
|
| 729 |
-
|
| 730 |
-
|
| 731 |
-
|
| 732 |
-
|
| 733 |
-
|
| 734 |
-
|
| 735 |
-
|
| 736 |
-
|
| 737 |
-
|
| 738 |
-
|
| 739 |
-
|
| 740 |
-
|
| 741 |
-
|
| 742 |
-
|
| 743 |
-
|
| 744 |
-
|
| 745 |
-
|
| 746 |
-
|
| 747 |
-
|
| 748 |
-
|
| 749 |
-
|
| 750 |
-
|
| 751 |
-
|
| 752 |
-
|
| 753 |
-
|
| 754 |
-
|
| 755 |
-
<div className="p-6 space-y-6">
|
| 756 |
-
{/* Recorder Settings - Toggleable Inline */}
|
| 757 |
-
{showConfigure && (
|
| 758 |
-
<div className="space-y-6">
|
| 759 |
-
{/* Hugging Face Settings */}
|
| 760 |
-
<div className="space-y-3">
|
| 761 |
-
<h3 className="text-lg font-semibold text-foreground">
|
| 762 |
-
Settings
|
| 763 |
-
</h3>
|
| 764 |
-
<div className="grid grid-cols-1 md:grid-cols-2 gap-4">
|
| 765 |
-
<div className="space-y-2">
|
| 766 |
-
<label className="text-sm text-muted-foreground">
|
| 767 |
-
Hugging Face API Key
|
| 768 |
-
</label>
|
| 769 |
-
<Input
|
| 770 |
-
placeholder="Enter your Hugging Face API key"
|
| 771 |
-
value={recorderSettings.huggingfaceApiKey}
|
| 772 |
onChange={(e) => {
|
| 773 |
const newSettings = {
|
| 774 |
...recorderSettings,
|
| 775 |
-
|
| 776 |
};
|
| 777 |
setRecorderSettings(newSettings);
|
| 778 |
saveRecorderSettings(newSettings);
|
| 779 |
}}
|
| 780 |
-
type="password"
|
| 781 |
-
className="bg-black/20 border-white/10"
|
| 782 |
/>
|
| 783 |
-
|
| 784 |
-
|
| 785 |
-
</p>
|
| 786 |
-
</div>
|
| 787 |
</div>
|
| 788 |
</div>
|
|
|
|
| 789 |
|
| 790 |
-
|
| 791 |
-
|
| 792 |
-
|
| 793 |
-
|
| 794 |
-
|
| 795 |
-
|
| 796 |
-
|
| 797 |
-
|
| 798 |
-
|
| 799 |
-
|
| 800 |
-
|
| 801 |
-
|
| 802 |
-
|
| 803 |
-
|
| 804 |
-
|
| 805 |
-
|
| 806 |
-
|
| 807 |
-
|
| 808 |
-
|
| 809 |
-
|
| 810 |
-
|
| 811 |
-
|
| 812 |
-
|
| 813 |
-
|
| 814 |
-
|
| 815 |
-
|
| 816 |
-
|
| 817 |
-
|
| 818 |
-
|
| 819 |
-
|
| 820 |
-
</div>
|
| 821 |
</div>
|
| 822 |
-
|
| 823 |
-
|
| 824 |
-
|
| 825 |
-
|
| 826 |
-
|
| 827 |
-
|
| 828 |
-
|
| 829 |
-
|
| 830 |
-
|
| 831 |
-
|
| 832 |
-
|
| 833 |
-
|
| 834 |
-
|
| 835 |
-
|
| 836 |
-
</
|
| 837 |
</div>
|
| 838 |
-
|
| 839 |
-
|
| 840 |
-
{/* Camera List with Refresh Button */}
|
| 841 |
-
{cameraPermissionState === "granted" &&
|
| 842 |
-
availableCameras.length > 0 && (
|
| 843 |
-
<div className="space-y-2">
|
| 844 |
-
<label className="text-sm text-white/70">
|
| 845 |
-
Select Camera:
|
| 846 |
-
</label>
|
| 847 |
-
<div className="flex items-center gap-2">
|
| 848 |
-
<Select
|
| 849 |
-
value={selectedCameraId}
|
| 850 |
-
onValueChange={switchCameraPreview}
|
| 851 |
-
disabled={hasRecordedFrames}
|
| 852 |
-
>
|
| 853 |
-
<SelectTrigger className="flex-1 bg-black/20 border-white/10">
|
| 854 |
-
<SelectValue placeholder="Choose a camera" />
|
| 855 |
-
</SelectTrigger>
|
| 856 |
-
<SelectContent>
|
| 857 |
-
{availableCameras.map((camera) => (
|
| 858 |
-
<SelectItem
|
| 859 |
-
key={camera.deviceId}
|
| 860 |
-
value={camera.deviceId}
|
| 861 |
-
>
|
| 862 |
-
{camera.label ||
|
| 863 |
-
`Camera ${camera.deviceId.slice(
|
| 864 |
-
0,
|
| 865 |
-
8
|
| 866 |
-
)}...`}
|
| 867 |
-
</SelectItem>
|
| 868 |
-
))}
|
| 869 |
-
</SelectContent>
|
| 870 |
-
</Select>
|
| 871 |
-
<Button
|
| 872 |
-
onClick={() => loadAvailableCameras(false)}
|
| 873 |
-
variant="ghost"
|
| 874 |
-
size="sm"
|
| 875 |
-
className="gap-2 text-white/70 hover:text-white"
|
| 876 |
-
disabled={isLoadingCameras}
|
| 877 |
-
>
|
| 878 |
-
<RefreshCw
|
| 879 |
-
className={`w-4 h-4 ${
|
| 880 |
-
isLoadingCameras ? "animate-spin" : ""
|
| 881 |
-
}`}
|
| 882 |
-
/>
|
| 883 |
-
Refresh
|
| 884 |
-
</Button>
|
| 885 |
-
</div>
|
| 886 |
-
</div>
|
| 887 |
-
)}
|
| 888 |
|
| 889 |
-
|
| 890 |
-
|
|
|
|
| 891 |
<div className="space-y-2">
|
| 892 |
<label className="text-sm text-white/70">
|
| 893 |
-
|
| 894 |
</label>
|
| 895 |
-
<
|
| 896 |
-
|
| 897 |
-
|
| 898 |
-
|
| 899 |
-
|
| 900 |
-
|
| 901 |
-
|
| 902 |
-
|
| 903 |
-
|
| 904 |
-
|
| 905 |
-
|
| 906 |
-
|
| 907 |
-
|
| 908 |
-
|
| 909 |
-
|
| 910 |
-
|
| 911 |
-
|
| 912 |
-
|
| 913 |
-
|
| 914 |
-
|
| 915 |
-
|
| 916 |
-
|
| 917 |
-
|
| 918 |
-
|
| 919 |
-
|
| 920 |
-
|
| 921 |
-
|
| 922 |
-
|
| 923 |
-
|
| 924 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 925 |
</div>
|
| 926 |
)}
|
| 927 |
-
</div>
|
| 928 |
|
| 929 |
-
{/*
|
| 930 |
-
|
| 931 |
-
<div className="
|
| 932 |
-
|
| 933 |
-
|
| 934 |
-
|
| 935 |
-
|
| 936 |
-
|
| 937 |
-
|
| 938 |
-
|
| 939 |
-
/
|
| 940 |
-
|
| 941 |
-
|
| 942 |
-
|
| 943 |
-
|
| 944 |
-
|
| 945 |
-
|
| 946 |
-
Request camera access to preview
|
| 947 |
-
</p>
|
| 948 |
-
) : cameraPermissionState === "denied" ? (
|
| 949 |
-
<p className="text-sm">Camera access denied</p>
|
| 950 |
-
) : availableCameras.length === 0 ? (
|
| 951 |
-
<p className="text-sm">No cameras available</p>
|
| 952 |
-
) : !selectedCameraId ? (
|
| 953 |
-
<p className="text-sm">
|
| 954 |
-
Select a camera to preview
|
| 955 |
-
</p>
|
| 956 |
-
) : (
|
| 957 |
-
<p className="text-sm">Loading preview...</p>
|
| 958 |
-
)}
|
| 959 |
-
</div>
|
| 960 |
-
</div>
|
| 961 |
-
)}
|
| 962 |
</div>
|
| 963 |
-
|
| 964 |
-
|
| 965 |
-
|
| 966 |
-
|
| 967 |
-
|
| 968 |
-
|
| 969 |
-
|
| 970 |
-
|
| 971 |
-
|
| 972 |
-
|
| 973 |
-
|
| 974 |
-
|
| 975 |
-
|
| 976 |
-
<div className="grid grid-cols-2 md:grid-cols-3 lg:grid-cols-4 gap-4">
|
| 977 |
-
{Object.entries(additionalCameras).map(([cameraName, stream]) => (
|
| 978 |
-
<div
|
| 979 |
-
key={cameraName}
|
| 980 |
-
className="bg-black/40 border border-white/20 rounded-lg p-3 space-y-2"
|
| 981 |
-
>
|
| 982 |
-
<div className="aspect-video bg-black/60 border border-white/10 rounded overflow-hidden">
|
| 983 |
-
<video
|
| 984 |
-
autoPlay
|
| 985 |
-
muted
|
| 986 |
-
playsInline
|
| 987 |
-
className="w-full h-full object-cover"
|
| 988 |
-
ref={(video) => {
|
| 989 |
-
if (video && stream) {
|
| 990 |
-
video.srcObject = stream;
|
| 991 |
}
|
| 992 |
-
}}
|
| 993 |
-
/>
|
| 994 |
-
</div>
|
| 995 |
-
<div className="flex items-center justify-between">
|
| 996 |
-
{editingCameraName === cameraName ? (
|
| 997 |
-
<div className="flex items-center gap-1 flex-1">
|
| 998 |
-
<Input
|
| 999 |
-
value={editingCameraNewName}
|
| 1000 |
-
onChange={(e) =>
|
| 1001 |
-
setEditingCameraNewName(e.target.value)
|
| 1002 |
-
}
|
| 1003 |
-
className="text-xs h-6 bg-black/20 border-white/10"
|
| 1004 |
-
onKeyDown={(e) => {
|
| 1005 |
-
if (e.key === "Enter") {
|
| 1006 |
-
handleConfirmCameraNameEdit(cameraName);
|
| 1007 |
-
} else if (e.key === "Escape") {
|
| 1008 |
-
handleCancelCameraNameEdit();
|
| 1009 |
-
}
|
| 1010 |
-
}}
|
| 1011 |
-
autoFocus
|
| 1012 |
-
/>
|
| 1013 |
-
<button
|
| 1014 |
-
onClick={() =>
|
| 1015 |
-
handleConfirmCameraNameEdit(cameraName)
|
| 1016 |
-
}
|
| 1017 |
-
className="text-green-400 hover:text-green-300 p-1"
|
| 1018 |
-
>
|
| 1019 |
-
<Check className="w-3 h-3" />
|
| 1020 |
-
</button>
|
| 1021 |
-
<button
|
| 1022 |
-
onClick={handleCancelCameraNameEdit}
|
| 1023 |
-
className="text-red-400 hover:text-red-300 p-1"
|
| 1024 |
-
>
|
| 1025 |
-
<X className="w-3 h-3" />
|
| 1026 |
-
</button>
|
| 1027 |
-
</div>
|
| 1028 |
-
) : (
|
| 1029 |
-
<button
|
| 1030 |
-
onClick={() => handleStartEditingCameraName(cameraName)}
|
| 1031 |
-
className="text-sm font-medium text-white/90 truncate hover:text-white cursor-pointer flex items-center gap-1 flex-1"
|
| 1032 |
-
disabled={hasRecordedFrames}
|
| 1033 |
>
|
| 1034 |
-
|
| 1035 |
-
|
| 1036 |
-
</
|
| 1037 |
-
|
| 1038 |
-
|
| 1039 |
-
onClick={() => handleRemoveCamera(cameraName)}
|
| 1040 |
-
className="text-red-400 hover:text-red-300 p-1 ml-2"
|
| 1041 |
-
disabled={hasRecordedFrames}
|
| 1042 |
-
title="Remove camera"
|
| 1043 |
-
>
|
| 1044 |
-
<X className="w-4 h-4" />
|
| 1045 |
-
</button>
|
| 1046 |
-
</div>
|
| 1047 |
</div>
|
| 1048 |
-
))}
|
| 1049 |
-
</div>
|
| 1050 |
-
</div>
|
| 1051 |
-
)}
|
| 1052 |
-
|
| 1053 |
-
{/* Episode Management & Dataset Actions */}
|
| 1054 |
-
<div className="flex justify-between items-center">
|
| 1055 |
-
<div className="flex items-center gap-2">
|
| 1056 |
-
<Button
|
| 1057 |
-
variant="outline"
|
| 1058 |
-
className="gap-2"
|
| 1059 |
-
onClick={handleResetFrames}
|
| 1060 |
-
disabled={isRecording || !hasRecordedFrames}
|
| 1061 |
-
>
|
| 1062 |
-
<Trash2 className="w-4 h-4" />
|
| 1063 |
-
Reset Frames
|
| 1064 |
-
</Button>
|
| 1065 |
-
<Button
|
| 1066 |
-
variant="outline"
|
| 1067 |
-
className="gap-2"
|
| 1068 |
-
onClick={handleNextEpisode}
|
| 1069 |
-
disabled={isRecording}
|
| 1070 |
-
>
|
| 1071 |
-
<PlusCircle className="w-4 h-4" />
|
| 1072 |
-
Next Episode
|
| 1073 |
-
</Button>
|
| 1074 |
-
</div>
|
| 1075 |
-
|
| 1076 |
-
<div className="flex items-center gap-2">
|
| 1077 |
-
<Button
|
| 1078 |
-
variant="outline"
|
| 1079 |
-
className="gap-2"
|
| 1080 |
-
onClick={handleDownloadZip}
|
| 1081 |
-
disabled={recorderRef.current?.teleoperatorData.length === 0 || isRecording}
|
| 1082 |
-
>
|
| 1083 |
-
<Download className="w-4 h-4" />
|
| 1084 |
-
Download as ZIP
|
| 1085 |
-
</Button>
|
| 1086 |
-
<Button
|
| 1087 |
-
variant="outline"
|
| 1088 |
-
className="gap-2"
|
| 1089 |
-
onClick={handleUploadToHuggingFace}
|
| 1090 |
-
disabled={
|
| 1091 |
-
recorderRef.current?.teleoperatorData.length === 0 ||
|
| 1092 |
-
isRecording ||
|
| 1093 |
-
!recorderSettings.huggingfaceApiKey
|
| 1094 |
-
}
|
| 1095 |
-
>
|
| 1096 |
-
<Upload className="w-4 h-4" />
|
| 1097 |
-
Upload to Hugging Face
|
| 1098 |
-
</Button>
|
| 1099 |
-
</div>
|
| 1100 |
-
</div>
|
| 1101 |
-
|
| 1102 |
-
<div className="border border-white/10 rounded-md overflow-hidden">
|
| 1103 |
-
<TeleoperatorEpisodesView teleoperatorData={recorderRef.current?.teleoperatorData} />
|
| 1104 |
-
</div>
|
| 1105 |
|
| 1106 |
-
|
| 1107 |
-
<div className="border-t border-white/10 pt-4 mt-4">
|
| 1108 |
-
<div className="flex items-center justify-between mb-4">
|
| 1109 |
-
<h3 className="text-lg font-semibold">Camera Setup</h3>
|
| 1110 |
-
<Button
|
| 1111 |
-
onClick={() => setShowCameraConfig(!showCameraConfig)}
|
| 1112 |
-
variant="outline"
|
| 1113 |
-
size="sm"
|
| 1114 |
-
className="gap-2"
|
| 1115 |
-
>
|
| 1116 |
-
<Camera className="w-4 h-4" />
|
| 1117 |
-
{showCameraConfig ? "Hide Camera Config" : "Configure Cameras"}
|
| 1118 |
-
</Button>
|
| 1119 |
-
</div>
|
| 1120 |
-
|
| 1121 |
-
{showCameraConfig && (
|
| 1122 |
-
<div className="bg-black/40 border border-white/20 rounded-lg p-6 mb-4">
|
| 1123 |
-
<div className="grid grid-cols-1 lg:grid-cols-2 gap-6">
|
| 1124 |
-
{/* Left Column: Camera Preview & Selection */}
|
| 1125 |
<div className="space-y-4">
|
| 1126 |
-
<h4 className="text-md font-semibold text-white/90">
|
| 1127 |
-
Camera Preview
|
| 1128 |
-
</h4>
|
| 1129 |
-
|
| 1130 |
-
{/* Camera Preview */}
|
| 1131 |
<div className="aspect-video bg-black/60 border border-white/20 rounded-lg overflow-hidden">
|
| 1132 |
{previewStream ? (
|
| 1133 |
<video
|
|
@@ -1141,178 +1051,270 @@ export function Recorder({
|
|
| 1141 |
<div className="w-full h-full flex items-center justify-center text-white/60">
|
| 1142 |
<div className="text-center">
|
| 1143 |
<Camera className="w-12 h-12 mx-auto mb-2 opacity-50" />
|
| 1144 |
-
{
|
| 1145 |
-
<p
|
|
|
|
|
|
|
| 1146 |
) : cameraPermissionState === "denied" ? (
|
| 1147 |
-
<
|
| 1148 |
-
<p>Camera access denied</p>
|
| 1149 |
-
<p className="text-xs mt-1">
|
| 1150 |
-
Please allow camera access and refresh
|
| 1151 |
-
</p>
|
| 1152 |
-
</div>
|
| 1153 |
) : availableCameras.length === 0 ? (
|
| 1154 |
-
<p
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1155 |
) : (
|
| 1156 |
-
<p>
|
| 1157 |
)}
|
| 1158 |
</div>
|
| 1159 |
</div>
|
| 1160 |
)}
|
| 1161 |
</div>
|
| 1162 |
-
|
| 1163 |
-
{/* Camera Controls */}
|
| 1164 |
-
<div className="space-y-3">
|
| 1165 |
-
<Button
|
| 1166 |
-
onClick={() => loadAvailableCameras(false)}
|
| 1167 |
-
variant="outline"
|
| 1168 |
-
className="w-full gap-2"
|
| 1169 |
-
disabled={hasRecordedFrames || isLoadingCameras}
|
| 1170 |
-
>
|
| 1171 |
-
<Camera className="w-4 h-4" />
|
| 1172 |
-
{isLoadingCameras
|
| 1173 |
-
? "Loading..."
|
| 1174 |
-
: availableCameras.length > 0
|
| 1175 |
-
? "Refresh Cameras"
|
| 1176 |
-
: "Load Cameras"}
|
| 1177 |
-
</Button>
|
| 1178 |
-
|
| 1179 |
-
{availableCameras.length > 0 && (
|
| 1180 |
-
<div className="space-y-2">
|
| 1181 |
-
<label className="text-sm text-white/70">
|
| 1182 |
-
Select Camera:
|
| 1183 |
-
</label>
|
| 1184 |
-
<Select
|
| 1185 |
-
value={selectedCameraId}
|
| 1186 |
-
onValueChange={switchCameraPreview}
|
| 1187 |
-
disabled={hasRecordedFrames}
|
| 1188 |
-
>
|
| 1189 |
-
<SelectTrigger className="w-full bg-black/20 border-white/10">
|
| 1190 |
-
<SelectValue placeholder="Choose a camera" />
|
| 1191 |
-
</SelectTrigger>
|
| 1192 |
-
<SelectContent>
|
| 1193 |
-
{availableCameras.map((camera) => (
|
| 1194 |
-
<SelectItem
|
| 1195 |
-
key={camera.deviceId}
|
| 1196 |
-
value={camera.deviceId}
|
| 1197 |
-
>
|
| 1198 |
-
{camera.label ||
|
| 1199 |
-
`Camera ${camera.deviceId.slice(0, 8)}...`}
|
| 1200 |
-
</SelectItem>
|
| 1201 |
-
))}
|
| 1202 |
-
</SelectContent>
|
| 1203 |
-
</Select>
|
| 1204 |
-
</div>
|
| 1205 |
-
)}
|
| 1206 |
-
</div>
|
| 1207 |
</div>
|
| 1208 |
-
|
| 1209 |
-
|
| 1210 |
-
|
| 1211 |
-
|
| 1212 |
-
|
| 1213 |
-
|
| 1214 |
-
|
| 1215 |
-
|
| 1216 |
-
|
| 1217 |
-
|
| 1218 |
-
|
| 1219 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1220 |
<Input
|
| 1221 |
-
|
| 1222 |
-
|
| 1223 |
-
|
| 1224 |
-
|
| 1225 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1226 |
/>
|
| 1227 |
-
<
|
| 1228 |
-
|
| 1229 |
-
|
| 1230 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1231 |
</div>
|
| 1232 |
-
|
| 1233 |
-
<
|
| 1234 |
-
onClick={
|
| 1235 |
-
className="
|
| 1236 |
-
disabled={
|
| 1237 |
-
hasRecordedFrames ||
|
| 1238 |
-
!cameraName.trim() ||
|
| 1239 |
-
!selectedCameraId ||
|
| 1240 |
-
!previewStream
|
| 1241 |
-
}
|
| 1242 |
>
|
| 1243 |
-
|
| 1244 |
-
|
| 1245 |
-
</
|
| 1246 |
-
|
| 1247 |
-
|
| 1248 |
-
|
| 1249 |
-
|
| 1250 |
-
|
| 1251 |
-
|
| 1252 |
-
|
| 1253 |
-
|
| 1254 |
-
|
| 1255 |
-
|
| 1256 |
-
|
| 1257 |
-
|
| 1258 |
-
|
| 1259 |
-
|
| 1260 |
-
|
| 1261 |
-
|
| 1262 |
-
|
| 1263 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1264 |
<button
|
| 1265 |
-
onClick={() => handleRemoveCamera(
|
| 1266 |
-
className="
|
| 1267 |
disabled={hasRecordedFrames}
|
|
|
|
| 1268 |
>
|
| 1269 |
-
|
| 1270 |
</button>
|
| 1271 |
-
</
|
| 1272 |
-
|
| 1273 |
</div>
|
| 1274 |
-
|
| 1275 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1276 |
</div>
|
| 1277 |
|
| 1278 |
-
<div className="flex items-center gap-
|
| 1279 |
-
<
|
| 1280 |
-
|
| 1281 |
-
|
| 1282 |
-
|
| 1283 |
-
|
| 1284 |
-
|
| 1285 |
-
|
| 1286 |
-
|
| 1287 |
-
|
| 1288 |
-
</
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1289 |
|
| 1290 |
-
|
| 1291 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1292 |
|
| 1293 |
-
|
| 1294 |
-
|
| 1295 |
-
|
| 1296 |
-
|
| 1297 |
-
|
| 1298 |
-
|
| 1299 |
-
|
| 1300 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1301 |
>
|
| 1302 |
-
|
| 1303 |
-
|
| 1304 |
-
|
| 1305 |
-
|
| 1306 |
-
|
| 1307 |
-
|
| 1308 |
-
value={huggingfaceApiKey}
|
| 1309 |
-
onChange={(e) => setHuggingfaceApiKey(e.target.value)}
|
| 1310 |
-
className="flex-1 bg-black/20 border-white/10"
|
| 1311 |
-
type="password"
|
| 1312 |
-
/>
|
| 1313 |
-
</div>
|
| 1314 |
-
</div>
|
| 1315 |
-
</div>
|
| 1316 |
-
</Card>
|
| 1317 |
);
|
| 1318 |
}
|
|
|
|
|
|
|
|
|
| 1 |
"use client";
|
| 2 |
|
| 3 |
+
import { useState, useEffect, useRef, useCallback, memo } from "react";
|
| 4 |
import { Button } from "@/components/ui/button";
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5 |
import { Input } from "@/components/ui/input";
|
| 6 |
import { Badge } from "@/components/ui/badge";
|
| 7 |
import {
|
|
|
|
| 26 |
Edit2,
|
| 27 |
Check,
|
| 28 |
} from "lucide-react";
|
| 29 |
+
import {
|
| 30 |
+
AlertDialog,
|
| 31 |
+
AlertDialogAction,
|
| 32 |
+
AlertDialogCancel,
|
| 33 |
+
AlertDialogContent,
|
| 34 |
+
AlertDialogDescription,
|
| 35 |
+
AlertDialogFooter,
|
| 36 |
+
AlertDialogHeader,
|
| 37 |
+
AlertDialogTitle,
|
| 38 |
+
} from "@/components/ui/alert-dialog";
|
| 39 |
+
import { record, LeRobotEpisode } from "@lerobot/web";
|
| 40 |
+
import type { RecordProcess } from "@lerobot/web";
|
| 41 |
import { TeleoperatorEpisodesView } from "./teleoperator-episodes-view";
|
| 42 |
+
import { uploadToHuggingFace } from "@/utils/dataset-uploader";
|
| 43 |
|
| 44 |
interface RecorderProps {
|
| 45 |
teleoperators: any[];
|
| 46 |
robot: any; // eslint-disable-line @typescript-eslint/no-explicit-any
|
| 47 |
onNeedsTeleoperation: () => Promise<boolean>;
|
| 48 |
+
showConfigure: boolean;
|
| 49 |
+
onRecorderReady?: (callbacks: {
|
| 50 |
+
startRecording: () => Promise<void>;
|
| 51 |
+
stopRecording: () => Promise<void>;
|
| 52 |
+
isRecording: boolean;
|
| 53 |
+
}) => void;
|
| 54 |
videoStreams?: { [key: string]: MediaStream };
|
| 55 |
}
|
| 56 |
|
|
|
|
|
|
|
| 57 |
interface RecorderSettings {
|
| 58 |
huggingfaceApiKey: string;
|
| 59 |
+
huggingfaceRepoName?: string;
|
| 60 |
+
huggingfacePrivate?: boolean;
|
| 61 |
cameraConfigs: {
|
| 62 |
[cameraName: string]: {
|
| 63 |
deviceId: string;
|
|
|
|
| 80 |
}
|
| 81 |
return {
|
| 82 |
huggingfaceApiKey: "",
|
| 83 |
+
huggingfaceRepoName: "",
|
| 84 |
+
huggingfacePrivate: true,
|
| 85 |
cameraConfigs: {},
|
| 86 |
};
|
| 87 |
}
|
|
|
|
| 98 |
teleoperators,
|
| 99 |
robot,
|
| 100 |
onNeedsTeleoperation,
|
| 101 |
+
showConfigure,
|
| 102 |
+
onRecorderReady,
|
| 103 |
}: RecorderProps) {
|
| 104 |
const [isRecording, setIsRecording] = useState(false);
|
| 105 |
const [currentEpisode, setCurrentEpisode] = useState(0);
|
| 106 |
+
const [persistedEpisodes, setPersistedEpisodes] = useState<any[]>([]);
|
| 107 |
+
const [uiTick, setUiTick] = useState(0);
|
| 108 |
+
const [showDeleteEpisodesDialog, setShowDeleteEpisodesDialog] =
|
| 109 |
+
useState(false);
|
| 110 |
// Use huggingfaceApiKey from recorderSettings instead of separate state
|
| 111 |
const [cameraName, setCameraName] = useState("");
|
| 112 |
const [additionalCameras, setAdditionalCameras] = useState<{
|
|
|
|
| 121 |
const [cameraPermissionState, setCameraPermissionState] = useState<
|
| 122 |
"unknown" | "granted" | "denied"
|
| 123 |
>("unknown");
|
| 124 |
+
|
|
|
|
| 125 |
const [recorderSettings, setRecorderSettings] = useState<RecorderSettings>(
|
| 126 |
() => getRecorderSettings()
|
| 127 |
);
|
|
|
|
| 130 |
null
|
| 131 |
);
|
| 132 |
const [editingCameraNewName, setEditingCameraNewName] = useState("");
|
| 133 |
+
const [sourceSelectorOpenFor, setSourceSelectorOpenFor] = useState<
|
| 134 |
+
string | null
|
| 135 |
+
>(null);
|
| 136 |
+
const [uploadState, setUploadState] = useState<
|
| 137 |
+
{ status: "idle" } | { status: "uploading" } | { status: "done" }
|
| 138 |
+
>({ status: "idle" });
|
| 139 |
+
|
| 140 |
+
const recordProcessRef = useRef<RecordProcess | null>(null);
|
| 141 |
const videoRef = useRef<HTMLVideoElement | null>(null);
|
| 142 |
const { toast } = useToast();
|
| 143 |
|
| 144 |
// Initialize the recorder when teleoperators are available
|
| 145 |
useEffect(() => {
|
| 146 |
+
if (teleoperators.length > 0 && !recordProcessRef.current) {
|
| 147 |
+
(async () => {
|
| 148 |
+
// Get robot type/family for metadata
|
| 149 |
+
let robotType = "unknown";
|
| 150 |
+
try {
|
| 151 |
+
const type = (robot?.robotType || "").toString();
|
| 152 |
+
robotType = type.split("_")[0] || type || "unknown";
|
| 153 |
+
} catch {}
|
| 154 |
+
|
| 155 |
+
// Create record process with upfront config
|
| 156 |
+
const recordProcess = await record({
|
| 157 |
+
teleoperator: teleoperators[0],
|
| 158 |
+
videoStreams: additionalCameras,
|
| 159 |
+
robotType,
|
| 160 |
+
options: {
|
| 161 |
+
fps: 30,
|
| 162 |
+
taskDescription: "Robot teleoperation recording",
|
| 163 |
+
},
|
| 164 |
+
});
|
| 165 |
+
recordProcessRef.current = recordProcess;
|
| 166 |
+
|
| 167 |
+
// Restore episodes if any were persisted
|
| 168 |
+
if (persistedEpisodes.length > 0) {
|
| 169 |
+
recordProcess.restoreEpisodes(persistedEpisodes);
|
| 170 |
+
setCurrentEpisode(persistedEpisodes.length - 1);
|
| 171 |
+
}
|
| 172 |
+
|
| 173 |
+
// Attach any cameras that were configured before control was enabled
|
| 174 |
+
for (const [key, stream] of Object.entries(additionalCameras)) {
|
| 175 |
+
try {
|
| 176 |
+
recordProcess.addCamera(key, stream as MediaStream);
|
| 177 |
+
} catch (e) {
|
| 178 |
+
console.warn("[Recorder] init: addCamera failed", key, e);
|
| 179 |
+
}
|
| 180 |
+
}
|
| 181 |
+
})();
|
| 182 |
}
|
| 183 |
+
}, [teleoperators, persistedEpisodes, additionalCameras]);
|
| 184 |
|
| 185 |
+
// Sync additional cameras into recorder without re-creating it
|
| 186 |
+
useEffect(() => {
|
| 187 |
+
if (!recordProcessRef.current) return;
|
| 188 |
+
if (isRecording) return; // don't change streams during recording
|
| 189 |
+
const recordProcess = recordProcessRef.current;
|
|
|
|
|
|
|
| 190 |
|
| 191 |
+
// Add any new cameras
|
| 192 |
+
for (const [key, stream] of Object.entries(additionalCameras)) {
|
| 193 |
+
try {
|
| 194 |
+
recordProcess.addCamera(key, stream);
|
| 195 |
+
} catch (e) {
|
| 196 |
+
console.warn("Failed to add camera", key, e);
|
|
|
|
|
|
|
| 197 |
}
|
| 198 |
+
}
|
| 199 |
+
}, [additionalCameras, isRecording]);
|
| 200 |
|
| 201 |
+
// Notify parent of recorder state changes
|
| 202 |
+
useEffect(() => {
|
| 203 |
+
if (onRecorderReady) {
|
| 204 |
+
onRecorderReady({
|
| 205 |
+
startRecording: handleStartRecordingClick,
|
| 206 |
+
stopRecording: handleStopRecording,
|
| 207 |
+
isRecording: isRecording,
|
| 208 |
+
});
|
| 209 |
}
|
| 210 |
+
}, [isRecording, onRecorderReady]);
|
| 211 |
|
| 212 |
+
// Simple recording start - just like the original working version
|
| 213 |
+
const handleStartRecordingClick = async () => {
|
| 214 |
+
if (!recordProcessRef.current) {
|
| 215 |
toast({
|
| 216 |
title: "Recording Error",
|
| 217 |
+
description: "Recorder not ready yet. Please enable control first.",
|
| 218 |
variant: "destructive",
|
| 219 |
});
|
| 220 |
return;
|
| 221 |
}
|
| 222 |
|
| 223 |
try {
|
|
|
|
|
|
|
|
|
|
|
|
|
| 224 |
// Start recording
|
| 225 |
+
recordProcessRef.current.start();
|
| 226 |
setIsRecording(true);
|
| 227 |
setHasRecordedFrames(true);
|
| 228 |
|
| 229 |
+
// Sync current episode to recorder's latest
|
| 230 |
+
setCurrentEpisode(
|
| 231 |
+
Math.max(0, (recordProcessRef.current.getEpisodeCount() || 1) - 1)
|
| 232 |
+
);
|
| 233 |
} catch (error) {
|
| 234 |
const errorMessage =
|
| 235 |
error instanceof Error ? error.message : "Failed to start recording";
|
|
|
|
| 242 |
};
|
| 243 |
|
| 244 |
const handleStopRecording = async () => {
|
| 245 |
+
if (!recordProcessRef.current || !isRecording) {
|
| 246 |
return;
|
| 247 |
}
|
| 248 |
|
| 249 |
try {
|
| 250 |
+
const result = await recordProcessRef.current.stop();
|
| 251 |
setIsRecording(false);
|
| 252 |
|
| 253 |
+
// No need to manually persist - episodes are managed by record process
|
| 254 |
+
|
| 255 |
+
// Force a small delay to ensure videoBlobs populated before export
|
| 256 |
+
await new Promise((r) => setTimeout(r, 50));
|
| 257 |
} catch (error) {
|
| 258 |
const errorMessage =
|
| 259 |
error instanceof Error ? error.message : "Failed to stop recording";
|
|
|
|
| 265 |
}
|
| 266 |
};
|
| 267 |
|
| 268 |
+
const handleDeleteEpisodes = async () => {
|
| 269 |
+
if (recordProcessRef.current) {
|
| 270 |
+
recordProcessRef.current.clearEpisodes();
|
|
|
|
| 271 |
}
|
| 272 |
|
| 273 |
+
// Clear persisted episodes only if not recording
|
| 274 |
+
if (!isRecording) {
|
| 275 |
+
setPersistedEpisodes([]);
|
| 276 |
+
}
|
| 277 |
|
| 278 |
+
setCurrentEpisode(0);
|
| 279 |
+
setHasRecordedFrames(isRecording); // Keep true if recording, false if not
|
| 280 |
+
setShowDeleteEpisodesDialog(false);
|
| 281 |
+
|
| 282 |
+
// No toast needed; dialog confirmation provides sufficient feedback
|
| 283 |
};
|
| 284 |
|
| 285 |
+
const handleNextEpisode = () => {
|
| 286 |
+
if (!isRecording || !recordProcessRef.current) {
|
| 287 |
+
return;
|
|
|
|
| 288 |
}
|
| 289 |
|
| 290 |
+
// Finalize current video segment and start a new one
|
| 291 |
+
recordProcessRef.current
|
| 292 |
+
.nextEpisode()
|
| 293 |
+
.then((newIndex: number) => setCurrentEpisode(newIndex))
|
| 294 |
+
.catch(() => {
|
| 295 |
+
/* noop */
|
|
|
|
| 296 |
});
|
| 297 |
+
};
|
| 298 |
+
|
| 299 |
+
// Force lightweight UI refresh while recording so episode table updates in near real-time
|
| 300 |
+
useEffect(() => {
|
| 301 |
+
if (!isRecording) return;
|
| 302 |
+
const id = setInterval(() => {
|
| 303 |
+
setUiTick((t) => (t + 1) % 1_000_000);
|
| 304 |
+
}, 500);
|
| 305 |
+
return () => clearInterval(id);
|
| 306 |
+
}, [isRecording]);
|
| 307 |
+
|
| 308 |
+
// (Removed) Reset frames button in favor of Delete Episodes with confirmation
|
| 309 |
|
| 310 |
// Load available cameras
|
| 311 |
const loadAvailableCameras = useCallback(
|
|
|
|
| 438 |
[previewStream, toast]
|
| 439 |
);
|
| 440 |
|
| 441 |
+
// Change camera source for an already-added camera card
|
| 442 |
+
const handleChangeCameraSourceForCard = useCallback(
|
| 443 |
+
async (cameraName: string, deviceId: string) => {
|
| 444 |
+
if (hasRecordedFrames) {
|
| 445 |
+
toast({
|
| 446 |
+
title: "Camera Error",
|
| 447 |
+
description:
|
| 448 |
+
"Cannot change camera source after recording has started",
|
| 449 |
+
variant: "destructive",
|
| 450 |
+
});
|
| 451 |
+
return;
|
| 452 |
+
}
|
| 453 |
+
|
| 454 |
+
try {
|
| 455 |
+
// Create new stream for selected device
|
| 456 |
+
const newStream = await navigator.mediaDevices.getUserMedia({
|
| 457 |
+
video: {
|
| 458 |
+
deviceId: { exact: deviceId },
|
| 459 |
+
width: { ideal: 1280 },
|
| 460 |
+
height: { ideal: 720 },
|
| 461 |
+
},
|
| 462 |
+
});
|
| 463 |
+
|
| 464 |
+
// Replace existing stream in additionalCameras
|
| 465 |
+
setAdditionalCameras((prev) => {
|
| 466 |
+
const next = { ...prev };
|
| 467 |
+
const old = next[cameraName];
|
| 468 |
+
if (old) {
|
| 469 |
+
old.getTracks().forEach((t) => t.stop());
|
| 470 |
+
}
|
| 471 |
+
next[cameraName] = newStream;
|
| 472 |
+
return next;
|
| 473 |
+
});
|
| 474 |
+
|
| 475 |
+
// Update persistent config
|
| 476 |
+
const selected = availableCameras.find((c) => c.deviceId === deviceId);
|
| 477 |
+
const newSettings = {
|
| 478 |
+
...recorderSettings,
|
| 479 |
+
cameraConfigs: { ...recorderSettings.cameraConfigs },
|
| 480 |
+
};
|
| 481 |
+
if (!newSettings.cameraConfigs[cameraName]) {
|
| 482 |
+
newSettings.cameraConfigs[cameraName] = {
|
| 483 |
+
deviceId,
|
| 484 |
+
deviceLabel: selected?.label || `Camera ${deviceId.slice(0, 8)}...`,
|
| 485 |
+
};
|
| 486 |
+
} else {
|
| 487 |
+
newSettings.cameraConfigs[cameraName].deviceId = deviceId;
|
| 488 |
+
newSettings.cameraConfigs[cameraName].deviceLabel =
|
| 489 |
+
selected?.label || `Camera ${deviceId.slice(0, 8)}...`;
|
| 490 |
+
}
|
| 491 |
+
setRecorderSettings(newSettings);
|
| 492 |
+
saveRecorderSettings(newSettings);
|
| 493 |
+
|
| 494 |
+
setSourceSelectorOpenFor(null);
|
| 495 |
+
toast({
|
| 496 |
+
title: "Camera Source Updated",
|
| 497 |
+
description: `\"${cameraName}\" now uses \"${
|
| 498 |
+
selected?.label || deviceId
|
| 499 |
+
}\"`,
|
| 500 |
+
});
|
| 501 |
+
} catch (error) {
|
| 502 |
+
toast({
|
| 503 |
+
title: "Camera Error",
|
| 504 |
+
description: `Failed to switch camera: ${
|
| 505 |
+
error instanceof Error ? error.message : String(error)
|
| 506 |
+
}`,
|
| 507 |
+
variant: "destructive",
|
| 508 |
+
});
|
| 509 |
+
}
|
| 510 |
+
},
|
| 511 |
+
[hasRecordedFrames, availableCameras, recorderSettings, toast]
|
| 512 |
+
);
|
| 513 |
+
|
| 514 |
// Add a new camera to the recorder
|
| 515 |
const handleAddCamera = useCallback(async () => {
|
| 516 |
if (!cameraName.trim()) {
|
|
|
|
| 758 |
}, [availableCameras, cameraPermissionState, restoreSavedCameras]);
|
| 759 |
|
| 760 |
const handleDownloadZip = async () => {
|
| 761 |
+
if (!recordProcessRef.current) return;
|
| 762 |
+
try {
|
| 763 |
+
if (isRecording) {
|
| 764 |
+
await handleStopRecording();
|
| 765 |
+
}
|
| 766 |
+
await recordProcessRef.current.exportForLeRobot("zip-download");
|
| 767 |
+
} catch (e) {
|
| 768 |
toast({
|
| 769 |
+
title: "Export Error",
|
| 770 |
+
description:
|
| 771 |
+
e instanceof Error ? e.message : "Failed to export the dataset",
|
| 772 |
variant: "destructive",
|
| 773 |
});
|
| 774 |
return;
|
| 775 |
}
|
| 776 |
+
// No toast; the browser download prompt is sufficient feedback
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 777 |
};
|
| 778 |
|
| 779 |
const handleUploadToHuggingFace = async () => {
|
| 780 |
+
if (!recordProcessRef.current) return;
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 781 |
|
| 782 |
if (!recorderSettings.huggingfaceApiKey) {
|
| 783 |
toast({
|
|
|
|
| 789 |
}
|
| 790 |
|
| 791 |
try {
|
| 792 |
+
// Use provided repo name or generate one
|
| 793 |
+
const repoName =
|
| 794 |
+
(recorderSettings.huggingfaceRepoName || "").trim() ||
|
| 795 |
+
`lerobot-recording-${Date.now()}`;
|
| 796 |
+
|
| 797 |
+
// Get blobs from recorder
|
| 798 |
+
const blobArray = await recordProcessRef.current.exportForLeRobot(
|
| 799 |
+
"blobs"
|
| 800 |
+
);
|
| 801 |
|
| 802 |
+
// Upload using demo utility
|
| 803 |
+
const uploader = await uploadToHuggingFace(
|
| 804 |
+
blobArray,
|
| 805 |
+
recorderSettings.huggingfaceApiKey,
|
| 806 |
+
repoName,
|
| 807 |
+
!!recorderSettings.huggingfacePrivate
|
| 808 |
);
|
| 809 |
|
| 810 |
+
// Progress UI next to button
|
| 811 |
+
setUploadState({ status: "uploading" });
|
| 812 |
+
const onProgress = (_event: Event) => {
|
| 813 |
+
// Spinner-only UI; keep uploading state
|
| 814 |
+
setUploadState({ status: "uploading" });
|
| 815 |
+
};
|
| 816 |
+
const onFinished = () => setUploadState({ status: "done" });
|
| 817 |
+
const onError = () => setUploadState({ status: "idle" });
|
| 818 |
+
uploader.addEventListener("progress", onProgress);
|
| 819 |
+
uploader.addEventListener("finished", onFinished);
|
| 820 |
+
uploader.addEventListener("error", onError);
|
| 821 |
+
// No fake progress; spinner indicates activity
|
| 822 |
} catch (error) {
|
| 823 |
const errorMessage =
|
| 824 |
error instanceof Error
|
|
|
|
| 832 |
}
|
| 833 |
};
|
| 834 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 835 |
return (
|
| 836 |
+
<div className="space-y-6">
|
| 837 |
+
{/* Recorder Settings - Toggleable Inline */}
|
| 838 |
+
{showConfigure && (
|
| 839 |
+
<div className="space-y-6">
|
| 840 |
+
{/* Hugging Face Settings */}
|
| 841 |
+
<div className="space-y-3">
|
| 842 |
+
<h3 className="text-lg font-semibold text-foreground">Settings</h3>
|
| 843 |
+
<div className="grid grid-cols-1 md:grid-cols-2 gap-4">
|
| 844 |
+
<div className="space-y-2">
|
| 845 |
+
<label className="text-sm text-muted-foreground">
|
| 846 |
+
Hugging Face API Key
|
| 847 |
+
</label>
|
| 848 |
+
<Input
|
| 849 |
+
placeholder="Enter your Hugging Face API key"
|
| 850 |
+
value={recorderSettings.huggingfaceApiKey}
|
| 851 |
+
onChange={(e) => {
|
| 852 |
+
const newSettings = {
|
| 853 |
+
...recorderSettings,
|
| 854 |
+
huggingfaceApiKey: e.target.value,
|
| 855 |
+
};
|
| 856 |
+
setRecorderSettings(newSettings);
|
| 857 |
+
saveRecorderSettings(newSettings);
|
| 858 |
+
}}
|
| 859 |
+
type="password"
|
| 860 |
+
className="bg-black/20 border-white/10"
|
| 861 |
+
/>
|
| 862 |
+
<p className="text-xs text-white/50">
|
| 863 |
+
Required to upload datasets to Hugging Face Hub
|
| 864 |
+
</p>
|
| 865 |
+
</div>
|
| 866 |
+
<div className="space-y-2">
|
| 867 |
+
<label className="text-sm text-muted-foreground">
|
| 868 |
+
Hugging Face Repo Name
|
| 869 |
+
</label>
|
| 870 |
+
<Input
|
| 871 |
+
placeholder="e.g. my-dataset-name"
|
| 872 |
+
value={recorderSettings.huggingfaceRepoName || ""}
|
| 873 |
+
onChange={(e) => {
|
| 874 |
+
const newSettings = {
|
| 875 |
+
...recorderSettings,
|
| 876 |
+
huggingfaceRepoName: e.target.value,
|
| 877 |
+
};
|
| 878 |
+
setRecorderSettings(newSettings);
|
| 879 |
+
saveRecorderSettings(newSettings);
|
| 880 |
+
}}
|
| 881 |
+
className="bg-black/20 border-white/10"
|
| 882 |
+
/>
|
| 883 |
+
<label className="flex items-center gap-2 text-sm text-muted-foreground">
|
| 884 |
+
<input
|
| 885 |
+
type="checkbox"
|
| 886 |
+
className="accent-white"
|
| 887 |
+
checked={!!recorderSettings.huggingfacePrivate}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 888 |
onChange={(e) => {
|
| 889 |
const newSettings = {
|
| 890 |
...recorderSettings,
|
| 891 |
+
huggingfacePrivate: e.target.checked,
|
| 892 |
};
|
| 893 |
setRecorderSettings(newSettings);
|
| 894 |
saveRecorderSettings(newSettings);
|
| 895 |
}}
|
|
|
|
|
|
|
| 896 |
/>
|
| 897 |
+
Private repository
|
| 898 |
+
</label>
|
|
|
|
|
|
|
| 899 |
</div>
|
| 900 |
</div>
|
| 901 |
+
</div>
|
| 902 |
|
| 903 |
+
{/* Camera Configuration */}
|
| 904 |
+
<div className="space-y-4">
|
| 905 |
+
<h3 className="text-lg font-semibold text-foreground">
|
| 906 |
+
Camera Setup
|
| 907 |
+
</h3>
|
| 908 |
+
<div className="bg-black/40 border border-white/20 p-6">
|
| 909 |
+
<div className="grid grid-cols-1 lg:grid-cols-2 gap-6">
|
| 910 |
+
{/* Left Column: Camera Selection & Adding */}
|
| 911 |
+
<div className="space-y-4">
|
| 912 |
+
{/* Camera Selection and Refresh */}
|
| 913 |
+
|
| 914 |
+
{/* Camera Access Request */}
|
| 915 |
+
{cameraPermissionState === "unknown" && (
|
| 916 |
+
<div className="space-y-3">
|
| 917 |
+
<div className="bg-black/60 border border-white/20 rounded-lg p-4 text-center">
|
| 918 |
+
<Camera className="w-8 h-8 mx-auto mb-2 opacity-50" />
|
| 919 |
+
<p className="text-sm text-white/70 mb-3">
|
| 920 |
+
Camera access needed to configure cameras
|
| 921 |
+
</p>
|
| 922 |
+
<Button
|
| 923 |
+
onClick={() => loadAvailableCameras(false)}
|
| 924 |
+
variant="outline"
|
| 925 |
+
className="gap-2"
|
| 926 |
+
disabled={isLoadingCameras}
|
| 927 |
+
>
|
| 928 |
+
<Camera className="w-4 h-4" />
|
| 929 |
+
{isLoadingCameras
|
| 930 |
+
? "Loading..."
|
| 931 |
+
: "Request Camera Access"}
|
| 932 |
+
</Button>
|
|
|
|
| 933 |
</div>
|
| 934 |
+
</div>
|
| 935 |
+
)}
|
| 936 |
+
|
| 937 |
+
{/* Camera Access Denied */}
|
| 938 |
+
{cameraPermissionState === "denied" && (
|
| 939 |
+
<div className="space-y-3">
|
| 940 |
+
<div className="bg-red-900/20 border border-red-500/20 rounded-lg p-4 text-center">
|
| 941 |
+
<Camera className="w-8 h-8 mx-auto mb-2 opacity-50 text-red-400" />
|
| 942 |
+
<p className="text-sm text-red-300 mb-1">
|
| 943 |
+
Camera access denied
|
| 944 |
+
</p>
|
| 945 |
+
<p className="text-xs text-red-400">
|
| 946 |
+
Please allow camera access in your browser settings
|
| 947 |
+
and refresh
|
| 948 |
+
</p>
|
| 949 |
</div>
|
| 950 |
+
</div>
|
| 951 |
+
)}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 952 |
|
| 953 |
+
{/* Camera List with Refresh Button */}
|
| 954 |
+
{cameraPermissionState === "granted" &&
|
| 955 |
+
availableCameras.length > 0 && (
|
| 956 |
<div className="space-y-2">
|
| 957 |
<label className="text-sm text-white/70">
|
| 958 |
+
Select Camera:
|
| 959 |
</label>
|
| 960 |
+
<div className="flex items-center gap-2">
|
| 961 |
+
<Select
|
| 962 |
+
value={selectedCameraId}
|
| 963 |
+
onValueChange={switchCameraPreview}
|
| 964 |
+
disabled={hasRecordedFrames}
|
| 965 |
+
>
|
| 966 |
+
<SelectTrigger className="flex-1 bg-black/20 border-white/10">
|
| 967 |
+
<SelectValue placeholder="Choose a camera" />
|
| 968 |
+
</SelectTrigger>
|
| 969 |
+
<SelectContent>
|
| 970 |
+
{availableCameras.map((camera) => (
|
| 971 |
+
<SelectItem
|
| 972 |
+
key={camera.deviceId}
|
| 973 |
+
value={camera.deviceId}
|
| 974 |
+
>
|
| 975 |
+
{camera.label ||
|
| 976 |
+
`Camera ${camera.deviceId.slice(0, 8)}...`}
|
| 977 |
+
</SelectItem>
|
| 978 |
+
))}
|
| 979 |
+
</SelectContent>
|
| 980 |
+
</Select>
|
| 981 |
+
<Button
|
| 982 |
+
onClick={() => loadAvailableCameras(false)}
|
| 983 |
+
variant="ghost"
|
| 984 |
+
size="sm"
|
| 985 |
+
className="gap-2 text-white/70 hover:text-white"
|
| 986 |
+
disabled={isLoadingCameras}
|
| 987 |
+
>
|
| 988 |
+
<RefreshCw
|
| 989 |
+
className={`w-4 h-4 ${
|
| 990 |
+
isLoadingCameras ? "animate-spin" : ""
|
| 991 |
+
}`}
|
| 992 |
+
/>
|
| 993 |
+
Refresh
|
| 994 |
+
</Button>
|
| 995 |
+
</div>
|
| 996 |
</div>
|
| 997 |
)}
|
|
|
|
| 998 |
|
| 999 |
+
{/* Camera Name Input */}
|
| 1000 |
+
{selectedCameraId && (
|
| 1001 |
+
<div className="space-y-2">
|
| 1002 |
+
<label className="text-sm text-white/70">
|
| 1003 |
+
Camera Name:
|
| 1004 |
+
</label>
|
| 1005 |
+
<Input
|
| 1006 |
+
placeholder="e.g., 'Overhead View', 'Side Angle', 'Close-up'"
|
| 1007 |
+
value={cameraName}
|
| 1008 |
+
onChange={(e) => setCameraName(e.target.value)}
|
| 1009 |
+
className="bg-black/20 border-white/10"
|
| 1010 |
+
disabled={hasRecordedFrames}
|
| 1011 |
+
/>
|
| 1012 |
+
<p className="text-xs text-white/50">
|
| 1013 |
+
Give this camera a descriptive name for your recording
|
| 1014 |
+
setup
|
| 1015 |
+
</p>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1016 |
</div>
|
| 1017 |
+
)}
|
| 1018 |
+
|
| 1019 |
+
{/* Add Camera Button */}
|
| 1020 |
+
{selectedCameraId && (
|
| 1021 |
+
<div className="flex justify-end">
|
| 1022 |
+
<Button
|
| 1023 |
+
onClick={handleAddCamera}
|
| 1024 |
+
className="gap-2"
|
| 1025 |
+
disabled={
|
| 1026 |
+
hasRecordedFrames ||
|
| 1027 |
+
!cameraName.trim() ||
|
| 1028 |
+
!selectedCameraId ||
|
| 1029 |
+
!previewStream
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1030 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1031 |
>
|
| 1032 |
+
<PlusCircle className="w-4 h-4" />
|
| 1033 |
+
Add Camera to Recorder
|
| 1034 |
+
</Button>
|
| 1035 |
+
</div>
|
| 1036 |
+
)}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1037 |
</div>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1038 |
|
| 1039 |
+
{/* Right Column: Camera Preview */}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1040 |
<div className="space-y-4">
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1041 |
<div className="aspect-video bg-black/60 border border-white/20 rounded-lg overflow-hidden">
|
| 1042 |
{previewStream ? (
|
| 1043 |
<video
|
|
|
|
| 1051 |
<div className="w-full h-full flex items-center justify-center text-white/60">
|
| 1052 |
<div className="text-center">
|
| 1053 |
<Camera className="w-12 h-12 mx-auto mb-2 opacity-50" />
|
| 1054 |
+
{cameraPermissionState === "unknown" ? (
|
| 1055 |
+
<p className="text-sm">
|
| 1056 |
+
Request camera access to preview
|
| 1057 |
+
</p>
|
| 1058 |
) : cameraPermissionState === "denied" ? (
|
| 1059 |
+
<p className="text-sm">Camera access denied</p>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1060 |
) : availableCameras.length === 0 ? (
|
| 1061 |
+
<p className="text-sm">No cameras available</p>
|
| 1062 |
+
) : !selectedCameraId ? (
|
| 1063 |
+
<p className="text-sm">
|
| 1064 |
+
Select a camera to preview
|
| 1065 |
+
</p>
|
| 1066 |
) : (
|
| 1067 |
+
<p className="text-sm">Loading preview...</p>
|
| 1068 |
)}
|
| 1069 |
</div>
|
| 1070 |
</div>
|
| 1071 |
)}
|
| 1072 |
</div>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1073 |
</div>
|
| 1074 |
+
</div>
|
| 1075 |
+
</div>
|
| 1076 |
+
</div>
|
| 1077 |
+
</div>
|
| 1078 |
+
)}
|
| 1079 |
+
|
| 1080 |
+
{/* Added Camera Previews */}
|
| 1081 |
+
{Object.keys(additionalCameras).length > 0 && (
|
| 1082 |
+
<div className="space-y-4">
|
| 1083 |
+
<h3 className="text-lg font-semibold text-foreground">
|
| 1084 |
+
Active Cameras
|
| 1085 |
+
</h3>
|
| 1086 |
+
<div className="grid grid-cols-2 md:grid-cols-3 lg:grid-cols-4 gap-4">
|
| 1087 |
+
{Object.entries(additionalCameras).map(([cameraName, stream]) => (
|
| 1088 |
+
<div
|
| 1089 |
+
key={cameraName}
|
| 1090 |
+
className="bg-black/40 border border-white/20 rounded-lg p-3 space-y-2"
|
| 1091 |
+
>
|
| 1092 |
+
<div className="aspect-video bg-black/60 border border-white/10 rounded overflow-hidden">
|
| 1093 |
+
<video
|
| 1094 |
+
autoPlay
|
| 1095 |
+
muted
|
| 1096 |
+
playsInline
|
| 1097 |
+
className="w-full h-full object-cover"
|
| 1098 |
+
ref={(video) => {
|
| 1099 |
+
if (video && stream) {
|
| 1100 |
+
const current = video.srcObject as MediaStream | null;
|
| 1101 |
+
if (current !== stream) {
|
| 1102 |
+
video.srcObject = stream;
|
| 1103 |
+
}
|
| 1104 |
+
}
|
| 1105 |
+
}}
|
| 1106 |
+
/>
|
| 1107 |
+
</div>
|
| 1108 |
+
<div className="flex items-center justify-between">
|
| 1109 |
+
{editingCameraName === cameraName ? (
|
| 1110 |
+
<div className="flex items-center gap-1 flex-1">
|
| 1111 |
<Input
|
| 1112 |
+
value={editingCameraNewName}
|
| 1113 |
+
onChange={(e) =>
|
| 1114 |
+
setEditingCameraNewName(e.target.value)
|
| 1115 |
+
}
|
| 1116 |
+
className="text-xs h-6 bg-black/20 border-white/10"
|
| 1117 |
+
onKeyDown={(e) => {
|
| 1118 |
+
if (e.key === "Enter") {
|
| 1119 |
+
handleConfirmCameraNameEdit(cameraName);
|
| 1120 |
+
} else if (e.key === "Escape") {
|
| 1121 |
+
handleCancelCameraNameEdit();
|
| 1122 |
+
}
|
| 1123 |
+
}}
|
| 1124 |
+
autoFocus
|
| 1125 |
/>
|
| 1126 |
+
<button
|
| 1127 |
+
onClick={() => handleConfirmCameraNameEdit(cameraName)}
|
| 1128 |
+
className="text-green-400 hover:text-green-300 p-1"
|
| 1129 |
+
>
|
| 1130 |
+
<Check className="w-3 h-3" />
|
| 1131 |
+
</button>
|
| 1132 |
+
<button
|
| 1133 |
+
onClick={handleCancelCameraNameEdit}
|
| 1134 |
+
className="text-red-400 hover:text-red-300 p-1"
|
| 1135 |
+
>
|
| 1136 |
+
<X className="w-3 h-3" />
|
| 1137 |
+
</button>
|
| 1138 |
</div>
|
| 1139 |
+
) : (
|
| 1140 |
+
<button
|
| 1141 |
+
onClick={() => handleStartEditingCameraName(cameraName)}
|
| 1142 |
+
className="text-sm font-medium text-white/90 truncate hover:text-white cursor-pointer flex items-center gap-1 flex-1"
|
| 1143 |
+
disabled={hasRecordedFrames}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1144 |
>
|
| 1145 |
+
{cameraName}
|
| 1146 |
+
<Edit2 className="w-3 h-3 opacity-50" />
|
| 1147 |
+
</button>
|
| 1148 |
+
)}
|
| 1149 |
+
<div className="flex items-center gap-2 ml-2">
|
| 1150 |
+
{/* Inline camera source selector trigger */}
|
| 1151 |
+
<div className="relative">
|
| 1152 |
+
<button
|
| 1153 |
+
onClick={() =>
|
| 1154 |
+
setSourceSelectorOpenFor((prev) =>
|
| 1155 |
+
prev === cameraName ? null : cameraName
|
| 1156 |
+
)
|
| 1157 |
+
}
|
| 1158 |
+
className="text-white/80 hover:text-white p-1"
|
| 1159 |
+
disabled={hasRecordedFrames}
|
| 1160 |
+
title="Change camera source"
|
| 1161 |
+
>
|
| 1162 |
+
<Camera className="w-4 h-4" />
|
| 1163 |
+
</button>
|
| 1164 |
+
{sourceSelectorOpenFor === cameraName && (
|
| 1165 |
+
<div className="absolute right-0 mt-1 z-20 bg-black/90 border border-white/20 rounded p-2 w-64">
|
| 1166 |
+
<label className="text-xs text-white/70">
|
| 1167 |
+
Camera Source
|
| 1168 |
+
</label>
|
| 1169 |
+
<Select
|
| 1170 |
+
value={
|
| 1171 |
+
recorderSettings.cameraConfigs[cameraName]
|
| 1172 |
+
?.deviceId || ""
|
| 1173 |
+
}
|
| 1174 |
+
onValueChange={(deviceId) =>
|
| 1175 |
+
handleChangeCameraSourceForCard(
|
| 1176 |
+
cameraName,
|
| 1177 |
+
deviceId
|
| 1178 |
+
)
|
| 1179 |
+
}
|
| 1180 |
+
disabled={hasRecordedFrames}
|
| 1181 |
+
>
|
| 1182 |
+
<SelectTrigger className="w-full h-8 bg-black/20 border-white/10">
|
| 1183 |
+
<SelectValue placeholder="Choose a camera" />
|
| 1184 |
+
</SelectTrigger>
|
| 1185 |
+
<SelectContent>
|
| 1186 |
+
{availableCameras.map((cam) => (
|
| 1187 |
+
<SelectItem
|
| 1188 |
+
key={cam.deviceId}
|
| 1189 |
+
value={cam.deviceId}
|
| 1190 |
+
>
|
| 1191 |
+
{cam.label ||
|
| 1192 |
+
`Camera ${cam.deviceId.slice(0, 8)}...`}
|
| 1193 |
+
</SelectItem>
|
| 1194 |
+
))}
|
| 1195 |
+
</SelectContent>
|
| 1196 |
+
</Select>
|
| 1197 |
+
</div>
|
| 1198 |
+
)}
|
| 1199 |
+
</div>
|
| 1200 |
<button
|
| 1201 |
+
onClick={() => handleRemoveCamera(cameraName)}
|
| 1202 |
+
className="text-red-400 hover:text-red-300 p-1"
|
| 1203 |
disabled={hasRecordedFrames}
|
| 1204 |
+
title="Remove camera"
|
| 1205 |
>
|
| 1206 |
+
<X className="w-4 h-4" />
|
| 1207 |
</button>
|
| 1208 |
+
</div>
|
| 1209 |
+
</div>
|
| 1210 |
</div>
|
| 1211 |
+
))}
|
| 1212 |
+
</div>
|
| 1213 |
+
</div>
|
| 1214 |
+
)}
|
| 1215 |
+
|
| 1216 |
+
{/* Episode Management & Dataset Actions */}
|
| 1217 |
+
<div className="flex justify-between items-center">
|
| 1218 |
+
<div className="flex items-center gap-2">
|
| 1219 |
+
<Button
|
| 1220 |
+
variant="outline"
|
| 1221 |
+
className="gap-2"
|
| 1222 |
+
onClick={handleNextEpisode}
|
| 1223 |
+
disabled={!isRecording}
|
| 1224 |
+
>
|
| 1225 |
+
<PlusCircle className="w-4 h-4" />
|
| 1226 |
+
Next Episode
|
| 1227 |
+
</Button>
|
| 1228 |
+
<Button
|
| 1229 |
+
variant="outline"
|
| 1230 |
+
className="gap-2"
|
| 1231 |
+
onClick={() => setShowDeleteEpisodesDialog(true)}
|
| 1232 |
+
disabled={
|
| 1233 |
+
persistedEpisodes.length === 0 &&
|
| 1234 |
+
(recordProcessRef.current?.getEpisodeCount() || 0) === 0
|
| 1235 |
+
}
|
| 1236 |
+
>
|
| 1237 |
+
<Trash2 className="w-4 h-4" />
|
| 1238 |
+
Delete Episodes
|
| 1239 |
+
</Button>
|
| 1240 |
</div>
|
| 1241 |
|
| 1242 |
+
<div className="flex items-center gap-2">
|
| 1243 |
+
<Button
|
| 1244 |
+
variant="outline"
|
| 1245 |
+
className="gap-2"
|
| 1246 |
+
onClick={handleDownloadZip}
|
| 1247 |
+
disabled={
|
| 1248 |
+
(recordProcessRef.current?.getEpisodeCount() || 0) === 0 ||
|
| 1249 |
+
isRecording
|
| 1250 |
+
}
|
| 1251 |
+
>
|
| 1252 |
+
<Download className="w-4 h-4" />
|
| 1253 |
+
Download as ZIP
|
| 1254 |
+
</Button>
|
| 1255 |
+
<Button
|
| 1256 |
+
variant="outline"
|
| 1257 |
+
className="gap-2 relative"
|
| 1258 |
+
onClick={handleUploadToHuggingFace}
|
| 1259 |
+
disabled={
|
| 1260 |
+
(recordProcessRef.current?.getEpisodeCount() || 0) === 0 ||
|
| 1261 |
+
isRecording ||
|
| 1262 |
+
!recorderSettings.huggingfaceApiKey ||
|
| 1263 |
+
uploadState.status === "uploading"
|
| 1264 |
+
}
|
| 1265 |
+
>
|
| 1266 |
+
<Upload className="w-4 h-4" />
|
| 1267 |
+
{uploadState.status === "uploading" ? (
|
| 1268 |
+
<span className="inline-flex items-center gap-2">
|
| 1269 |
+
Uploading…
|
| 1270 |
+
<span className="inline-block w-4 h-4 border-2 border-white/50 border-t-white rounded-full animate-spin" />
|
| 1271 |
+
</span>
|
| 1272 |
+
) : uploadState.status === "done" ? (
|
| 1273 |
+
<span>Uploaded ✓</span>
|
| 1274 |
+
) : (
|
| 1275 |
+
<span>Upload to Hugging Face</span>
|
| 1276 |
+
)}
|
| 1277 |
+
</Button>
|
| 1278 |
+
</div>
|
| 1279 |
+
</div>
|
| 1280 |
|
| 1281 |
+
<div className="border border-white/10 rounded-md overflow-hidden">
|
| 1282 |
+
<TeleoperatorEpisodesView
|
| 1283 |
+
teleoperatorData={
|
| 1284 |
+
(uiTick,
|
| 1285 |
+
recordProcessRef.current?.getEpisodes() || persistedEpisodes)
|
| 1286 |
+
}
|
| 1287 |
+
isRecording={isRecording}
|
| 1288 |
+
refreshTick={uiTick}
|
| 1289 |
+
/>
|
| 1290 |
+
</div>
|
| 1291 |
|
| 1292 |
+
{/* Delete Episodes Confirmation Dialog */}
|
| 1293 |
+
<AlertDialog
|
| 1294 |
+
open={showDeleteEpisodesDialog}
|
| 1295 |
+
onOpenChange={setShowDeleteEpisodesDialog}
|
| 1296 |
+
>
|
| 1297 |
+
<AlertDialogContent>
|
| 1298 |
+
<AlertDialogHeader>
|
| 1299 |
+
<AlertDialogTitle>Delete All Episodes</AlertDialogTitle>
|
| 1300 |
+
<AlertDialogDescription>
|
| 1301 |
+
Are you sure you want to delete all recorded episodes? This action
|
| 1302 |
+
cannot be undone.
|
| 1303 |
+
</AlertDialogDescription>
|
| 1304 |
+
</AlertDialogHeader>
|
| 1305 |
+
<AlertDialogFooter>
|
| 1306 |
+
<AlertDialogCancel>Cancel</AlertDialogCancel>
|
| 1307 |
+
<AlertDialogAction
|
| 1308 |
+
onClick={handleDeleteEpisodes}
|
| 1309 |
+
className="bg-destructive text-destructive-foreground hover:bg-destructive/90"
|
| 1310 |
>
|
| 1311 |
+
Delete Episodes
|
| 1312 |
+
</AlertDialogAction>
|
| 1313 |
+
</AlertDialogFooter>
|
| 1314 |
+
</AlertDialogContent>
|
| 1315 |
+
</AlertDialog>
|
| 1316 |
+
</div>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1317 |
);
|
| 1318 |
}
|
| 1319 |
+
|
| 1320 |
+
export const MemoRecorder = memo(Recorder);
|
|
@@ -0,0 +1,598 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"use client";
|
| 2 |
+
|
| 3 |
+
import {
|
| 4 |
+
useState,
|
| 5 |
+
useEffect,
|
| 6 |
+
useMemo,
|
| 7 |
+
useRef,
|
| 8 |
+
useCallback,
|
| 9 |
+
startTransition,
|
| 10 |
+
} from "react";
|
| 11 |
+
import { Card } from "@/components/ui/card";
|
| 12 |
+
import { Badge } from "@/components/ui/badge";
|
| 13 |
+
import { Button } from "@/components/ui/button";
|
| 14 |
+
import { cn } from "@/lib/utils";
|
| 15 |
+
import {
|
| 16 |
+
Gamepad2,
|
| 17 |
+
Keyboard,
|
| 18 |
+
Settings,
|
| 19 |
+
Square,
|
| 20 |
+
Disc as Record,
|
| 21 |
+
Check,
|
| 22 |
+
} from "lucide-react";
|
| 23 |
+
import {
|
| 24 |
+
Select,
|
| 25 |
+
SelectContent,
|
| 26 |
+
SelectItem,
|
| 27 |
+
SelectTrigger,
|
| 28 |
+
SelectValue,
|
| 29 |
+
} from "@/components/ui/select";
|
| 30 |
+
import {
|
| 31 |
+
teleoperate,
|
| 32 |
+
type TeleoperationProcess,
|
| 33 |
+
type TeleoperationState,
|
| 34 |
+
type TeleoperateConfig,
|
| 35 |
+
type RobotConnection,
|
| 36 |
+
} from "@lerobot/web";
|
| 37 |
+
import { getUnifiedRobotData } from "@/lib/unified-storage";
|
| 38 |
+
import { MemoRecorder as Recorder } from "@/components/recorder";
|
| 39 |
+
|
| 40 |
+
const UI_UPDATE_INTERVAL_MS = 100;
|
| 41 |
+
|
| 42 |
+
interface RecordingViewProps {
|
| 43 |
+
robot: RobotConnection;
|
| 44 |
+
}
|
| 45 |
+
|
| 46 |
+
export function RecordingView({ robot }: RecordingViewProps) {
|
| 47 |
+
const [teleopState, setTeleopState] = useState<TeleoperationState>({
|
| 48 |
+
isActive: false,
|
| 49 |
+
motorConfigs: [],
|
| 50 |
+
lastUpdate: 0,
|
| 51 |
+
keyStates: {},
|
| 52 |
+
});
|
| 53 |
+
|
| 54 |
+
const [isInitialized, setIsInitialized] = useState(false);
|
| 55 |
+
const [controlEnabled, setControlEnabled] = useState(false);
|
| 56 |
+
const [selectedTeleoperatorType, setSelectedTeleoperatorType] = useState<
|
| 57 |
+
"direct" | "keyboard"
|
| 58 |
+
>("keyboard");
|
| 59 |
+
const [showConfigure, setShowConfigure] = useState(false);
|
| 60 |
+
const [recorderCallbacks, setRecorderCallbacks] = useState<{
|
| 61 |
+
startRecording: () => Promise<void>;
|
| 62 |
+
stopRecording: () => Promise<void>;
|
| 63 |
+
isRecording: boolean;
|
| 64 |
+
} | null>(null);
|
| 65 |
+
|
| 66 |
+
// Current step tracking
|
| 67 |
+
const currentStep = useMemo(() => {
|
| 68 |
+
if (!robot.isConnected) return 0; // No steps active if robot offline
|
| 69 |
+
if (!controlEnabled) return 1; // Step 1: Enable Control
|
| 70 |
+
if (!recorderCallbacks?.isRecording) return 2; // Step 2: Start Recording
|
| 71 |
+
return 3; // Step 3: Move Robot (recording active)
|
| 72 |
+
}, [robot.isConnected, controlEnabled, recorderCallbacks?.isRecording]);
|
| 73 |
+
const keyboardProcessRef = useRef<TeleoperationProcess | null>(null);
|
| 74 |
+
const directProcessRef = useRef<TeleoperationProcess | null>(null);
|
| 75 |
+
const lastUiUpdateRef = useRef(0);
|
| 76 |
+
|
| 77 |
+
// Load calibration data from unified storage
|
| 78 |
+
const calibrationData = useMemo(() => {
|
| 79 |
+
if (!robot.serialNumber) return undefined;
|
| 80 |
+
|
| 81 |
+
const data = getUnifiedRobotData(robot.serialNumber);
|
| 82 |
+
if (data?.calibration) {
|
| 83 |
+
return data.calibration;
|
| 84 |
+
}
|
| 85 |
+
|
| 86 |
+
// Return undefined if no calibration data - let library handle defaults
|
| 87 |
+
return undefined;
|
| 88 |
+
}, [robot.serialNumber]);
|
| 89 |
+
|
| 90 |
+
// Initialize teleoperation for recording
|
| 91 |
+
const initializeTeleoperation = useCallback(async () => {
|
| 92 |
+
if (!robot || !robot.robotType) {
|
| 93 |
+
return false;
|
| 94 |
+
}
|
| 95 |
+
|
| 96 |
+
try {
|
| 97 |
+
// Create keyboard teleoperation process
|
| 98 |
+
const keyboardConfig: TeleoperateConfig = {
|
| 99 |
+
robot: robot,
|
| 100 |
+
teleop: {
|
| 101 |
+
type: "keyboard",
|
| 102 |
+
},
|
| 103 |
+
calibrationData,
|
| 104 |
+
onStateUpdate: (state: TeleoperationState) => {
|
| 105 |
+
const now = performance.now();
|
| 106 |
+
if (now - lastUiUpdateRef.current >= UI_UPDATE_INTERVAL_MS) {
|
| 107 |
+
lastUiUpdateRef.current = now;
|
| 108 |
+
startTransition(() => setTeleopState(state));
|
| 109 |
+
}
|
| 110 |
+
},
|
| 111 |
+
};
|
| 112 |
+
const keyboardProcess = await teleoperate(keyboardConfig);
|
| 113 |
+
|
| 114 |
+
// Create direct teleoperation process
|
| 115 |
+
const directConfig: TeleoperateConfig = {
|
| 116 |
+
robot: robot,
|
| 117 |
+
teleop: {
|
| 118 |
+
type: "direct",
|
| 119 |
+
},
|
| 120 |
+
calibrationData,
|
| 121 |
+
onStateUpdate: (state: TeleoperationState) => {
|
| 122 |
+
const now = performance.now();
|
| 123 |
+
if (now - lastUiUpdateRef.current >= UI_UPDATE_INTERVAL_MS) {
|
| 124 |
+
lastUiUpdateRef.current = now;
|
| 125 |
+
startTransition(() => setTeleopState(state));
|
| 126 |
+
}
|
| 127 |
+
},
|
| 128 |
+
};
|
| 129 |
+
const directProcess = await teleoperate(directConfig);
|
| 130 |
+
|
| 131 |
+
keyboardProcessRef.current = keyboardProcess;
|
| 132 |
+
directProcessRef.current = directProcess;
|
| 133 |
+
setTeleopState(directProcess.getState());
|
| 134 |
+
|
| 135 |
+
setIsInitialized(true);
|
| 136 |
+
|
| 137 |
+
return true;
|
| 138 |
+
} catch (error) {
|
| 139 |
+
const errorMessage =
|
| 140 |
+
error instanceof Error
|
| 141 |
+
? error.message
|
| 142 |
+
: "Failed to initialize teleoperation for recording";
|
| 143 |
+
toast({
|
| 144 |
+
title: "Teleoperation Error",
|
| 145 |
+
description: errorMessage,
|
| 146 |
+
variant: "destructive",
|
| 147 |
+
});
|
| 148 |
+
return false;
|
| 149 |
+
}
|
| 150 |
+
}, [robot, robot.robotType, calibrationData]);
|
| 151 |
+
|
| 152 |
+
// Enable robot control for recording
|
| 153 |
+
const enableControl = useCallback(async () => {
|
| 154 |
+
if (!robot.isConnected) {
|
| 155 |
+
return false;
|
| 156 |
+
}
|
| 157 |
+
|
| 158 |
+
const success = await initializeTeleoperation();
|
| 159 |
+
if (success) {
|
| 160 |
+
// Start the appropriate teleoperator based on selection
|
| 161 |
+
if (
|
| 162 |
+
selectedTeleoperatorType === "keyboard" &&
|
| 163 |
+
keyboardProcessRef.current
|
| 164 |
+
) {
|
| 165 |
+
keyboardProcessRef.current.start();
|
| 166 |
+
} else if (
|
| 167 |
+
selectedTeleoperatorType === "direct" &&
|
| 168 |
+
directProcessRef.current
|
| 169 |
+
) {
|
| 170 |
+
directProcessRef.current.start();
|
| 171 |
+
}
|
| 172 |
+
|
| 173 |
+
setControlEnabled(true);
|
| 174 |
+
return true;
|
| 175 |
+
}
|
| 176 |
+
return false;
|
| 177 |
+
}, [robot.isConnected, initializeTeleoperation, selectedTeleoperatorType]);
|
| 178 |
+
|
| 179 |
+
// Disable robot control
|
| 180 |
+
const disableControl = useCallback(async () => {
|
| 181 |
+
if (keyboardProcessRef.current) {
|
| 182 |
+
keyboardProcessRef.current.stop();
|
| 183 |
+
}
|
| 184 |
+
if (directProcessRef.current) {
|
| 185 |
+
directProcessRef.current.stop();
|
| 186 |
+
}
|
| 187 |
+
setControlEnabled(false);
|
| 188 |
+
}, [selectedTeleoperatorType]);
|
| 189 |
+
|
| 190 |
+
// Keyboard event handlers (guarded to not interfere with inputs/shortcuts)
|
| 191 |
+
const handleKeyDown = useCallback(
|
| 192 |
+
(event: KeyboardEvent) => {
|
| 193 |
+
if (!teleopState.isActive || !keyboardProcessRef.current) return;
|
| 194 |
+
|
| 195 |
+
// Ignore when user is typing in inputs/textareas or contenteditable elements
|
| 196 |
+
const target = event.target as HTMLElement | null;
|
| 197 |
+
const isEditableTarget = !!(
|
| 198 |
+
target &&
|
| 199 |
+
(target.tagName === "INPUT" ||
|
| 200 |
+
target.tagName === "TEXTAREA" ||
|
| 201 |
+
target.tagName === "SELECT" ||
|
| 202 |
+
target.isContentEditable ||
|
| 203 |
+
target.closest(
|
| 204 |
+
'[role="textbox"], [contenteditable="true"], input, textarea, select'
|
| 205 |
+
))
|
| 206 |
+
);
|
| 207 |
+
if (isEditableTarget) return;
|
| 208 |
+
|
| 209 |
+
// Allow browser/system shortcuts (e.g. Ctrl/Cmd+R, Ctrl/Cmd+L, etc.)
|
| 210 |
+
if (event.metaKey || event.ctrlKey || event.altKey) return;
|
| 211 |
+
|
| 212 |
+
// Only handle specific teleop keys
|
| 213 |
+
const rawKey = event.key;
|
| 214 |
+
const normalizedKey = rawKey.length === 1 ? rawKey.toLowerCase() : rawKey;
|
| 215 |
+
const allowedKeys = new Set([
|
| 216 |
+
"ArrowUp",
|
| 217 |
+
"ArrowDown",
|
| 218 |
+
"ArrowLeft",
|
| 219 |
+
"ArrowRight",
|
| 220 |
+
"w",
|
| 221 |
+
"a",
|
| 222 |
+
"s",
|
| 223 |
+
"d",
|
| 224 |
+
"q",
|
| 225 |
+
"e",
|
| 226 |
+
"o",
|
| 227 |
+
"c",
|
| 228 |
+
"Escape",
|
| 229 |
+
]);
|
| 230 |
+
if (!allowedKeys.has(normalizedKey)) return;
|
| 231 |
+
|
| 232 |
+
event.preventDefault();
|
| 233 |
+
const keyboardTeleoperator = keyboardProcessRef.current.teleoperator;
|
| 234 |
+
if (keyboardTeleoperator && "updateKeyState" in keyboardTeleoperator) {
|
| 235 |
+
(
|
| 236 |
+
keyboardTeleoperator as {
|
| 237 |
+
updateKeyState: (key: string, pressed: boolean) => void;
|
| 238 |
+
}
|
| 239 |
+
).updateKeyState(normalizedKey, true);
|
| 240 |
+
}
|
| 241 |
+
},
|
| 242 |
+
[teleopState.isActive]
|
| 243 |
+
);
|
| 244 |
+
|
| 245 |
+
const handleKeyUp = useCallback(
|
| 246 |
+
(event: KeyboardEvent) => {
|
| 247 |
+
if (!teleopState.isActive || !keyboardProcessRef.current) return;
|
| 248 |
+
|
| 249 |
+
// Ignore when user is typing in inputs/textareas or contenteditable elements
|
| 250 |
+
const target = event.target as HTMLElement | null;
|
| 251 |
+
const isEditableTarget = !!(
|
| 252 |
+
target &&
|
| 253 |
+
(target.tagName === "INPUT" ||
|
| 254 |
+
target.tagName === "TEXTAREA" ||
|
| 255 |
+
target.tagName === "SELECT" ||
|
| 256 |
+
target.isContentEditable ||
|
| 257 |
+
target.closest(
|
| 258 |
+
'[role="textbox"], [contenteditable="true"], input, textarea, select'
|
| 259 |
+
))
|
| 260 |
+
);
|
| 261 |
+
if (isEditableTarget) return;
|
| 262 |
+
|
| 263 |
+
// Allow browser/system shortcuts (e.g. Ctrl/Cmd+R, Ctrl/Cmd+L, etc.)
|
| 264 |
+
if (event.metaKey || event.ctrlKey || event.altKey) return;
|
| 265 |
+
|
| 266 |
+
// Only handle specific teleop keys
|
| 267 |
+
const rawKey = event.key;
|
| 268 |
+
const normalizedKey = rawKey.length === 1 ? rawKey.toLowerCase() : rawKey;
|
| 269 |
+
const allowedKeys = new Set([
|
| 270 |
+
"ArrowUp",
|
| 271 |
+
"ArrowDown",
|
| 272 |
+
"ArrowLeft",
|
| 273 |
+
"ArrowRight",
|
| 274 |
+
"w",
|
| 275 |
+
"a",
|
| 276 |
+
"s",
|
| 277 |
+
"d",
|
| 278 |
+
"q",
|
| 279 |
+
"e",
|
| 280 |
+
"o",
|
| 281 |
+
"c",
|
| 282 |
+
"Escape",
|
| 283 |
+
]);
|
| 284 |
+
if (!allowedKeys.has(normalizedKey)) return;
|
| 285 |
+
|
| 286 |
+
event.preventDefault();
|
| 287 |
+
const keyboardTeleoperator = keyboardProcessRef.current.teleoperator;
|
| 288 |
+
if (keyboardTeleoperator && "updateKeyState" in keyboardTeleoperator) {
|
| 289 |
+
(
|
| 290 |
+
keyboardTeleoperator as {
|
| 291 |
+
updateKeyState: (key: string, pressed: boolean) => void;
|
| 292 |
+
}
|
| 293 |
+
).updateKeyState(normalizedKey, false);
|
| 294 |
+
}
|
| 295 |
+
},
|
| 296 |
+
[teleopState.isActive]
|
| 297 |
+
);
|
| 298 |
+
|
| 299 |
+
// Register keyboard events
|
| 300 |
+
useEffect(() => {
|
| 301 |
+
if (teleopState.isActive && selectedTeleoperatorType === "keyboard") {
|
| 302 |
+
window.addEventListener("keydown", handleKeyDown);
|
| 303 |
+
window.addEventListener("keyup", handleKeyUp);
|
| 304 |
+
|
| 305 |
+
return () => {
|
| 306 |
+
window.removeEventListener("keydown", handleKeyDown);
|
| 307 |
+
window.removeEventListener("keyup", handleKeyUp);
|
| 308 |
+
};
|
| 309 |
+
}
|
| 310 |
+
}, [
|
| 311 |
+
teleopState.isActive,
|
| 312 |
+
selectedTeleoperatorType,
|
| 313 |
+
handleKeyDown,
|
| 314 |
+
handleKeyUp,
|
| 315 |
+
]);
|
| 316 |
+
|
| 317 |
+
// Cleanup on unmount
|
| 318 |
+
useEffect(() => {
|
| 319 |
+
return () => {
|
| 320 |
+
const cleanup = async () => {
|
| 321 |
+
try {
|
| 322 |
+
if (keyboardProcessRef.current) {
|
| 323 |
+
await keyboardProcessRef.current.disconnect();
|
| 324 |
+
keyboardProcessRef.current = null;
|
| 325 |
+
}
|
| 326 |
+
if (directProcessRef.current) {
|
| 327 |
+
await directProcessRef.current.disconnect();
|
| 328 |
+
directProcessRef.current = null;
|
| 329 |
+
}
|
| 330 |
+
} catch (error) {
|
| 331 |
+
console.warn("Error during teleoperation cleanup:", error);
|
| 332 |
+
}
|
| 333 |
+
};
|
| 334 |
+
cleanup();
|
| 335 |
+
};
|
| 336 |
+
}, []);
|
| 337 |
+
|
| 338 |
+
// Memoize teleoperators array for the Recorder component
|
| 339 |
+
const memoizedTeleoperators = useMemo(() => {
|
| 340 |
+
if (!controlEnabled) return [];
|
| 341 |
+
|
| 342 |
+
// Return the active teleoperator based on selection
|
| 343 |
+
const activeTeleoperator =
|
| 344 |
+
selectedTeleoperatorType === "keyboard"
|
| 345 |
+
? keyboardProcessRef.current?.teleoperator
|
| 346 |
+
: directProcessRef.current?.teleoperator;
|
| 347 |
+
|
| 348 |
+
return activeTeleoperator ? [activeTeleoperator] : [];
|
| 349 |
+
}, [controlEnabled, selectedTeleoperatorType]);
|
| 350 |
+
|
| 351 |
+
return (
|
| 352 |
+
<div className="space-y-6">
|
| 353 |
+
{/* Robot Movement Recorder Header */}
|
| 354 |
+
<Card className="border-0 rounded-none">
|
| 355 |
+
<div className="p-4 border-b border-white/10">
|
| 356 |
+
<div className="flex items-center justify-between">
|
| 357 |
+
<div className="flex items-center gap-4">
|
| 358 |
+
<div className="w-1 h-8 bg-primary"></div>
|
| 359 |
+
<div>
|
| 360 |
+
<h3 className="text-xl font-bold text-foreground font-mono tracking-wider uppercase">
|
| 361 |
+
robot movement recorder
|
| 362 |
+
</h3>
|
| 363 |
+
<p className="text-sm text-muted-foreground font-mono">
|
| 364 |
+
dataset{" "}
|
| 365 |
+
<span className="text-muted-foreground">recording</span>{" "}
|
| 366 |
+
interface
|
| 367 |
+
</p>
|
| 368 |
+
</div>
|
| 369 |
+
</div>
|
| 370 |
+
<div className="flex items-center gap-6">
|
| 371 |
+
<div className="border-l border-white/10 pl-6 flex items-center gap-4">
|
| 372 |
+
<Button
|
| 373 |
+
variant="outline"
|
| 374 |
+
size="lg"
|
| 375 |
+
className="gap-2"
|
| 376 |
+
onClick={() => setShowConfigure(!showConfigure)}
|
| 377 |
+
>
|
| 378 |
+
<Settings className="w-5 h-5" />
|
| 379 |
+
Configure
|
| 380 |
+
</Button>
|
| 381 |
+
|
| 382 |
+
<Button
|
| 383 |
+
variant={
|
| 384 |
+
recorderCallbacks?.isRecording ? "destructive" : "default"
|
| 385 |
+
}
|
| 386 |
+
size="lg"
|
| 387 |
+
className="gap-2"
|
| 388 |
+
disabled={!robot.isConnected || !recorderCallbacks}
|
| 389 |
+
onClick={
|
| 390 |
+
recorderCallbacks?.isRecording
|
| 391 |
+
? recorderCallbacks.stopRecording
|
| 392 |
+
: recorderCallbacks?.startRecording
|
| 393 |
+
}
|
| 394 |
+
>
|
| 395 |
+
{recorderCallbacks?.isRecording ? (
|
| 396 |
+
<>
|
| 397 |
+
<Square className="w-5 h-5" />
|
| 398 |
+
Stop Recording
|
| 399 |
+
</>
|
| 400 |
+
) : (
|
| 401 |
+
<>
|
| 402 |
+
<Record className="w-5 h-5" />
|
| 403 |
+
Start Recording
|
| 404 |
+
</>
|
| 405 |
+
)}
|
| 406 |
+
</Button>
|
| 407 |
+
</div>
|
| 408 |
+
<div className="flex items-center gap-2">
|
| 409 |
+
<span className="text-sm font-mono text-muted-foreground uppercase">
|
| 410 |
+
robot:
|
| 411 |
+
</span>
|
| 412 |
+
<Badge
|
| 413 |
+
variant="outline"
|
| 414 |
+
className={cn(
|
| 415 |
+
"border-primary/50 bg-primary/20 text-primary font-mono text-xs",
|
| 416 |
+
robot.isConnected
|
| 417 |
+
? "border-green-500/50 bg-green-500/20 text-green-400"
|
| 418 |
+
: "border-red-500/50 bg-red-500/20 text-red-400"
|
| 419 |
+
)}
|
| 420 |
+
>
|
| 421 |
+
{robot.isConnected ? "ONLINE" : "OFFLINE"}
|
| 422 |
+
</Badge>
|
| 423 |
+
</div>
|
| 424 |
+
</div>
|
| 425 |
+
</div>
|
| 426 |
+
</div>
|
| 427 |
+
<div className="p-6 space-y-6">
|
| 428 |
+
{/* Step-by-Step Recording Guide */}
|
| 429 |
+
<div className="bg-black/40 border border-white/20 rounded-lg p-6">
|
| 430 |
+
<h4 className="text-lg font-semibold text-foreground mb-4">
|
| 431 |
+
How to Record Robot Data
|
| 432 |
+
</h4>
|
| 433 |
+
<div className="space-y-4">
|
| 434 |
+
{/* Step 1 */}
|
| 435 |
+
<div className="flex items-start gap-4">
|
| 436 |
+
<div
|
| 437 |
+
className={cn(
|
| 438 |
+
"flex-shrink-0 w-8 h-8 border rounded-full flex items-center justify-center text-sm font-mono",
|
| 439 |
+
currentStep > 1
|
| 440 |
+
? "bg-green-500/20 border-green-500/50 text-green-400" // Completed
|
| 441 |
+
: currentStep === 1
|
| 442 |
+
? "bg-yellow-500/20 border-yellow-500/50 text-yellow-400" // Active
|
| 443 |
+
: "bg-muted/20 border-muted/50 text-muted-foreground" // Inactive
|
| 444 |
+
)}
|
| 445 |
+
>
|
| 446 |
+
{currentStep > 1 ? <Check className="w-4 h-4" /> : "1"}
|
| 447 |
+
</div>
|
| 448 |
+
<div className="flex-1">
|
| 449 |
+
<h5 className="font-semibold text-foreground">
|
| 450 |
+
Enable Robot Control
|
| 451 |
+
</h5>
|
| 452 |
+
<p className="text-sm text-muted-foreground mb-3">
|
| 453 |
+
Choose how you want to control the robot during recording.
|
| 454 |
+
</p>
|
| 455 |
+
|
| 456 |
+
{!controlEnabled && robot.isConnected && (
|
| 457 |
+
<div className="space-y-3">
|
| 458 |
+
<div className="flex items-center gap-3">
|
| 459 |
+
<label className="text-sm text-muted-foreground min-w-0">
|
| 460 |
+
Control Type:
|
| 461 |
+
</label>
|
| 462 |
+
<Select
|
| 463 |
+
value={selectedTeleoperatorType}
|
| 464 |
+
onValueChange={(value: "direct" | "keyboard") =>
|
| 465 |
+
setSelectedTeleoperatorType(value)
|
| 466 |
+
}
|
| 467 |
+
>
|
| 468 |
+
<SelectTrigger className="w-48 bg-black/20 border-white/10">
|
| 469 |
+
<SelectValue />
|
| 470 |
+
</SelectTrigger>
|
| 471 |
+
<SelectContent>
|
| 472 |
+
<SelectItem value="direct">
|
| 473 |
+
<div className="flex items-center gap-2">
|
| 474 |
+
<Gamepad2 className="w-4 h-4" />
|
| 475 |
+
Direct Control
|
| 476 |
+
</div>
|
| 477 |
+
</SelectItem>
|
| 478 |
+
<SelectItem value="keyboard">
|
| 479 |
+
<div className="flex items-center gap-2">
|
| 480 |
+
<Keyboard className="w-4 h-4" />
|
| 481 |
+
Keyboard Control
|
| 482 |
+
</div>
|
| 483 |
+
</SelectItem>
|
| 484 |
+
</SelectContent>
|
| 485 |
+
</Select>
|
| 486 |
+
</div>
|
| 487 |
+
<p className="text-xs text-muted-foreground">
|
| 488 |
+
{selectedTeleoperatorType === "direct"
|
| 489 |
+
? "Programmatic control (use teleoperation page for manual sliders)"
|
| 490 |
+
: "Move robot using keyboard keys (WASD, arrows, Q/E, O/C)"}
|
| 491 |
+
</p>
|
| 492 |
+
</div>
|
| 493 |
+
)}
|
| 494 |
+
|
| 495 |
+
<div className="mt-3">
|
| 496 |
+
<Button
|
| 497 |
+
variant={controlEnabled ? "destructive" : "default"}
|
| 498 |
+
onClick={controlEnabled ? disableControl : enableControl}
|
| 499 |
+
disabled={!robot.isConnected}
|
| 500 |
+
className="gap-2"
|
| 501 |
+
size="sm"
|
| 502 |
+
>
|
| 503 |
+
{selectedTeleoperatorType === "keyboard" ? (
|
| 504 |
+
<Keyboard className="w-4 h-4" />
|
| 505 |
+
) : (
|
| 506 |
+
<Gamepad2 className="w-4 h-4" />
|
| 507 |
+
)}
|
| 508 |
+
{controlEnabled
|
| 509 |
+
? `${selectedTeleoperatorType} Control Active`
|
| 510 |
+
: `Enable ${selectedTeleoperatorType} Control`}
|
| 511 |
+
</Button>
|
| 512 |
+
</div>
|
| 513 |
+
</div>
|
| 514 |
+
</div>
|
| 515 |
+
|
| 516 |
+
{/* Step 2 */}
|
| 517 |
+
<div className="flex items-start gap-4">
|
| 518 |
+
<div
|
| 519 |
+
className={cn(
|
| 520 |
+
"flex-shrink-0 w-8 h-8 border rounded-full flex items-center justify-center text-sm font-mono",
|
| 521 |
+
currentStep > 2
|
| 522 |
+
? "bg-green-500/20 border-green-500/50 text-green-400" // Completed
|
| 523 |
+
: currentStep === 2
|
| 524 |
+
? "bg-yellow-500/20 border-yellow-500/50 text-yellow-400" // Active
|
| 525 |
+
: "bg-muted/20 border-muted/50 text-muted-foreground" // Inactive
|
| 526 |
+
)}
|
| 527 |
+
>
|
| 528 |
+
{currentStep > 2 ? <Check className="w-4 h-4" /> : "2"}
|
| 529 |
+
</div>
|
| 530 |
+
<div>
|
| 531 |
+
<h5
|
| 532 |
+
className={cn(
|
| 533 |
+
"font-semibold",
|
| 534 |
+
controlEnabled
|
| 535 |
+
? "text-foreground"
|
| 536 |
+
: "text-muted-foreground"
|
| 537 |
+
)}
|
| 538 |
+
>
|
| 539 |
+
Start Recording
|
| 540 |
+
</h5>
|
| 541 |
+
<p className="text-sm text-muted-foreground">
|
| 542 |
+
Click "Start Recording" to begin capturing robot movements.
|
| 543 |
+
</p>
|
| 544 |
+
{null}
|
| 545 |
+
</div>
|
| 546 |
+
</div>
|
| 547 |
+
|
| 548 |
+
{/* Step 3 */}
|
| 549 |
+
<div className="flex items-start gap-4">
|
| 550 |
+
<div
|
| 551 |
+
className={cn(
|
| 552 |
+
"flex-shrink-0 w-8 h-8 border rounded-full flex items-center justify-center text-sm font-mono",
|
| 553 |
+
currentStep === 3
|
| 554 |
+
? "bg-yellow-500/20 border-yellow-500/50 text-yellow-400 animate-pulse" // Active with pulse
|
| 555 |
+
: "bg-muted/20 border-muted/50 text-muted-foreground" // Inactive
|
| 556 |
+
)}
|
| 557 |
+
>
|
| 558 |
+
3
|
| 559 |
+
</div>
|
| 560 |
+
<div>
|
| 561 |
+
<h5
|
| 562 |
+
className={cn(
|
| 563 |
+
"font-semibold",
|
| 564 |
+
currentStep === 3
|
| 565 |
+
? "text-foreground"
|
| 566 |
+
: "text-muted-foreground"
|
| 567 |
+
)}
|
| 568 |
+
>
|
| 569 |
+
Move the Robot
|
| 570 |
+
</h5>
|
| 571 |
+
<p className="text-sm text-muted-foreground">
|
| 572 |
+
{currentStep === 3 ? (
|
| 573 |
+
<span className="text-yellow-400">
|
| 574 |
+
🎯 Recording active! Move the robot to demonstrate your
|
| 575 |
+
task.
|
| 576 |
+
</span>
|
| 577 |
+
) : (
|
| 578 |
+
"Move the robot manually to demonstrate the task you want to teach. Recording will capture your movements."
|
| 579 |
+
)}
|
| 580 |
+
</p>
|
| 581 |
+
</div>
|
| 582 |
+
</div>
|
| 583 |
+
</div>
|
| 584 |
+
</div>
|
| 585 |
+
|
| 586 |
+
{/* Robot Movement Recorder */}
|
| 587 |
+
<Recorder
|
| 588 |
+
teleoperators={memoizedTeleoperators}
|
| 589 |
+
robot={robot}
|
| 590 |
+
onNeedsTeleoperation={enableControl}
|
| 591 |
+
showConfigure={showConfigure}
|
| 592 |
+
onRecorderReady={setRecorderCallbacks}
|
| 593 |
+
/>
|
| 594 |
+
</div>
|
| 595 |
+
</Card>
|
| 596 |
+
</div>
|
| 597 |
+
);
|
| 598 |
+
}
|
|
@@ -36,14 +36,14 @@ const roadmapItems: RoadmapItem[] = [
|
|
| 36 |
description: "Node.js CLI tools for robot control and automation scripts",
|
| 37 |
status: "completed",
|
| 38 |
},
|
| 39 |
-
{
|
| 40 |
-
title: "SO-100 leader arm",
|
| 41 |
-
description: "Leader arm teleoperation support for intuitive robot control",
|
| 42 |
-
status: "in_progress",
|
| 43 |
-
},
|
| 44 |
{
|
| 45 |
title: "record",
|
| 46 |
description: "Record robot trajectories and sensor data to create datasets",
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 47 |
status: "in_progress",
|
| 48 |
},
|
| 49 |
{
|
|
|
|
| 36 |
description: "Node.js CLI tools for robot control and automation scripts",
|
| 37 |
status: "completed",
|
| 38 |
},
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 39 |
{
|
| 40 |
title: "record",
|
| 41 |
description: "Record robot trajectories and sensor data to create datasets",
|
| 42 |
+
status: "completed",
|
| 43 |
+
},
|
| 44 |
+
{
|
| 45 |
+
title: "SO-100 leader arm",
|
| 46 |
+
description: "Leader arm teleoperation support for intuitive robot control",
|
| 47 |
status: "in_progress",
|
| 48 |
},
|
| 49 |
{
|
|
@@ -22,7 +22,6 @@ import {
|
|
| 22 |
} from "@lerobot/web";
|
| 23 |
import { getUnifiedRobotData } from "@/lib/unified-storage";
|
| 24 |
import VirtualKey from "@/components/VirtualKey";
|
| 25 |
-
import { Recorder } from "@/components/recorder";
|
| 26 |
import { Canvas } from "@react-three/fiber";
|
| 27 |
import { Physics } from "@react-three/cannon";
|
| 28 |
import * as THREE from "three";
|
|
@@ -196,12 +195,48 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
|
|
| 196 |
};
|
| 197 |
}, []);
|
| 198 |
|
| 199 |
-
// Keyboard event handlers
|
| 200 |
const handleKeyDown = useCallback(
|
| 201 |
(event: KeyboardEvent) => {
|
| 202 |
if (!teleopState.isActive || !keyboardProcessRef.current) return;
|
| 203 |
|
| 204 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 205 |
event.preventDefault();
|
| 206 |
|
| 207 |
const keyboardTeleoperator = keyboardProcessRef.current.teleoperator;
|
|
@@ -210,7 +245,7 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
|
|
| 210 |
keyboardTeleoperator as {
|
| 211 |
updateKeyState: (key: string, pressed: boolean) => void;
|
| 212 |
}
|
| 213 |
-
).updateKeyState(
|
| 214 |
}
|
| 215 |
},
|
| 216 |
[teleopState.isActive]
|
|
@@ -220,7 +255,43 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
|
|
| 220 |
(event: KeyboardEvent) => {
|
| 221 |
if (!teleopState.isActive || !keyboardProcessRef.current) return;
|
| 222 |
|
| 223 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 224 |
event.preventDefault();
|
| 225 |
|
| 226 |
const keyboardTeleoperator = keyboardProcessRef.current.teleoperator;
|
|
@@ -229,7 +300,7 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
|
|
| 229 |
keyboardTeleoperator as {
|
| 230 |
updateKeyState: (key: string, pressed: boolean) => void;
|
| 231 |
}
|
| 232 |
-
).updateKeyState(
|
| 233 |
}
|
| 234 |
},
|
| 235 |
[teleopState.isActive]
|
|
@@ -414,15 +485,6 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
|
|
| 414 |
const keyStates = teleopState?.keyStates || {};
|
| 415 |
const controls = SO100_KEYBOARD_CONTROLS;
|
| 416 |
|
| 417 |
-
// Memoize teleoperators array to prevent unnecessary re-renders of the Recorder component
|
| 418 |
-
const memoizedTeleoperators = useMemo(() => {
|
| 419 |
-
if (!directProcessRef.current) return [];
|
| 420 |
-
|
| 421 |
-
return [
|
| 422 |
-
directProcessRef.current?.teleoperator,
|
| 423 |
-
].filter(Boolean);
|
| 424 |
-
}, [directProcessRef.current]);
|
| 425 |
-
|
| 426 |
return (
|
| 427 |
<>
|
| 428 |
<Card className="border-0 rounded-none">
|
|
@@ -456,19 +518,37 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
|
|
| 456 |
<Power className="w-5 h-5 mr-2" /> Control Robot
|
| 457 |
</Button>
|
| 458 |
)}
|
| 459 |
-
<div className="flex items-center gap-
|
| 460 |
-
<
|
| 461 |
-
|
| 462 |
-
|
| 463 |
-
|
| 464 |
-
|
| 465 |
-
|
| 466 |
-
|
| 467 |
-
|
| 468 |
-
|
| 469 |
-
|
| 470 |
-
|
| 471 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 472 |
</div>
|
| 473 |
</div>
|
| 474 |
</div>
|
|
@@ -759,13 +839,6 @@ export function TeleoperationView({ robot }: TeleoperationViewProps) {
|
|
| 759 |
</div>
|
| 760 |
</div>
|
| 761 |
</Card>
|
| 762 |
-
|
| 763 |
-
{/* Robot Movement Recorder - Always show UI */}
|
| 764 |
-
<Recorder
|
| 765 |
-
teleoperators={memoizedTeleoperators}
|
| 766 |
-
robot={robot}
|
| 767 |
-
onNeedsTeleoperation={initializeTeleoperation}
|
| 768 |
-
/>
|
| 769 |
</>
|
| 770 |
);
|
| 771 |
}
|
|
|
|
| 22 |
} from "@lerobot/web";
|
| 23 |
import { getUnifiedRobotData } from "@/lib/unified-storage";
|
| 24 |
import VirtualKey from "@/components/VirtualKey";
|
|
|
|
| 25 |
import { Canvas } from "@react-three/fiber";
|
| 26 |
import { Physics } from "@react-three/cannon";
|
| 27 |
import * as THREE from "three";
|
|
|
|
| 195 |
};
|
| 196 |
}, []);
|
| 197 |
|
| 198 |
+
// Keyboard event handlers (guarded to not interfere with inputs/shortcuts)
|
| 199 |
const handleKeyDown = useCallback(
|
| 200 |
(event: KeyboardEvent) => {
|
| 201 |
if (!teleopState.isActive || !keyboardProcessRef.current) return;
|
| 202 |
|
| 203 |
+
// Ignore when user is typing in inputs/textareas or contenteditable elements
|
| 204 |
+
const target = event.target as HTMLElement | null;
|
| 205 |
+
const isEditableTarget = !!(
|
| 206 |
+
target &&
|
| 207 |
+
(target.tagName === "INPUT" ||
|
| 208 |
+
target.tagName === "TEXTAREA" ||
|
| 209 |
+
target.tagName === "SELECT" ||
|
| 210 |
+
target.isContentEditable ||
|
| 211 |
+
target.closest(
|
| 212 |
+
'[role="textbox"], [contenteditable="true"], input, textarea, select'
|
| 213 |
+
))
|
| 214 |
+
);
|
| 215 |
+
if (isEditableTarget) return;
|
| 216 |
+
|
| 217 |
+
// Allow browser/system shortcuts (e.g. Ctrl/Cmd+R, Ctrl/Cmd+L, etc.)
|
| 218 |
+
if (event.metaKey || event.ctrlKey || event.altKey) return;
|
| 219 |
+
|
| 220 |
+
// Only handle specific teleop keys
|
| 221 |
+
const rawKey = event.key;
|
| 222 |
+
const normalizedKey = rawKey.length === 1 ? rawKey.toLowerCase() : rawKey;
|
| 223 |
+
const allowedKeys = new Set([
|
| 224 |
+
"ArrowUp",
|
| 225 |
+
"ArrowDown",
|
| 226 |
+
"ArrowLeft",
|
| 227 |
+
"ArrowRight",
|
| 228 |
+
"w",
|
| 229 |
+
"a",
|
| 230 |
+
"s",
|
| 231 |
+
"d",
|
| 232 |
+
"q",
|
| 233 |
+
"e",
|
| 234 |
+
"o",
|
| 235 |
+
"c",
|
| 236 |
+
"Escape",
|
| 237 |
+
]);
|
| 238 |
+
if (!allowedKeys.has(normalizedKey)) return;
|
| 239 |
+
|
| 240 |
event.preventDefault();
|
| 241 |
|
| 242 |
const keyboardTeleoperator = keyboardProcessRef.current.teleoperator;
|
|
|
|
| 245 |
keyboardTeleoperator as {
|
| 246 |
updateKeyState: (key: string, pressed: boolean) => void;
|
| 247 |
}
|
| 248 |
+
).updateKeyState(normalizedKey, true);
|
| 249 |
}
|
| 250 |
},
|
| 251 |
[teleopState.isActive]
|
|
|
|
| 255 |
(event: KeyboardEvent) => {
|
| 256 |
if (!teleopState.isActive || !keyboardProcessRef.current) return;
|
| 257 |
|
| 258 |
+
// Ignore when user is typing in inputs/textareas or contenteditable elements
|
| 259 |
+
const target = event.target as HTMLElement | null;
|
| 260 |
+
const isEditableTarget = !!(
|
| 261 |
+
target &&
|
| 262 |
+
(target.tagName === "INPUT" ||
|
| 263 |
+
target.tagName === "TEXTAREA" ||
|
| 264 |
+
target.tagName === "SELECT" ||
|
| 265 |
+
target.isContentEditable ||
|
| 266 |
+
target.closest(
|
| 267 |
+
'[role="textbox"], [contenteditable="true"], input, textarea, select'
|
| 268 |
+
))
|
| 269 |
+
);
|
| 270 |
+
if (isEditableTarget) return;
|
| 271 |
+
|
| 272 |
+
// Allow browser/system shortcuts (e.g. Ctrl/Cmd+R, Ctrl/Cmd+L, etc.)
|
| 273 |
+
if (event.metaKey || event.ctrlKey || event.altKey) return;
|
| 274 |
+
|
| 275 |
+
// Only handle specific teleop keys
|
| 276 |
+
const rawKey = event.key;
|
| 277 |
+
const normalizedKey = rawKey.length === 1 ? rawKey.toLowerCase() : rawKey;
|
| 278 |
+
const allowedKeys = new Set([
|
| 279 |
+
"ArrowUp",
|
| 280 |
+
"ArrowDown",
|
| 281 |
+
"ArrowLeft",
|
| 282 |
+
"ArrowRight",
|
| 283 |
+
"w",
|
| 284 |
+
"a",
|
| 285 |
+
"s",
|
| 286 |
+
"d",
|
| 287 |
+
"q",
|
| 288 |
+
"e",
|
| 289 |
+
"o",
|
| 290 |
+
"c",
|
| 291 |
+
"Escape",
|
| 292 |
+
]);
|
| 293 |
+
if (!allowedKeys.has(normalizedKey)) return;
|
| 294 |
+
|
| 295 |
event.preventDefault();
|
| 296 |
|
| 297 |
const keyboardTeleoperator = keyboardProcessRef.current.teleoperator;
|
|
|
|
| 300 |
keyboardTeleoperator as {
|
| 301 |
updateKeyState: (key: string, pressed: boolean) => void;
|
| 302 |
}
|
| 303 |
+
).updateKeyState(normalizedKey, false);
|
| 304 |
}
|
| 305 |
},
|
| 306 |
[teleopState.isActive]
|
|
|
|
| 485 |
const keyStates = teleopState?.keyStates || {};
|
| 486 |
const controls = SO100_KEYBOARD_CONTROLS;
|
| 487 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 488 |
return (
|
| 489 |
<>
|
| 490 |
<Card className="border-0 rounded-none">
|
|
|
|
| 518 |
<Power className="w-5 h-5 mr-2" /> Control Robot
|
| 519 |
</Button>
|
| 520 |
)}
|
| 521 |
+
<div className="flex items-center gap-4">
|
| 522 |
+
<div className="flex items-center gap-2">
|
| 523 |
+
<span className="text-sm font-mono text-muted-foreground uppercase">
|
| 524 |
+
robot:
|
| 525 |
+
</span>
|
| 526 |
+
<Badge
|
| 527 |
+
variant="outline"
|
| 528 |
+
className={cn(
|
| 529 |
+
"border-primary/50 bg-primary/20 text-primary font-mono text-xs",
|
| 530 |
+
robot.isConnected
|
| 531 |
+
? "border-green-500/50 bg-green-500/20 text-green-400"
|
| 532 |
+
: "border-red-500/50 bg-red-500/20 text-red-400"
|
| 533 |
+
)}
|
| 534 |
+
>
|
| 535 |
+
{robot.isConnected ? "ONLINE" : "OFFLINE"}
|
| 536 |
+
</Badge>
|
| 537 |
+
</div>
|
| 538 |
+
<div className="flex items-center gap-2">
|
| 539 |
+
<span className="text-sm font-mono text-muted-foreground uppercase">
|
| 540 |
+
control:
|
| 541 |
+
</span>
|
| 542 |
+
<Badge
|
| 543 |
+
variant="outline"
|
| 544 |
+
className={cn(
|
| 545 |
+
"border-primary/50 bg-primary/20 text-primary font-mono text-xs",
|
| 546 |
+
teleopState?.isActive && "animate-pulse-slow"
|
| 547 |
+
)}
|
| 548 |
+
>
|
| 549 |
+
{teleopState?.isActive ? "ACTIVE" : "STOPPED"}
|
| 550 |
+
</Badge>
|
| 551 |
+
</div>
|
| 552 |
</div>
|
| 553 |
</div>
|
| 554 |
</div>
|
|
|
|
| 839 |
</div>
|
| 840 |
</div>
|
| 841 |
</Card>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 842 |
</>
|
| 843 |
);
|
| 844 |
}
|
|
@@ -7,23 +7,38 @@ import { Button } from "./ui/button";
|
|
| 7 |
|
| 8 |
interface TeleoperatorEpisodesViewProps {
|
| 9 |
teleoperatorData?: LeRobotEpisode[];
|
|
|
|
|
|
|
| 10 |
}
|
| 11 |
|
| 12 |
-
export function TeleoperatorEpisodesView({
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
// State to track which episodes are expanded
|
| 14 |
-
const [expandedEpisodes, setExpandedEpisodes] = useState<
|
| 15 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 16 |
// Toggle expanded state for an episode
|
| 17 |
const toggleEpisode = (index: number) => {
|
| 18 |
-
setExpandedEpisodes(prev => ({
|
| 19 |
...prev,
|
| 20 |
-
[index]: !prev[index]
|
| 21 |
}));
|
| 22 |
};
|
| 23 |
-
|
| 24 |
return (
|
| 25 |
<div className="flex flex-col gap-2">
|
| 26 |
-
<div className="text-sm text-center text-muted-foreground mb-2">
|
|
|
|
|
|
|
| 27 |
<div className="flex flex-col gap-1">
|
| 28 |
{/* Header */}
|
| 29 |
<div className="flex flex-row font-medium text-sm">
|
|
@@ -31,7 +46,7 @@ export function TeleoperatorEpisodesView({ teleoperatorData }: TeleoperatorEpiso
|
|
| 31 |
<div className="flex-1 px-4 py-2">time length</div>
|
| 32 |
<div className="flex-1 px-4 py-2">frames</div>
|
| 33 |
</div>
|
| 34 |
-
|
| 35 |
{/* Body */}
|
| 36 |
{teleoperatorData && teleoperatorData.length > 0 ? (
|
| 37 |
teleoperatorData.map((episode: LeRobotEpisode, i: number) => (
|
|
@@ -39,26 +54,37 @@ export function TeleoperatorEpisodesView({ teleoperatorData }: TeleoperatorEpiso
|
|
| 39 |
{/* Episode row */}
|
| 40 |
<div className="flex flex-row">
|
| 41 |
<div className="flex-1 px-4 py-2 font-mono">{i}</div>
|
| 42 |
-
<div className="flex-1 px-4 py-2 font-mono">
|
| 43 |
-
|
| 44 |
-
<div
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 48 |
</div>
|
| 49 |
</div>
|
| 50 |
-
|
| 51 |
{/* Frames (collapsible) */}
|
| 52 |
{expandedEpisodes[i] && (
|
| 53 |
-
<TeleoperatorFramesView
|
|
|
|
|
|
|
|
|
|
|
|
|
| 54 |
)}
|
| 55 |
</div>
|
| 56 |
))
|
| 57 |
) : (
|
| 58 |
<div className="flex flex-row border-t border-gray-700">
|
| 59 |
-
<div
|
| 60 |
-
className="flex-1 px-4 py-4 text-center text-muted-foreground"
|
| 61 |
-
>
|
| 62 |
No episodes recorded yet. Click "Start Recording" to begin.
|
| 63 |
</div>
|
| 64 |
</div>
|
|
|
|
| 7 |
|
| 8 |
interface TeleoperatorEpisodesViewProps {
|
| 9 |
teleoperatorData?: LeRobotEpisode[];
|
| 10 |
+
isRecording?: boolean;
|
| 11 |
+
refreshTick?: number;
|
| 12 |
}
|
| 13 |
|
| 14 |
+
export function TeleoperatorEpisodesView({
|
| 15 |
+
teleoperatorData,
|
| 16 |
+
isRecording,
|
| 17 |
+
refreshTick,
|
| 18 |
+
}: TeleoperatorEpisodesViewProps) {
|
| 19 |
// State to track which episodes are expanded
|
| 20 |
+
const [expandedEpisodes, setExpandedEpisodes] = useState<
|
| 21 |
+
Record<number, boolean>
|
| 22 |
+
>({});
|
| 23 |
+
|
| 24 |
+
const formatSeconds = (s: number): string => {
|
| 25 |
+
if (!Number.isFinite(s)) return String(s);
|
| 26 |
+
return s.toFixed(3);
|
| 27 |
+
};
|
| 28 |
+
|
| 29 |
// Toggle expanded state for an episode
|
| 30 |
const toggleEpisode = (index: number) => {
|
| 31 |
+
setExpandedEpisodes((prev) => ({
|
| 32 |
...prev,
|
| 33 |
+
[index]: !prev[index],
|
| 34 |
}));
|
| 35 |
};
|
| 36 |
+
|
| 37 |
return (
|
| 38 |
<div className="flex flex-col gap-2">
|
| 39 |
+
<div className="text-sm text-center text-muted-foreground mb-2">
|
| 40 |
+
List of recorded episodes
|
| 41 |
+
</div>
|
| 42 |
<div className="flex flex-col gap-1">
|
| 43 |
{/* Header */}
|
| 44 |
<div className="flex flex-row font-medium text-sm">
|
|
|
|
| 46 |
<div className="flex-1 px-4 py-2">time length</div>
|
| 47 |
<div className="flex-1 px-4 py-2">frames</div>
|
| 48 |
</div>
|
| 49 |
+
|
| 50 |
{/* Body */}
|
| 51 |
{teleoperatorData && teleoperatorData.length > 0 ? (
|
| 52 |
teleoperatorData.map((episode: LeRobotEpisode, i: number) => (
|
|
|
|
| 54 |
{/* Episode row */}
|
| 55 |
<div className="flex flex-row">
|
| 56 |
<div className="flex-1 px-4 py-2 font-mono">{i}</div>
|
| 57 |
+
<div className="flex-1 px-4 py-2 font-mono">
|
| 58 |
+
{formatSeconds(episode.timespan)}
|
| 59 |
+
</div>
|
| 60 |
+
<div className="flex-1 px-4 py-2 font-mono">
|
| 61 |
+
{episode.length}
|
| 62 |
+
</div>
|
| 63 |
+
<div
|
| 64 |
+
className="px-4 py-2 cursor-pointer"
|
| 65 |
+
onClick={() => toggleEpisode(i)}
|
| 66 |
+
>
|
| 67 |
+
{expandedEpisodes[i] ? (
|
| 68 |
+
<Button>hide frames</Button>
|
| 69 |
+
) : (
|
| 70 |
+
<Button>show frames</Button>
|
| 71 |
+
)}
|
| 72 |
</div>
|
| 73 |
</div>
|
| 74 |
+
|
| 75 |
{/* Frames (collapsible) */}
|
| 76 |
{expandedEpisodes[i] && (
|
| 77 |
+
<TeleoperatorFramesView
|
| 78 |
+
frames={episode.frames}
|
| 79 |
+
isRecording={isRecording}
|
| 80 |
+
refreshTick={refreshTick}
|
| 81 |
+
/>
|
| 82 |
)}
|
| 83 |
</div>
|
| 84 |
))
|
| 85 |
) : (
|
| 86 |
<div className="flex flex-row border-t border-gray-700">
|
| 87 |
+
<div className="flex-1 px-4 py-4 text-center text-muted-foreground">
|
|
|
|
|
|
|
| 88 |
No episodes recorded yet. Click "Start Recording" to begin.
|
| 89 |
</div>
|
| 90 |
</div>
|
|
@@ -1,21 +1,28 @@
|
|
| 1 |
"use client";
|
| 2 |
|
|
|
|
| 3 |
import { NonIndexedLeRobotDatasetRow } from "@lerobot/web";
|
| 4 |
import { TeleoperatorJointGraph } from "./teleoperator-joint-graph";
|
| 5 |
|
| 6 |
interface TeleoperatorFramesViewProps {
|
| 7 |
frames: NonIndexedLeRobotDatasetRow[];
|
|
|
|
|
|
|
| 8 |
}
|
| 9 |
|
| 10 |
-
export
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
// Joint names in the order they appear in the arrays
|
| 12 |
const jointNames = [
|
| 13 |
-
"shoulder_pan",
|
| 14 |
"shoulder_lift",
|
| 15 |
"elbow_flex",
|
| 16 |
"wrist_flex",
|
| 17 |
"wrist_roll",
|
| 18 |
-
"gripper"
|
| 19 |
];
|
| 20 |
|
| 21 |
// Helper function to format an object as a column of key-value pairs with joint names
|
|
@@ -24,55 +31,77 @@ export function TeleoperatorFramesView({ frames }: TeleoperatorFramesViewProps)
|
|
| 24 |
.map(([key, value]) => {
|
| 25 |
// Convert numeric key to joint name if possible
|
| 26 |
const index = parseInt(key);
|
| 27 |
-
const jointName =
|
| 28 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 29 |
})
|
| 30 |
-
.join(
|
| 31 |
};
|
| 32 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
return (
|
| 34 |
<div className="ml-8 mr-4 mb-2">
|
| 35 |
{/* Joint visualization graph */}
|
| 36 |
-
<TeleoperatorJointGraph
|
| 37 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 38 |
{/* Frames container with horizontal scroll */}
|
| 39 |
<div className="bg-gray-800/50 rounded-md overflow-hidden">
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
</tr>
|
| 50 |
-
</thead>
|
| 51 |
-
|
| 52 |
-
{/* Frame rows */}
|
| 53 |
-
<tbody className="max-h-60 overflow-y-auto">
|
| 54 |
-
{frames.map((frame: NonIndexedLeRobotDatasetRow, frameIndex: number) => (
|
| 55 |
-
<tr key={frameIndex} className="text-xs border-t border-gray-700/50">
|
| 56 |
-
<td className="w-16 px-2 py-1 font-mono whitespace-nowrap">{frameIndex}</td>
|
| 57 |
-
<td className="w-64 px-2 py-1 font-mono whitespace-nowrap">
|
| 58 |
-
{frame.timestamp}
|
| 59 |
-
</td>
|
| 60 |
-
<td className="w-[200px] px-2 py-1 font-mono whitespace-pre-wrap align-top">
|
| 61 |
-
{Object.keys(frame.action).length > 0 ?
|
| 62 |
-
formatArrayAsColumn(frame.action) :
|
| 63 |
-
'-'}
|
| 64 |
-
</td>
|
| 65 |
-
<td className="w-[300px] px-2 py-1 font-mono whitespace-pre-wrap align-top">
|
| 66 |
-
{Object.keys(frame["observation.state"]).length > 0 ?
|
| 67 |
-
formatArrayAsColumn(frame["observation.state"]) :
|
| 68 |
-
'-'}
|
| 69 |
-
</td>
|
| 70 |
</tr>
|
| 71 |
-
|
| 72 |
-
|
| 73 |
-
|
| 74 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 75 |
</div>
|
| 76 |
</div>
|
| 77 |
);
|
| 78 |
-
}
|
|
|
|
| 1 |
"use client";
|
| 2 |
|
| 3 |
+
import React, { memo, useMemo } from "react";
|
| 4 |
import { NonIndexedLeRobotDatasetRow } from "@lerobot/web";
|
| 5 |
import { TeleoperatorJointGraph } from "./teleoperator-joint-graph";
|
| 6 |
|
| 7 |
interface TeleoperatorFramesViewProps {
|
| 8 |
frames: NonIndexedLeRobotDatasetRow[];
|
| 9 |
+
isRecording?: boolean;
|
| 10 |
+
refreshTick?: number;
|
| 11 |
}
|
| 12 |
|
| 13 |
+
export const TeleoperatorFramesView = memo(function TeleoperatorFramesView({
|
| 14 |
+
frames,
|
| 15 |
+
isRecording,
|
| 16 |
+
refreshTick,
|
| 17 |
+
}: TeleoperatorFramesViewProps) {
|
| 18 |
// Joint names in the order they appear in the arrays
|
| 19 |
const jointNames = [
|
| 20 |
+
"shoulder_pan",
|
| 21 |
"shoulder_lift",
|
| 22 |
"elbow_flex",
|
| 23 |
"wrist_flex",
|
| 24 |
"wrist_roll",
|
| 25 |
+
"gripper",
|
| 26 |
];
|
| 27 |
|
| 28 |
// Helper function to format an object as a column of key-value pairs with joint names
|
|
|
|
| 31 |
.map(([key, value]) => {
|
| 32 |
// Convert numeric key to joint name if possible
|
| 33 |
const index = parseInt(key);
|
| 34 |
+
const jointName =
|
| 35 |
+
!isNaN(index) && index < jointNames.length ? jointNames[index] : key;
|
| 36 |
+
const formatted = Number.isFinite(value)
|
| 37 |
+
? value.toFixed(2)
|
| 38 |
+
: String(value);
|
| 39 |
+
return `${jointName}: ${formatted}`;
|
| 40 |
})
|
| 41 |
+
.join("\n");
|
| 42 |
};
|
| 43 |
+
|
| 44 |
+
const visibleFrames = useMemo(
|
| 45 |
+
() => frames || ([] as NonIndexedLeRobotDatasetRow[]),
|
| 46 |
+
[frames, refreshTick]
|
| 47 |
+
);
|
| 48 |
+
|
| 49 |
return (
|
| 50 |
<div className="ml-8 mr-4 mb-2">
|
| 51 |
{/* Joint visualization graph */}
|
| 52 |
+
<TeleoperatorJointGraph
|
| 53 |
+
frames={visibleFrames}
|
| 54 |
+
refreshTick={refreshTick}
|
| 55 |
+
isRecording={isRecording}
|
| 56 |
+
/>
|
| 57 |
+
|
| 58 |
{/* Frames container with horizontal scroll */}
|
| 59 |
<div className="bg-gray-800/50 rounded-md overflow-hidden">
|
| 60 |
+
<div className="overflow-x-auto">
|
| 61 |
+
{/* Frames header */}
|
| 62 |
+
<table className="w-full min-w-max table-fixed">
|
| 63 |
+
<thead>
|
| 64 |
+
<tr className="text-xs font-medium bg-gray-800/80 text-gray-300">
|
| 65 |
+
<th className="w-16 px-2 py-1 text-left">Frame</th>
|
| 66 |
+
<th className="w-64 px-2 py-1 text-left">Timestamp</th>
|
| 67 |
+
<th className="w-[300px] px-2 py-1 text-left">Action</th>
|
| 68 |
+
<th className="w-[500px] px-2 py-1 text-left">State</th>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 69 |
</tr>
|
| 70 |
+
</thead>
|
| 71 |
+
|
| 72 |
+
{/* Frame rows */}
|
| 73 |
+
<tbody className="max-h-60 overflow-y-auto">
|
| 74 |
+
{visibleFrames.map(
|
| 75 |
+
(frame: NonIndexedLeRobotDatasetRow, frameIndex: number) => (
|
| 76 |
+
<tr
|
| 77 |
+
key={frameIndex}
|
| 78 |
+
className="text-xs border-t border-gray-700/50"
|
| 79 |
+
>
|
| 80 |
+
<td className="w-16 px-2 py-1 font-mono whitespace-nowrap">
|
| 81 |
+
{frameIndex}
|
| 82 |
+
</td>
|
| 83 |
+
<td className="w-64 px-2 py-1 font-mono whitespace-nowrap">
|
| 84 |
+
{Number.isFinite(frame.timestamp)
|
| 85 |
+
? frame.timestamp.toFixed(3)
|
| 86 |
+
: String(frame.timestamp)}
|
| 87 |
+
</td>
|
| 88 |
+
<td className="w-[200px] px-2 py-1 font-mono whitespace-pre-wrap align-top">
|
| 89 |
+
{Object.keys(frame.action).length > 0
|
| 90 |
+
? formatArrayAsColumn(frame.action)
|
| 91 |
+
: "-"}
|
| 92 |
+
</td>
|
| 93 |
+
<td className="w-[300px] px-2 py-1 font-mono whitespace-pre-wrap align-top">
|
| 94 |
+
{Object.keys(frame["observation.state"]).length > 0
|
| 95 |
+
? formatArrayAsColumn(frame["observation.state"])
|
| 96 |
+
: "-"}
|
| 97 |
+
</td>
|
| 98 |
+
</tr>
|
| 99 |
+
)
|
| 100 |
+
)}
|
| 101 |
+
</tbody>
|
| 102 |
+
</table>
|
| 103 |
+
</div>
|
| 104 |
</div>
|
| 105 |
</div>
|
| 106 |
);
|
| 107 |
+
});
|
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
import React from "react";
|
| 2 |
import {
|
| 3 |
LineChart,
|
| 4 |
Line,
|
|
@@ -7,15 +7,21 @@ import {
|
|
| 7 |
CartesianGrid,
|
| 8 |
Tooltip,
|
| 9 |
Legend,
|
| 10 |
-
ResponsiveContainer
|
| 11 |
} from "recharts";
|
| 12 |
import { NonIndexedLeRobotDatasetRow } from "@lerobot/web";
|
| 13 |
|
| 14 |
interface TeleoperatorJointGraphProps {
|
| 15 |
frames: NonIndexedLeRobotDatasetRow[];
|
|
|
|
| 16 |
}
|
| 17 |
|
| 18 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
// Skip rendering if no frames
|
| 20 |
if (!frames || frames.length === 0) {
|
| 21 |
return null;
|
|
@@ -23,100 +29,129 @@ export function TeleoperatorJointGraph({ frames }: TeleoperatorJointGraphProps)
|
|
| 23 |
|
| 24 |
// Use hardcoded joint names that match the LeRobot dataset format
|
| 25 |
const jointNames = [
|
| 26 |
-
"shoulder_pan",
|
| 27 |
"shoulder_lift",
|
| 28 |
"elbow_flex",
|
| 29 |
"wrist_flex",
|
| 30 |
"wrist_roll",
|
| 31 |
-
"gripper"
|
| 32 |
];
|
| 33 |
-
|
| 34 |
// Generate a color palette for the joints
|
| 35 |
const colors = [
|
| 36 |
-
"#8884d8",
|
| 37 |
-
"#
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 38 |
];
|
| 39 |
-
|
| 40 |
// Prepare data for the chart - handling arrays
|
| 41 |
-
const chartData =
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 54 |
}
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
}
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 71 |
// Create lines for each joint
|
| 72 |
-
const linesToRender = jointNames.flatMap(jointName => [
|
| 73 |
{
|
| 74 |
key: `action_${jointName}`,
|
| 75 |
dataKey: `action_${jointName}`,
|
| 76 |
name: `Action: ${jointName}`,
|
| 77 |
-
isDotted: true
|
| 78 |
},
|
| 79 |
{
|
| 80 |
key: `state_${jointName}`,
|
| 81 |
dataKey: `state_${jointName}`,
|
| 82 |
name: `State: ${jointName}`,
|
| 83 |
-
isDotted: false
|
| 84 |
-
}
|
| 85 |
]);
|
| 86 |
|
| 87 |
return (
|
| 88 |
<div className="w-full bg-gray-800/50 rounded-md p-4 mb-4">
|
| 89 |
-
<h3 className="text-sm font-medium text-gray-300 mb-2">
|
|
|
|
|
|
|
| 90 |
<ResponsiveContainer width="100%" height={300}>
|
| 91 |
<LineChart
|
| 92 |
-
data={
|
| 93 |
margin={{
|
| 94 |
top: 5,
|
| 95 |
right: 30,
|
| 96 |
left: 20,
|
| 97 |
-
bottom: 5
|
| 98 |
}}
|
| 99 |
>
|
| 100 |
<CartesianGrid strokeDasharray="3 3" stroke="#444" />
|
| 101 |
-
<XAxis
|
| 102 |
-
dataKey="name"
|
| 103 |
-
label={{
|
| 104 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 105 |
/>
|
| 106 |
<YAxis stroke="#aaa" />
|
| 107 |
-
<Tooltip
|
| 108 |
-
contentStyle={{ backgroundColor:
|
| 109 |
-
labelStyle={{ color:
|
| 110 |
-
itemStyle={{ color:
|
| 111 |
/>
|
| 112 |
<Legend />
|
| 113 |
-
|
| 114 |
{/* Render all lines */}
|
| 115 |
{linesToRender.map((lineConfig, index) => {
|
| 116 |
-
const jointName = lineConfig.dataKey.replace(
|
|
|
|
|
|
|
|
|
|
| 117 |
const jointIndex = jointNames.indexOf(jointName);
|
| 118 |
-
const colorIndex =
|
| 119 |
-
|
|
|
|
| 120 |
return (
|
| 121 |
<Line
|
| 122 |
key={lineConfig.key}
|
|
@@ -134,4 +169,4 @@ export function TeleoperatorJointGraph({ frames }: TeleoperatorJointGraphProps)
|
|
| 134 |
</ResponsiveContainer>
|
| 135 |
</div>
|
| 136 |
);
|
| 137 |
-
}
|
|
|
|
| 1 |
+
import React, { memo, useMemo } from "react";
|
| 2 |
import {
|
| 3 |
LineChart,
|
| 4 |
Line,
|
|
|
|
| 7 |
CartesianGrid,
|
| 8 |
Tooltip,
|
| 9 |
Legend,
|
| 10 |
+
ResponsiveContainer,
|
| 11 |
} from "recharts";
|
| 12 |
import { NonIndexedLeRobotDatasetRow } from "@lerobot/web";
|
| 13 |
|
| 14 |
interface TeleoperatorJointGraphProps {
|
| 15 |
frames: NonIndexedLeRobotDatasetRow[];
|
| 16 |
+
refreshTick?: number;
|
| 17 |
}
|
| 18 |
|
| 19 |
+
const MAX_POINTS = 5000;
|
| 20 |
+
|
| 21 |
+
export const TeleoperatorJointGraph = memo(function TeleoperatorJointGraph({
|
| 22 |
+
frames,
|
| 23 |
+
refreshTick,
|
| 24 |
+
}: TeleoperatorJointGraphProps) {
|
| 25 |
// Skip rendering if no frames
|
| 26 |
if (!frames || frames.length === 0) {
|
| 27 |
return null;
|
|
|
|
| 29 |
|
| 30 |
// Use hardcoded joint names that match the LeRobot dataset format
|
| 31 |
const jointNames = [
|
| 32 |
+
"shoulder_pan",
|
| 33 |
"shoulder_lift",
|
| 34 |
"elbow_flex",
|
| 35 |
"wrist_flex",
|
| 36 |
"wrist_roll",
|
| 37 |
+
"gripper",
|
| 38 |
];
|
| 39 |
+
|
| 40 |
// Generate a color palette for the joints
|
| 41 |
const colors = [
|
| 42 |
+
"#8884d8",
|
| 43 |
+
"#82ca9d",
|
| 44 |
+
"#ffc658",
|
| 45 |
+
"#ff8042",
|
| 46 |
+
"#0088fe",
|
| 47 |
+
"#00C49F",
|
| 48 |
+
"#FFBB28",
|
| 49 |
+
"#FF8042",
|
| 50 |
+
"#a4de6c",
|
| 51 |
+
"#d0ed57",
|
| 52 |
];
|
| 53 |
+
|
| 54 |
// Prepare data for the chart - handling arrays
|
| 55 |
+
const chartData = useMemo(
|
| 56 |
+
() =>
|
| 57 |
+
frames.map((frame, index) => {
|
| 58 |
+
// Create base data point with index
|
| 59 |
+
const dataPoint: any = {
|
| 60 |
+
name: index,
|
| 61 |
+
timestamp: frame.timestamp,
|
| 62 |
+
};
|
| 63 |
+
|
| 64 |
+
// Add action values (assuming action is an array)
|
| 65 |
+
if (Array.isArray(frame.action)) {
|
| 66 |
+
// Map each array index to the corresponding joint name
|
| 67 |
+
jointNames.forEach((jointName, i) => {
|
| 68 |
+
if (i < frame.action.length) {
|
| 69 |
+
dataPoint[`action_${jointName}`] = frame.action[i];
|
| 70 |
+
}
|
| 71 |
+
});
|
| 72 |
}
|
| 73 |
+
|
| 74 |
+
// Add observation state values (assuming observation.state is an array)
|
| 75 |
+
if (Array.isArray(frame["observation.state"])) {
|
| 76 |
+
// Map each array index to the corresponding joint name
|
| 77 |
+
jointNames.forEach((jointName, i) => {
|
| 78 |
+
if (i < frame["observation.state"].length) {
|
| 79 |
+
dataPoint[`state_${jointName}`] = frame["observation.state"][i];
|
| 80 |
+
}
|
| 81 |
+
});
|
| 82 |
}
|
| 83 |
+
|
| 84 |
+
return dataPoint;
|
| 85 |
+
}),
|
| 86 |
+
[frames, refreshTick]
|
| 87 |
+
);
|
| 88 |
+
|
| 89 |
+
const limitedData = useMemo(() => {
|
| 90 |
+
if (!chartData) return [] as typeof chartData;
|
| 91 |
+
const len = chartData.length;
|
| 92 |
+
if (len <= MAX_POINTS) return chartData;
|
| 93 |
+
return chartData.slice(len - MAX_POINTS, len);
|
| 94 |
+
}, [chartData]);
|
| 95 |
+
|
| 96 |
// Create lines for each joint
|
| 97 |
+
const linesToRender = jointNames.flatMap((jointName) => [
|
| 98 |
{
|
| 99 |
key: `action_${jointName}`,
|
| 100 |
dataKey: `action_${jointName}`,
|
| 101 |
name: `Action: ${jointName}`,
|
| 102 |
+
isDotted: true,
|
| 103 |
},
|
| 104 |
{
|
| 105 |
key: `state_${jointName}`,
|
| 106 |
dataKey: `state_${jointName}`,
|
| 107 |
name: `State: ${jointName}`,
|
| 108 |
+
isDotted: false,
|
| 109 |
+
},
|
| 110 |
]);
|
| 111 |
|
| 112 |
return (
|
| 113 |
<div className="w-full bg-gray-800/50 rounded-md p-4 mb-4">
|
| 114 |
+
<h3 className="text-sm font-medium text-gray-300 mb-2">
|
| 115 |
+
Joint Positions Over Time
|
| 116 |
+
</h3>
|
| 117 |
<ResponsiveContainer width="100%" height={300}>
|
| 118 |
<LineChart
|
| 119 |
+
data={limitedData}
|
| 120 |
margin={{
|
| 121 |
top: 5,
|
| 122 |
right: 30,
|
| 123 |
left: 20,
|
| 124 |
+
bottom: 5,
|
| 125 |
}}
|
| 126 |
>
|
| 127 |
<CartesianGrid strokeDasharray="3 3" stroke="#444" />
|
| 128 |
+
<XAxis
|
| 129 |
+
dataKey="name"
|
| 130 |
+
label={{
|
| 131 |
+
value: "Frame Index",
|
| 132 |
+
position: "insideBottomRight",
|
| 133 |
+
offset: -10,
|
| 134 |
+
}}
|
| 135 |
+
stroke="#aaa"
|
| 136 |
/>
|
| 137 |
<YAxis stroke="#aaa" />
|
| 138 |
+
<Tooltip
|
| 139 |
+
contentStyle={{ backgroundColor: "#333", borderColor: "#555" }}
|
| 140 |
+
labelStyle={{ color: "#eee" }}
|
| 141 |
+
itemStyle={{ color: "#eee" }}
|
| 142 |
/>
|
| 143 |
<Legend />
|
| 144 |
+
|
| 145 |
{/* Render all lines */}
|
| 146 |
{linesToRender.map((lineConfig, index) => {
|
| 147 |
+
const jointName = lineConfig.dataKey.replace(
|
| 148 |
+
/^(action|state)_/,
|
| 149 |
+
""
|
| 150 |
+
);
|
| 151 |
const jointIndex = jointNames.indexOf(jointName);
|
| 152 |
+
const colorIndex =
|
| 153 |
+
jointIndex >= 0 ? jointIndex : index % colors.length;
|
| 154 |
+
|
| 155 |
return (
|
| 156 |
<Line
|
| 157 |
key={lineConfig.key}
|
|
|
|
| 169 |
</ResponsiveContainer>
|
| 170 |
</div>
|
| 171 |
);
|
| 172 |
+
});
|
|
@@ -0,0 +1,243 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import * as hub from "@huggingface/hub";
|
| 2 |
+
import { S3Client, HeadBucketCommand } from "@aws-sdk/client-s3";
|
| 3 |
+
import { Upload } from "@aws-sdk/lib-storage";
|
| 4 |
+
|
| 5 |
+
type FileArray = Array<{ path: string; content: Blob | Uint8Array }>;
|
| 6 |
+
|
| 7 |
+
/**
|
| 8 |
+
* Uploads a leRobot dataset to Hugging Face
|
| 9 |
+
*
|
| 10 |
+
* @param files Array of files to upload
|
| 11 |
+
* @param accessToken Hugging Face access token
|
| 12 |
+
* @param repoName Repository name (will be created if it doesn't exist)
|
| 13 |
+
* @param privateRepo Whether the repo should be private (default: false)
|
| 14 |
+
* @returns EventTarget that emits 'repoCreated', 'progress', 'finished', and 'error' events
|
| 15 |
+
*/
|
| 16 |
+
export async function uploadToHuggingFace(
|
| 17 |
+
files: FileArray,
|
| 18 |
+
accessToken: string,
|
| 19 |
+
repoName: string,
|
| 20 |
+
privateRepo: boolean = false
|
| 21 |
+
): Promise<EventTarget> {
|
| 22 |
+
const eventTarget = new EventTarget();
|
| 23 |
+
|
| 24 |
+
// Run upload asynchronously so UI can subscribe to events immediately
|
| 25 |
+
(async () => {
|
| 26 |
+
try {
|
| 27 |
+
// Get username from token
|
| 28 |
+
const { name: username } = await hub.whoAmI({ accessToken });
|
| 29 |
+
|
| 30 |
+
const repoDesignation = {
|
| 31 |
+
name: `${username}/${repoName}`,
|
| 32 |
+
type: "dataset" as const,
|
| 33 |
+
};
|
| 34 |
+
|
| 35 |
+
// Try to create repo; if it already exists (409), continue and upload
|
| 36 |
+
try {
|
| 37 |
+
await hub.createRepo({
|
| 38 |
+
repo: repoDesignation,
|
| 39 |
+
accessToken,
|
| 40 |
+
license: "mit",
|
| 41 |
+
private: privateRepo,
|
| 42 |
+
});
|
| 43 |
+
eventTarget.dispatchEvent(
|
| 44 |
+
new CustomEvent("repoCreated", { detail: repoDesignation })
|
| 45 |
+
);
|
| 46 |
+
} catch (error: any) {
|
| 47 |
+
const message = (error && (error.message || `${error}`)) as string;
|
| 48 |
+
const isConflict =
|
| 49 |
+
message?.includes("409") ||
|
| 50 |
+
message?.toLowerCase()?.includes("already created") ||
|
| 51 |
+
message?.toLowerCase()?.includes("already exists");
|
| 52 |
+
if (!isConflict) {
|
| 53 |
+
eventTarget.dispatchEvent(new CustomEvent("error", { detail: error }));
|
| 54 |
+
throw error;
|
| 55 |
+
}
|
| 56 |
+
// Repo exists: proceed as created
|
| 57 |
+
eventTarget.dispatchEvent(
|
| 58 |
+
new CustomEvent("repoCreated", { detail: repoDesignation })
|
| 59 |
+
);
|
| 60 |
+
}
|
| 61 |
+
|
| 62 |
+
// Upload files to v2.1 branch, fallback to main if branch doesn't exist
|
| 63 |
+
let uploadedBranch = "v2.1";
|
| 64 |
+
try {
|
| 65 |
+
await uploadFilesWithProgress(
|
| 66 |
+
files,
|
| 67 |
+
accessToken,
|
| 68 |
+
repoDesignation,
|
| 69 |
+
uploadedBranch,
|
| 70 |
+
eventTarget
|
| 71 |
+
);
|
| 72 |
+
} catch (error: any) {
|
| 73 |
+
const message = (error && (error.message || `${error}`)) as string;
|
| 74 |
+
const invalidRev = message?.toLowerCase()?.includes("invalid rev id");
|
| 75 |
+
if (invalidRev) {
|
| 76 |
+
console.warn(
|
| 77 |
+
"v2.1 branch not available. Falling back to main branch."
|
| 78 |
+
);
|
| 79 |
+
uploadedBranch = "main";
|
| 80 |
+
await uploadFilesWithProgress(
|
| 81 |
+
files,
|
| 82 |
+
accessToken,
|
| 83 |
+
repoDesignation,
|
| 84 |
+
uploadedBranch,
|
| 85 |
+
eventTarget
|
| 86 |
+
);
|
| 87 |
+
} else {
|
| 88 |
+
throw error;
|
| 89 |
+
}
|
| 90 |
+
}
|
| 91 |
+
|
| 92 |
+
console.log(
|
| 93 |
+
`Successfully uploaded dataset to ${username}/${repoName} (${uploadedBranch})`
|
| 94 |
+
);
|
| 95 |
+
eventTarget.dispatchEvent(
|
| 96 |
+
new CustomEvent("finished", { detail: { branch: uploadedBranch } })
|
| 97 |
+
);
|
| 98 |
+
} catch (error) {
|
| 99 |
+
console.error("Error uploading to Hugging Face:", error);
|
| 100 |
+
eventTarget.dispatchEvent(new CustomEvent("error", { detail: error }));
|
| 101 |
+
}
|
| 102 |
+
})();
|
| 103 |
+
|
| 104 |
+
return eventTarget;
|
| 105 |
+
}
|
| 106 |
+
|
| 107 |
+
/**
|
| 108 |
+
* Uploads a leRobot dataset to Amazon S3
|
| 109 |
+
*
|
| 110 |
+
* @param files Array of files to upload
|
| 111 |
+
* @param bucketName S3 bucket name
|
| 112 |
+
* @param accessKeyId AWS access key ID
|
| 113 |
+
* @param secretAccessKey AWS secret access key
|
| 114 |
+
* @param region AWS region (default: us-east-1)
|
| 115 |
+
* @param prefix Optional prefix/folder for uploaded files
|
| 116 |
+
* @returns EventTarget that emits 'bucketVerified', 'progress', 'finished', and 'error' events
|
| 117 |
+
*/
|
| 118 |
+
export async function uploadToS3(
|
| 119 |
+
files: FileArray,
|
| 120 |
+
bucketName: string,
|
| 121 |
+
accessKeyId: string,
|
| 122 |
+
secretAccessKey: string,
|
| 123 |
+
region: string = "us-east-1",
|
| 124 |
+
prefix: string = ""
|
| 125 |
+
): Promise<EventTarget> {
|
| 126 |
+
const eventTarget = new EventTarget();
|
| 127 |
+
|
| 128 |
+
// Run upload asynchronously
|
| 129 |
+
(async () => {
|
| 130 |
+
try {
|
| 131 |
+
const s3Client = new S3Client({
|
| 132 |
+
region,
|
| 133 |
+
credentials: {
|
| 134 |
+
accessKeyId,
|
| 135 |
+
secretAccessKey,
|
| 136 |
+
},
|
| 137 |
+
});
|
| 138 |
+
|
| 139 |
+
// Verify bucket exists
|
| 140 |
+
try {
|
| 141 |
+
await s3Client.send(
|
| 142 |
+
new HeadBucketCommand({ Bucket: bucketName })
|
| 143 |
+
);
|
| 144 |
+
eventTarget.dispatchEvent(
|
| 145 |
+
new CustomEvent("bucketVerified", {
|
| 146 |
+
detail: { bucketName, region },
|
| 147 |
+
})
|
| 148 |
+
);
|
| 149 |
+
} catch (error: any) {
|
| 150 |
+
const message = error?.message || `${error}`;
|
| 151 |
+
if (message.includes("404") || message.includes("NotFound")) {
|
| 152 |
+
throw new Error(
|
| 153 |
+
`S3 bucket "${bucketName}" not found in region "${region}"`
|
| 154 |
+
);
|
| 155 |
+
}
|
| 156 |
+
throw error;
|
| 157 |
+
}
|
| 158 |
+
|
| 159 |
+
// Upload files
|
| 160 |
+
for (const file of files) {
|
| 161 |
+
const key = prefix ? `${prefix}/${file.path}` : file.path;
|
| 162 |
+
|
| 163 |
+
const upload = new Upload({
|
| 164 |
+
client: s3Client,
|
| 165 |
+
params: {
|
| 166 |
+
Bucket: bucketName,
|
| 167 |
+
Key: key,
|
| 168 |
+
Body:
|
| 169 |
+
file.content instanceof Blob
|
| 170 |
+
? Buffer.from(await file.content.arrayBuffer())
|
| 171 |
+
: Buffer.from(file.content),
|
| 172 |
+
},
|
| 173 |
+
});
|
| 174 |
+
|
| 175 |
+
upload.on("httpUploadProgress", (progress) => {
|
| 176 |
+
eventTarget.dispatchEvent(
|
| 177 |
+
new CustomEvent("progress", { detail: { file: file.path, progress } })
|
| 178 |
+
);
|
| 179 |
+
});
|
| 180 |
+
|
| 181 |
+
await upload.done();
|
| 182 |
+
console.log(`Uploaded ${key}`);
|
| 183 |
+
}
|
| 184 |
+
|
| 185 |
+
console.log(
|
| 186 |
+
`Successfully uploaded dataset to S3 bucket: ${bucketName}${prefix ? `/${prefix}` : ""}`
|
| 187 |
+
);
|
| 188 |
+
eventTarget.dispatchEvent(
|
| 189 |
+
new CustomEvent("finished", {
|
| 190 |
+
detail: { bucketName, prefix, filesCount: files.length },
|
| 191 |
+
})
|
| 192 |
+
);
|
| 193 |
+
} catch (error) {
|
| 194 |
+
console.error("Error uploading to S3:", error);
|
| 195 |
+
eventTarget.dispatchEvent(new CustomEvent("error", { detail: error }));
|
| 196 |
+
}
|
| 197 |
+
})();
|
| 198 |
+
|
| 199 |
+
return eventTarget;
|
| 200 |
+
}
|
| 201 |
+
|
| 202 |
+
/**
|
| 203 |
+
* Helper function to upload files to Hugging Face with progress tracking
|
| 204 |
+
*/
|
| 205 |
+
async function uploadFilesWithProgress(
|
| 206 |
+
files: FileArray,
|
| 207 |
+
accessToken: string,
|
| 208 |
+
repoDesignation: { name: string; type: "dataset" },
|
| 209 |
+
branch: string,
|
| 210 |
+
eventTarget: EventTarget
|
| 211 |
+
): Promise<void> {
|
| 212 |
+
const referenceId = `lerobot-upload-${Date.now()}`;
|
| 213 |
+
|
| 214 |
+
// Upload each file
|
| 215 |
+
for (const file of files) {
|
| 216 |
+
let blob: Blob;
|
| 217 |
+
|
| 218 |
+
if (file.content instanceof Blob) {
|
| 219 |
+
blob = file.content;
|
| 220 |
+
} else {
|
| 221 |
+
blob = new Blob([file.content]);
|
| 222 |
+
}
|
| 223 |
+
|
| 224 |
+
await hub.uploadFile({
|
| 225 |
+
repo: repoDesignation,
|
| 226 |
+
credentials: { accessToken },
|
| 227 |
+
file: {
|
| 228 |
+
content: blob,
|
| 229 |
+
path: file.path,
|
| 230 |
+
},
|
| 231 |
+
revision: branch,
|
| 232 |
+
});
|
| 233 |
+
|
| 234 |
+
eventTarget.dispatchEvent(
|
| 235 |
+
new CustomEvent("progress", {
|
| 236 |
+
detail: { file: file.path, referenceId },
|
| 237 |
+
})
|
| 238 |
+
);
|
| 239 |
+
|
| 240 |
+
console.log(`Uploaded ${file.path}`);
|
| 241 |
+
}
|
| 242 |
+
}
|
| 243 |
+
|
|
@@ -1,7 +1,7 @@
|
|
| 1 |
{
|
| 2 |
"name": "lerobot",
|
| 3 |
"version": "0.0.0",
|
| 4 |
-
"description": "
|
| 5 |
"type": "module",
|
| 6 |
"workspaces": [
|
| 7 |
"packages/*"
|
|
|
|
| 1 |
{
|
| 2 |
"name": "lerobot",
|
| 3 |
"version": "0.0.0",
|
| 4 |
+
"description": "Control your robots with JS, inspired by LeRobot (python)",
|
| 5 |
"type": "module",
|
| 6 |
"workspaces": [
|
| 7 |
"packages/*"
|
|
@@ -20,7 +20,13 @@ yarn add @lerobot/web
|
|
| 20 |
## Quick Start
|
| 21 |
|
| 22 |
```typescript
|
| 23 |
-
import {
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 24 |
|
| 25 |
// 1. find available robot ports (shows browser port dialog)
|
| 26 |
console.log("🔍 finding available robot ports...");
|
|
@@ -387,72 +393,68 @@ await releaseMotors(robot, [1, 2, 3]);
|
|
| 387 |
|
| 388 |
---
|
| 389 |
|
| 390 |
-
##
|
| 391 |
|
| 392 |
-
|
| 393 |
-
|
| 394 |
-
### `LeRobotDatasetRecorder`
|
| 395 |
-
|
| 396 |
-
Records teleoperator movements and camera streams, then exports them in the LeRobot dataset format.
|
| 397 |
|
| 398 |
```typescript
|
| 399 |
-
import {
|
| 400 |
-
|
| 401 |
-
// Create a recorder with teleoperator and video streams
|
| 402 |
-
const recorder = new LeRobotDatasetRecorder(
|
| 403 |
-
[teleoperator], // Array of teleoperators to record, currently only supports 1 teleoperator
|
| 404 |
-
{ "main": videoStream }, // Video streams by camera key
|
| 405 |
-
30, // Target FPS
|
| 406 |
-
"Pick and place task" // Task description
|
| 407 |
-
);
|
| 408 |
-
|
| 409 |
-
// Start recording
|
| 410 |
-
await recorder.startRecording();
|
| 411 |
|
| 412 |
-
// .
|
| 413 |
-
|
| 414 |
-
|
| 415 |
-
|
| 416 |
-
|
| 417 |
-
// Export the dataset in various formats
|
| 418 |
-
// 1. As a downloadable zip file
|
| 419 |
-
await recorder.exportForLeRobot('zip-download');
|
| 420 |
-
|
| 421 |
-
// 2. Upload to Hugging Face
|
| 422 |
-
const hfUploader = await recorder.exportForLeRobot('huggingface', {
|
| 423 |
-
repoName: 'my-robot-dataset',
|
| 424 |
-
accessToken: 'hf_...',
|
| 425 |
});
|
| 426 |
|
| 427 |
-
//
|
| 428 |
-
const
|
| 429 |
-
|
| 430 |
-
|
| 431 |
-
|
| 432 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 433 |
});
|
| 434 |
-
```
|
| 435 |
|
| 436 |
-
|
|
|
|
|
|
|
| 437 |
|
| 438 |
-
|
| 439 |
-
|
| 440 |
-
- **Multiple Export Formats**: Supports local download, Hugging Face, and S3 upload
|
| 441 |
-
- **LeRobot Dataset Format**: Follows the standard format for compatibility with ML models
|
| 442 |
|
| 443 |
-
|
|
|
|
|
|
|
|
|
|
| 444 |
|
| 445 |
-
####
|
| 446 |
|
| 447 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 448 |
|
| 449 |
-
|
| 450 |
-
/data/chunk-000/file-000.parquet # Teleoperator data
|
| 451 |
-
/videos/observation.images.{camera-key}/chunk-000/file-000.mp4 # Video data
|
| 452 |
-
/metadata.json # Dataset metadata
|
| 453 |
-
/statistics.json # Dataset statistics (currently incorrect)
|
| 454 |
-
/README.md # Dataset documentation
|
| 455 |
-
```
|
| 456 |
|
| 457 |
## Browser Requirements
|
| 458 |
|
|
|
|
| 20 |
## Quick Start
|
| 21 |
|
| 22 |
```typescript
|
| 23 |
+
import {
|
| 24 |
+
findPort,
|
| 25 |
+
releaseMotors,
|
| 26 |
+
calibrate,
|
| 27 |
+
teleoperate,
|
| 28 |
+
record,
|
| 29 |
+
} from "@lerobot/web";
|
| 30 |
|
| 31 |
// 1. find available robot ports (shows browser port dialog)
|
| 32 |
console.log("🔍 finding available robot ports...");
|
|
|
|
| 393 |
|
| 394 |
---
|
| 395 |
|
| 396 |
+
### `record(config): Promise<RecordProcess>`
|
| 397 |
|
| 398 |
+
Records robot motor positions and teleoperation data using a clean function API that matches the patterns established by `calibrate()` and `teleoperate()`.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 399 |
|
| 400 |
```typescript
|
| 401 |
+
import { teleoperate, record } from "@lerobot/web";
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 402 |
|
| 403 |
+
// 1. Create teleoperation first
|
| 404 |
+
const teleoperationProcess = await teleoperate({
|
| 405 |
+
robot: connectedRobot,
|
| 406 |
+
teleop: { type: "keyboard" },
|
| 407 |
+
calibrationData: calibrationData,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 408 |
});
|
| 409 |
|
| 410 |
+
// 2. Create recording with teleoperator
|
| 411 |
+
const recordProcess = await record({
|
| 412 |
+
teleoperator: teleoperationProcess.teleoperator,
|
| 413 |
+
videoStreams: {
|
| 414 |
+
main: mainCameraStream,
|
| 415 |
+
},
|
| 416 |
+
robotType: "so100",
|
| 417 |
+
options: {
|
| 418 |
+
fps: 30,
|
| 419 |
+
taskDescription: "Pick and place task",
|
| 420 |
+
},
|
| 421 |
});
|
|
|
|
| 422 |
|
| 423 |
+
// 3. Start both processes
|
| 424 |
+
teleoperationProcess.start();
|
| 425 |
+
recordProcess.start();
|
| 426 |
|
| 427 |
+
// 4. Manage recording during operation
|
| 428 |
+
recordProcess.nextEpisode(); // Start new episode if needed
|
|
|
|
|
|
|
| 429 |
|
| 430 |
+
// 5. Stop recording and export
|
| 431 |
+
const robotData = await recordProcess.stop();
|
| 432 |
+
await recordProcess.exportForLeRobot("zip-download");
|
| 433 |
+
```
|
| 434 |
|
| 435 |
+
#### Options
|
| 436 |
|
| 437 |
+
- `config: RecordConfig`
|
| 438 |
+
- `teleoperator: WebTeleoperator` - The teleoperator to record from
|
| 439 |
+
- `videoStreams?: { [name: string]: MediaStream }` - Optional camera streams (e.g., `{ main: stream1, wrist: stream2 }`)
|
| 440 |
+
- `robotType?: string` - Robot metadata (e.g., "so100")
|
| 441 |
+
- `options?: RecordOptions` - Optional recording configuration:
|
| 442 |
+
- `fps?: number` - Target frames per second (default: 30)
|
| 443 |
+
- `taskDescription?: string` - Task description
|
| 444 |
+
- `onStateUpdate?: (state: RecordingState) => void` - Recording state changes
|
| 445 |
+
|
| 446 |
+
#### Returns: `RecordProcess`
|
| 447 |
+
|
| 448 |
+
- `start(): void` - Start recording
|
| 449 |
+
- `stop(): Promise<RobotRecordingData>` - Stop recording and get data
|
| 450 |
+
- `getState(): RecordingState` - Current recording state
|
| 451 |
+
- `getEpisodeCount(): number` - Get total episodes
|
| 452 |
+
- `nextEpisode(): Promise<number>` - Start new episode
|
| 453 |
+
- `clearEpisodes(): void` - Delete all episodes
|
| 454 |
+
- `addCamera(name: string, stream: MediaStream): void` - Add camera dynamically
|
| 455 |
+
- `exportForLeRobot(format?: "blobs" | "zip" | "zip-download"): Promise<any>` - Export dataset
|
| 456 |
|
| 457 |
+
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 458 |
|
| 459 |
## Browser Requirements
|
| 460 |
|
|
@@ -68,9 +68,6 @@
|
|
| 68 |
"access": "public"
|
| 69 |
},
|
| 70 |
"dependencies": {
|
| 71 |
-
"@aws-sdk/client-s3": "^3.856.0",
|
| 72 |
-
"@aws-sdk/lib-storage": "^3.856.0",
|
| 73 |
-
"@huggingface/hub": "^2.4.0",
|
| 74 |
"apache-arrow": "^21.0.0",
|
| 75 |
"jszip": "^3.10.1",
|
| 76 |
"parquet-wasm": "^0.6.1"
|
|
|
|
| 68 |
"access": "public"
|
| 69 |
},
|
| 70 |
"dependencies": {
|
|
|
|
|
|
|
|
|
|
| 71 |
"apache-arrow": "^21.0.0",
|
| 72 |
"jszip": "^3.10.1",
|
| 73 |
"parquet-wasm": "^0.6.1"
|
|
@@ -1,86 +0,0 @@
|
|
| 1 |
-
import * as hub from "@huggingface/hub";
|
| 2 |
-
import type { RepoDesignation } from "@huggingface/hub";
|
| 3 |
-
import type { ContentSource } from "@huggingface/hub";
|
| 4 |
-
|
| 5 |
-
type FileArray = Array<URL | File | { path: string; content: ContentSource }>;
|
| 6 |
-
/**
|
| 7 |
-
* Uploads a leRobot dataset to huggingface
|
| 8 |
-
*/
|
| 9 |
-
|
| 10 |
-
export class LeRobotHFUploader extends EventTarget {
|
| 11 |
-
private _repoDesignation: RepoDesignation;
|
| 12 |
-
private _uploaded : boolean;
|
| 13 |
-
private _created_repo : boolean;
|
| 14 |
-
|
| 15 |
-
constructor(username: string, repoName: string) {
|
| 16 |
-
super();
|
| 17 |
-
this._repoDesignation = {
|
| 18 |
-
name : `${username}/${repoName}`,
|
| 19 |
-
type : "dataset"
|
| 20 |
-
};
|
| 21 |
-
|
| 22 |
-
this._uploaded = false;
|
| 23 |
-
this._created_repo = false;
|
| 24 |
-
}
|
| 25 |
-
|
| 26 |
-
/**
|
| 27 |
-
* Returns whether the repository has been successfully created
|
| 28 |
-
*/
|
| 29 |
-
get createdRepo(): boolean {
|
| 30 |
-
return this._created_repo;
|
| 31 |
-
}
|
| 32 |
-
|
| 33 |
-
get uploaded() : boolean {
|
| 34 |
-
return this._uploaded;
|
| 35 |
-
}
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
/**
|
| 39 |
-
* Uploads the dataset to huggingface
|
| 40 |
-
*
|
| 41 |
-
* A referenceId is used to be able to track progress of the upload,
|
| 42 |
-
* this provides a progressEvent from huggingface.js (see : https://github.com/huggingface/huggingface.js/blob/main/packages/hub/README.md#usage)
|
| 43 |
-
*
|
| 44 |
-
* both this and huggingface.js have pretty bad documentation, some exploration will be required (sorry!, I have university, and work and I already feel guilty procastinating those to to write this)
|
| 45 |
-
*
|
| 46 |
-
* @param dataset The dataset to upload
|
| 47 |
-
* @param accessToken The access token for huggingface
|
| 48 |
-
* @param referenceId The reference id for the upload, to track it (optional)
|
| 49 |
-
*/
|
| 50 |
-
async createRepoAndUploadFiles(files : FileArray, accessToken : string, referenceId : string = "") {
|
| 51 |
-
await hub.createRepo({
|
| 52 |
-
repo: this._repoDesignation,
|
| 53 |
-
accessToken: accessToken,
|
| 54 |
-
license: "mit"
|
| 55 |
-
});
|
| 56 |
-
|
| 57 |
-
this._created_repo = true;
|
| 58 |
-
this.dispatchEvent(new CustomEvent("repoCreated", { detail: this._repoDesignation }));
|
| 59 |
-
|
| 60 |
-
const uploadPromises : Promise<void>[] = [];
|
| 61 |
-
uploadPromises.push(this.uploadFilesWithProgress(files, accessToken, referenceId));
|
| 62 |
-
|
| 63 |
-
await Promise.all(uploadPromises);
|
| 64 |
-
}
|
| 65 |
-
|
| 66 |
-
/**
|
| 67 |
-
* Uploads files to huggingface with progress events
|
| 68 |
-
*
|
| 69 |
-
* @param files The files to upload
|
| 70 |
-
* @param accessToken The access token for huggingface
|
| 71 |
-
* @param referenceId The reference id for the upload, to track it (optional)
|
| 72 |
-
*/
|
| 73 |
-
async uploadFilesWithProgress(files : FileArray, accessToken : string, referenceId : string = "") {
|
| 74 |
-
for await (const progressEvent of hub.uploadFilesWithProgress({
|
| 75 |
-
repo: this._repoDesignation,
|
| 76 |
-
accessToken: accessToken,
|
| 77 |
-
files: files,
|
| 78 |
-
})) {
|
| 79 |
-
this.dispatchEvent(new CustomEvent("progress", { detail: {
|
| 80 |
-
progressEvent,
|
| 81 |
-
repoDesignation: this._repoDesignation,
|
| 82 |
-
referenceId
|
| 83 |
-
} }));
|
| 84 |
-
}
|
| 85 |
-
}
|
| 86 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@@ -9,6 +9,7 @@ export { calibrate } from "./calibrate.js";
|
|
| 9 |
export { teleoperate } from "./teleoperate.js";
|
| 10 |
export { findPort } from "./find_port.js";
|
| 11 |
export { releaseMotors } from "./release_motors.js";
|
|
|
|
| 12 |
|
| 13 |
// Browser support utilities
|
| 14 |
export {
|
|
@@ -51,6 +52,14 @@ export type {
|
|
| 51 |
KeyboardControl,
|
| 52 |
} from "./types/robot-config.js";
|
| 53 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 54 |
// Utilities (advanced users)
|
| 55 |
export { WebSerialPortWrapper } from "./utils/serial-port-wrapper.js";
|
| 56 |
export {
|
|
@@ -64,5 +73,4 @@ export {
|
|
| 64 |
export { KEYBOARD_TELEOPERATOR_DEFAULTS } from "./teleoperators/index.js";
|
| 65 |
|
| 66 |
// Record
|
| 67 |
-
export { LeRobotDatasetRecorder } from "./record.js";
|
| 68 |
-
export { LeRobotHFUploader } from "./hf_uploader.js";
|
|
|
|
| 9 |
export { teleoperate } from "./teleoperate.js";
|
| 10 |
export { findPort } from "./find_port.js";
|
| 11 |
export { releaseMotors } from "./release_motors.js";
|
| 12 |
+
export { record } from "./record.js";
|
| 13 |
|
| 14 |
// Browser support utilities
|
| 15 |
export {
|
|
|
|
| 52 |
KeyboardControl,
|
| 53 |
} from "./types/robot-config.js";
|
| 54 |
|
| 55 |
+
export type {
|
| 56 |
+
RecordConfig,
|
| 57 |
+
RecordProcess,
|
| 58 |
+
RecordingState,
|
| 59 |
+
RecordingData,
|
| 60 |
+
RobotRecordingData,
|
| 61 |
+
} from "./types/recording.js";
|
| 62 |
+
|
| 63 |
// Utilities (advanced users)
|
| 64 |
export { WebSerialPortWrapper } from "./utils/serial-port-wrapper.js";
|
| 65 |
export {
|
|
|
|
| 73 |
export { KEYBOARD_TELEOPERATOR_DEFAULTS } from "./teleoperators/index.js";
|
| 74 |
|
| 75 |
// Record
|
| 76 |
+
export { LeRobotDatasetRecorder, LeRobotEpisode } from "./record.js";
|
|
|
|
@@ -3,12 +3,7 @@ import { MotorConfig } from "./types/teleoperation";
|
|
| 3 |
import * as parquet from "parquet-wasm";
|
| 4 |
import * as arrow from "apache-arrow";
|
| 5 |
import JSZip from "jszip";
|
| 6 |
-
import getMetadataInfo from "./utils/record/metadataInfo";
|
| 7 |
-
import type { VideoInfo } from "./utils/record/metadataInfo";
|
| 8 |
-
import getStats from "./utils/record/stats";
|
| 9 |
import generateREADME from "./utils/record/generateREADME";
|
| 10 |
-
import { LeRobotHFUploader } from "./hf_uploader";
|
| 11 |
-
import { LeRobotS3Uploader } from "./s3_uploader";
|
| 12 |
|
| 13 |
// declare a type leRobot action that's basically an array of numbers
|
| 14 |
interface LeRobotAction {
|
|
@@ -338,12 +333,18 @@ export class LeRobotDatasetRecorder {
|
|
| 338 |
mediaRecorders: { [key: string]: MediaRecorder };
|
| 339 |
videoChunks: { [key: string]: Blob[] };
|
| 340 |
videoBlobs: { [key: string]: Blob };
|
|
|
|
|
|
|
|
|
|
|
|
|
| 341 |
teleoperatorData: LeRobotEpisode[];
|
| 342 |
private _isRecording: boolean;
|
| 343 |
private episodeIndex: number = 0;
|
| 344 |
private taskIndex: number = 0;
|
|
|
|
| 345 |
fps: number;
|
| 346 |
taskDescription: string;
|
|
|
|
| 347 |
|
| 348 |
/**
|
| 349 |
* Ensures BlobPart compatibility across environments by converting Uint8Array
|
|
@@ -382,17 +383,45 @@ export class LeRobotDatasetRecorder {
|
|
| 382 |
this.mediaRecorders = {};
|
| 383 |
this.videoChunks = {};
|
| 384 |
this.videoBlobs = {};
|
|
|
|
| 385 |
this.videoStreams = {};
|
|
|
|
| 386 |
this.teleoperatorData = [];
|
| 387 |
this._isRecording = false;
|
| 388 |
this.fps = fps;
|
| 389 |
this.taskDescription = taskDescription;
|
|
|
|
| 390 |
|
| 391 |
for (const [key, stream] of Object.entries(videoStreams)) {
|
| 392 |
this.addVideoStream(key, stream);
|
| 393 |
}
|
| 394 |
}
|
| 395 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 396 |
get isRecording(): boolean {
|
| 397 |
return this._isRecording;
|
| 398 |
}
|
|
@@ -407,21 +436,18 @@ export class LeRobotDatasetRecorder {
|
|
| 407 |
* @param stream The media stream to record from
|
| 408 |
*/
|
| 409 |
addVideoStream(key: string, stream: MediaStream) {
|
| 410 |
-
console.log("Adding video stream", key);
|
| 411 |
if (this._isRecording) {
|
| 412 |
throw new Error("Cannot add video streams while recording");
|
| 413 |
}
|
| 414 |
|
| 415 |
// Add to video streams dictionary
|
| 416 |
this.videoStreams[key] = stream;
|
| 417 |
-
|
| 418 |
-
// Initialize MediaRecorder for this stream
|
| 419 |
-
this.mediaRecorders[key] = new MediaRecorder(stream, {
|
| 420 |
-
mimeType: "video/mp4",
|
| 421 |
-
});
|
| 422 |
-
|
| 423 |
-
// add a video chunk array for this stream
|
| 424 |
this.videoChunks[key] = [];
|
|
|
|
|
|
|
|
|
|
|
|
|
| 425 |
}
|
| 426 |
|
| 427 |
/**
|
|
@@ -466,7 +492,6 @@ export class LeRobotDatasetRecorder {
|
|
| 466 |
* Starts recording for all teleoperators and video streams
|
| 467 |
*/
|
| 468 |
startRecording() {
|
| 469 |
-
console.log("Starting recording");
|
| 470 |
if (this._isRecording) {
|
| 471 |
console.warn("Recording already in progress");
|
| 472 |
return;
|
|
@@ -474,19 +499,26 @@ export class LeRobotDatasetRecorder {
|
|
| 474 |
|
| 475 |
this._isRecording = true;
|
| 476 |
|
| 477 |
-
//
|
|
|
|
| 478 |
this.teleoperatorData.push(new LeRobotEpisode());
|
|
|
|
| 479 |
|
| 480 |
// Start recording video streams
|
| 481 |
Object.entries(this.videoStreams).forEach(([key, stream]) => {
|
| 482 |
-
//
|
| 483 |
-
const
|
| 484 |
-
|
| 485 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 486 |
|
| 487 |
// Handle data available events
|
| 488 |
mediaRecorder.ondataavailable = (event) => {
|
| 489 |
-
console.log("data available for", key);
|
| 490 |
if (event.data && event.data.size > 0) {
|
| 491 |
this.videoChunks[key].push(event.data);
|
| 492 |
}
|
|
@@ -495,8 +527,6 @@ export class LeRobotDatasetRecorder {
|
|
| 495 |
// Save the recorder and start recording
|
| 496 |
this.mediaRecorders[key] = mediaRecorder;
|
| 497 |
mediaRecorder.start(1000); // Capture in 1-second chunks
|
| 498 |
-
|
| 499 |
-
console.log(`Started recording video stream: ${key}`);
|
| 500 |
});
|
| 501 |
}
|
| 502 |
|
|
@@ -560,8 +590,17 @@ export class LeRobotDatasetRecorder {
|
|
| 560 |
recorder.onstop = () => {
|
| 561 |
// Combine all chunks into a single blob
|
| 562 |
const chunks = this.videoChunks[key] || [];
|
| 563 |
-
const
|
|
|
|
| 564 |
this.videoBlobs[key] = blob;
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 565 |
resolve();
|
| 566 |
};
|
| 567 |
|
|
@@ -579,12 +618,95 @@ export class LeRobotDatasetRecorder {
|
|
| 579 |
};
|
| 580 |
}
|
| 581 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 582 |
/**
|
| 583 |
* Clears the teleoperator data and video blobs
|
| 584 |
*/
|
| 585 |
clearRecording() {
|
| 586 |
this.teleoperatorData = [];
|
| 587 |
this.videoBlobs = {};
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 588 |
}
|
| 589 |
|
| 590 |
/**
|
|
@@ -939,68 +1061,17 @@ export class LeRobotDatasetRecorder {
|
|
| 939 |
* Generates metadata for the dataset
|
| 940 |
* @returns Metadata object for the LeRobot dataset
|
| 941 |
*/
|
| 942 |
-
|
| 943 |
-
|
| 944 |
-
|
| 945 |
-
const total_frames = data.length;
|
| 946 |
-
let total_tasks = 0;
|
| 947 |
-
|
| 948 |
-
for (const row of data) {
|
| 949 |
-
total_episodes = Math.max(total_episodes, row.episode_index);
|
| 950 |
-
total_tasks = Math.max(total_tasks, row.task_index);
|
| 951 |
-
}
|
| 952 |
-
|
| 953 |
-
// Create video info objects for each video stream
|
| 954 |
-
const videos_info: VideoInfo[] = Object.keys(this.videoBlobs).map((key) => {
|
| 955 |
-
// Default values - in a production environment, you would extract
|
| 956 |
-
// these from the actual video metadata using the key to identify the video source
|
| 957 |
-
console.log(`Generating metadata for video stream: ${key}`);
|
| 958 |
-
return {
|
| 959 |
-
height: 480,
|
| 960 |
-
width: 640,
|
| 961 |
-
channels: 3,
|
| 962 |
-
codec: "h264",
|
| 963 |
-
pix_fmt: "yuv420p",
|
| 964 |
-
is_depth_map: false,
|
| 965 |
-
has_audio: false,
|
| 966 |
-
};
|
| 967 |
-
});
|
| 968 |
-
|
| 969 |
-
// Calculate approximate file sizes in MB
|
| 970 |
-
const data_files_size_in_mb = Math.round(data.length * 0.001); // Estimate
|
| 971 |
-
|
| 972 |
-
// Calculate video size by summing the sizes of all video blobs and converting to MB
|
| 973 |
-
let video_files_size_in_mb = 0;
|
| 974 |
-
for (const blob of Object.values(this.videoBlobs)) {
|
| 975 |
-
video_files_size_in_mb += blob.size / (1024 * 1024);
|
| 976 |
-
}
|
| 977 |
-
video_files_size_in_mb = Math.round(video_files_size_in_mb);
|
| 978 |
-
|
| 979 |
-
// Generate and return the metadata
|
| 980 |
-
return getMetadataInfo({
|
| 981 |
-
total_episodes,
|
| 982 |
-
total_frames,
|
| 983 |
-
total_tasks,
|
| 984 |
-
chunks_size: 1000, // Default chunk size
|
| 985 |
-
fps: this.fps,
|
| 986 |
-
splits: { train: `0:${total_episodes}` }, // All episodes in train split
|
| 987 |
-
features: {}, // Additional features can be added here
|
| 988 |
-
videos_info,
|
| 989 |
-
data_files_size_in_mb,
|
| 990 |
-
video_files_size_in_mb,
|
| 991 |
-
});
|
| 992 |
}
|
| 993 |
|
| 994 |
/**
|
| 995 |
* Generates statistics for the dataset
|
| 996 |
* @returns Statistics object for the LeRobot dataset
|
| 997 |
*/
|
| 998 |
-
async getStatistics(
|
| 999 |
-
|
| 1000 |
-
const cameraKeys = Object.keys(this.videoBlobs);
|
| 1001 |
-
|
| 1002 |
-
// Generate stats using the data and camera keys
|
| 1003 |
-
return getStats(data, cameraKeys);
|
| 1004 |
}
|
| 1005 |
|
| 1006 |
/**
|
|
@@ -1057,299 +1128,8 @@ export class LeRobotDatasetRecorder {
|
|
| 1057 |
* Creates the episodes statistics parquet file
|
| 1058 |
* @returns A Uint8Array blob containing the parquet data
|
| 1059 |
*/
|
| 1060 |
-
async getEpisodeStatistics(
|
| 1061 |
-
|
| 1062 |
-
const statistics = await this.getStatistics(data);
|
| 1063 |
-
|
| 1064 |
-
// Calculate total episodes and frames
|
| 1065 |
-
let total_episodes = 0;
|
| 1066 |
-
|
| 1067 |
-
for (let row of data) {
|
| 1068 |
-
total_episodes = Math.max(total_episodes, row.episode_index);
|
| 1069 |
-
}
|
| 1070 |
-
|
| 1071 |
-
total_episodes += 1; // +1 since episodes start from 0
|
| 1072 |
-
|
| 1073 |
-
const episodes: any[] = [];
|
| 1074 |
-
|
| 1075 |
-
// we'll create one row per episode
|
| 1076 |
-
for (
|
| 1077 |
-
let episode_index = 0;
|
| 1078 |
-
episode_index < total_episodes;
|
| 1079 |
-
episode_index++
|
| 1080 |
-
) {
|
| 1081 |
-
// Get data for this episode only
|
| 1082 |
-
const episodeData = data.filter(
|
| 1083 |
-
(row) => row.episode_index === episode_index
|
| 1084 |
-
);
|
| 1085 |
-
|
| 1086 |
-
// Extract timestamps for this episode
|
| 1087 |
-
const timestamps = episodeData.map((row) => row.timestamp);
|
| 1088 |
-
let min_timestamp = Infinity;
|
| 1089 |
-
let max_timestamp = -Infinity;
|
| 1090 |
-
|
| 1091 |
-
for (let timestamp of timestamps) {
|
| 1092 |
-
min_timestamp = Math.min(min_timestamp, timestamp);
|
| 1093 |
-
max_timestamp = Math.max(max_timestamp, timestamp);
|
| 1094 |
-
}
|
| 1095 |
-
|
| 1096 |
-
// Camera keys from video blobs
|
| 1097 |
-
const cameraKeys = Object.keys(this.videoBlobs);
|
| 1098 |
-
|
| 1099 |
-
// Create entry for this episode
|
| 1100 |
-
const episodeEntry: any = {
|
| 1101 |
-
// Basic episode information
|
| 1102 |
-
episode_index: episode_index,
|
| 1103 |
-
"data/chunk_index": 0,
|
| 1104 |
-
"data/file_index": 0,
|
| 1105 |
-
dataset_from_index: 0,
|
| 1106 |
-
dataset_to_index: episodeData.length - 1,
|
| 1107 |
-
length: episodeData.length,
|
| 1108 |
-
tasks: [0], // Task index 0, could be extended for multiple tasks
|
| 1109 |
-
|
| 1110 |
-
// Meta information
|
| 1111 |
-
"meta/episodes/chunk_index": 0,
|
| 1112 |
-
"meta/episodes/file_index": 0,
|
| 1113 |
-
};
|
| 1114 |
-
|
| 1115 |
-
// Add video information for each camera
|
| 1116 |
-
cameraKeys.forEach((key) => {
|
| 1117 |
-
episodeEntry[`videos/observation.images.${key}/chunk_index`] = 0;
|
| 1118 |
-
episodeEntry[`videos/observation.images.${key}/file_index`] = 0;
|
| 1119 |
-
episodeEntry[`videos/observation.images.${key}/from_timestamp`] =
|
| 1120 |
-
min_timestamp;
|
| 1121 |
-
episodeEntry[`videos/observation.images.${key}/to_timestamp`] =
|
| 1122 |
-
max_timestamp;
|
| 1123 |
-
});
|
| 1124 |
-
|
| 1125 |
-
// Add statistics for each field
|
| 1126 |
-
// This is a simplified approach - in a real implementation, you'd calculate
|
| 1127 |
-
// these values for each episode individually
|
| 1128 |
-
|
| 1129 |
-
// Add timestamp statistics
|
| 1130 |
-
episodeEntry["stats/timestamp/min"] = [statistics.timestamp.min];
|
| 1131 |
-
episodeEntry["stats/timestamp/max"] = [statistics.timestamp.max];
|
| 1132 |
-
episodeEntry["stats/timestamp/mean"] = [statistics.timestamp.mean];
|
| 1133 |
-
episodeEntry["stats/timestamp/std"] = [statistics.timestamp.std];
|
| 1134 |
-
episodeEntry["stats/timestamp/count"] = [statistics.timestamp.count];
|
| 1135 |
-
|
| 1136 |
-
// Add frame_index statistics
|
| 1137 |
-
episodeEntry["stats/frame_index/min"] = [statistics.frame_index.min];
|
| 1138 |
-
episodeEntry["stats/frame_index/max"] = [statistics.frame_index.max];
|
| 1139 |
-
episodeEntry["stats/frame_index/mean"] = [statistics.frame_index.mean];
|
| 1140 |
-
episodeEntry["stats/frame_index/std"] = [statistics.frame_index.std];
|
| 1141 |
-
episodeEntry["stats/frame_index/count"] = [statistics.frame_index.count];
|
| 1142 |
-
|
| 1143 |
-
// Add episode_index statistics
|
| 1144 |
-
episodeEntry["stats/episode_index/min"] = [statistics.episode_index.min];
|
| 1145 |
-
episodeEntry["stats/episode_index/max"] = [statistics.episode_index.max];
|
| 1146 |
-
episodeEntry["stats/episode_index/mean"] = [
|
| 1147 |
-
statistics.episode_index.mean,
|
| 1148 |
-
];
|
| 1149 |
-
episodeEntry["stats/episode_index/std"] = [statistics.episode_index.std];
|
| 1150 |
-
episodeEntry["stats/episode_index/count"] = [
|
| 1151 |
-
statistics.episode_index.count,
|
| 1152 |
-
];
|
| 1153 |
-
|
| 1154 |
-
// Add task_index statistics
|
| 1155 |
-
episodeEntry["stats/task_index/min"] = [statistics.task_index.min];
|
| 1156 |
-
episodeEntry["stats/task_index/max"] = [statistics.task_index.max];
|
| 1157 |
-
episodeEntry["stats/task_index/mean"] = [statistics.task_index.mean];
|
| 1158 |
-
episodeEntry["stats/task_index/std"] = [statistics.task_index.std];
|
| 1159 |
-
episodeEntry["stats/task_index/count"] = [statistics.task_index.count];
|
| 1160 |
-
|
| 1161 |
-
// Add index statistics
|
| 1162 |
-
episodeEntry["stats/index/min"] = [0];
|
| 1163 |
-
episodeEntry["stats/index/max"] = [episodeData.length - 1];
|
| 1164 |
-
episodeEntry["stats/index/mean"] = [episodeData.length / 2];
|
| 1165 |
-
episodeEntry["stats/index/std"] = [episodeData.length / 4]; // Approximate std
|
| 1166 |
-
episodeEntry["stats/index/count"] = [episodeData.length];
|
| 1167 |
-
|
| 1168 |
-
// Add action statistics (placeholder)
|
| 1169 |
-
episodeEntry["stats/action/min"] = [0.0];
|
| 1170 |
-
episodeEntry["stats/action/max"] = [1.0];
|
| 1171 |
-
episodeEntry["stats/action/mean"] = [0.5];
|
| 1172 |
-
episodeEntry["stats/action/std"] = [0.2];
|
| 1173 |
-
episodeEntry["stats/action/count"] = [episodeData.length];
|
| 1174 |
-
|
| 1175 |
-
// Add observation.state statistics (placeholder)
|
| 1176 |
-
episodeEntry["stats/observation.state/min"] = [0.0];
|
| 1177 |
-
episodeEntry["stats/observation.state/max"] = [1.0];
|
| 1178 |
-
episodeEntry["stats/observation.state/mean"] = [0.5];
|
| 1179 |
-
episodeEntry["stats/observation.state/std"] = [0.2];
|
| 1180 |
-
episodeEntry["stats/observation.state/count"] = [episodeData.length];
|
| 1181 |
-
|
| 1182 |
-
// Add observation.images statistics for each camera
|
| 1183 |
-
cameraKeys.forEach((key) => {
|
| 1184 |
-
// Get the image statistics from the overall statistics
|
| 1185 |
-
const imageStats = statistics[`observation.images.${key}`] || {
|
| 1186 |
-
min: [[[0.0]], [[0.0]], [[0.0]]],
|
| 1187 |
-
max: [[[255.0]], [[255.0]], [[255.0]]],
|
| 1188 |
-
mean: [[[127.5]], [[127.5]], [[127.5]]],
|
| 1189 |
-
std: [[[50.0]], [[50.0]], [[50.0]]],
|
| 1190 |
-
count: [[[episodeData.length * 3]]],
|
| 1191 |
-
};
|
| 1192 |
-
|
| 1193 |
-
episodeEntry[`stats/observation.images.${key}/min`] = imageStats.min;
|
| 1194 |
-
episodeEntry[`stats/observation.images.${key}/max`] = imageStats.max;
|
| 1195 |
-
episodeEntry[`stats/observation.images.${key}/mean`] = imageStats.mean;
|
| 1196 |
-
episodeEntry[`stats/observation.images.${key}/std`] = imageStats.std;
|
| 1197 |
-
episodeEntry[`stats/observation.images.${key}/count`] =
|
| 1198 |
-
imageStats.count;
|
| 1199 |
-
});
|
| 1200 |
-
|
| 1201 |
-
episodes.push(episodeEntry);
|
| 1202 |
-
}
|
| 1203 |
-
|
| 1204 |
-
// Create vector arrays for each column
|
| 1205 |
-
const columns: any = {};
|
| 1206 |
-
|
| 1207 |
-
// Define column names and default types
|
| 1208 |
-
const columnNames = [
|
| 1209 |
-
"episode_index",
|
| 1210 |
-
"data/chunk_index",
|
| 1211 |
-
"data/file_index",
|
| 1212 |
-
"dataset_from_index",
|
| 1213 |
-
"dataset_to_index",
|
| 1214 |
-
"length",
|
| 1215 |
-
"meta/episodes/chunk_index",
|
| 1216 |
-
"meta/episodes/file_index",
|
| 1217 |
-
"tasks",
|
| 1218 |
-
];
|
| 1219 |
-
|
| 1220 |
-
// Add camera-specific columns
|
| 1221 |
-
const cameraKeys = Object.keys(this.videoBlobs);
|
| 1222 |
-
cameraKeys.forEach((key) => {
|
| 1223 |
-
columnNames.push(
|
| 1224 |
-
`videos/observation.images.${key}/chunk_index`,
|
| 1225 |
-
`videos/observation.images.${key}/file_index`,
|
| 1226 |
-
`videos/observation.images.${key}/from_timestamp`,
|
| 1227 |
-
`videos/observation.images.${key}/to_timestamp`
|
| 1228 |
-
);
|
| 1229 |
-
});
|
| 1230 |
-
|
| 1231 |
-
// Add statistic columns for each field
|
| 1232 |
-
const statFields = [
|
| 1233 |
-
"timestamp",
|
| 1234 |
-
"frame_index",
|
| 1235 |
-
"episode_index",
|
| 1236 |
-
"task_index",
|
| 1237 |
-
"index",
|
| 1238 |
-
"action",
|
| 1239 |
-
"observation.state",
|
| 1240 |
-
];
|
| 1241 |
-
statFields.forEach((field) => {
|
| 1242 |
-
columnNames.push(
|
| 1243 |
-
`stats/${field}/min`,
|
| 1244 |
-
`stats/${field}/max`,
|
| 1245 |
-
`stats/${field}/mean`,
|
| 1246 |
-
`stats/${field}/std`,
|
| 1247 |
-
`stats/${field}/count`
|
| 1248 |
-
);
|
| 1249 |
-
});
|
| 1250 |
-
|
| 1251 |
-
// Add image statistic columns for each camera
|
| 1252 |
-
cameraKeys.forEach((key) => {
|
| 1253 |
-
columnNames.push(
|
| 1254 |
-
`stats/observation.images.${key}/min`,
|
| 1255 |
-
`stats/observation.images.${key}/max`,
|
| 1256 |
-
`stats/observation.images.${key}/mean`,
|
| 1257 |
-
`stats/observation.images.${key}/std`,
|
| 1258 |
-
`stats/observation.images.${key}/count`
|
| 1259 |
-
);
|
| 1260 |
-
});
|
| 1261 |
-
|
| 1262 |
-
// Create vector arrays for each column
|
| 1263 |
-
columnNames.forEach((columnName) => {
|
| 1264 |
-
const values = episodes.map((ep) => ep[columnName] || 0);
|
| 1265 |
-
|
| 1266 |
-
// Check if the column is an array type and needs special handling
|
| 1267 |
-
if (columnName.includes("stats/") || columnName === "tasks") {
|
| 1268 |
-
// Handle different types of array columns based on their naming pattern
|
| 1269 |
-
if (columnName.includes("/count")) {
|
| 1270 |
-
// Bigint arrays for count fields
|
| 1271 |
-
// @ts-ignore
|
| 1272 |
-
columns[columnName] = vectorFromArray(
|
| 1273 |
-
values.map((v) => Number(v)),
|
| 1274 |
-
new arrow.List(new arrow.Field("item", new arrow.Int64()))
|
| 1275 |
-
);
|
| 1276 |
-
} else if (
|
| 1277 |
-
columnName.includes("/min") ||
|
| 1278 |
-
columnName.includes("/max") ||
|
| 1279 |
-
columnName.includes("/mean") ||
|
| 1280 |
-
columnName.includes("/std")
|
| 1281 |
-
) {
|
| 1282 |
-
// Double arrays for min, max, mean, std fields
|
| 1283 |
-
if (
|
| 1284 |
-
columnName.includes("observation.images") &&
|
| 1285 |
-
(columnName.includes("/min") ||
|
| 1286 |
-
columnName.includes("/max") ||
|
| 1287 |
-
columnName.includes("/mean") ||
|
| 1288 |
-
columnName.includes("/std"))
|
| 1289 |
-
) {
|
| 1290 |
-
// These are 3D arrays [[[value]]]
|
| 1291 |
-
// For 3D arrays, we need nested Lists
|
| 1292 |
-
// @ts-ignore
|
| 1293 |
-
columns[columnName] = vectorFromArray(
|
| 1294 |
-
values,
|
| 1295 |
-
new arrow.List(
|
| 1296 |
-
new arrow.Field(
|
| 1297 |
-
"item",
|
| 1298 |
-
new arrow.List(
|
| 1299 |
-
new arrow.Field(
|
| 1300 |
-
"subitem",
|
| 1301 |
-
new arrow.List(
|
| 1302 |
-
new arrow.Field("value", new arrow.Float64())
|
| 1303 |
-
)
|
| 1304 |
-
)
|
| 1305 |
-
)
|
| 1306 |
-
)
|
| 1307 |
-
)
|
| 1308 |
-
);
|
| 1309 |
-
} else {
|
| 1310 |
-
// These are normal arrays [value]
|
| 1311 |
-
// @ts-ignore
|
| 1312 |
-
columns[columnName] = vectorFromArray(
|
| 1313 |
-
values,
|
| 1314 |
-
new arrow.List(new arrow.Field("item", new arrow.Float64()))
|
| 1315 |
-
);
|
| 1316 |
-
}
|
| 1317 |
-
} else {
|
| 1318 |
-
// Default to Float64 List for other array types
|
| 1319 |
-
// @ts-ignore
|
| 1320 |
-
columns[columnName] = vectorFromArray(
|
| 1321 |
-
values,
|
| 1322 |
-
new arrow.List(new arrow.Field("item", new arrow.Float64()))
|
| 1323 |
-
);
|
| 1324 |
-
}
|
| 1325 |
-
} else {
|
| 1326 |
-
// For non-array columns, use regular vectorFromArray
|
| 1327 |
-
// @ts-ignore
|
| 1328 |
-
columns[columnName] = vectorFromArray(values);
|
| 1329 |
-
}
|
| 1330 |
-
});
|
| 1331 |
-
|
| 1332 |
-
// Create the table with all columns
|
| 1333 |
-
const table = arrow.tableFromArrays(columns);
|
| 1334 |
-
|
| 1335 |
-
// Initialize the WASM module
|
| 1336 |
-
const wasmUrl =
|
| 1337 |
-
"https://cdn.jsdelivr.net/npm/parquet-wasm@0.6.1/esm/parquet_wasm_bg.wasm";
|
| 1338 |
-
const initWasm = parquet.default;
|
| 1339 |
-
await initWasm(wasmUrl);
|
| 1340 |
-
|
| 1341 |
-
// Convert Arrow table to Parquet WASM table
|
| 1342 |
-
const wasmTable = parquet.Table.fromIPCStream(
|
| 1343 |
-
arrow.tableToIPC(table, "stream")
|
| 1344 |
-
);
|
| 1345 |
-
|
| 1346 |
-
// Set compression properties
|
| 1347 |
-
const writerProperties = new parquet.WriterPropertiesBuilder()
|
| 1348 |
-
.setCompression(parquet.Compression.UNCOMPRESSED)
|
| 1349 |
-
.build();
|
| 1350 |
-
|
| 1351 |
-
// Write the Parquet file
|
| 1352 |
-
return parquet.writeParquet(wasmTable, writerProperties);
|
| 1353 |
}
|
| 1354 |
|
| 1355 |
generateREADME(metaInfo: string) {
|
|
@@ -1363,59 +1143,236 @@ export class LeRobotDatasetRecorder {
|
|
| 1363 |
* @private
|
| 1364 |
*/
|
| 1365 |
async _exportForLeRobotBlobs() {
|
| 1366 |
-
const
|
|
|
|
|
|
|
| 1367 |
const parquetEpisodeDataFiles = await this._exportEpisodesToBlob(
|
| 1368 |
-
|
| 1369 |
);
|
| 1370 |
-
|
| 1371 |
-
|
| 1372 |
-
const
|
| 1373 |
-
|
| 1374 |
-
|
| 1375 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1376 |
);
|
| 1377 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1378 |
|
| 1379 |
-
//
|
| 1380 |
-
const
|
| 1381 |
-
.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1382 |
{
|
| 1383 |
path: "meta/info.json",
|
| 1384 |
-
content: new Blob([JSON.stringify(
|
| 1385 |
type: "application/json",
|
| 1386 |
}),
|
| 1387 |
},
|
| 1388 |
{
|
| 1389 |
-
path: "meta/
|
| 1390 |
-
content: new Blob([
|
| 1391 |
-
type: "application/
|
| 1392 |
}),
|
| 1393 |
},
|
| 1394 |
{
|
| 1395 |
-
path: "meta/tasks.
|
| 1396 |
-
content: new Blob([
|
| 1397 |
-
|
| 1398 |
-
|
| 1399 |
},
|
| 1400 |
{
|
| 1401 |
-
path: "meta/
|
| 1402 |
-
content: new Blob([
|
| 1403 |
-
|
| 1404 |
-
|
| 1405 |
},
|
| 1406 |
{
|
| 1407 |
path: "README.md",
|
| 1408 |
content: new Blob([readme], { type: "text/markdown" }),
|
| 1409 |
-
}
|
| 1410 |
-
|
| 1411 |
|
| 1412 |
-
//
|
| 1413 |
-
|
| 1414 |
-
|
| 1415 |
-
|
| 1416 |
-
|
| 1417 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1418 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1419 |
|
| 1420 |
return blobArray;
|
| 1421 |
}
|
|
@@ -1452,130 +1409,15 @@ export class LeRobotDatasetRecorder {
|
|
| 1452 |
return await zip.generateAsync({ type: "blob" });
|
| 1453 |
}
|
| 1454 |
|
| 1455 |
-
/**
|
| 1456 |
-
* Uploads the LeRobot dataset to Hugging Face
|
| 1457 |
-
*
|
| 1458 |
-
* @param username Hugging Face username
|
| 1459 |
-
* @param repoName Repository name for the dataset
|
| 1460 |
-
* @param accessToken Hugging Face access token
|
| 1461 |
-
* @returns The LeRobotHFUploader instance used for upload
|
| 1462 |
-
*/
|
| 1463 |
-
async _exportForLeRobotHuggingface(
|
| 1464 |
-
username: string,
|
| 1465 |
-
repoName: string,
|
| 1466 |
-
accessToken: string
|
| 1467 |
-
) {
|
| 1468 |
-
// Create the blobs array for upload
|
| 1469 |
-
const blobArray = await this._exportForLeRobotBlobs();
|
| 1470 |
-
|
| 1471 |
-
// Create the uploader
|
| 1472 |
-
const uploader = new LeRobotHFUploader(username, repoName);
|
| 1473 |
-
|
| 1474 |
-
// Convert blobs to File objects for HF uploader
|
| 1475 |
-
const files = blobArray.map((item) => {
|
| 1476 |
-
return {
|
| 1477 |
-
path: item.path,
|
| 1478 |
-
content: item.content,
|
| 1479 |
-
};
|
| 1480 |
-
});
|
| 1481 |
-
|
| 1482 |
-
// Generate a unique reference ID for tracking the upload
|
| 1483 |
-
const referenceId = `lerobot-upload-${Date.now()}`;
|
| 1484 |
-
|
| 1485 |
-
try {
|
| 1486 |
-
// Start the upload process
|
| 1487 |
-
uploader.createRepoAndUploadFiles(files, accessToken, referenceId);
|
| 1488 |
-
console.log(`Successfully uploaded dataset to ${username}/${repoName}`);
|
| 1489 |
-
return uploader;
|
| 1490 |
-
} catch (error) {
|
| 1491 |
-
console.error("Error uploading to Hugging Face:", error);
|
| 1492 |
-
throw error;
|
| 1493 |
-
}
|
| 1494 |
-
}
|
| 1495 |
-
|
| 1496 |
-
/**
|
| 1497 |
-
* Uploads the LeRobot dataset to Amazon S3
|
| 1498 |
-
*
|
| 1499 |
-
* @param bucketName S3 bucket name
|
| 1500 |
-
* @param accessKeyId AWS access key ID
|
| 1501 |
-
* @param secretAccessKey AWS secret access key
|
| 1502 |
-
* @param region AWS region (default: us-east-1)
|
| 1503 |
-
* @param prefix Optional prefix (folder) to upload files to within the bucket
|
| 1504 |
-
* @returns The LeRobotS3Uploader instance used for upload
|
| 1505 |
-
*/
|
| 1506 |
-
async _exportForLeRobotS3(
|
| 1507 |
-
bucketName: string,
|
| 1508 |
-
accessKeyId: string,
|
| 1509 |
-
secretAccessKey: string,
|
| 1510 |
-
region: string = "us-east-1",
|
| 1511 |
-
prefix: string = ""
|
| 1512 |
-
) {
|
| 1513 |
-
// Create the blobs array for upload
|
| 1514 |
-
const blobArray = await this._exportForLeRobotBlobs();
|
| 1515 |
-
|
| 1516 |
-
// Create the uploader
|
| 1517 |
-
const uploader = new LeRobotS3Uploader(bucketName, region);
|
| 1518 |
-
|
| 1519 |
-
// Convert blobs to File objects for S3 uploader
|
| 1520 |
-
const files = blobArray.map((item) => {
|
| 1521 |
-
return {
|
| 1522 |
-
path: item.path,
|
| 1523 |
-
content: item.content,
|
| 1524 |
-
};
|
| 1525 |
-
});
|
| 1526 |
-
|
| 1527 |
-
// Generate a unique reference ID for tracking the upload
|
| 1528 |
-
const referenceId = `lerobot-s3-upload-${Date.now()}`;
|
| 1529 |
-
|
| 1530 |
-
try {
|
| 1531 |
-
// Start the upload process
|
| 1532 |
-
uploader.checkBucketAndUploadFiles(
|
| 1533 |
-
files,
|
| 1534 |
-
accessKeyId,
|
| 1535 |
-
secretAccessKey,
|
| 1536 |
-
prefix,
|
| 1537 |
-
referenceId
|
| 1538 |
-
);
|
| 1539 |
-
console.log(`Successfully uploaded dataset to S3 bucket: ${bucketName}`);
|
| 1540 |
-
return uploader;
|
| 1541 |
-
} catch (error) {
|
| 1542 |
-
console.error("Error uploading to S3:", error);
|
| 1543 |
-
throw error;
|
| 1544 |
-
}
|
| 1545 |
-
}
|
| 1546 |
-
|
| 1547 |
/**
|
| 1548 |
* Exports the LeRobot dataset in various formats
|
| 1549 |
*
|
| 1550 |
-
* @param format The export format - 'blobs', 'zip', 'zip-download'
|
| 1551 |
-
* @param options Additional options
|
| 1552 |
-
* @
|
| 1553 |
-
* @param options.repoName Hugging Face repository name (required for 'huggingface' format)
|
| 1554 |
-
* @param options.accessToken Hugging Face access token (required for 'huggingface' format)
|
| 1555 |
-
* @param options.bucketName S3 bucket name (required for 's3' format)
|
| 1556 |
-
* @param options.accessKeyId AWS access key ID (required for 's3' format)
|
| 1557 |
-
* @param options.secretAccessKey AWS secret access key (required for 's3' format)
|
| 1558 |
-
* @param options.region AWS region (optional for 's3' format, default: us-east-1)
|
| 1559 |
-
* @param options.prefix S3 prefix/folder (optional for 's3' format)
|
| 1560 |
-
* @returns The exported data in the requested format or the uploader instance for 'huggingface'/'s3' formats
|
| 1561 |
*/
|
| 1562 |
async exportForLeRobot(
|
| 1563 |
-
format:
|
| 1564 |
-
| "blobs"
|
| 1565 |
-
| "zip"
|
| 1566 |
-
| "zip-download"
|
| 1567 |
-
| "huggingface"
|
| 1568 |
-
| "s3" = "zip-download",
|
| 1569 |
-
options?: {
|
| 1570 |
-
username?: string;
|
| 1571 |
-
repoName?: string;
|
| 1572 |
-
accessToken?: string;
|
| 1573 |
-
bucketName?: string;
|
| 1574 |
-
accessKeyId?: string;
|
| 1575 |
-
secretAccessKey?: string;
|
| 1576 |
-
region?: string;
|
| 1577 |
-
prefix?: string;
|
| 1578 |
-
}
|
| 1579 |
) {
|
| 1580 |
switch (format) {
|
| 1581 |
case "blobs":
|
|
@@ -1584,49 +1426,6 @@ export class LeRobotDatasetRecorder {
|
|
| 1584 |
case "zip":
|
| 1585 |
return this._exportForLeRobotZip();
|
| 1586 |
|
| 1587 |
-
case "huggingface":
|
| 1588 |
-
// Validate required options for Hugging Face upload
|
| 1589 |
-
if (!options || !options.repoName || !options.accessToken) {
|
| 1590 |
-
throw new Error(
|
| 1591 |
-
"Hugging Face upload requires repoName, and accessToken options"
|
| 1592 |
-
);
|
| 1593 |
-
}
|
| 1594 |
-
|
| 1595 |
-
if (!options.username) {
|
| 1596 |
-
const hub = await import("@huggingface/hub");
|
| 1597 |
-
const { name: username } = await hub.whoAmI({
|
| 1598 |
-
accessToken: options.accessToken,
|
| 1599 |
-
});
|
| 1600 |
-
options.username = username;
|
| 1601 |
-
}
|
| 1602 |
-
|
| 1603 |
-
return this._exportForLeRobotHuggingface(
|
| 1604 |
-
options.username,
|
| 1605 |
-
options.repoName,
|
| 1606 |
-
options.accessToken
|
| 1607 |
-
);
|
| 1608 |
-
|
| 1609 |
-
case "s3":
|
| 1610 |
-
// Validate required options for S3 upload
|
| 1611 |
-
if (
|
| 1612 |
-
!options ||
|
| 1613 |
-
!options.bucketName ||
|
| 1614 |
-
!options.accessKeyId ||
|
| 1615 |
-
!options.secretAccessKey
|
| 1616 |
-
) {
|
| 1617 |
-
throw new Error(
|
| 1618 |
-
"S3 upload requires bucketName, accessKeyId, and secretAccessKey options"
|
| 1619 |
-
);
|
| 1620 |
-
}
|
| 1621 |
-
|
| 1622 |
-
return this._exportForLeRobotS3(
|
| 1623 |
-
options.bucketName,
|
| 1624 |
-
options.accessKeyId,
|
| 1625 |
-
options.secretAccessKey,
|
| 1626 |
-
options.region,
|
| 1627 |
-
options.prefix
|
| 1628 |
-
);
|
| 1629 |
-
|
| 1630 |
case "zip-download":
|
| 1631 |
default:
|
| 1632 |
// Get the zip blob
|
|
@@ -1654,3 +1453,185 @@ export class LeRobotDatasetRecorder {
|
|
| 1654 |
}
|
| 1655 |
}
|
| 1656 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
import * as parquet from "parquet-wasm";
|
| 4 |
import * as arrow from "apache-arrow";
|
| 5 |
import JSZip from "jszip";
|
|
|
|
|
|
|
|
|
|
| 6 |
import generateREADME from "./utils/record/generateREADME";
|
|
|
|
|
|
|
| 7 |
|
| 8 |
// declare a type leRobot action that's basically an array of numbers
|
| 9 |
interface LeRobotAction {
|
|
|
|
| 333 |
mediaRecorders: { [key: string]: MediaRecorder };
|
| 334 |
videoChunks: { [key: string]: Blob[] };
|
| 335 |
videoBlobs: { [key: string]: Blob };
|
| 336 |
+
private videoBlobsByEpisode: {
|
| 337 |
+
[episodeIndex: number]: { [key: string]: Blob };
|
| 338 |
+
};
|
| 339 |
+
private videoMimeByKey: { [key: string]: { mime: string; ext: string } };
|
| 340 |
teleoperatorData: LeRobotEpisode[];
|
| 341 |
private _isRecording: boolean;
|
| 342 |
private episodeIndex: number = 0;
|
| 343 |
private taskIndex: number = 0;
|
| 344 |
+
private currentVideoSegmentEpisodeIndex: number | null = null;
|
| 345 |
fps: number;
|
| 346 |
taskDescription: string;
|
| 347 |
+
private robotLabel?: string;
|
| 348 |
|
| 349 |
/**
|
| 350 |
* Ensures BlobPart compatibility across environments by converting Uint8Array
|
|
|
|
| 383 |
this.mediaRecorders = {};
|
| 384 |
this.videoChunks = {};
|
| 385 |
this.videoBlobs = {};
|
| 386 |
+
this.videoBlobsByEpisode = {};
|
| 387 |
this.videoStreams = {};
|
| 388 |
+
this.videoMimeByKey = {};
|
| 389 |
this.teleoperatorData = [];
|
| 390 |
this._isRecording = false;
|
| 391 |
this.fps = fps;
|
| 392 |
this.taskDescription = taskDescription;
|
| 393 |
+
this.robotLabel = undefined;
|
| 394 |
|
| 395 |
for (const [key, stream] of Object.entries(videoStreams)) {
|
| 396 |
this.addVideoStream(key, stream);
|
| 397 |
}
|
| 398 |
}
|
| 399 |
|
| 400 |
+
setRobotLabel(label: string) {
|
| 401 |
+
this.robotLabel = label;
|
| 402 |
+
}
|
| 403 |
+
|
| 404 |
+
private static getSupportedRecorderType(): { mime: string; ext: string } {
|
| 405 |
+
// Prefer H.264 MP4 for viewer compatibility; fall back to WebM
|
| 406 |
+
const candidates: { mime: string; ext: string }[] = [
|
| 407 |
+
{ mime: "video/mp4;codecs=h264", ext: "mp4" },
|
| 408 |
+
{ mime: "video/mp4", ext: "mp4" },
|
| 409 |
+
{ mime: "video/webm;codecs=vp9", ext: "webm" },
|
| 410 |
+
{ mime: "video/webm;codecs=vp8", ext: "webm" },
|
| 411 |
+
{ mime: "video/webm", ext: "webm" },
|
| 412 |
+
];
|
| 413 |
+
for (const c of candidates) {
|
| 414 |
+
if (
|
| 415 |
+
(window as any).MediaRecorder &&
|
| 416 |
+
MediaRecorder.isTypeSupported &&
|
| 417 |
+
MediaRecorder.isTypeSupported(c.mime)
|
| 418 |
+
) {
|
| 419 |
+
return c;
|
| 420 |
+
}
|
| 421 |
+
}
|
| 422 |
+
return { mime: "video/webm", ext: "webm" };
|
| 423 |
+
}
|
| 424 |
+
|
| 425 |
get isRecording(): boolean {
|
| 426 |
return this._isRecording;
|
| 427 |
}
|
|
|
|
| 436 |
* @param stream The media stream to record from
|
| 437 |
*/
|
| 438 |
addVideoStream(key: string, stream: MediaStream) {
|
|
|
|
| 439 |
if (this._isRecording) {
|
| 440 |
throw new Error("Cannot add video streams while recording");
|
| 441 |
}
|
| 442 |
|
| 443 |
// Add to video streams dictionary
|
| 444 |
this.videoStreams[key] = stream;
|
| 445 |
+
// Initialize chunks storage
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 446 |
this.videoChunks[key] = [];
|
| 447 |
+
|
| 448 |
+
// Pre-warm container selection for consistent extension even if added before start
|
| 449 |
+
const { mime, ext } = LeRobotDatasetRecorder.getSupportedRecorderType();
|
| 450 |
+
this.videoMimeByKey[key] = { mime, ext };
|
| 451 |
}
|
| 452 |
|
| 453 |
/**
|
|
|
|
| 492 |
* Starts recording for all teleoperators and video streams
|
| 493 |
*/
|
| 494 |
startRecording() {
|
|
|
|
| 495 |
if (this._isRecording) {
|
| 496 |
console.warn("Recording already in progress");
|
| 497 |
return;
|
|
|
|
| 499 |
|
| 500 |
this._isRecording = true;
|
| 501 |
|
| 502 |
+
// Always start a brand new episode using the next available index
|
| 503 |
+
this.episodeIndex = this.teleoperatorData.length;
|
| 504 |
this.teleoperatorData.push(new LeRobotEpisode());
|
| 505 |
+
this.currentVideoSegmentEpisodeIndex = this.episodeIndex;
|
| 506 |
|
| 507 |
// Start recording video streams
|
| 508 |
Object.entries(this.videoStreams).forEach(([key, stream]) => {
|
| 509 |
+
// Pick a supported mime/container pair per browser
|
| 510 |
+
const supported =
|
| 511 |
+
this.videoMimeByKey[key] ||
|
| 512 |
+
LeRobotDatasetRecorder.getSupportedRecorderType();
|
| 513 |
+
const { mime, ext } = supported;
|
| 514 |
+
this.videoMimeByKey[key] = { mime, ext };
|
| 515 |
+
|
| 516 |
+
// Reset chunks for a clean segment start
|
| 517 |
+
this.videoChunks[key] = [];
|
| 518 |
+
const mediaRecorder = new MediaRecorder(stream, { mimeType: mime });
|
| 519 |
|
| 520 |
// Handle data available events
|
| 521 |
mediaRecorder.ondataavailable = (event) => {
|
|
|
|
| 522 |
if (event.data && event.data.size > 0) {
|
| 523 |
this.videoChunks[key].push(event.data);
|
| 524 |
}
|
|
|
|
| 527 |
// Save the recorder and start recording
|
| 528 |
this.mediaRecorders[key] = mediaRecorder;
|
| 529 |
mediaRecorder.start(1000); // Capture in 1-second chunks
|
|
|
|
|
|
|
| 530 |
});
|
| 531 |
}
|
| 532 |
|
|
|
|
| 590 |
recorder.onstop = () => {
|
| 591 |
// Combine all chunks into a single blob
|
| 592 |
const chunks = this.videoChunks[key] || [];
|
| 593 |
+
const mime = this.videoMimeByKey[key]?.mime || "video/webm";
|
| 594 |
+
const blob = new Blob(chunks, { type: mime });
|
| 595 |
this.videoBlobs[key] = blob;
|
| 596 |
+
const segmentEpisodeIndex =
|
| 597 |
+
this.currentVideoSegmentEpisodeIndex ?? this.episodeIndex;
|
| 598 |
+
if (!this.videoBlobsByEpisode[segmentEpisodeIndex]) {
|
| 599 |
+
this.videoBlobsByEpisode[segmentEpisodeIndex] = {} as any;
|
| 600 |
+
}
|
| 601 |
+
this.videoBlobsByEpisode[segmentEpisodeIndex][key] = blob;
|
| 602 |
+
// Prepare for any subsequent recording
|
| 603 |
+
this.videoChunks[key] = [];
|
| 604 |
resolve();
|
| 605 |
};
|
| 606 |
|
|
|
|
| 618 |
};
|
| 619 |
}
|
| 620 |
|
| 621 |
+
/**
|
| 622 |
+
* Finalizes the current video segment and immediately starts a new one
|
| 623 |
+
* while continuing the recording session. Also advances to the next
|
| 624 |
+
* episode and begins collecting frames under the new episode index.
|
| 625 |
+
*
|
| 626 |
+
* @returns The new episode index
|
| 627 |
+
*/
|
| 628 |
+
async nextEpisodeSegment(): Promise<number> {
|
| 629 |
+
if (!this._isRecording) {
|
| 630 |
+
console.warn("nextEpisodeSegment() called while not recording");
|
| 631 |
+
// Ensure episode index points to last episode if any
|
| 632 |
+
this.episodeIndex = Math.max(0, this.teleoperatorData.length - 1);
|
| 633 |
+
return this.episodeIndex;
|
| 634 |
+
}
|
| 635 |
+
|
| 636 |
+
const oldSegmentEpisodeIndex =
|
| 637 |
+
this.currentVideoSegmentEpisodeIndex ?? this.episodeIndex;
|
| 638 |
+
|
| 639 |
+
// Stop current media recorders and persist the segment blobs under the old episode index
|
| 640 |
+
const stopPromises = Object.entries(this.mediaRecorders).map(
|
| 641 |
+
([key, recorder]) => {
|
| 642 |
+
return new Promise<void>((resolve) => {
|
| 643 |
+
if (recorder.state === "inactive") {
|
| 644 |
+
resolve();
|
| 645 |
+
return;
|
| 646 |
+
}
|
| 647 |
+
recorder.onstop = () => {
|
| 648 |
+
const chunks = this.videoChunks[key] || [];
|
| 649 |
+
const mime = this.videoMimeByKey[key]?.mime || "video/webm";
|
| 650 |
+
const blob = new Blob(chunks, { type: mime });
|
| 651 |
+
if (!this.videoBlobsByEpisode[oldSegmentEpisodeIndex]) {
|
| 652 |
+
this.videoBlobsByEpisode[oldSegmentEpisodeIndex] = {} as any;
|
| 653 |
+
}
|
| 654 |
+
this.videoBlobsByEpisode[oldSegmentEpisodeIndex][key] = blob;
|
| 655 |
+
// Reset chunks for the next segment
|
| 656 |
+
this.videoChunks[key] = [];
|
| 657 |
+
resolve();
|
| 658 |
+
};
|
| 659 |
+
recorder.stop();
|
| 660 |
+
});
|
| 661 |
+
}
|
| 662 |
+
);
|
| 663 |
+
|
| 664 |
+
await Promise.all(stopPromises);
|
| 665 |
+
|
| 666 |
+
// Advance to the next episode
|
| 667 |
+
const newEpisodeIndex = this.teleoperatorData.length;
|
| 668 |
+
this.episodeIndex = newEpisodeIndex;
|
| 669 |
+
this.teleoperatorData.push(new LeRobotEpisode());
|
| 670 |
+
|
| 671 |
+
// Start new media recorders for the next segment
|
| 672 |
+
Object.entries(this.videoStreams).forEach(([key, stream]) => {
|
| 673 |
+
const supported =
|
| 674 |
+
this.videoMimeByKey[key] ||
|
| 675 |
+
LeRobotDatasetRecorder.getSupportedRecorderType();
|
| 676 |
+
const { mime, ext } = supported;
|
| 677 |
+
this.videoMimeByKey[key] = { mime, ext };
|
| 678 |
+
|
| 679 |
+
// Ensure a fresh chunk buffer
|
| 680 |
+
this.videoChunks[key] = [];
|
| 681 |
+
const mediaRecorder = new MediaRecorder(stream, { mimeType: mime });
|
| 682 |
+
mediaRecorder.ondataavailable = (event) => {
|
| 683 |
+
if (event.data && event.data.size > 0) {
|
| 684 |
+
this.videoChunks[key].push(event.data);
|
| 685 |
+
}
|
| 686 |
+
};
|
| 687 |
+
this.mediaRecorders[key] = mediaRecorder;
|
| 688 |
+
mediaRecorder.start(1000);
|
| 689 |
+
});
|
| 690 |
+
|
| 691 |
+
// Point subsequent stop() to the new episode index
|
| 692 |
+
this.currentVideoSegmentEpisodeIndex = newEpisodeIndex;
|
| 693 |
+
|
| 694 |
+
return newEpisodeIndex;
|
| 695 |
+
}
|
| 696 |
+
|
| 697 |
/**
|
| 698 |
* Clears the teleoperator data and video blobs
|
| 699 |
*/
|
| 700 |
clearRecording() {
|
| 701 |
this.teleoperatorData = [];
|
| 702 |
this.videoBlobs = {};
|
| 703 |
+
this.videoBlobsByEpisode = {} as any;
|
| 704 |
+
// Reset video chunk buffers so future segments don't include cleared data
|
| 705 |
+
for (const key of Object.keys(this.videoChunks)) {
|
| 706 |
+
this.videoChunks[key] = [];
|
| 707 |
+
}
|
| 708 |
+
this.episodeIndex = 0;
|
| 709 |
+
this.currentVideoSegmentEpisodeIndex = null;
|
| 710 |
}
|
| 711 |
|
| 712 |
/**
|
|
|
|
| 1061 |
* Generates metadata for the dataset
|
| 1062 |
* @returns Metadata object for the LeRobot dataset
|
| 1063 |
*/
|
| 1064 |
+
// Deprecated for v2.1 exporter; left for backwards-compat APIs
|
| 1065 |
+
async generateMetadata(_data: any[]): Promise<any> {
|
| 1066 |
+
return {};
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1067 |
}
|
| 1068 |
|
| 1069 |
/**
|
| 1070 |
* Generates statistics for the dataset
|
| 1071 |
* @returns Statistics object for the LeRobot dataset
|
| 1072 |
*/
|
| 1073 |
+
async getStatistics(_data: any[]): Promise<any> {
|
| 1074 |
+
return {};
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1075 |
}
|
| 1076 |
|
| 1077 |
/**
|
|
|
|
| 1128 |
* Creates the episodes statistics parquet file
|
| 1129 |
* @returns A Uint8Array blob containing the parquet data
|
| 1130 |
*/
|
| 1131 |
+
async getEpisodeStatistics(_data: any[]): Promise<Uint8Array> {
|
| 1132 |
+
return new Uint8Array();
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1133 |
}
|
| 1134 |
|
| 1135 |
generateREADME(metaInfo: string) {
|
|
|
|
| 1143 |
* @private
|
| 1144 |
*/
|
| 1145 |
async _exportForLeRobotBlobs() {
|
| 1146 |
+
const regularizedEpisodes = (await this.exportEpisodes("json")) as any[];
|
| 1147 |
+
|
| 1148 |
+
// Build episodes parquet files under data/chunk-000/episode_<id>.parquet
|
| 1149 |
const parquetEpisodeDataFiles = await this._exportEpisodesToBlob(
|
| 1150 |
+
regularizedEpisodes
|
| 1151 |
);
|
| 1152 |
+
|
| 1153 |
+
// Rewrite parquet file paths to v2.1 layout with chunk folder
|
| 1154 |
+
const rewrittenParquet = parquetEpisodeDataFiles.map((file, idx) => {
|
| 1155 |
+
return {
|
| 1156 |
+
path: `data/chunk-000/episode_${idx
|
| 1157 |
+
.toString()
|
| 1158 |
+
.padStart(6, "0")}.parquet`,
|
| 1159 |
+
content: file.content,
|
| 1160 |
+
};
|
| 1161 |
+
});
|
| 1162 |
+
|
| 1163 |
+
// Videos: videos/chunk-000/observation.images.<camera>/episode_<id>.<ext>
|
| 1164 |
+
const blobArray: { path: string; content: Blob }[] = [...rewrittenParquet];
|
| 1165 |
+
|
| 1166 |
+
const allEpisodeIndices = regularizedEpisodes.map((_: any, i: number) => i);
|
| 1167 |
+
const cameraKeySet = new Set<string>();
|
| 1168 |
+
const episodesVideoMap: { [ep: number]: { [cam: string]: string } } = {};
|
| 1169 |
+
|
| 1170 |
+
for (const epIdx of allEpisodeIndices) {
|
| 1171 |
+
const byCam = this.videoBlobsByEpisode[epIdx] || {};
|
| 1172 |
+
episodesVideoMap[epIdx] = {};
|
| 1173 |
+
for (const [key, blob] of Object.entries(byCam)) {
|
| 1174 |
+
const ext = this.videoMimeByKey[key]?.ext || "mp4";
|
| 1175 |
+
const episodeId = epIdx.toString().padStart(6, "0");
|
| 1176 |
+
const path = `videos/chunk-000/observation.images.${key}/episode_${episodeId}.${ext}`;
|
| 1177 |
+
cameraKeySet.add(key);
|
| 1178 |
+
episodesVideoMap[epIdx][key] = path;
|
| 1179 |
+
blobArray.push({ path, content: blob });
|
| 1180 |
+
}
|
| 1181 |
+
}
|
| 1182 |
+
|
| 1183 |
+
// info.json (v2.1)
|
| 1184 |
+
const cameras = Array.from(cameraKeySet);
|
| 1185 |
+
const numEpisodes = regularizedEpisodes.length;
|
| 1186 |
+
const totalFrames = regularizedEpisodes.reduce(
|
| 1187 |
+
(sum: number, ep: any) => sum + ep.frames.length,
|
| 1188 |
+
0
|
| 1189 |
);
|
| 1190 |
+
// Determine a default video extension for video_path pattern
|
| 1191 |
+
let defaultVideoExt = "mp4";
|
| 1192 |
+
if (cameras.length > 0) {
|
| 1193 |
+
const firstKey = cameras[0];
|
| 1194 |
+
const ext = this.videoMimeByKey[firstKey]?.ext;
|
| 1195 |
+
if (ext) defaultVideoExt = ext;
|
| 1196 |
+
}
|
| 1197 |
+
|
| 1198 |
+
// Build features object
|
| 1199 |
+
const features: Record<string, any> = {
|
| 1200 |
+
action: {
|
| 1201 |
+
dtype: "float32",
|
| 1202 |
+
shape: [6],
|
| 1203 |
+
names: [
|
| 1204 |
+
"main_shoulder_pan",
|
| 1205 |
+
"main_shoulder_lift",
|
| 1206 |
+
"main_elbow_flex",
|
| 1207 |
+
"main_wrist_flex",
|
| 1208 |
+
"main_wrist_roll",
|
| 1209 |
+
"main_gripper",
|
| 1210 |
+
],
|
| 1211 |
+
},
|
| 1212 |
+
"observation.state": {
|
| 1213 |
+
dtype: "float32",
|
| 1214 |
+
shape: [6],
|
| 1215 |
+
names: [
|
| 1216 |
+
"main_shoulder_pan",
|
| 1217 |
+
"main_shoulder_lift",
|
| 1218 |
+
"main_elbow_flex",
|
| 1219 |
+
"main_wrist_flex",
|
| 1220 |
+
"main_wrist_roll",
|
| 1221 |
+
"main_gripper",
|
| 1222 |
+
],
|
| 1223 |
+
},
|
| 1224 |
+
timestamp: { dtype: "float32", shape: [1], names: null },
|
| 1225 |
+
frame_index: { dtype: "int64", shape: [1], names: null },
|
| 1226 |
+
episode_index: { dtype: "int64", shape: [1], names: null },
|
| 1227 |
+
index: { dtype: "int64", shape: [1], names: null },
|
| 1228 |
+
task_index: { dtype: "int64", shape: [1], names: null },
|
| 1229 |
+
};
|
| 1230 |
+
for (const cam of cameras) {
|
| 1231 |
+
// Map mime to codec
|
| 1232 |
+
const mime = this.videoMimeByKey[cam]?.mime || "video/mp4";
|
| 1233 |
+
let codec = "avc1";
|
| 1234 |
+
if (mime.includes("vp9")) codec = "vp9";
|
| 1235 |
+
else if (mime.includes("vp8")) codec = "vp8";
|
| 1236 |
+
features[`observation.images.${cam}`] = {
|
| 1237 |
+
dtype: "video",
|
| 1238 |
+
shape: [480, 640, 3],
|
| 1239 |
+
names: ["height", "width", "channels"],
|
| 1240 |
+
info: {
|
| 1241 |
+
"video.fps": this.fps,
|
| 1242 |
+
"video.height": 480,
|
| 1243 |
+
"video.width": 640,
|
| 1244 |
+
"video.channels": 3,
|
| 1245 |
+
"video.codec": codec,
|
| 1246 |
+
"video.pix_fmt": "yuv420p",
|
| 1247 |
+
"video.is_depth_map": false,
|
| 1248 |
+
has_audio: false,
|
| 1249 |
+
},
|
| 1250 |
+
};
|
| 1251 |
+
}
|
| 1252 |
+
|
| 1253 |
+
const infoJson = {
|
| 1254 |
+
version: "2.1",
|
| 1255 |
+
// Compatibility fields consumed by visualizers
|
| 1256 |
+
total_episodes: numEpisodes,
|
| 1257 |
+
total_frames: totalFrames,
|
| 1258 |
+
fps: this.fps,
|
| 1259 |
+
data_path:
|
| 1260 |
+
"data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
|
| 1261 |
+
video_path: `videos/chunk-{episode_chunk:03d}/{video_key}/episode_{episode_index:06d}.${defaultVideoExt}`,
|
| 1262 |
+
features,
|
| 1263 |
+
// Extra descriptive fields
|
| 1264 |
+
name: `lerobot_dataset_${new Date().toISOString().slice(0, 10)}`,
|
| 1265 |
+
robot: this.robotLabel || "unknown",
|
| 1266 |
+
cameras,
|
| 1267 |
+
action_space: "joint_position",
|
| 1268 |
+
frame_rate: this.fps,
|
| 1269 |
+
num_episodes: numEpisodes,
|
| 1270 |
+
created_by: "lerobot.js",
|
| 1271 |
+
created_at: new Date().toISOString(),
|
| 1272 |
+
} as any;
|
| 1273 |
+
|
| 1274 |
+
// episodes.jsonl
|
| 1275 |
+
const episodesJsonlLines: string[] = [];
|
| 1276 |
+
for (let epIdx = 0; epIdx < numEpisodes; epIdx++) {
|
| 1277 |
+
const episodeId = epIdx.toString().padStart(6, "0");
|
| 1278 |
+
const length = regularizedEpisodes[epIdx]?.frames.length || 0;
|
| 1279 |
+
const videos: any = {};
|
| 1280 |
+
Object.entries(episodesVideoMap[epIdx] || {}).forEach(([cam, path]) => {
|
| 1281 |
+
videos[cam] = path;
|
| 1282 |
+
});
|
| 1283 |
+
const row = {
|
| 1284 |
+
episode_id: episodeId,
|
| 1285 |
+
task: this.taskDescription || "default",
|
| 1286 |
+
length,
|
| 1287 |
+
videos,
|
| 1288 |
+
};
|
| 1289 |
+
episodesJsonlLines.push(JSON.stringify(row));
|
| 1290 |
+
}
|
| 1291 |
|
| 1292 |
+
// tasks.jsonl (single task)
|
| 1293 |
+
const tasksJsonl = JSON.stringify({
|
| 1294 |
+
task: this.taskDescription || "default",
|
| 1295 |
+
description: this.taskDescription || "default",
|
| 1296 |
+
});
|
| 1297 |
+
|
| 1298 |
+
// stats.json (minimal)
|
| 1299 |
+
const lengths = regularizedEpisodes.map((ep: any) => ep.frames.length);
|
| 1300 |
+
const epMin = lengths.length ? Math.min(...lengths) : 0;
|
| 1301 |
+
const epMax = lengths.length ? Math.max(...lengths) : 0;
|
| 1302 |
+
const epMean = lengths.length
|
| 1303 |
+
? lengths.reduce((a, b) => a + b, 0) / lengths.length
|
| 1304 |
+
: 0;
|
| 1305 |
+
const statsJson = {
|
| 1306 |
+
total_frames: totalFrames,
|
| 1307 |
+
episode_lengths: { min: epMin, max: epMax, mean: Math.round(epMean) },
|
| 1308 |
+
} as any;
|
| 1309 |
+
|
| 1310 |
+
const readme = this.generateREADME(JSON.stringify(infoJson));
|
| 1311 |
+
|
| 1312 |
+
blobArray.push(
|
| 1313 |
{
|
| 1314 |
path: "meta/info.json",
|
| 1315 |
+
content: new Blob([JSON.stringify(infoJson, null, 2)], {
|
| 1316 |
type: "application/json",
|
| 1317 |
}),
|
| 1318 |
},
|
| 1319 |
{
|
| 1320 |
+
path: "meta/episodes.jsonl",
|
| 1321 |
+
content: new Blob([episodesJsonlLines.join("\n") + "\n"], {
|
| 1322 |
+
type: "application/jsonlines",
|
| 1323 |
}),
|
| 1324 |
},
|
| 1325 |
{
|
| 1326 |
+
path: "meta/tasks.jsonl",
|
| 1327 |
+
content: new Blob([tasksJsonl + "\n"], {
|
| 1328 |
+
type: "application/jsonlines",
|
| 1329 |
+
}),
|
| 1330 |
},
|
| 1331 |
{
|
| 1332 |
+
path: "meta/stats.json",
|
| 1333 |
+
content: new Blob([JSON.stringify(statsJson, null, 2)], {
|
| 1334 |
+
type: "application/json",
|
| 1335 |
+
}),
|
| 1336 |
},
|
| 1337 |
{
|
| 1338 |
path: "README.md",
|
| 1339 |
content: new Blob([readme], { type: "text/markdown" }),
|
| 1340 |
+
}
|
| 1341 |
+
);
|
| 1342 |
|
| 1343 |
+
// episodes_stats.jsonl (v2.1)
|
| 1344 |
+
const episodesStatsLines: string[] = [];
|
| 1345 |
+
for (let epIdx = 0; epIdx < numEpisodes; epIdx++) {
|
| 1346 |
+
const episodeId = epIdx.toString().padStart(6, "0");
|
| 1347 |
+
const length = regularizedEpisodes[epIdx]?.frames.length || 0;
|
| 1348 |
+
const timestamps = (regularizedEpisodes[epIdx]?.frames || []).map(
|
| 1349 |
+
(f: any) => f.timestamp
|
| 1350 |
+
);
|
| 1351 |
+
const fromTs = timestamps.length ? Math.min(...timestamps) : 0;
|
| 1352 |
+
const toTs = timestamps.length ? Math.max(...timestamps) : 0;
|
| 1353 |
+
|
| 1354 |
+
const row: any = {
|
| 1355 |
+
episode_id: episodeId,
|
| 1356 |
+
"data/chunk_index": 0,
|
| 1357 |
+
"data/file": `data/chunk-000/episode_${episodeId}.parquet`,
|
| 1358 |
+
length,
|
| 1359 |
+
};
|
| 1360 |
+
// add per-camera video fields
|
| 1361 |
+
const map = episodesVideoMap[epIdx] || {};
|
| 1362 |
+
for (const [cam, path] of Object.entries(map)) {
|
| 1363 |
+
row[`videos/observation.images.${cam}/chunk_index`] = 0;
|
| 1364 |
+
row[`videos/observation.images.${cam}/file`] = path;
|
| 1365 |
+
row[`videos/observation.images.${cam}/from_timestamp`] = fromTs;
|
| 1366 |
+
row[`videos/observation.images.${cam}/to_timestamp`] = toTs;
|
| 1367 |
+
}
|
| 1368 |
+
episodesStatsLines.push(JSON.stringify(row));
|
| 1369 |
}
|
| 1370 |
+
blobArray.push({
|
| 1371 |
+
path: "meta/episodes_stats.jsonl",
|
| 1372 |
+
content: new Blob([episodesStatsLines.join("\n") + "\n"], {
|
| 1373 |
+
type: "application/jsonlines",
|
| 1374 |
+
}),
|
| 1375 |
+
});
|
| 1376 |
|
| 1377 |
return blobArray;
|
| 1378 |
}
|
|
|
|
| 1409 |
return await zip.generateAsync({ type: "blob" });
|
| 1410 |
}
|
| 1411 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1412 |
/**
|
| 1413 |
* Exports the LeRobot dataset in various formats
|
| 1414 |
*
|
| 1415 |
+
* @param format The export format - 'blobs', 'zip', or 'zip-download'
|
| 1416 |
+
* @param options Additional options (currently unused)
|
| 1417 |
+
* @returns The exported data in the requested format
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1418 |
*/
|
| 1419 |
async exportForLeRobot(
|
| 1420 |
+
format: "blobs" | "zip" | "zip-download" = "zip-download"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1421 |
) {
|
| 1422 |
switch (format) {
|
| 1423 |
case "blobs":
|
|
|
|
| 1426 |
case "zip":
|
| 1427 |
return this._exportForLeRobotZip();
|
| 1428 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1429 |
case "zip-download":
|
| 1430 |
default:
|
| 1431 |
// Get the zip blob
|
|
|
|
| 1453 |
}
|
| 1454 |
}
|
| 1455 |
}
|
| 1456 |
+
|
| 1457 |
+
// Simple record() function API - wraps LeRobotDatasetRecorder
|
| 1458 |
+
import type {
|
| 1459 |
+
RecordConfig,
|
| 1460 |
+
RecordProcess,
|
| 1461 |
+
RecordingState,
|
| 1462 |
+
RecordingData,
|
| 1463 |
+
RobotRecordingData,
|
| 1464 |
+
} from "./types/recording.js";
|
| 1465 |
+
|
| 1466 |
+
/**
|
| 1467 |
+
* Simple recording function that follows LeRobot.js conventions
|
| 1468 |
+
*
|
| 1469 |
+
* Records robot motor positions and teleoperation data using a clean function API
|
| 1470 |
+
* that matches the patterns established by calibrate() and teleoperate().
|
| 1471 |
+
*
|
| 1472 |
+
* @param config Recording configuration with explicit teleoperator dependency
|
| 1473 |
+
* @returns RecordProcess with start(), stop(), getState(), and result
|
| 1474 |
+
*
|
| 1475 |
+
* @example
|
| 1476 |
+
* ```typescript
|
| 1477 |
+
* // 1. Create teleoperation
|
| 1478 |
+
* const teleoperationProcess = await teleoperate({
|
| 1479 |
+
* robot: connectedRobot,
|
| 1480 |
+
* teleop: { type: "keyboard" },
|
| 1481 |
+
* calibrationData: calibrationData,
|
| 1482 |
+
* });
|
| 1483 |
+
*
|
| 1484 |
+
* // 2. Create recording with explicit teleoperator dependency
|
| 1485 |
+
* const recordProcess = await record({
|
| 1486 |
+
* teleoperator: teleoperationProcess.teleoperator,
|
| 1487 |
+
* options: {
|
| 1488 |
+
* fps: 30,
|
| 1489 |
+
* taskDescription: "Pick and place task",
|
| 1490 |
+
* onDataUpdate: (data) => console.log(`Recorded ${data.frameCount} frames`),
|
| 1491 |
+
* }
|
| 1492 |
+
* });
|
| 1493 |
+
*
|
| 1494 |
+
* // 3. Start both processes
|
| 1495 |
+
* teleoperationProcess.start();
|
| 1496 |
+
* recordProcess.start();
|
| 1497 |
+
*
|
| 1498 |
+
* // 4. Stop recording
|
| 1499 |
+
* const robotData = await recordProcess.stop();
|
| 1500 |
+
* ```
|
| 1501 |
+
*/
|
| 1502 |
+
export async function record(config: RecordConfig): Promise<RecordProcess> {
|
| 1503 |
+
// Use the provided teleoperator (explicit dependency - good architecture!)
|
| 1504 |
+
const recorder = new LeRobotDatasetRecorder(
|
| 1505 |
+
[config.teleoperator],
|
| 1506 |
+
config.videoStreams || {},
|
| 1507 |
+
config.options?.fps || 30,
|
| 1508 |
+
config.options?.taskDescription || "Robot recording"
|
| 1509 |
+
);
|
| 1510 |
+
|
| 1511 |
+
// Set robot metadata if provided
|
| 1512 |
+
if (config.robotType) {
|
| 1513 |
+
(recorder as any).setRobotLabel?.(config.robotType);
|
| 1514 |
+
}
|
| 1515 |
+
|
| 1516 |
+
let startTime = 0;
|
| 1517 |
+
let resultPromise: Promise<RobotRecordingData> | null = null;
|
| 1518 |
+
let stateUpdateInterval: NodeJS.Timeout | null = null;
|
| 1519 |
+
|
| 1520 |
+
const recordProcess: RecordProcess = {
|
| 1521 |
+
start(): void {
|
| 1522 |
+
startTime = Date.now();
|
| 1523 |
+
recorder.startRecording();
|
| 1524 |
+
|
| 1525 |
+
// Set up state update polling for callbacks
|
| 1526 |
+
if (config.options?.onStateUpdate || config.options?.onDataUpdate) {
|
| 1527 |
+
stateUpdateInterval = setInterval(() => {
|
| 1528 |
+
if (recorder.isRecording) {
|
| 1529 |
+
const state = recordProcess.getState();
|
| 1530 |
+
|
| 1531 |
+
if (config.options?.onStateUpdate) {
|
| 1532 |
+
config.options.onStateUpdate(state);
|
| 1533 |
+
}
|
| 1534 |
+
|
| 1535 |
+
if (config.options?.onDataUpdate) {
|
| 1536 |
+
config.options.onDataUpdate({
|
| 1537 |
+
frameCount: state.frameCount,
|
| 1538 |
+
currentEpisode: state.episodeCount,
|
| 1539 |
+
recentFrames: [],
|
| 1540 |
+
});
|
| 1541 |
+
}
|
| 1542 |
+
}
|
| 1543 |
+
}, 100);
|
| 1544 |
+
}
|
| 1545 |
+
},
|
| 1546 |
+
|
| 1547 |
+
async stop(): Promise<RobotRecordingData> {
|
| 1548 |
+
if (stateUpdateInterval) {
|
| 1549 |
+
clearInterval(stateUpdateInterval);
|
| 1550 |
+
stateUpdateInterval = null;
|
| 1551 |
+
}
|
| 1552 |
+
|
| 1553 |
+
const result = await recorder.stopRecording();
|
| 1554 |
+
|
| 1555 |
+
const robotData: RobotRecordingData = {
|
| 1556 |
+
episodes: recorder.episodes.map((episode) => episode.frames),
|
| 1557 |
+
metadata: {
|
| 1558 |
+
fps: config.options?.fps || 30,
|
| 1559 |
+
robotType: config.robotType || "unknown",
|
| 1560 |
+
startTime: startTime,
|
| 1561 |
+
endTime: Date.now(),
|
| 1562 |
+
totalFrames: recorder.teleoperatorData.reduce(
|
| 1563 |
+
(sum, ep) => sum + ep.length,
|
| 1564 |
+
0
|
| 1565 |
+
),
|
| 1566 |
+
totalEpisodes: recorder.teleoperatorData.length,
|
| 1567 |
+
},
|
| 1568 |
+
};
|
| 1569 |
+
|
| 1570 |
+
return robotData;
|
| 1571 |
+
},
|
| 1572 |
+
|
| 1573 |
+
getState(): RecordingState {
|
| 1574 |
+
return {
|
| 1575 |
+
isActive: recorder.isRecording,
|
| 1576 |
+
frameCount: recorder.teleoperatorData.reduce(
|
| 1577 |
+
(sum, ep) => sum + ep.length,
|
| 1578 |
+
0
|
| 1579 |
+
),
|
| 1580 |
+
episodeCount: recorder.teleoperatorData.length,
|
| 1581 |
+
duration: recorder.isRecording ? Date.now() - startTime : 0,
|
| 1582 |
+
lastUpdate: Date.now(),
|
| 1583 |
+
};
|
| 1584 |
+
},
|
| 1585 |
+
|
| 1586 |
+
get result(): Promise<RobotRecordingData> {
|
| 1587 |
+
if (!resultPromise) {
|
| 1588 |
+
resultPromise = new Promise((resolve) => {
|
| 1589 |
+
const originalStop = recordProcess.stop;
|
| 1590 |
+
recordProcess.stop = async () => {
|
| 1591 |
+
const data = await originalStop();
|
| 1592 |
+
resolve(data);
|
| 1593 |
+
return data;
|
| 1594 |
+
};
|
| 1595 |
+
});
|
| 1596 |
+
}
|
| 1597 |
+
return resultPromise;
|
| 1598 |
+
},
|
| 1599 |
+
|
| 1600 |
+
getEpisodeCount(): number {
|
| 1601 |
+
return recorder.teleoperatorData.length;
|
| 1602 |
+
},
|
| 1603 |
+
|
| 1604 |
+
getEpisodes(): any[] {
|
| 1605 |
+
return recorder.teleoperatorData;
|
| 1606 |
+
},
|
| 1607 |
+
|
| 1608 |
+
clearEpisodes(): void {
|
| 1609 |
+
(recorder as any).clearRecording();
|
| 1610 |
+
},
|
| 1611 |
+
|
| 1612 |
+
async nextEpisode(): Promise<number> {
|
| 1613 |
+
return (recorder as any).nextEpisodeSegment();
|
| 1614 |
+
},
|
| 1615 |
+
|
| 1616 |
+
restoreEpisodes(episodes: any[]): void {
|
| 1617 |
+
(recorder as any).teleoperatorData = [...episodes];
|
| 1618 |
+
},
|
| 1619 |
+
|
| 1620 |
+
addCamera(name: string, stream: MediaStream): void {
|
| 1621 |
+
(recorder as any).addVideoStream(name, stream);
|
| 1622 |
+
},
|
| 1623 |
+
|
| 1624 |
+
removeCamera(name: string): void {
|
| 1625 |
+
const videoStreams = (recorder as any).videoStreams || {};
|
| 1626 |
+
delete videoStreams[name];
|
| 1627 |
+
},
|
| 1628 |
+
|
| 1629 |
+
async exportForLeRobot(
|
| 1630 |
+
format: "blobs" | "zip" | "zip-download" = "zip-download"
|
| 1631 |
+
): Promise<any> {
|
| 1632 |
+
return recorder.exportForLeRobot(format);
|
| 1633 |
+
},
|
| 1634 |
+
};
|
| 1635 |
+
|
| 1636 |
+
return recordProcess;
|
| 1637 |
+
}
|
|
@@ -1,172 +0,0 @@
|
|
| 1 |
-
// Note: To use this module, you'll need to install the AWS SDK:
|
| 2 |
-
// npm install @aws-sdk/client-s3 @aws-sdk/lib-storage
|
| 3 |
-
|
| 4 |
-
import { S3Client, HeadBucketCommand } from "@aws-sdk/client-s3";
|
| 5 |
-
import { Upload } from "@aws-sdk/lib-storage";
|
| 6 |
-
|
| 7 |
-
// Define ContentSource type locally to avoid HuggingFace dependency
|
| 8 |
-
type ContentSource = Blob | ArrayBuffer | Uint8Array | string;
|
| 9 |
-
type FileArray = Array<URL | File | { path: string; content: ContentSource }>;
|
| 10 |
-
|
| 11 |
-
/**
|
| 12 |
-
* Uploads a leRobot dataset to Amazon S3
|
| 13 |
-
*/
|
| 14 |
-
export class LeRobotS3Uploader extends EventTarget {
|
| 15 |
-
private _bucketName: string;
|
| 16 |
-
private _region: string;
|
| 17 |
-
private _uploaded: boolean;
|
| 18 |
-
private _bucket_exists: boolean;
|
| 19 |
-
private _s3Client: S3Client | null;
|
| 20 |
-
|
| 21 |
-
constructor(bucketName: string, region: string = "us-east-1") {
|
| 22 |
-
super();
|
| 23 |
-
this._bucketName = bucketName;
|
| 24 |
-
this._region = region;
|
| 25 |
-
this._uploaded = false;
|
| 26 |
-
this._bucket_exists = false;
|
| 27 |
-
this._s3Client = null;
|
| 28 |
-
}
|
| 29 |
-
|
| 30 |
-
/**
|
| 31 |
-
* Returns whether the bucket has been successfully checked/created
|
| 32 |
-
*/
|
| 33 |
-
get bucketExists(): boolean {
|
| 34 |
-
return this._bucket_exists;
|
| 35 |
-
}
|
| 36 |
-
|
| 37 |
-
get uploaded(): boolean {
|
| 38 |
-
return this._uploaded;
|
| 39 |
-
}
|
| 40 |
-
|
| 41 |
-
/**
|
| 42 |
-
* Initialize the S3 client with credentials
|
| 43 |
-
*
|
| 44 |
-
* @param accessKeyId AWS access key ID
|
| 45 |
-
* @param secretAccessKey AWS secret access key
|
| 46 |
-
*/
|
| 47 |
-
initializeClient(accessKeyId: string, secretAccessKey: string): void {
|
| 48 |
-
this._s3Client = new S3Client({
|
| 49 |
-
region: this._region,
|
| 50 |
-
credentials: {
|
| 51 |
-
accessKeyId,
|
| 52 |
-
secretAccessKey
|
| 53 |
-
}
|
| 54 |
-
});
|
| 55 |
-
}
|
| 56 |
-
|
| 57 |
-
/**
|
| 58 |
-
* Checks if the bucket exists and uploads files to it
|
| 59 |
-
*
|
| 60 |
-
* @param files The files to upload
|
| 61 |
-
* @param accessKeyId AWS access key ID
|
| 62 |
-
* @param secretAccessKey AWS secret access key
|
| 63 |
-
* @param prefix Optional prefix (folder) to upload files to within the bucket
|
| 64 |
-
* @param referenceId The reference id for the upload, to track it (optional)
|
| 65 |
-
*/
|
| 66 |
-
async checkBucketAndUploadFiles(
|
| 67 |
-
files: FileArray,
|
| 68 |
-
accessKeyId: string,
|
| 69 |
-
secretAccessKey: string,
|
| 70 |
-
prefix: string = "",
|
| 71 |
-
referenceId: string = ""
|
| 72 |
-
): Promise<void> {
|
| 73 |
-
// Initialize the client if not already done
|
| 74 |
-
if (!this._s3Client) {
|
| 75 |
-
this.initializeClient(accessKeyId, secretAccessKey);
|
| 76 |
-
}
|
| 77 |
-
|
| 78 |
-
// Check if bucket exists
|
| 79 |
-
try {
|
| 80 |
-
await this._s3Client!.send(new HeadBucketCommand({ Bucket: this._bucketName }));
|
| 81 |
-
this._bucket_exists = true;
|
| 82 |
-
this.dispatchEvent(new CustomEvent("bucketExists", {
|
| 83 |
-
detail: { bucketName: this._bucketName }
|
| 84 |
-
}));
|
| 85 |
-
} catch (error) {
|
| 86 |
-
throw new Error(`Bucket ${this._bucketName} does not exist or you don't have permission to access it`);
|
| 87 |
-
}
|
| 88 |
-
|
| 89 |
-
// Upload files
|
| 90 |
-
const uploadPromises: Promise<void>[] = [];
|
| 91 |
-
for (const file of files) {
|
| 92 |
-
uploadPromises.push(this.uploadFileWithProgress([file], prefix, referenceId));
|
| 93 |
-
}
|
| 94 |
-
|
| 95 |
-
await Promise.all(uploadPromises);
|
| 96 |
-
this._uploaded = true;
|
| 97 |
-
}
|
| 98 |
-
|
| 99 |
-
/**
|
| 100 |
-
* Uploads files to S3 with progress events
|
| 101 |
-
*
|
| 102 |
-
* @param files The files to upload
|
| 103 |
-
* @param prefix Optional prefix (folder) to upload files to within the bucket
|
| 104 |
-
* @param referenceId The reference id for the upload, to track it (optional)
|
| 105 |
-
*/
|
| 106 |
-
async uploadFileWithProgress(
|
| 107 |
-
files: FileArray,
|
| 108 |
-
prefix: string = "",
|
| 109 |
-
referenceId: string = ""
|
| 110 |
-
): Promise<void> {
|
| 111 |
-
if (!this._s3Client) {
|
| 112 |
-
throw new Error("S3 client not initialized. Call initializeClient first.");
|
| 113 |
-
}
|
| 114 |
-
|
| 115 |
-
for (const file of files) {
|
| 116 |
-
let key: string;
|
| 117 |
-
let body: any;
|
| 118 |
-
|
| 119 |
-
if (file instanceof URL) {
|
| 120 |
-
const response = await fetch(file);
|
| 121 |
-
body = await response.blob();
|
| 122 |
-
key = `${prefix}${prefix ? '/' : ''}${file.pathname.split('/').pop()}`;
|
| 123 |
-
} else if (file instanceof File) {
|
| 124 |
-
body = file;
|
| 125 |
-
key = `${prefix}${prefix ? '/' : ''}${file.name}`;
|
| 126 |
-
} else {
|
| 127 |
-
body = file.content;
|
| 128 |
-
key = `${prefix}${prefix ? '/' : ''}${file.path}`;
|
| 129 |
-
}
|
| 130 |
-
|
| 131 |
-
const upload = new Upload({
|
| 132 |
-
client: this._s3Client,
|
| 133 |
-
params: {
|
| 134 |
-
Bucket: this._bucketName,
|
| 135 |
-
Key: key,
|
| 136 |
-
Body: body,
|
| 137 |
-
},
|
| 138 |
-
});
|
| 139 |
-
|
| 140 |
-
// Set up progress tracking
|
| 141 |
-
upload.on('httpUploadProgress', (progress) => {
|
| 142 |
-
this.dispatchEvent(new CustomEvent("progress", {
|
| 143 |
-
detail: {
|
| 144 |
-
progressEvent: progress,
|
| 145 |
-
bucketName: this._bucketName,
|
| 146 |
-
key,
|
| 147 |
-
referenceId
|
| 148 |
-
}
|
| 149 |
-
}));
|
| 150 |
-
});
|
| 151 |
-
|
| 152 |
-
await upload.done();
|
| 153 |
-
}
|
| 154 |
-
}
|
| 155 |
-
|
| 156 |
-
/**
|
| 157 |
-
* Generates a pre-signed URL for downloading a file from S3
|
| 158 |
-
*
|
| 159 |
-
* @param _key The key (path) of the file in the S3 bucket
|
| 160 |
-
* @param _expiresIn The number of seconds until the URL expires (default: 3600 = 1 hour)
|
| 161 |
-
* @returns A pre-signed URL for downloading the file
|
| 162 |
-
*/
|
| 163 |
-
async generatePresignedUrl(_key: string, _expiresIn: number = 3600): Promise<string> {
|
| 164 |
-
if (!this._s3Client) {
|
| 165 |
-
throw new Error("S3 client not initialized. Call initializeClient first.");
|
| 166 |
-
}
|
| 167 |
-
|
| 168 |
-
// Note: This requires the @aws-sdk/s3-request-presigner package
|
| 169 |
-
// Implementation would go here
|
| 170 |
-
throw new Error("generatePresignedUrl not implemented. Requires @aws-sdk/s3-request-presigner package.");
|
| 171 |
-
}
|
| 172 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@@ -0,0 +1,122 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/**
|
| 2 |
+
* Types for the simple record() function API
|
| 3 |
+
*/
|
| 4 |
+
|
| 5 |
+
import type { WebTeleoperator } from "../teleoperators/base-teleoperator.js";
|
| 6 |
+
import type { LeRobotDatasetRecorder } from "../record.js";
|
| 7 |
+
|
| 8 |
+
/**
|
| 9 |
+
* Configuration for the record() function
|
| 10 |
+
* Supports both upfront configuration and runtime management
|
| 11 |
+
*/
|
| 12 |
+
export interface RecordConfig {
|
| 13 |
+
/** The teleoperator to record from (explicit dependency) */
|
| 14 |
+
teleoperator: WebTeleoperator;
|
| 15 |
+
|
| 16 |
+
/** Optional: video streams to record by camera name (e.g. { main: videoStream, wrist: videoStream }) */
|
| 17 |
+
videoStreams?: { [cameraName: string]: MediaStream };
|
| 18 |
+
|
| 19 |
+
/** Optional: robot type/model name for metadata (e.g. "so100") */
|
| 20 |
+
robotType?: string;
|
| 21 |
+
|
| 22 |
+
/** Optional recording configuration */
|
| 23 |
+
options?: {
|
| 24 |
+
/** Target frames per second (default: 30) */
|
| 25 |
+
fps?: number;
|
| 26 |
+
/** Task description for the recording */
|
| 27 |
+
taskDescription?: string;
|
| 28 |
+
/** Callback for real-time recording data updates */
|
| 29 |
+
onDataUpdate?: (data: RecordingData) => void;
|
| 30 |
+
/** Callback for recording state changes */
|
| 31 |
+
onStateUpdate?: (state: RecordingState) => void;
|
| 32 |
+
};
|
| 33 |
+
}
|
| 34 |
+
|
| 35 |
+
/**
|
| 36 |
+
* Process interface returned by record() function
|
| 37 |
+
* Supports flexible upfront config and runtime management
|
| 38 |
+
*/
|
| 39 |
+
export interface RecordProcess {
|
| 40 |
+
// Recording control
|
| 41 |
+
/** Start recording */
|
| 42 |
+
start(): void;
|
| 43 |
+
/** Stop recording and return the result */
|
| 44 |
+
stop(): Promise<RobotRecordingData>;
|
| 45 |
+
/** Get current recording state */
|
| 46 |
+
getState(): RecordingState;
|
| 47 |
+
/** Promise that resolves when recording is stopped with the data */
|
| 48 |
+
result: Promise<RobotRecordingData>;
|
| 49 |
+
|
| 50 |
+
// Episode management (runtime)
|
| 51 |
+
/** Get total number of episodes recorded */
|
| 52 |
+
getEpisodeCount(): number;
|
| 53 |
+
/** Get raw episode data for viewing/analysis */
|
| 54 |
+
getEpisodes(): any[];
|
| 55 |
+
/** Delete all recorded episodes */
|
| 56 |
+
clearEpisodes(): void;
|
| 57 |
+
/** Start a new episode segment and get the new episode index */
|
| 58 |
+
nextEpisode(): Promise<number>;
|
| 59 |
+
/** Restore previously recorded episodes */
|
| 60 |
+
restoreEpisodes(episodes: any[]): void;
|
| 61 |
+
|
| 62 |
+
// Camera management (runtime)
|
| 63 |
+
/** Add a camera stream for recording */
|
| 64 |
+
addCamera(name: string, stream: MediaStream): void;
|
| 65 |
+
/** Remove a camera from recording */
|
| 66 |
+
removeCamera(name: string): void;
|
| 67 |
+
|
| 68 |
+
// Export
|
| 69 |
+
/** Export the recorded dataset in various formats */
|
| 70 |
+
exportForLeRobot(format?: "blobs" | "zip" | "zip-download"): Promise<any>;
|
| 71 |
+
}
|
| 72 |
+
|
| 73 |
+
/**
|
| 74 |
+
* Current state of the recording process
|
| 75 |
+
*/
|
| 76 |
+
export interface RecordingState {
|
| 77 |
+
/** Whether recording is currently active */
|
| 78 |
+
isActive: boolean;
|
| 79 |
+
/** Total number of frames recorded */
|
| 80 |
+
frameCount: number;
|
| 81 |
+
/** Number of episodes recorded */
|
| 82 |
+
episodeCount: number;
|
| 83 |
+
/** Duration of current recording in milliseconds */
|
| 84 |
+
duration: number;
|
| 85 |
+
/** Timestamp of last update */
|
| 86 |
+
lastUpdate: number;
|
| 87 |
+
}
|
| 88 |
+
|
| 89 |
+
/**
|
| 90 |
+
* Real-time recording data for UI feedback
|
| 91 |
+
*/
|
| 92 |
+
export interface RecordingData {
|
| 93 |
+
/** Total frames recorded */
|
| 94 |
+
frameCount: number;
|
| 95 |
+
/** Current episode number */
|
| 96 |
+
currentEpisode: number;
|
| 97 |
+
/** Recent frames for preview (last few frames) */
|
| 98 |
+
recentFrames: any[];
|
| 99 |
+
}
|
| 100 |
+
|
| 101 |
+
/**
|
| 102 |
+
* Final robot recording data (hardware only, no video)
|
| 103 |
+
*/
|
| 104 |
+
export interface RobotRecordingData {
|
| 105 |
+
/** Recorded episodes with motor position data */
|
| 106 |
+
episodes: any[];
|
| 107 |
+
/** Recording metadata */
|
| 108 |
+
metadata: {
|
| 109 |
+
/** Frames per second */
|
| 110 |
+
fps: number;
|
| 111 |
+
/** Robot type if available */
|
| 112 |
+
robotType: string;
|
| 113 |
+
/** Recording start time */
|
| 114 |
+
startTime: number;
|
| 115 |
+
/** Recording end time */
|
| 116 |
+
endTime: number;
|
| 117 |
+
/** Total frames recorded */
|
| 118 |
+
totalFrames: number;
|
| 119 |
+
/** Total episodes recorded */
|
| 120 |
+
totalEpisodes: number;
|
| 121 |
+
};
|
| 122 |
+
}
|
|
@@ -1,159 +0,0 @@
|
|
| 1 |
-
export interface VideoInfo {
|
| 2 |
-
height: number;
|
| 3 |
-
width: number;
|
| 4 |
-
channels: number;
|
| 5 |
-
codec: string;
|
| 6 |
-
pix_fmt: string;
|
| 7 |
-
is_depth_map: boolean;
|
| 8 |
-
has_audio: boolean;
|
| 9 |
-
}
|
| 10 |
-
|
| 11 |
-
/**
|
| 12 |
-
* Metadata parameters interface
|
| 13 |
-
*/
|
| 14 |
-
interface MetadataParams {
|
| 15 |
-
total_episodes: number;
|
| 16 |
-
total_frames: number;
|
| 17 |
-
total_tasks: number;
|
| 18 |
-
chunks_size: number;
|
| 19 |
-
fps: number;
|
| 20 |
-
splits: { [key: string]: string };
|
| 21 |
-
features: { [key: string]: any };
|
| 22 |
-
videos_info: VideoInfo[];
|
| 23 |
-
data_files_size_in_mb: number;
|
| 24 |
-
video_files_size_in_mb: number;
|
| 25 |
-
}
|
| 26 |
-
|
| 27 |
-
/**
|
| 28 |
-
* Generates and returns a metadata information dictionary
|
| 29 |
-
* Needs some named parameters passed as parameters
|
| 30 |
-
*/
|
| 31 |
-
function getMetadataInfo(params: MetadataParams) {
|
| 32 |
-
return {
|
| 33 |
-
"codebase_version": "v2.1",
|
| 34 |
-
"robot_type": "so100",
|
| 35 |
-
"total_episodes": params.total_episodes,
|
| 36 |
-
"total_frames": params.total_frames,
|
| 37 |
-
"total_tasks": params.total_tasks,
|
| 38 |
-
"total_videos": params.videos_info.length,
|
| 39 |
-
"total_chunks": 1,
|
| 40 |
-
"chunks_size": params.chunks_size,
|
| 41 |
-
"fps": params.fps,
|
| 42 |
-
"splits": {
|
| 43 |
-
"train": `0:${params.total_episodes}`
|
| 44 |
-
},
|
| 45 |
-
"data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
|
| 46 |
-
"video_path": "videos/chunk-{episode_chunk:03d}/{video_key}/episode_{episode_index:06d}.mp4",
|
| 47 |
-
"features": {
|
| 48 |
-
"action": {
|
| 49 |
-
"dtype": "float32",
|
| 50 |
-
"shape": [
|
| 51 |
-
6
|
| 52 |
-
],
|
| 53 |
-
"names": [
|
| 54 |
-
"main_shoulder_pan",
|
| 55 |
-
"main_shoulder_lift",
|
| 56 |
-
"main_elbow_flex",
|
| 57 |
-
"main_wrist_flex",
|
| 58 |
-
"main_wrist_roll",
|
| 59 |
-
"main_gripper"
|
| 60 |
-
],
|
| 61 |
-
"fps": params.fps
|
| 62 |
-
},
|
| 63 |
-
"observation.state": {
|
| 64 |
-
"dtype": "float32",
|
| 65 |
-
"shape": [
|
| 66 |
-
6
|
| 67 |
-
],
|
| 68 |
-
"names": [
|
| 69 |
-
"main_shoulder_pan",
|
| 70 |
-
"main_shoulder_lift",
|
| 71 |
-
"main_elbow_flex",
|
| 72 |
-
"main_wrist_flex",
|
| 73 |
-
"main_wrist_roll",
|
| 74 |
-
"main_gripper"
|
| 75 |
-
],
|
| 76 |
-
"fps": params.fps
|
| 77 |
-
},
|
| 78 |
-
"observation.images.front": {
|
| 79 |
-
"dtype": "video",
|
| 80 |
-
"shape": [
|
| 81 |
-
480,
|
| 82 |
-
640,
|
| 83 |
-
3
|
| 84 |
-
],
|
| 85 |
-
"names": [
|
| 86 |
-
"height",
|
| 87 |
-
"width",
|
| 88 |
-
"channels"
|
| 89 |
-
],
|
| 90 |
-
"info": {
|
| 91 |
-
"video.fps": params.fps,
|
| 92 |
-
"video.height": 480,
|
| 93 |
-
"video.width": 640,
|
| 94 |
-
"video.channels": 3,
|
| 95 |
-
"video.codec": "av1",
|
| 96 |
-
"video.pix_fmt": "yuv420p",
|
| 97 |
-
"video.is_depth_map": false,
|
| 98 |
-
"has_audio": false
|
| 99 |
-
}
|
| 100 |
-
},
|
| 101 |
-
"timestamp": {
|
| 102 |
-
"dtype": "float32",
|
| 103 |
-
"shape": [
|
| 104 |
-
1
|
| 105 |
-
],
|
| 106 |
-
"names": null,
|
| 107 |
-
"fps": params.fps
|
| 108 |
-
},
|
| 109 |
-
"frame_index": {
|
| 110 |
-
"dtype": "int64",
|
| 111 |
-
"shape": [
|
| 112 |
-
1
|
| 113 |
-
],
|
| 114 |
-
"names": null,
|
| 115 |
-
"fps": params.fps
|
| 116 |
-
},
|
| 117 |
-
"episode_index": {
|
| 118 |
-
"dtype": "int64",
|
| 119 |
-
"shape": [
|
| 120 |
-
1
|
| 121 |
-
],
|
| 122 |
-
"names": null,
|
| 123 |
-
"fps": params.fps
|
| 124 |
-
},
|
| 125 |
-
"index": {
|
| 126 |
-
"dtype": "int64",
|
| 127 |
-
"shape": [
|
| 128 |
-
1
|
| 129 |
-
],
|
| 130 |
-
"names": null,
|
| 131 |
-
"fps": params.fps
|
| 132 |
-
},
|
| 133 |
-
"task_index": {
|
| 134 |
-
"dtype": "int64",
|
| 135 |
-
"shape": [
|
| 136 |
-
1
|
| 137 |
-
],
|
| 138 |
-
"names": null,
|
| 139 |
-
"fps": params.fps
|
| 140 |
-
}
|
| 141 |
-
},
|
| 142 |
-
"data_files_size_in_mb": 100,
|
| 143 |
-
"video_files_size_in_mb": 500
|
| 144 |
-
}
|
| 145 |
-
}
|
| 146 |
-
|
| 147 |
-
export function getVideoInfo(width: number, height: number): VideoInfo {
|
| 148 |
-
return {
|
| 149 |
-
height,
|
| 150 |
-
width,
|
| 151 |
-
channels: 3,
|
| 152 |
-
codec: "h264",
|
| 153 |
-
pix_fmt: "yuv420p",
|
| 154 |
-
is_depth_map: false,
|
| 155 |
-
has_audio: false
|
| 156 |
-
};
|
| 157 |
-
}
|
| 158 |
-
|
| 159 |
-
export default getMetadataInfo;
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@@ -1,79 +0,0 @@
|
|
| 1 |
-
/**
|
| 2 |
-
* Calculates basic statistics (min, max, mean, std, count) for a numeric array
|
| 3 |
-
*/
|
| 4 |
-
function calculateStats(values: number[]): { min: number[], max: number[], mean: number[], std: number[], count: number[] } {
|
| 5 |
-
const count = values.length;
|
| 6 |
-
if (count === 0) {
|
| 7 |
-
return {
|
| 8 |
-
min: [0],
|
| 9 |
-
max: [0],
|
| 10 |
-
mean: [0],
|
| 11 |
-
std: [0],
|
| 12 |
-
count: [0]
|
| 13 |
-
};
|
| 14 |
-
}
|
| 15 |
-
|
| 16 |
-
let min = Infinity;
|
| 17 |
-
let max = -Infinity;
|
| 18 |
-
|
| 19 |
-
for(let value of values){
|
| 20 |
-
min = Math.min(min, value);
|
| 21 |
-
max = Math.max(max, value);
|
| 22 |
-
}
|
| 23 |
-
|
| 24 |
-
const sum = values.reduce((acc, val) => acc + val, 0);
|
| 25 |
-
const mean = sum / count;
|
| 26 |
-
|
| 27 |
-
// Calculate standard deviation
|
| 28 |
-
const squareDiffs = values.map(value => Math.pow(value - mean, 2));
|
| 29 |
-
const avgSquareDiff = squareDiffs.reduce((acc, val) => acc + val, 0) / count;
|
| 30 |
-
const std = Math.sqrt(avgSquareDiff);
|
| 31 |
-
|
| 32 |
-
return {
|
| 33 |
-
min: [min],
|
| 34 |
-
max: [max],
|
| 35 |
-
mean: [mean],
|
| 36 |
-
std: [std],
|
| 37 |
-
count: [count]
|
| 38 |
-
};
|
| 39 |
-
}
|
| 40 |
-
|
| 41 |
-
/**
|
| 42 |
-
* Generates statistics for a dataset
|
| 43 |
-
* @param data The dataset to analyze
|
| 44 |
-
* @param cameraKeys Array of camera keys for dynamic observation.images entries
|
| 45 |
-
*/
|
| 46 |
-
export function getStats(data: any[], cameraKeys: string[] = []): any {
|
| 47 |
-
// Extract timestamp and episode_index values
|
| 48 |
-
const timestamps = data.map(item => item.timestamp);
|
| 49 |
-
const episodeIndices = data.map(item => item.episode_index);
|
| 50 |
-
|
| 51 |
-
// Extract other common fields if they exist
|
| 52 |
-
const frameIndices = data.map(item => item.frame_index || 0);
|
| 53 |
-
const taskIndices = data.map(item => item.task_index || 0);
|
| 54 |
-
|
| 55 |
-
const stats: any = {
|
| 56 |
-
// Standard fields
|
| 57 |
-
"timestamp": calculateStats(timestamps),
|
| 58 |
-
"episode_index": calculateStats(episodeIndices),
|
| 59 |
-
"frame_index": calculateStats(frameIndices),
|
| 60 |
-
"task_index": calculateStats(taskIndices),
|
| 61 |
-
};
|
| 62 |
-
|
| 63 |
-
// Add observation.images entries for each camera key
|
| 64 |
-
cameraKeys.forEach(key => {
|
| 65 |
-
// In a real implementation, you would calculate actual stats from video data
|
| 66 |
-
// Since we don't have actual video frame data to analyze, we'll use placeholder values
|
| 67 |
-
stats[`observation.images.${key}`] = {
|
| 68 |
-
"min": [[[0.0]], [[0.0]], [[0.0]]], // R,G,B channels min
|
| 69 |
-
"max": [[[255.0]], [[255.0]], [[255.0]]], // R,G,B channels max
|
| 70 |
-
"mean": [[[127.5]], [[127.5]], [[127.5]]], // R,G,B channels mean
|
| 71 |
-
"std": [[[50.0]], [[50.0]], [[50.0]]], // R,G,B channels std
|
| 72 |
-
"count": [[[data.length * 3]]] // Number of pixels × 3 channels
|
| 73 |
-
};
|
| 74 |
-
});
|
| 75 |
-
|
| 76 |
-
return stats;
|
| 77 |
-
}
|
| 78 |
-
|
| 79 |
-
export default getStats;
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|