Exploring Conditions for Diffusion models in Robotic Control
While pre-trained visual representations have significantly advanced
imitation learning, they are often task-agnostic as they remain frozen during
policy learning. In this work, we explore leveraging pre-trained text-to-image
diffusion models to obtain task-adaptive visual representations for robotic
control, without fine-tuning the model itself. However, we find that naively
applying textual conditions - a successful strategy in other vision domains -
yields minimal or even negative gains in control tasks. We attribute this to
the domain gap between the diffusion model's training data and robotic control
environments, leading us to argue for conditions that consider the specific,
dynamic visual information required for control. To this end, we propose ORCA,
which introduces learnable task prompts that adapt to the control environment
and visual prompts that capture fine-grained, frame-specific details. Through
facilitating task-adaptive representations with our newly devised conditions,
our approach achieves state-of-the-art performance on various robotic control
benchmarks, significantly surpassing prior methods.
68feef866cdff8b857f471d1
Heeseong Shin
claimed_verified
2025-10-28T15:38:00.777Z
66f989359ebb4cf074a1f9f7
/avatars/d0e899f579e0897a849e79688bf19c8a.svg
Heeseong Shin
hsshin98
user
hsshin98
68feef866cdff8b857f471d2
Byeongho Heo
claimed_verified
2025-10-31T14:28:34.237Z
64b9feed96676e40d0fa89a7
/avatars/2154a6ceb87677ad2c9d9620de5b18ec.svg
Byeongho Heo
bhheo
user
bhheo
68feef866cdff8b857f471d3
Dongyoon Han
claimed_verified
2026-02-17T15:48:13.720Z
660f8cc1a61244f3df3d4426
/avatars/45d59766122bb3482f6dd7f9d98aa87a.svg
Dongyoon Han
calintz
user
calintz
68feef866cdff8b857f471d4
Seungryong Kim
68feef866cdff8b857f471d5
Taekyung Kim
claimed_verified
2025-10-27T10:25:08.822Z
67c6a1e75e2443d7d5f85cb3
/avatars/0569b368520411ab828d46725bc3896a.svg
Taekyung Kim
taekyung-k
user
taekyung-k
Exploring Conditions for Diffusion models in Robotic Control
While pre-trained visual representations have significantly advanced
imitation learning, they are often task-agnostic as they remain frozen during
policy learning. In this work, we explore leveraging pre-trained text-to-image
diffusion models to obtain task-adaptive visual representations for robotic
control, without fine-tuning the model itself. However, we find that naively
applying textual conditions - a successful strategy in other vision domains -
yields minimal or even negative gains in control tasks. We attribute this to
the domain gap between the diffusion model's training data and robotic control
environments, leading us to argue for conditions that consider the specific,
dynamic visual information required for control. To this end, we propose ORCA,
which introduces learnable task prompts that adapt to the control environment
and visual prompts that capture fine-grained, frame-specific details. Through
facilitating task-adaptive representations with our newly devised conditions,
our approach achieves state-of-the-art performance on various robotic control
benchmarks, significantly surpassing prior methods.
ORCA uses learnable task and visual prompts to adapt pre-trained text-to-image diffusion models for robotic control, achieving state-of-the-art performance on benchmarks.
text-to-image diffusion models task-adaptive visual representations robotic control domain gap learnable task prompts visual prompts frame-specific details