LEGO-Eval: Towards Fine-Grained Evaluation on Synthesizing 3D Embodied
  Environments with Tool Augmentation
Despite recent progress in using Large Language Models (LLMs) for
automatically generating 3D scenes, generated scenes often lack realistic
spatial layouts and object attributes found in real-world environments. As this
problem stems from insufficiently detailed, coarse-grained instructions,
advancing 3D scene synthesis guided by more detailed, fine-grained instructions
that reflect real-world environments becomes crucial. Without such realistic
scenes, training embodied agents in unrealistic environments can lead them to
learn priors that diverge significantly from real-world physics and semantics,
degrading their performance when deployed. Thus, verifying the alignment
between the fine-grained instruction and the generated scene is essential for
effective learning. However, current evaluation methods, such as CLIPScore and
vision-language models (VLMs), often fail to reliably assess such alignment.
This shortcoming arises primarily from their shallow understanding of 3D
scenes, which often leads to improperly grounded scene components. To address
this, we introduce LEGO-Eval, an evaluation framework equipped with diverse
tools designed to explicitly ground scene components, enabling more accurate
alignment assessments. We also present LEGO-Bench, a benchmark of detailed
instructions that specify complex layouts and attributes of real-world
environments. Experiments demonstrate that LEGO-Eval outperforms VLM-as-a-judge
by 0.41 F1 score in assessing scene-instruction alignment. Benchmarking with
LEGO-Bench reveals significant limitations in current generation methods.
Across all evaluated approaches, success rates reached at most 10% in
generating scenes that fully align with fine-grained instructions.
690c0e1c60494e4fa767563a
Gyeom Hwangbo
claimed_verified
2025-11-06T11:48:07.609Z
66bb326534295e9cf08df4e2
/avatars/5dc3225be1194467b30691b5d33c7b19.svg
Gyeom hwangbo
aerojohn1223
user
aerojohn1223
690c0e1c60494e4fa767563b
Hyungjoo Chae
claimed_verified
2025-11-10T09:32:13.916Z
64c8f4cec547ed5243ebd0a8
https://cdn-avatars.huggingface.co/v1/production/uploads/64c8f4cec547ed5243ebd0a8/MiOH5YbMg8Gh9KYlQsLmX.jpeg
Hyungjoo Chae
hyungjoochae
user
hyungjoochae
690c0e1c60494e4fa767563c
Minseok Kang
690c0e1c60494e4fa767563d
Hyeonjong Ju
690c0e1c60494e4fa767563e
Soohyun Oh
690c0e1c60494e4fa767563f
Jinyoung Yeo
LEGO-Eval: Towards Fine-Grained Evaluation on Synthesizing 3D Embodied
  Environments with Tool Augmentation
Despite recent progress in using Large Language Models (LLMs) for
automatically generating 3D scenes, generated scenes often lack realistic
spatial layouts and object attributes found in real-world environments. As this
problem stems from insufficiently detailed, coarse-grained instructions,
advancing 3D scene synthesis guided by more detailed, fine-grained instructions
that reflect real-world environments becomes crucial. Without such realistic
scenes, training embodied agents in unrealistic environments can lead them to
learn priors that diverge significantly from real-world physics and semantics,
degrading their performance when deployed. Thus, verifying the alignment
between the fine-grained instruction and the generated scene is essential for
effective learning. However, current evaluation methods, such as CLIPScore and
vision-language models (VLMs), often fail to reliably assess such alignment.
This shortcoming arises primarily from their shallow understanding of 3D
scenes, which often leads to improperly grounded scene components. To address
this, we introduce LEGO-Eval, an evaluation framework equipped with diverse
tools designed to explicitly ground scene components, enabling more accurate
alignment assessments. We also present LEGO-Bench, a benchmark of detailed
instructions that specify complex layouts and attributes of real-world
environments. Experiments demonstrate that LEGO-Eval outperforms VLM-as-a-judge
by 0.41 F1 score in assessing scene-instruction alignment. Benchmarking with
LEGO-Bench reveals significant limitations in current generation methods.
Across all evaluated approaches, success rates reached at most 10% in
generating scenes that fully align with fine-grained instructions.
LEGO-Eval and LEGO-Bench improve the evaluation and generation of realistic 3D scenes by aligning detailed instructions with scene components, outperforming existing methods.
Large Language Models 3D scene synthesis fine-grained instructions real-world environments CLIPScore vision-language models LEGO-Eval LEGO-Bench scene-instruction alignment F1 score