Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States

📖 Project Page   |   📄 Paper   |   💻 GitHub   |   🤗 Models   |   🗂️ Dataset   |   🧪 Supplementary

A world model should do more than generate plausible pixels. It should know where the camera is in the physical world, understand the geometry beneath an observation, and generate new viewpoints while remaining consistent with gravity and scene structure.

We introduce Puffin-World, a unified multimodal world model that perceives, simulates, generates, and reconstructs the 3D world within one framework. Instead of treating a world as a sequence of RGB frames alone, Puffin-World represents it through three complementary native world states:

• Physics: gravity field and latitude, anchoring each observation to the real world.
• Geometry: depth, describing the underlying 3D spatial structure.
• Appearance: images and sequences, describing what an observer sees.

In general, Puffin-World supports camera-to-world understanding, camera-controllable text-to-image generation, image- and text-to-3D world generation, challenging camera trajectories, native geometry prediction, and 3D reconstruction—without external offline perception or reconstruction modules.

In short, Puffin-World moves multimodal unification from 2D semantics toward physically grounded 3D worlds.

Why appearance alone is not enough

Recent generative world models can produce visually impressive future observations, and unified multimodal models can combine image understanding and generation in one network. Yet most systems still represent the world primarily at the appearance level. They can predict how a frame may look, but the physical orientation of the camera and the geometry of the scene usually remain implicit.

This becomes a serious limitation when the camera performs large rotations, follows a long trajectory, or starts from a single uncalibrated image. Relative camera controls can describe motion between views, but they do not define which way is up in the real world. As a result, the same relative trajectory may correspond to multiple absolute orientations. The horizon can drift, scene uprightness can change, and generated observations may no longer share a persistent physical frame.

Puffin-World addresses three coupled questions:

1. How can one camera representation support both absolute physical grounding and continuous cross-view motion?
2. How can the physical knowledge inferred from an observed view remain consistent at unseen viewpoints?
3. How can these capabilities be scaled with data that contains accurate camera labels and genuinely challenging motion?

Our answer is to model physics, geometry, and appearance natively, connect them within one model, and scale the entire framework with our constructed Puffin-16M and the large-scale public datasets.

Three native 3D world states

The physical world is organized at multiple interconnected levels. Puffin-World makes three of them explicit:

Physics World State

The physics state describes how an observation is oriented with respect to the real world. In the current model, it contains a gravity field and a latitude map. These dense quantities encode scene uprightness, the horizon, and each viewing ray's elevation relative to the horizontal plane.

Geometry World State

The geometry state is represented by depth. It exposes the scene structure underlying the visible appearance and provides a direct bridge from generated multi-view observations to 3D reconstruction.

Appearance World State

The appearance state contains RGB images and sequences. It captures visual content while being generated jointly with camera and geometry conditions rather than in isolation.

Starting from a single image or a text prompt, Puffin-World generates camera-controlled multi-view appearances together with geometry. The resulting views remain coherent across challenging trajectories and can be consolidated directly into a 3D reconstruction.

Omni-Camera: one representation for physical grounding and flexible camera motion

Camera representations used in vision typically fall into two groups. Relative representations are effective for multi-view geometry and continuous motion, but lack a global physical anchor. Absolute representations capture gravity, uprightness, and horizon cues, but are difficult to obtain from monocular images and less convenient for cross-view transitions.

Omni-Camera combines both. At every pixel x, the camera condition concatenates a three-channel absolute field with a six-channel relative ray field:

$${\mathbf{c}}_{\mathbf{x}}=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}({\mathbf{a}}_{\mathbf{x}},{\mathbf{p}}_{\mathbf{x}})\in {\mathbb{R}}^9\mathrm{.}\backslash mathbf\{c\}\_\{\backslash mathbf\{x\}\}\; =\; \backslash mathrm\{Concat\}(\backslash mathbf\{a\}\_\{\backslash mathbf\{x\}\},\backslash mathbf\{p\}\_\{\backslash mathbf\{x\}\})\; \backslash in\; \backslash mathbb\{R\}^\{9\}.$$

The absolute component a(x) contains a two-dimensional up vector and a latitude angle. The relative component p(x) contains the ray origin and direction. Together they encode both the physical frame of the observation and its position within a continuous multi-view trajectory.

This single dense condition supports:

• monocular camera-to-world understanding;
• camera-controllable text-to-image generation;
• image-conditioned multi-view generation;
• in-place roll, pitch, and yaw rotation;
• camera translation and compound motion;
• joint appearance and geometry generation.

One framework, three levels of unification

Puffin-World is unified at three levels.

• Representation unification. RGB images and depth maps share the same VAE latent space, while Omni-Camera describes absolute cameras, relative motion, rotation, and translation. A compact role mask identifies reference views, target views, image conditions, and geometry views.

• Modality unification. A geometry-aligned vision encoder and a large language model yield autoregressive understanding outputs. The same language-model hidden states are transformed through learnable queries and a lightweight connector to condition the diffusion generator. Understanding and generation are complementary outputs of one model rather than separate pipelines.

• Task unification. A task is determined by its available inputs—text, target cameras, reference images, and role masks. Changing this composition lets the same framework perform physical-world perception, free-viewpoint image generation, 3D world generation, and joint appearance-geometry reconstruction.

Perceiving and propagating physics

Given a single image, Puffin-World estimates roll, pitch, vertical field-of-view, and a semantic scene description. We formulate camera-to-world understanding as autoregressive multimodal sequence modeling: the model first reasons about holistic cues such as the horizon, vertical structures, foreground composition, and camera height, then predicts numerical absolute camera parameters.

Perception alone is not enough for long trajectories. The absolute physical state inferred from the reference image must also remain valid after the camera moves. Physics propagation transfers the reference gravity direction through the known relative rotation at every future view. The propagated gravity is converted into a target-view gravity field and latitude map, forming an absolute physical condition for generation.

Physics propagation is defined as

$$\mathcal{G}={\left\{{\mathbf{g}}_t\right\}}_{t=0}^{T-1}={\left\{{\mathbf{R}}_{t\leftarrow 0}^{\mathrm{r}\mathrm{e}\mathrm{l}}{\mathbf{g}}_0\right\}}_{t=0}^{T-1}\mathrm{.}\backslash mathcal\{G\}\; =\; \backslash left\backslash \{\backslash mathbf\{g\}\_t\backslash right\backslash \}\_\{t=0\}^\{T-1\}\; =\; \backslash left\backslash \{\backslash mathbf\{R\}^\{\backslash mathrm\{rel\}\}\_\{t\backslash leftarrow0\}\backslash mathbf\{g\}\_0\backslash right\backslash \}\_\{t=0\}^\{T-1\}.$$

Here, g₀ is the gravity direction perceived from the reference view, while Rᵣₑₗ(t←0) rotates a vector from the reference camera frame into the coordinate frame of view t. The resulting gₜ gives each target view an absolute gravity direction, keeping the entire generated trajectory anchored to one coherent physical world frame.

This distinction matters. Relative motion tells the model how the camera moved. Physics propagation also tells it how the new camera is oriented with respect to the real world.

Scaling with Puffin-16M

Existing camera and 3D datasets are often biased toward level cameras, modest pitch changes, and translation-dominant capture. That distribution is not sufficient for an agent expected to explore from arbitrary viewpoints.

Puffin-16M contains two complementary subsets:

• Puffin-Cam-15M: 15 million vision-language-camera triplets rendered from 900,000 source panoramas. It covers diverse aspect ratios, roll and pitch from −45° to 45°, and vertical FoV from 20° to 105°. Captions include scene semantics and structured spatial and physical reasoning.
• Puffin-Traj-1M: one million trajectories with continuous look-up/look-down motion, clockwise and counterclockwise rotation, and full 360° exploration. Roll and pitch span −45° to 45°, while yaw spans 0° to 360°.

Beyond Puffin-16M, we use Puffin-World to annotate 28 widely used public datasets with absolute roll, pitch, and vertical FoV. The release covers approximately 44.5 million images across recognition, detection, segmentation, pose estimation, generative pretraining, and sequential 3D data. These annotations make it possible to analyze viewpoint bias, curate camera-diverse training sets, and add gravity-aware physical grounding to existing corpora.

Explore Puffin-16M · Browse all camera-annotated datasets

Results

Physical-world perception from one image

Across four benchmarks, Puffin-World consistently delivers the strongest absolute camera-parameter estimation. This shows that scaling unified multimodal training turns single-image camera-to-world understanding into a reliable physical perception capability.

Given one image, the model can output dense perspective fields that reveal gravity and latitude throughout the scene, providing an absolute camera-to-world reference.

Free-viewpoint spatial simulation

Puffin-World combines a text prompt with an explicit camera specification to simulate the requested viewpoint. It provides precise physical camera control while maintaining strong visual quality across diverse scenes and camera configurations.

Across large viewpoint changes, the generated results follow the requested camera motion while preserving scene identity and visual coherence.

3D world generation and reconstruction

Puffin-World supports image-to-3D and text-to-3D generation, long trajectories, extreme rotations, compound motion, native world-state prediction, and direct 3D reconstruction. Joint appearance and geometry generation produces coherent views while remaining aligned with challenging camera trajectories.

The examples visualize aligned appearance, geometry, and physics states together with their reconstructed 3D scenes.

Multi-task synergy: closing the loop

The value of unification becomes especially clear when perception, reasoning, generation, and reconstruction must interact.

Mimic world exploration expands different 3D worlds from a shared initial viewpoint while following the same prescribed camera trajectory. This couples camera-to-world understanding with consistent multi-view generation.

Self-calibrated world exploration starts from a gravity-misaligned observation. Puffin-World reasons about its current physical state, predicts a corrective camera action, and imagines the resulting target observation. The same model therefore forms an interleaved perception-action-generation loop without an auxiliary calibrator or generator.

These early closed-loop behaviors point toward applications in interactive 3D environments, virtual reality, embodied intelligence, and World-Action Models.

Open resources

We are releasing the full Puffin ecosystem to support research on multimodal spatial intelligence and physical AI:

• Puffin-World project page
• Puffin-World paper
• Code repository
• Puffin-World models
• Puffin-16M dataset
• Puffin-World Hugging Face collection
• Interactive and supplementary results

Puffin-World currently focuses on static scenes and models physics primarily through gravity and latitude. Extending it to dynamic environments, longer horizons, richer interactions, and broader physical states is an important next step. We hope that grounding multimodal models in a shared physical and geometric representation will provide a practical path from models that generate images to agents that can perceive, build, and interact with worlds.

Citation

If Puffin-World, Puffin-16M, or the Puffin model series supports your research, please consider citing our work:

@article{liao2026puffinworld,
  title   = {Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States},
  author  = {Liao, Kang and Luo, Yihang and Wu, Xiao-Ming and Jin, Linyi and Wu, Size and Lin, Chunyu and Zhao, Yao and Wang, Fei and Li, Wei and Loy, Chen Change},
  journal = {arXiv preprint arXiv:2609.04196},
  year    = {2026}
}