Summary: Researchers led by J. Wang have introduced 3D World Simulator, an action-conditioned multimodal world model that combines particle and voxel representations to predict 3D scene geometry, multi-view images, and contact-based tactile signals within a unified spatial framework. Operating at up to 50 Hz, the system supports novel-view synthesis, wrist-mounted camera rendering, tactile simulation, robot imitation learning, simulation-based policy evaluation, and model-predictive planning with previously unseen objects.

Figure 1: Overview of 3D World Simulator. 

Robotic world models are becoming a critical foundation for intelligent manipulation. Most existing action-conditioned world models follow a 2D video-generation approach. Although they can produce visually realistic sequences, they lack explicit 3D scene representations. This limitation often leads to spatial inconsistencies across viewpoints during long-horizon interactions, camera movements, and object occlusions. Meanwhile, tactile signals are typically predicted independently, making it difficult to achieve accurate spatial alignment between touch information and object contact geometry. These challenges limit the practical deployment of vision-tactile fusion in robotic manipulation.

To address these problems, the research team developed 3D World Simulator, a multimodal simulation framework designed specifically for robotic interaction. Its central innovation is the use of a unified 3D spatial representation to jointly model visual and tactile information, enabling consistent predictions of how objects look, move, and respond to physical contact.

Figure 2: Overview of Model Architecture. 

The framework consists of two interconnected components: a hybrid particle-voxel autoencoder and an action-conditioned 3D latent dynamics model. The autoencoder takes point clouds that integrate visual and tactile observations as input. Sparse 3D convolutions first compress the scene into compact voxel-based latent representations. Its decoder then applies a coarse-to-fine upsampling strategy to progressively restore voxel resolution before reconstructing dense 3D particles. Each visual particle carries 3D Gaussian parameters, including color, scale, rotation, and opacity, enabling rendering from arbitrary viewpoints. Tactile observations are represented within the same 3D coordinate system, naturally aligning touch signals with contact surfaces and reducing inconsistencies between modalities.

The dynamics model is built on a 3D U-Net architecture incorporating spatial attention and causal temporal attention. It takes historical latent states, the current scene representation, and robot actions as inputs to autoregressively predict the next voxel-based latent state. To prevent prediction errors from accumulating during open-loop simulation, the researchers introduced multi-step autoregressive supervision and a curriculum learning strategy that gradually increases prediction horizons. Gaussian noise is also added to historical latent variables to improve long-term stability.

Training was conducted on a single NVIDIA RTX 5090 GPU. The autoencoder and dynamics model each required approximately 16 hours of training. During inference, the system achieved stable interactive multimodal prediction at up to 50 Hz, supporting long-horizon robotic simulation.

Figure 3: Qualitative comparison of open-loop multi-view predictions.

The researchers evaluated the model using an ALOHA dual-arm robotic platform equipped with FlexiTac fingertip tactile sensors. Experiments covered six manipulation tasks: component assembly, cloth folding, T-shaped object pushing, box packing, rope threading, and marker insertion. These tasks included rigid objects, articulated structures, and highly deformable materials such as fabric.

Across an average evaluation involving 200-frame open-loop predictions, 3D World Simulator outperformed both 2D video-based baselines, including Cosmos, Robot4DGen, and IWS, and 3D particle-dynamics approaches such as PGND, PointWorld, and ParticleFormer. Improvements were observed across all evaluated metrics, including image quality, depth accuracy, point-cloud geometry, and tactile L1 error.

The researchers found that 2D video-based models frequently produced inconsistencies across viewpoints and distorted object structures. Conventional 3D particle-based methods, meanwhile, rely on continuous point-tracking supervision. When objects become occluded, their geometric reconstruction quality can deteriorate rapidly over time.

To evaluate scalability, the team further tested the model on Deform360, a large-scale vision-tactile dataset containing 198 everyday object categories and 1,980 interaction sequences, with more than 215 hours of multi-camera, dual-arm robotic interaction data.

For objects encountered during training, the model achieved better overall geometric reconstruction and rendering performance than particle-based baselines. For previously unseen objects, it maintained an advantage in geometric prediction, although its image-rendering metrics were slightly lower. The authors attributed this difference to the baselines' use of previously fitted Gaussian trajectories, whereas their approach generates visual appearance directly from latent states without relying on pre-fitted trajectories.

Figure 4: Correlation between simulation and real-world policy performance. 

The researchers also demonstrated three practical applications for robotic learning and deployment.

The first is synthetic training data generation. The team evaluated four representative imitation-learning algorithms: ACT, Diffusion Policy, π0.5, and Cosmos Policy. Policies trained exclusively on vision-tactile, multi-view, and novel-view data generated by 3D World Simulator achieved success rates close to those trained on real-world collected data. In the component assembly task, a policy using simulated tactile inputs even outperformed one relying exclusively on real visual observations.

The second application is simulation-based robot policy evaluation. Each policy was tested 20 times in both the simulation environment and the real world. The results showed a strong positive correlation between simulated success rates and actual robotic performance. This suggests that 3D World Simulator could serve as a cost-effective evaluation tool, allowing researchers to screen and refine manipulation policies before conducting expensive real-robot experiments.

According to the project page, the video plays at 20x speed, showing how a strategy trained using only data generated from a world model runs on a real robot.

The third application is zero-shot model-based planning. Using a model pretrained on Deform360 and combined with Model Predictive Path Integral (MPPI) control, the researchers demonstrated interaction planning on a previously unseen single-arm robot. The system successfully handled four categories of unfamiliar objects: fabric, ropes, foam, and dolls. It autonomously optimized action sequences to guide these objects toward desired target configurations without task-specific retraining.

Despite these capabilities, the researchers clearly identified several limitations. The learned dynamics currently apply only within a fixed workspace. Scene backgrounds and robot bodies must be integrated separately using external 3D Gaussian assets and kinematic models. During Deform360 training, the model reconstructs objects themselves rather than the robot or surrounding environment. Rendering from wrist-mounted camera viewpoints also requires a separately fine-tuned prediction head.

Importantly, 3D World Simulator is not a physics engine that strictly enforces physical equations. It cannot generalize without limitations to arbitrary rooms, robots, or objects.

Compared with conventional 2D video-based world models, the framework offers a more natural solution to multi-view consistency. Compared with traditional particle-based dynamics models, it eliminates the need for continuous particle correspondence tracking, making it more adaptable to diverse object-interaction scenarios.

Its key contribution is a shift in robotic world modeling from simply generating video sequences to predicting a queryable, renderable 3D interaction environment that incorporates physical contact and tactile information. This approach could open new possibilities for robot imitation learning, vision-tactile manipulation, synthetic data generation, and model-predictive control. The research team plans to expand the framework to larger environments, including multi-room settings and mobile manipulation scenarios.