Summary: The EgoExplorer dataset combines first-person video, full-body motion, fine-grained hand movements, and tactile signals across 100 hours of human demonstrations, giving researchers new data to train humanoid robots for complex real-world tasks. Developed by a joint team from Shanghai AI Laboratory, Zhejiang University, and Shanghai Jiao Tong University, the dataset also demonstrates motion transfer to a simulated Unitree G1 humanoid robot.
Humanoid robots are making progress in walking, balancing, and object manipulation, but learning to perform complex tasks like humans remains a major challenge. To work effectively in homes, factories, and other real-world environments, robots must coordinate whole-body movement, hand dexterity, and physical contact with objects. A research team from Shanghai AI Laboratory, Zhejiang University, and Shanghai Jiao Tong University has introduced EgoExplorer, a multimodal human demonstration dataset designed to help bridge this gap.
Overview of EgoExplorer.
The dataset contains 100 hours of human activity recordings captured through a lightweight wearable system. A head-mounted GoPro camera records first-person video, while five motion trackers and sensor-equipped gloves capture body posture, finger movements, and tactile signals. The recordings also include step-by-step textual descriptions, linking human actions to their visual, physical, and semantic context.
Unlike datasets that capture only video or motion, EgoExplorer aligns multiple data streams from the same demonstrations. This provides researchers with a more comprehensive foundation for training humanoid robots to understand how people move, manipulate objects, and respond to physical contact during practical tasks.
A glimpse of EgoExplorer. It shows human loco-manipulation across workstations, with reconstructed body motion overlaid on the scene point cloud. Each inset is linked to a corresponding body pose along with synchronized egocentric view, finger articulation, and tactile observation.
To reconstruct complete human motion from a limited number of sensors, the researchers developed a three-stage pipeline called Calibrate-Complete-Constrain. It first calibrates human body parameters and sensor placement, then estimates unobserved joint movements using prior knowledge of human motion, and finally constrains the reconstructed trajectories to maintain smoothness and anatomical plausibility. Compared with the strongest baseline evaluated in the study, the method reduced human joint-position error by 71.9%, while improving hand-motion stability and object-contact estimation.
The team also evaluated the dataset through multimodal generation and robot motion retargeting experiments. Jointly generating video, motion, and tactile signals produced better results than generating individual modalities separately. In simulation, human demonstrations were successfully transferred to a Unitree G1 humanoid robot, which reproduced the demonstrated movements stably without falling.
However, these results remain an early research milestone. The robot demonstrations were tested in simulation rather than on a physical robot, and the current system does not yet establish precise real-world object interaction or reliable generalization to unfamiliar tasks.
EgoExplorer addresses a key bottleneck in humanoid robotics: the shortage of synchronized, multimodal human demonstrations that capture both whole-body behavior and detailed hand-object interactions. By combining first-person vision, full-body motion, finger-level movement, tactile sensing, and textual task descriptions, the dataset could support future research into imitation learning, robot manipulation, and embodied AI.
As researchers work toward more capable general-purpose humanoid robots, datasets such as EgoExplorer may help machines learn not only how humans move, but also how coordinated movements and physical interactions enable people to complete everyday and professional tasks.
