Introduction: Researchers at the Massachusetts Institute of Technology (MIT) have developed UQ-DAAAM, a robotic memory system that quantifies uncertainty in object descriptions and proactively gathers additional visual information to correct labeling errors. The technology aims to prevent potentially life-threatening mistakes, such as misidentifying a defibrillator as a fire extinguisher.

 

---

In an emergency, a robot that mistakes a wall-mounted defibrillator for a fire extinguisher could fail to locate a life-saving medical device when it is needed most. To address this problem, researchers at MIT have developed a new memory system called UQ-DAAAM, designed to help robots identify potentially inaccurate records in their memory and automatically correct object descriptions.

 

Credit: MIT News

 

UQ-DAAAM stands for "Uncertainty Quantification for Describe Anything, Anywhere, at Any Moment." The system works by comparing multiple descriptions of the same object generated from different viewing angles to quantify uncertainty in the robot's observations. When uncertainty exceeds a predefined threshold, it selects additional viewpoints that are most likely to provide useful visual information. It then combines descriptions from multiple observations to generate a more accurate representation of the object and update the robot's memory.

 

The system builds on the team's earlier DAAAM framework, which uses vision-language models to generate descriptions of objects captured by a robot's camera and stores text summaries in a 3D map-based representation of the surrounding environment. However, when a robot captures only part of an object, observes it from a distance, or encounters blurry images or poor lighting, DAAAM can generate incorrect descriptions. For example, it might mistake a gray chimney for a squirrel on a rooftop. Such errors can then persist in the robot's memory.

 

UQ-DAAAM addresses this limitation by introducing an uncertainty quantification mechanism. It first estimates the confidence of descriptions generated by the vision-language model, then compares descriptions across different viewpoints to evaluate their consistency. These measurements are combined into an overall reliability score.

 

The system uses heuristic rules to prioritize images that are most likely to reduce uncertainty, such as selecting a frame showing smoke emerging from a chimney. Mathematical analysis demonstrates that this selection strategy can identify the most informative image without repeatedly running the vision-language model, thereby reducing computational overhead.

 

After selecting additional visual evidence, the system uses a large language model (LLM) to combine multiple descriptions, assigning greater weight to clearer and less ambiguous observations. Tests showed that the approach resolved identification ambiguity for approximately 92% of objects in the environment, requiring an average of only two additional descriptions per object.

 

Compared with the best existing approach, UQ-DAAAM improved object-description accuracy by 5%. It also outperformed comparison systems in question-answering tasks involving objects, their locations, and the times they were last observed.

 

 

Luca Carlone, an associate professor in MIT's Department of Aeronautics and Astronautics and the project's lead researcher, said that AI systems operating in the physical world cannot rely solely on disclaimers warning users about potential errors. Robots must be capable of "knowing what they do not know," a critical requirement for the safe deployment of Physical AI.

 

The paper's co-first authors are MIT graduate students Harry Zhang and Nicolas Gorlo. The research will be presented at the Conference on Neural Information Processing Systems (NeurIPS).

 

The team plans to deploy UQ-DAAAM on a wider range of mobile manipulation robots and expand the application of uncertainty quantification. Future work will also explore enabling robots to revisit areas with high uncertainty, improve their understanding of surrounding environments, and predict how those environments may change over time.