Introduction

A robot needs a map that keeps up with changes in its surroundings and corrects mistakes from earlier observations. Objects may move after they are first mapped, leaving their stored locations out of date. Even when nothing moves, imperfect perception can introduce false objects or inaccurate geometry. If these errors persist, the robot may search in the wrong place or draw incorrect conclusions about the scene. Keeping the map reliable therefore requires continually revisiting past observations as new evidence becomes available.

Our approach centers on three requirements: continuously adapting to environmental changes, progressively correcting perception errors, and serving as online spatial memory for embodied agents executing vision-language navigation (VLN) tasks, including object navigation (ObjNav) and embodied question answering (EQA).

Method

Our framework builds an object-centric semantic map from a stream of posed RGB-D observations. For each frame, open-vocabulary detections and segmentation masks identify object regions, which are lifted into 3D and associated with existing map instances using semantic similarity and geometric overlap. Unmatched detections initialize new instances. Each instance stores its geometry, semantic features, and a confidence score for every point, allowing later observations to revise what the map has already recorded.

Before updating these scores, the system checks how existing points relate to the current camera view. Points supported by the observed surface and the associated segmentation mask receive positive evidence. Visible points outside the mask, or points contradicted by the depth measurement, lose confidence. Occluded points remain unchanged, while an unmatched instance is penalized only when it is sufficiently visible to be detectable. Evidence accumulates through log-odds updates, and points with low confidence are pruned. This process progressively corrects perception artifacts and removes outdated geometry without discarding valid objects simply because they are temporarily hidden.

As geometry changes, the map must also preserve the identity of each physical object. Periodic fragment merging combines partial instances when their geometry and visual features agree. BoT-SORT associates detections of objects moving within the camera’s view, keeping their map identities consistent as their positions change. For objects relocated outside the view, visual re-identification matches newly observed instances to stored records of removed objects. Together, these mechanisms keep object identities consistent while point-level refinement updates their geometry.

Results

1. Continuously Adapting to Environmental Changes

We evaluate adaptation under two kinds of object motion. In the relocation experiment (left), objects are moved outside the camera’s view and observed again at new locations. The mug, orange, and carton examples show how the map removes outdated geometry and re-identifies the relocated objects, preserving their identities across the change.

In the in-view tracking experiment (right), a red bowl moves from a chair to a table while remaining visible. The image sequence and map snapshots below show the same object instance following the bowl to its new position, with stale points at the previous location removed.

2. Progressively Correcting Perception Errors

We examine whether additional observations can correct errors introduced earlier in mapping. In the chair experiment (left), the camera observes a wooden chair from multiple viewpoints. Imperfect segmentation masks fill gaps between its thin parts and introduce spurious geometry. The comparison below shows DualMap retaining these artifacts, while our confidence updates progressively suppress inconsistent points and recover the chair’s open structure.

The false-detection experiment (right) tests a different error: a plastic bag is initially recognized as a pillow, creating an incorrect instance. As the camera gathers more observations, the false instance accumulates negative evidence and is removed. The actual pillow remains in the map, including when it temporarily leaves the camera’s view.

3. Online Spatial Memory for ObjNav and EQA

We integrate the mapping system into VL-Nav and deploy it on Spot to evaluate online spatial memory for language-guided tasks in hallway, office, and outdoor environments. The agent queries the map for objects, locations, and spatial context to support both ObjNav and EQA. When the available information is insufficient, it explores further, using new observations to update the same memory that guides subsequent actions and answers.

For ObjNav, the agent retrieves matching objects and their locations to guide navigation, as illustrated by the instructions to find a black helmet and a trash can below. For EQA, it gathers relevant object and spatial information before answering questions, such as how many robots are in the lab and whether a rocket model is present. These experiments demonstrate how a shared, continuously updated map supports both locating objects and reasoning about the scene.

Publications

1. 
   Point-Level Refinable Semantic Mapping for Spatial Understanding.

   Taimeng Fu, Yi Du, Zitong Zhan, Qiwei Du, Shaoshu Su, Zhipeng Zhao, Chen Wang.

   arXiv preprint arXiv:2610.31476, 2026.

   BibTex Cite Continuously refinable semantic maps as spatial memory for embodied agents

   @article{fu2026point,
     title = {Point-Level Refinable Semantic Mapping for Spatial Understanding},
     author = {Fu, Taimeng and Du, Yi and Zhan, Zitong and Du, Qiwei and Su, Shaoshu and Zhao, Zhipeng and Wang, Chen},
     journal = {arXiv preprint arXiv:2610.31476},
     year = {2026},
     url = {https://arxiv.org/abs/2610.31476},
     website = {https://sairlab.org/point/},
     cover = {/img/posts/2026-10-31-point/point_cover.gif},
     addendum = {Continuously refinable semantic maps as spatial memory for embodied agents}
   }
   

   Fu, Taimeng and Du, Yi and Zhan, Zitong and Du, Qiwei and Su, Shaoshu and Zhao, Zhipeng and Wang, Chen, "Point-Level Refinable Semantic Mapping for Spatial Understanding," arXiv preprint arXiv:2610.31476, 2026.