Simultaneous localization and mapping (SLAM) is a fundamental task for numerous applications such as autonomous navigation and exploration. Despite many SLAM datasets have been released, current SLAM solutions still struggle to have sustained and resilient performance. One major issue is the absence of high-quality datasets including diverse all-weather conditions and a reliable metric for assessing robustness. This limitation significantly restricts the scalability and generalizability of SLAM technologies, impacting their development, validation, and deployment.

To bridge this gap and push SLAM towards all-weather environments, we present an extremely challenging dataset, SubT-MRS, including scenarios featuring various sensor degradation, aggressive locomotions, and extreme-weather conditions. The SubT-MRS dataset comprises 3 years of data from the DARPA Subterranean (SubT) Challenge (2019-2021) and extends with an additional 2 years of diverse environments (2022-2023), containing mixed indoor and outdoor settings, including long corridors, off-road scenario, tunnels, caves, deserts, forests, and bushlands.

Cumulatively, this forms a 5-year dataset encompassing over 2000 hours and 300 kilometers of terrain subjected to multimodal sensors including LiDAR, fisheye cameras, depth cameras, thermal cameras, and IMU; heterogeneous platforms including RC cars, legged robots, aerial robots, and wheeled robots; and extreme obscurant conditions such as dense fog, dust, smoke, and heavy snow.

We develop accuracy and robustness evaluation tracks for SLAM and introduced novel robustness metrics. Comprehensive studies are performed, revealing new observations, challenges, and opportunities for future research.

All-weather Environments

• Spanning 5 years, SubT-MRS comprises over 2000 hours and 300 kilometers of accurately measured trajectories. SubT-MRS is the first real-world dataset that specifically addresses failure scenarios of SLAM by incorporating a variety of degraded conditions, multiple robotic platforms, and diverse sets of multimodal sensors.

High-precision Ground Truth Maps

To evaluate the actual SLAM performance running on robots, we propose a robustness metric, which, to the best of our knowledge, is the first metric that can evaluate the reliability, safety, and resilience of SLAM systems.

Publication

1. SubT-MRS Dataset: Pushing SLAM Towards All-weather Environments.

   Shibo Zhao, Yuanjun Gao, Tianhao Wu, Damanpreet Singh, Rushan Jiang, Haoxiang Sun, Mansi Sarawata, Warren C Whittaker, Ian Higgins, Shaoshu Su, Yi Du, Can Xu, John Keller, Jay Karhade, Lucas Nogueira, Sourojit Saha, Yuheng Qiu, Ji Zhang, Wenshan Wang, Chen Wang, Sebastian Scherer.

   IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
   BibTex Cite

   @inproceedings{zhao2024subt,
     title = {{SubT-MRS} Dataset: Pushing SLAM Towards All-weather Environments},
     author = {Zhao, Shibo and Gao, Yuanjun and Wu, Tianhao and Singh, Damanpreet and Jiang, Rushan and Sun, Haoxiang and Sarawata, Mansi and Whittaker, Warren C and Higgins, Ian and Su, Shaoshu and Du, Yi and Xu, Can and Keller, John and Karhade, Jay and Nogueira, Lucas and Saha, Sourojit and Qiu, Yuheng and Zhang, Ji and Wang, Wenshan and Wang, Chen and Scherer, Sebastian},
     booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
     year = {2024},
     url = {https://arxiv.org/abs/2307.07607},
     video = {https://youtu.be/mkN72Lv8S7A},
     website = {https://sairlab.org/datasets/subtmrs}
   }
   

   Zhao, Shibo and Gao, Yuanjun and Wu, Tianhao and Singh, Damanpreet and Jiang, Rushan and Sun, Haoxiang and Sarawata, Mansi and Whittaker, Warren C and Higgins, Ian and Su, Shaoshu and Du, Yi and Xu, Can and Keller, John and Karhade, Jay and Nogueira, Lucas and Saha, Sourojit and Qiu, Yuheng and Zhang, Ji and Wang, Wenshan and Wang, Chen and Scherer, Sebastian, "SubT-MRS Dataset: Pushing SLAM Towards All-weather Environments," IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.