Immersive NeRF-based teleoperation with robots for inspection and maintenance

State-of-the-art mobile manipulating robots can perform maintenance tasks like fuse replacements and interaction with buttons and switches. In such safety-critical electrical maintenance work, robot autonomy is undesirable. Instead, a robot should be supervised by a human teleoperator. Currently, available solutions for visualizing the robot's surroundings for the teleoperator, such as live camera streams or LIDAR maps, are insufficient in providing the necessary situational awareness required for remote robot operation. To enhance teleoperation capabilities, one practical approach is to use the Neural Radiance Field (NeRF) for scene representation. NeRF is a technique that employs neural networks to model complex 3D environments using a collection of 2D images. It can generate high-quality novel views of the scene. By integrating NeRF derived from real-time sensor data into the teleoperation system, the objective is to augment operator situational awareness, control precision, and overall robot performance, including collision detection in the motion planner.

This research proposal focuses on the use of NeRFs in real-time robot teleoperation for maintenance operations, e.g. switchboard cabinets in the Gotthard Base Tunnel. This research aims to:

1. Develop a real-time NeRF-based framework that accurately represents the robots surrounding 3D. This framework will utilize live sensor data to create a detailed scene.
2. Create a teleoperation system that seamlessly integrates NeRF-rendered scenes with real-time robot control to provide an immersive and responsive experience for the operator.
3. Explore the potential of NeRFs for collision checking with the environment or objects in the robot's motion planner to enhance safety and operational efficiency. 

The proposed research will be evaluated through comprehensive user studies, comparing the performance and effectiveness of the proposed system against conventional teleoperation methods. We strongly believe that tunnel maintenance will greatly benefit from real-time NeRF-based teleoperation. Furthermore, the research findings can be extended to other domains that require immersive teleoperation and advanced robot capabilities. Some of these domains include industrial inspection and maintenance, space exploration, disaster response, and many others.

Partner

• SBB

Roadmap

10.2024 - 09.2027 (48 months)