Immersive teleoperation relies heavily on the data received from remote manipulators for accurate control and aided decision making process. Current eXtended Reality (XR) technologies are using images, videos, and 3D models inside fully virtual environment, or to augment a real-world location. Moreover, usage of heavily processed data such as photogrammetric reconstruction, point clouds, or Digital Twin models increases immersivity and provide important link between XR environment and a real-world places.
Bringing lightfield data into XR environment will further raise immersion, and provide more precise view into remote area. Lightfield is a digital representation of all the light rays passing through a certain volume. Those rays can be computed from the images captured by robotically controlled camera. By having a sufficient image dataset, it is possible to compute new perspective views for every point in the said volume. Lightfield data viewed in XR environment can be compared to an immersive 360° or 180° photo of a real world, but with benefit of position tracking, correct parallax, reflections and shadows. Example: https://youtu.be/OUU2yGHgPQY.
We propose to build and evaluate a Proof-of-Concept (PoC) system for automatic image acquisition with robotically controlled camera, processing those images into lightfield data, and immersive teleoperation environment which can utilize lightfield data for viewing, analysis, and annotation.
Example use case: operator starts image data acquisition at the Point Of Interest (POI) with a robotic arm (on ISS or planetary rover); after the lightfield is created as result of processing it is sent to the XR environments at both ground control and ISS. Ground control can analyze lightfield data and create annotated instructions for ISS personnel, which in turn can control robotic arm to perform actions at POI. Project will evaluate which set of medias are giving most suitable situation awareness for the teleoperation in XR.
