Collaborating on physical objects remotely has always been a challenging task, but a new system is set to change that narrative. SharedNeRF, a remote conferencing system developed by Mose Sakashita, a doctoral student in the field of information science, is breaking new ground. This cutting-edge system allows users to manipulate a 3D view of a scene, enabling them to engage in complex tasks such as debugging intricate hardware, even when not physically present in the same location.
SharedNeRF is a game-changer as it combines two graphics rendering techniques to provide users with a dynamic and interactive experience. One technique is slow but delivers photorealistic results, while the other is faster but offers less precision. The system aims to bridge the gap between the physical space of collaborators, allowing remote users to engage in tasks that were previously challenging to convey through traditional video-based systems with static viewpoints.
A Paradigm Shift in Remote Collaboration
Mose Sakashita worked on developing SharedNeRF as an intern at Microsoft, in collaboration with Andrew Wilson, a former computer science major at Cornell. The system, set to be presented at the ACM CHI conference on Human Factors in Computing Systems, promises to revolutionize the way people collaborate remotely. Wilson highlights the clunkiness of current video conferencing systems when it comes to tasks involving physical objects and praises the innovative use of computer graphics and rendering techniques in SharedNeRF.
Sakashita’s work in the lab of François Guimbretière focuses on developing new technology to support remote collaboration. SharedNeRF leverages a graphics rendering method known as neural radiance field (NeRF) to construct a 3D representation of a scene using 2D images. This method results in incredibly realistic depictions, complete with reflections, transparent objects, and accurate textures, providing users with a comprehensive view from any angle.
The SharedNeRF system utilizes a head-mounted camera worn by the local collaborator to record the scene, which is then fed into a NeRF deep learning model to render the 3D scene for the remote user. This approach allows the remote collaborator to rotate the viewpoint as desired, albeit with a slight delay in updating the scene. To mitigate this delay, Sakashita’s team combined the detailed visuals of NeRF with point cloud rendering, a faster technology that captures real-time movements in the scene, such as moving hands.
Seven volunteers tested SharedNeRF by engaging in a collaborative flower-arranging project with a partner. The results were overwhelmingly positive, with five volunteers preferring SharedNeRF over traditional video conferencing tools or point cloud rendering alone. Users appreciated the ability to independently change viewpoints, zoom in and out on details, and avoid the need to verbally direct the local collaborator on their desired perspective.
Future Prospects and Expansion
Although currently designed for one-on-one collaboration, the researchers behind SharedNeRF envision extending the system to accommodate multiple users. Future work will focus on enhancing image quality and exploring immersive experiences through virtual reality or augmented reality techniques. SharedNeRF represents a significant step forward in remote collaboration, offering users a more dynamic, interactive, and engaging platform for collaborative tasks.
Leave a Reply