CDE Research Engineer Catherine Taylor tells us about her research, creating exciting new VR experiences with partner company experimental art collective Marshmallow Laser Feast.
My name is Catherine Taylor and I am an Engineering Doctorate (EngD) student with the EPSRC-funded Centre for Digital Entertainment at the University of Bath. I’m carrying out my research with Prof Darren Cosker, Director of CAMERA at the University of Bath (Centre for the Analysis of Motion, Entertainment Research and Applications) who applies visual computing and motion capture expertise to a variety of real-world problems. I collaborated with the team at CAMERA to create data capture opportunities to further my research.
My research is in collaboration with experiential art collective Marshmallow Laser Feast who specialise in interactive location-based experiences where participants can explore exciting virtual worlds using technology such as VR, AR, motion capture, hand tracking, breath sensing and heart rate tracking. At Marshmallow Laser Feast I have been supervised by one of the company co-founders, Robin McNicholas, and executive producer, Nell Whitley. Their recent work includes ‘Dream’ a collaboration with the Royal Shakespeare Company, Manchester International Festival and Philharmonia Orchestra where audiences encountered a live, interactive, online performance based on A Midsummer Night’s Dream.
My research looks at non-rigid object tracking for VR/AR. The way most of us interact in VR is by using some kind of controller, which is typically cold and hard. We are interested in creating exciting new experiences with novel, tactile interactions using soft, squishy objects, like for example a spongy ball. If you squeezed the ball with your hand the object you were squeezing in VR would look like it was squashing. This is not possible at the moment. To do this, you need to track the behaviour of a real-world object and then use the tracked behaviour to animate a CG model of the object.
We have experimented with more traditional computer vision approaches, hardware solutions and deep learning. I have designed a pipeline for taking everyday real-world objects and turning them into interactive VR props using a deep neural network (DNN) based tracking approach. With Marshmallow Laser Feast, I’ve had access to state-of-the-art technology, such as the Vicon Origin mocap system and the latest VR headsets. I have also been working with a team focused on producing user-ready content so I’ve got to experience the fast pace required to produce their VR experiences. I’ve also been part of the Research and Development team on several Innovate UK projects which have brought many valuable research findings to the team. It’s been great to be part of a growing team at Marshmallow Laser Feast and contribute towards an exciting project.
We have found out the effectiveness of interacting with non-rigid objects in VR/AR. This allows novel, immersive interactions which are much more engaging than traditional rigid ones. We have also found out participants tend to not notice if there is a slight disconnect between the real and virtual object when in VR. For example, if the animate deformations are not the same as the real behaviour, it is not noticed as long as it’s plausible.
This technology can be used to design brand-new ways of interacting in VR and AR experiences and given the growing popularity of these mediums, there are many applications which would benefit from an immersive interaction tool. Marshmallow Laser Feast will continue integrating deformable objects into their experiences, allowing participants to explore virtual worlds like never before. In the wider context, this kind of interaction could be used for virtual storytelling, games and connecting people in VR as well as being very beneficial for virtual training environments. The potential for engaging, immersive virtual experiences which closely link the physical and the virtual is really exciting!
The collaboration between academia and industry has allowed me the brilliant opportunity to work with a growing team with a range of backgrounds, from technical to creative, so I’ve had a chance to see things from a different perspective and learn how an immersive art company operates. I’ve been able to help them explore challenging ideas through my research and highlight what is possible.
Catherine’s research has been published at several international visual computing conferences including at ISMAR/ IEEE VR and Siggraph. View all Catherine’s published work here.
Catherine Taylor | LinkedIn,Catherine Taylor (@CatherineT2) / Twitter
MARSHMALLOW LASER FEAST: Overview | LinkedIn,MLF (@marshmallowlf) / Twitter,https://www.instagram.com/marshmallowlaserfeast/