CDE Research Engineer Javier DeHesa tells us about his research ‘Using data to improve human-character interaction in VR’ with partner company Ninja Theory……

My name is Javier Dehesa and I am an EngD student with the Centre for Digital Entertainment at the University of Bath, carrying out my research at the video games studio Ninja Theory, the game developer behind the award winning game Hellblade.

My research aims to increase realism in gaming interactions such as sword fighting by generating responsive animations and behaviours from characters. Our model means a game character can anticipate all possible fight situations and react to them resulting in a more enjoyable and immersive experience. The opportunity to partner with Ninja Theory brings academic research into an environment where my work can have a direct ‘real-world’ impact on the games industry and the products being designed right now.

My research is bringing “data-driven models” into different aspects of video game development. This means instead of manually programming what exactly has to happen in a game in given a situation, you give the computer a lot of examples of similar situations and it automatically learns the model that tells you what to do. We are working to apply this idea to the domains of interaction and animation. In our most recent publication, we created a sword-fighting scenario for virtual reality. The animation of the opponent sword fighter, as well as the recognition of different directional strikes from the player, are not “programmed” as such, but instead are performed by models trained on motion capture and gestural data. This is a great advantage for developers because they do not need to consider all the possible interactions that may come up within a sword fight in virtual reality – which can be a lot! At its core, our system uses neural networks, which is a powerful mathematical tool to implement such data-driven models. We are currently researching other scenarios to which we may apply the same concepts, such as automatically producing a controller for a character from clips of locomotion data.

My industrial partner, Ninja Theory, has provided me with the fundamental tools and support to carry out my research. Since the first day I got to the studio, my industrial supervisor, Andrew Vidler, has always made sure that I felt completely integrated and could count on the help of their experts. Luckily for me, Ninja Theory has its motion capture facilities and equipment, which has been critical to the viability of my work. Conversations with programmers, animators and designers have helped me understand the actual needs of a commercial studio, from performance requirements to quality evaluation or usability concerns. Andrew is also very technically savvy, so I rarely get stuck with issues without support.

The industrial impact of my work feeds directly into my partner company Ninja Theory who are excited about putting that research into some of their projects. They pride themselves in being at the edge of technical innovation and strongly advocate a “doing a lot with little” mindset, which is very much what my work is about. Other big studios, like Ubisoft, have also been exploring and implementing data-driven animation techniques in recent years, and it seems to be a strong trend that will become standard shortly.

With the level of realism expected from the upcoming generation of video games, animation is taking more effort and resources than ever, and any tool or technique designed to alleviate that load can have a very significant impact on development costs, scalability and maintainability. This partnership with the CDE has allowed them to dive into research areas where they had less experience. Personally, I find it really exciting to be working in something that is both very useful but also very visual and appealing and count myself lucky to have been given the opportunity to do this research with the CDE and Ninja Theory.

My background is mainly in computer science, completing an MSc back in Spain and then working as a developer for some time in Spain and Germany. I had been interested in academic research for some time before joining the CDE, with occasional minor research contributions during my undergrad years, but I was hesitant to be disconnected from the industry. When I learned about the Engineering Doctorate (EngD) at the Centre for Digital Entertainment I saw the perfect opportunity for me to learn, upgrade my qualifications and contribute some research to an area I am passionate about, all while keeping alive my professional career. Right now, I am in the final stage of my studies, and not too far from completing my thesis.

I have weekly calls with my academic supervisors at the University of Bath and CAMERA, Dr Julian Padget and Dr Christoph Lutteroth, who in addition to specific research advice, give me the guidance I need to wrap my technical results into an “academic envelope”. For example, for our last publication, I had to learn a lot about writing for a human-computer interaction venue, something I did not previously have any experience of.

Last year this research was featured in a paper I had accepted at CHI 2020 in April. You can view the paper ‘presentation’ video here:https://youtu.be/QY0Rae0S4do

Javier’s EngD research is supported by funding from EPSRC.