Sophia-in-Audition: Virtual Production

with a Robot Performer

Taotao Zhou*1,2, Teng Xu*1,2, Dong Zhang*1,2, Yuyang Jiao1, Peijun Xu1, Yaoyu He1,
Lan Xu1, Jingyi Yu1
1ShanghaiTech University, 2LumiAni Technology
Teaser Image

We present a new practice of virtual production: we deploy the humanoid robot Sophia as a virtual performer inside a virtual production studio, in our case, an UltraStage composed of a controllable lighting dome analogous to Light Stage coupled with multi-camera video shooting. We call this setup Sophia-in-Audition or SiA which allows for simultaneous controls over performance, lighting, and camera movements.

Abstract

We present Sophia-in-Audition (SiA), a new frontier in virtual production, by employing the humanoid robot Sophia within an UltraStage environment composed of a controllable lighting dome coupled with multiple cameras. We demonstrate Sophia's capability to replicate iconic film segments, follow real performers, and perform a variety of motions and expressions, showcasing her versatility as a virtual actor. Key to this process is the integration of facial motion transfer algorithms and the UltraStage's controllable lighting and multi-camera setup, enabling dynamic performances that align with the director's vision. Our comprehensive user studies indicate positive audience reception towards Sophia's performances, highlighting her potential to reduce the uncanny valley effect in virtual acting. Additionally, the immersive lighting in dynamic clips was highly rated for its naturalness and its ability to mirror professional film standards. The paper presents a first-of-its-kind multi-view robot performance video dataset with dynamic lighting, offering valuable insights for future enhancements in humanoid robotic performers and virtual production techniques. This research contributes significantly to the field by presenting a unique virtual production setup, developing tools for sophisticated performance control, and providing a comprehensive dataset and user study analysis for diverse applications.

Video

Overview

Our Sophia-in-Audition (SiA) system exploits the latest advances in humanoid robots and virtual production. In this section, we address specific challenges to incorporate the Sophia robot as a virtual performer. We also discuss how SiA can serve as a virtual audition solution by enabling simultaneous controls over the performance, lighting, and camera movement.

Overview Image