Remedy Entertainment has partnered up with NVIDIA to streamline production-level facial performance capture using deep convolutional neural networks.

　　As reported by Ars Technica, NVIDIA presented the “deep learning framework” with Quantum Break developer Remedy Entertainment during this year’s Siggraph event. A video was shared as well, which can be watched below:

　　We present a real-time deep learning framework for video-based facial performance capture—the dense 3D tracking of an actor’s face given a monocular video. Our pipeline begins with accurately capturing a subject using a high-end production facial capture pipeline based on multi-view stereo tracking and artist-enhanced animations. With 5–10 minutes of captured footage, we train a convolutional neural network to produce high-quality output, including self-occluded regions, from a monocular video sequence of that subject. Since this 3D facial performance capture is fully automated, our system can drastically reduce the amount of labor involved in the development of modern narrative-driven video games or films involving realistic digital doubles of actors and potentially hours of animated dialogue per character. We compare our results with several state-of-the-art monocular real-time facial capture techniques and demonstrate compelling animation inference in challenging areas such as eyes and lips.

　　2 of 9

　　While the technique appears to have limitations, the presented is able to achieve “high levels of accuracy for facial performance capture while drastically reducing the amount of manual work involved in a production setting”. NVIDIA and Remedy write that the technique has "proven accurate enough to be applied in an upcoming game production based on thorough pre-production testing while also requiring much less labor than other current facial performance capture pipelines used in game production.”

　　More about this technique can be found in the paper that has been submitted by Remedy and NVIDIA.