Creating a VFX Sequence Using Live Footage and Unreal Engine 5

During the first semester of my masters degree at Abertay Univeristy I created a VFX sequence using Unreal Engine 5. I wanted to utilise the power of Unreal for film making and decided to blend drone footage with CG elements. Although the final composition was rendered using path tracer I utilised the power of Unreal for real-time compositing for the previs of the final shots.

The project served as a great introduction to film making in Unreal as I pushed myself to use a veriety of techniques and technologies to a high standard. First and foremost, I needed a good sequence of drone footage to act as a backplate for my final composition. To shoot this footage I headed over the bridge to Tayport where I could safely fly outside of the Dundee Airport flight restriction. You can read more about the shoot in a previous blog post including the use of Google Earth VR to previs shots. Once I had captured footage for my sequence I tracked the footage using After Effects and Cinema 4D allowing me to repeat the same camera movements in Unreal Engine creating a realistic blend of CG elements and live action footage.

Creating realistic digital doubles of the actors was a key challenge I faced during the project and I used a miriad of technologies to acheive this. I utilised Abertay's Vicon motion capture stage to record full body motion of my actors and retargeted them to Unreal Engine 5 metahumans. The Metahumans played a pivitol role in the success of the project as they allowed me to create high fidelity doubles of my actors in a short space of time. To create good Metahumans I used Reality Scan to create photogrammetry scans of my actors which were then used to create the Metahuman face mesh. As well as full body mocap I used Live Link to stream facial animation data to Unreal. The Live Link animation worked really well as the actors were controlling a rig based on their own facial structure.

I used an image plate in the Unreal Editor to composite my sequence in real time allowing me to carefully plan and put together shots. Once I had all the individual elements for the composition I was able to sequence the shots and animate CG elements such as the helicopter and Metahuman actors. A final touch that elevated the final sequence was the use of simulated anamorphic camera lenses. These lens files helped to add realistic lens distortion and helped mask the "gamey" feel of Unreal Engine.