This page and the supporting PDF document, provide an in-depth guide to getting started with Unreal Engine Virtual Production and Composure.

It was not my initial intention to create this page, but after going through the process of virtual production I found that the information required was not all in one place, so I decided to document this process into one concise page and a supporting doc.

The aim is to provide a guide to get you up and running without a bunch of headaches. It does not go into every detail, but does provide the key steps that along with your own knowledge will be enough to get you started.

This guide is a little more suited towards Indie filmmaking, but large scale, larger budget productions will also find this a helpful starting point. This page provides an overview of the process and also different ways in which to input and output video.

The information contained in this page and doc focuses on Unreal Engine and setting up virtual production functions within the software program (as opposed to actual physical studio/stage set up). For Unreal Engine technical instructions please refer to the supporting PDF document that can be downloaded, for free, via the button below.

Free PDF Supporting Document on Virtual Production:

Unreal Engine – Virtual Production Getting Started Guide

Unreal Engine – Virtual Production Getting Started Guide

Click Here for PDF Download183 Downloads

Composure Inputs and Outputs

What I have found is that once you have familiarised yourself with Unreal Engine Inputs (video) and Outputs (output composites) everything will fall into place. The Inputs and Outputs are set up in the Comp (Composure addon) and whilst at first can seem a little confusing and complex. They are actually quite straight forward.

So whether you are outputting to a LED screen, or compositing everything together using a green screen, once you have got to grips with inputs and outputs it should be a straightforward process.

Virtual Production using Unreal Engine, involves the set up of green screen or a large LED screen array (Multiple LED screens joined together to make one big screen – also known as a ‘volume’).

Environments are built in Unreal Engine using high quality 3D assets and/or high-res HDRI images. The environment from Unreal Engine is then fed into an LED screen array and the actors/live footage is recorded in front of it. Or a green screen background / studio is keyed out with a 3D environment from Unreal Engine and then live actors and supporting props are all composited together for a final video output.

LED screen displays will provide higher quality results and provide many benefits over green screen. They remove any need for post process colour correction removing any green tint from the recorded footage. Also, any real world reflective surfaces (such as metal or glass) will reflect the LED 3D environment, where as with green screen, reflective surfaces will reflect the green material and again this would have to be corrected in post production.

To put it diplomatically, green screen is feeling somewhat archaic and anything you can do get away from green screen is beneficial to your production. However, you still get great results from green screen and it is far more cost effective. It also can be set up in a variety of different places, where as LED screens are far more expensive and require much more attention and time to set up. Although the price of LED screens for this kind of work is reducing, so will become more accessible to all productions in the years ahead.

Set Up:

1. Create a new Project and Enable all required Plugins.

2. Plan and build the required environment(s) with any animations and environmental effects.

3. Place Virtual Cine-Camera(s) in the Environment.

Create a Composite with Camera Input:

4. Create a Media Bundle.

5. Connect your Video Capture Card or Device to the PC / Unreal Engine.

6. Create Timecode Sync (Sync the timecode from the real world camera to the virtual camera).

7. Create Gen Lock (Prevent dropped or skipped frames whilst outputting video).

8. Create a new Comp (Composure).

9. Add the Media Bundle Texture into The Composure Media Plate Slot (this adds your live video footage to the comp).

10. Create Composure material so Unreal Engine knows what elements to composite.

11. Camera Tracking (Link the movement of a real world camera to the Unreal Engine Virtual Camera)

Exporting Footage / Output:

12. Offline Video Exporting: Exporting your Composited Footage via Sequencer.

13. Online Video Exporting: Output and record composited Footage via a Media capture device or card.

Miscellaneous:

14. Example Products for Virtual Production.

15. Further Documentation.

16. Thanks.

17. Further Resources.

Note:

This guide assumes you have Unreal Engine 5 installed and you have some prior working knowledge of the program and its user interface. If not it is recommend to familiarise yourself further with the program, but moving on to Virtual production.

The overview here and supporting document detail ways in which you can set up virtual production, but there are different ways in which this can be done and Unreal Engine is a versatile and flexible system.

1. Create New Project Enable all required Plugins in Unreal Engine.

When you start Unreal Engine 5 you are presented with an array of different starter project templates. For Virtual Production I have found using > Film / Video & Live Events > Then the blank template to be for most efficient and clutter free template to get started with.

Plugins:

Using this template should activate all the plugins required for virtual production, however if some are not activated then go to Edit > Plugins and activate the required plugins. The plugins below are the key plugins required for getting started with virtual production.

Unreal Engine Virtual Production Required Plugins

2. Plan and build the required enviornment(s) with any animations and environmental effects.

This is something that must happen before you go into your virtual production. It can take some time to do this and will require planning, storyboarding, pre-visuals and input from the director and environment artists and designers.

This section doesn’t go into any great detail regarding creating an evironment in Unreal Engineas, but there are many resources online for how to do this. Also there are many skilled environment artists and designers who do this kind of work for a living. However, if you have to do this yourself, the good news is you can get up and running fairly quickly, so don’t worry if you are just starting out. For example here is one link to a recommended video for getting started building environments in Unreal Engine.

Needless to say this can be a time consuming process, so make sure this part is completed before you go to virtual produciton.

Note: If you just getting up and running with virtual production, it is recommended to carry out some tests before moving in a production and before any detailed environment creation. So, don’t worry about creating a detailed environment for your first ever test. Start with a simple environment and make sure you can get your virual production set up running first, before going any further.

3. Place Virtual Cine-Camera(s) in the Environment.

Unreal Engine has a built in ‘Cine-Camera’ that is used for creating animaitons, movies or in game cinematics. You can add multiple Cine-Cameras to your environment and these Cine-Cameras are also used in Virtual Production.

Place a Cine-Camera into the scene ready for production. You can, of course, move these around in the environment and would work with the director or if you are the director – then would place these in the required starting position(s).

Note: If you have more then one Cine-Camera you would need to set (although it is not essential) each one up with an individual tracking device. So the physical real camera moves the virtual camera in Unreal Engine. This does take a bit more set up, so if working on your first virtual production, it’s a good idea to stick with one camera.

Please note: The process in the video above uses an ‘offline’ process to create the final video. Meaning the actor is recorded in front of the green screen with the green screen being recorded as well. Then the Unreal Engine 3D environment footage is exported out seperatly. Then finally the actor with the green screen footage is composited with the 3D enviroment footage from Unreal Engine in Adobe After Effects.

Please note: The process in the video above uses an online process where the actors and real world footage is composited with the Unreal Engine 3D environment and then outputted to a media capture device. So there is no post production compositing of the footage and it is all done in the set.

4. Create a Media Bundle

Note: Unreal Engine has a feature called Composure – this is simply where you can create ‘Comps’ (compositions) and add elements to them for composition.

The elements in comps primarily used are CG Elements and Media Plate Elements. CG elements are any 3D, particle or enviroment elements in the level and media plate elements are media players that play video or connect to camera feeds.

You can create a Media Bundle and this bundle will contain (most) of the required assets needed for connecting and compsiting live or pre-recorded footage. You don’t acutally have to create a media bundle and can just create a comp, media player etc… individually, but creating a media bundle saves time. Please note that media bundles do not come with a Comp and these have to be created in the Composure Panel.

5. Connect your Video Capture Card or Device to the PC / Unreal Engine.

Open the Media Bundle source file and select the blackmagic configuration build (this is for example assuming you are using a Blackmagic capture card or device) > In the file select the drop down and select the settings that match the real world camera.

6. Create Timecode Sync.

This will create a time code blueprint that will sync the time code in Unreal Engine to the time code in your camera. Allowing for seemless editing of footage with a matching time code from the camera and in Unreal Engine. This is not a complex process and is fairly straightforward to set up.

See Supporting PDF Doc: Section 21

7. Create Gen Lock.

Unreal Engine can drop frames when recording and exporting video in virtual production, so by creating a Gen Lock Blueprint this will enable the lossless recording of video without any skipping of frames. Again this is not a complex process and is fairly straight forward to set up.

See Supporting PDF Doc: Section 22

8. Create a new Comp (Composure)

To composite different elements together such as a video feed and a 3D background you can use a Comp. To do this go to the Composure Compositing panel (if not there – go to Window > Virtual Production > and select Composure Compositing). In the panel right click and create a new comp.

9. Add the Media Bundle Texture into The Composure Media Plate Slot.

Create a comp in the level > In this comp add a Media Plate element.

From the Content Broswer > Drag in the media player texture and place in the media plate > media source field of the Comp that was just created (this texture will have been created by default when the media bundle was created). This will then add the video feed (that was set up in the source file – point 5 above) into the composition.

10. Create Composure material so Unreal Engine knows what elements to composite from the comp.

For your comp to work you need to create composure material. This is a straight forward material and a screen shot of this is entered below.

See Supporting PDF Doc: Section 5

Above is a great video from Andy blondin on how to set up the composure material. Even though the video is getting a little old now, it still is relevant and works. The only part that no longer appears to apply is the ‘Fog’ Flag. This appears to have no effect if ticked on or off. Instead use the ‘Atmosphere’ flag to great rid of any unwanted or doubled up layers of fog.

Note: Composure does not recognise exponential height fog and only recognises Atmospheric Fog (which is now a legecy elements in Unreal Engine as Exponential height fog replaced it. So if you want to use a layer of fog you have to use Atmoshpehric Fog. Although I have found composure does not hangle engine fog well so if you can avoid it all together that is that best approach)

11. Camera Tracking (Link the movement of the real world camera to the Unreal Engine Virtual Camera).

Using a real world tracking device, such as a Vive Tracker and a Vive Mars. You can link a tracker device to the virtual camera. So when you move the real world camera the virtual camera will move as well. Thus linking your camera movement to the Unreal Engine Environment.

Outputting Video Note:

Offline and Online: There are two man ways you can output composited video from Unreal Engine.

The first is record your actor(s) in front of green screen and then export your 3D footage out of Unreal Engine. Then you can composite these together in a program such as Adobe After Effects or Davinci Resolve.

The second and recommended way (if you have the hardware) is to output both the live footage and 3D environment back into your PC / media capture card / device (all compositing is done in Unreal Engine).

12. Offline Video Exporting: Exporting your Composited Footage via Sequencer.

You can add your comp and any media player video into sequencer and then export out via the Movie Render Queue. When I first tried virtual production it was this part I got a little stuck on to begin with, but the process, whilst does have a few steps to go though is not that difficult once done.

See Supporting PDF Document Section 13 for how to do this.

For further information on how to use Sequencer please see the Unreal Engine Sequencer information here and also how to render your footage without outputting via a video capture device. This is especially handy and required if you don’t have a media capture device for outputted footage.

Note: Unreal Engine has a feature called Level Sequencer or also known as Sequencer. This is where you can animate characters, objects and environment effects inside your level. It can also be used with Movie Render Queue to export out footage and comps.

13. Online Video Exporting: Output and record composited Footage via a Media capture device or card.

You can Output your Mixed Composite (live camera feed and 3D environment) back into your PC / Media capture device. This requires the creation of a Media Output file and once set up you can configure it to match the details of your media capture/output device.

See Supporting PDF Document Section 19 for how to do this.

I have found this to be a little tricky at times, so make sure you have this set up before doing any virtual production work.

Example Set Up Below: Virtual Production Real Time Capture. Using (Black Magic – BM) ‘Video Asset Monitor/Capture Device.’

The video above, from Bennet Win, provides an excellent overview for setting up a camera, inputting live video and then outputing a final composite with the Unreal Engine 3D environment into a black magic video capture card.

Another great video on virtual production. This is an in depth and live overview of virtual production that goes into detail reagrding the set up and processes required. The processes in the video are detailed in this page, and also the supporting doc.

Example Camera Products for Media Output / Input and Capture

Black Magic Video Asset – Video Input and Output Monitor and Capture Device In One:

https://www.blackmagicdesign.com/products/blackmagicvideoassist

Black Magic Playback and Capture Products:

https://www.blackmagicdesign.com/products

AJA Capture Cards:

https://www.aja.com/products/u-tap-hdmi

Vive Mars: This Product is specifically for Virtual Production and Camera Tracking:

Vive Mars

Timecode Deity Products: Keeping and Syncing Time Codes between Camera and UE:

https://deitymic.com/products/#timecode

Key links to futher and supporting documentaiton

Unreal Engine: Compositing / Composure Tools Set Up

Setting up Unreal Engine Real Time Compositing Tools

Unreal Engine: Media Capture Cards: Contains information about getting professional quality video into and out of the Unreal Engine.

Link to AJA and Blackmagic Card media capture documentation

Unreal Engine: Hub For Integrating Medi: General info no integrating your pre-recorded media into your project.

Integrating Media

Epic Games Virtual Production Hub:

Virutal Production Hub

Virtual Production Indie Film Guide – Resource for Virtual Production

https://vpifg.com/guides/unreal-composure/

I would not have been able to learn so much about Virtual production in regards to Unreal Engine it it were not for Epic Games, Richard Frantzén, Bennett Win, Andy Blondin, Cinematography Database and Cinecom.net. So a big thanks to all of these people and organisations.

If you wish to learn more about Unreal Engine Please do see the 3D / CG Resources page that has many more links to tutorials, informaiton and also free resources in relation to Unreal Engine and also other 3D related programs as well.

The information on this page and the supporting PDF document are provided for free, but please consider supporting my work and for further updates please join, subscribe or donate to a channel below.

1. YouTube: https://www.youtube.com/channel/UCOSSd3JlWqHB5Op9ocHiVJQ

2. Discord: https://discord.gg/gKfujq5zmb

3. Ko-Fi: https://ko-fi.com/motionforge