Video Background

There are a number of reasons you may want to use a video as a background in UE4, maybe you want to use it for compositing your graphics in real-time or maybe you just want a cool effect for your game. In either case getting it set up isn't a long process and, while not being a beginner-level tutorial, this should help you work your way through it and understand the process.

The project worked through here is available in this Github repo

In order to allow this project to focus on the video elements we're going to start with an initial simple test scene which is set up with no skybox or similar in order that there's nothing in the background where we're going to be rendering the video, and aren't going to cover this as it's not really part of the video background technique.

Initial Scene

We can build the video background in UE4 incrementally in three stages:

Create Post-Process Material

To create a Post-Process Material we start as though building a standard Material, right-clicking in the Content Browser and choosing 'Material' in the 'Create Basic Assets' to create the Material and then right clicking and choosing 'Rename' to give it a meaningful name.

We can now start to edit it by double-clicking on it and the first thing we're going to do is turn this into a Post-Process Material rather than a standard Material we'd use for objects. To do this look in the 'Details' pane to the left and find the 'Material Domain' dropdown which should be 'Surface' by default and change this to 'Post Process'. Most of the pins on the Material output node should now dim apart from 'Emissive Color', which is because that's all that's relevant for a Post-Process Material.

Newly-Created Post-Process Material

In order to make this Material do the extreme basics create a Constant3Vector node and plug the output from it into the 'Emissive Color' pin, then double-click on the node and choose a recognisable colour and save the Material.

Basic Post-Process Material

To activate this we need to go back to the main editor and create a 'Post Process Volume', we can find this in the 'Volumes' section of the 'Place Actors' panel.

Adding A Post-Process Volume

Now we have a Post-Process Volume we'll make it apply over the entire scene, although for more complex setups we may want different volumes covering different parts of the scene. To do this select the PostProcessVolume in the 'World Outliner' and find the 'Infinite Extents (Unbound)' option and check it.

Making Post-Process Volume Unbound

Now the Volume applies to the entire scene we need to add the Material we've created to it. Find the 'Post Process Materials' in the PostProcessVolume's 'Details' pane and click the plus symbol to add a new Material then choose 'Asset reference' from the dropdown which should appear to allow us to reference the Material we've built, and then another drop-down will appear (Displaying 'None' by default) and will allow us to select the Material we've built.

When you choose your Material it will apply and the entire viewport should go blank with the colour you built into your Material.

Selecting Material in Post-Process

Adding a Video to the Post-Process Material

In order to play a video we need a video and so we'll do that next.

In the 'Content Browser' create a Media Source, this could be a 'File Media Source' if you want to import a video directly into your project and play it but for this demo we'll create a 'Stream Media Source' to allow a video to be played from a URL.

As with all new assets Rename it to a better name and double-click to edit it. The important bit here is the 'Stream URL' at the bottom which is the video we're going to play back and here I'm going to enter http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4 , which is the URL for the Blender Foundation's Big Bucks Bunny short film.

Setting video source

Now we have the video source we need to get it played, and for that we'll create a 'Media Player' which can be found in the 'Media' section of Create Advanced Assets. When we create this it'll ask if we want to automatically create a MediaTexture asset for the player and we do as this'll be the actual texture we plug into our Material.

Once we've Renamed both of the new Assets double click to edit the MediaPlayer, the preview here should be black as there's currently no videos loaded. Add the Source you've created by double-clicking on it in the list at the bottom of the screen and it'll be added to the Playlist, and while you're here toggle 'Loop' to True in the 'Details' pane so that the video will play in an endless loop- useful for demonstration.

We also need to save the playlist as this isn't stored internally in the MediaPlayer. Click on the 'Save This Playlist' button in the Playlist pane.

Setting up video player

Now we've got the video source, the player, and the texture we should see a preview of the video on the texture in the Content Browser to show everything is connected properly and that we can now return to the Post-Process Material.

Video texture preview

UE4 Problem: Empty playlist in MediaPlayer

At this point I hit a UE4 problem- it kept blanking the playlist in the MediaPlayer and so it wouldn't play. If you don't have this problem then skip this stage.

Create a Blueprint which has a MediaPlayer variable called 'MediaPlayer', and a MediaPlaylist variable called 'MediaPlaylist'. In a BeginPlay event it calls the 'Open Playlist' method on the MediaPlayer variable using the MediaPlayer variable.

Set Playlist BP

Adding this Blueprint to the level and plugging the Big Bucks Bunny player and playlist in as the variables fixed the problem.

Set Playlist BP details

Open up the Post-Process Material we built before and add a TextureSampleParameter2D node to allow us to use a texture and also to (outside of this tutorial's scope) change the video we're playing using Material Instances. Select the Big Bucks Bunny texture for the sampler and then plug it into the 'Emissive Color' pin, deleting the now unused 'Constant3Vector'.

Post-Process Material with Video

Go back and run the project and we should now see the video playing, although we won't currently see the rest of the scene as we're just replacing the entire rendered level with the video. Small steps!

Video playing in level

Displaying Video Behind Scene

Now we have the video rendering all we need is to make it so it only renders where we don't already have any geometry and for this we need to go into a bit more complexity with Post-Process Materials.

We firstly need to access the original rendered scene and for this go back into editing the Post-Process Material and, without deleting the TextureSampleParameter2D node (we'll need it again in a moment) create a SceneTexture node. This is used to access the various textures UE4 uses for the rendering process and for our use set the 'Scene Texture Id' to 'PostProcessInput0', this arcanely-named texture is actually the final rendered version of the scene. As a test plug this into the 'Emissive Pin', save and run, and the video should have disappeared and you'll see the scene as though we had no Post-Process running.

Accessing Rendered Scene in Post-Process Material

With access to both the video and the rendered scene all that's left is to blend the two and for this we're going to want to access the depth buffer (also known as the Z-Buffer). This is another of those internal rendering textures and is a greyscale image storeing details on how far each rendered pixel is from the camera, and we're going to want to check this against a depth limit and if it's further away then render the video otherwise keep the original rendered scene.

Create another SceneTexture node and this time for 'Scene Texture Id' choose 'Scene Depth' in order to access the depth buffer. Create a ScalarParameter which will be used to store the depth limit we're checking against and give it a decent default value such as 10000 to have the limit at 10000UU, or 100m at standard 1UU=1cm scale.

To compare two values in a Material we create an 'If' node, which compares the values on the A and B pins and returns the values from the 'A > B', 'A == B' and 'A < B' pins based on the comparison.

For the values of A and B for the 'If' node the compare we want to use the Scene Depth and the Depth Limit but first need to convert the Scene Depth to a single value as UE4 thinks that (because it's a texture) it's a Vector3 storing a colour, and for this we create a 'Component Mask' node and choose only 'R' (The red chnnael), and then plug the Scene Depth node into this. We can now plug the output of this 'Component Mask' into the A pin of the 'If' and the ScalarParameter depth limit node into B pin.

As we want to display the video when we're further than the limit we can plug the TextureSampleParameter2D for the video into the 'A > B' pin of the 'If' node, and as we want the rendered scene closer than the limit and so want to plug the SceneDepth:PostProcessInput0 into the 'A < B' pin, but we'll need another 'Component Mask' to drop the alpha and so will plug the SceneDepth:PostProcess0 node into the new 'Component Mask' and set it to include 'R G B' (Red, Green, Blue) channels and plug the result of this into the 'A < B' pin.

We're not really that bothered about which of them is displayed at the exact limit and so can leave the 'A == B' pin empty.

Blending video and scene in Post-Process

Now if you go back to the level and press play then you should see your video composited behind the rendered scene, just as we wanted!

Video rendered behind scene

Further stuff!

This is a fairly simple example of this technique and you can do more advanced things with it, and if I get time I may revisit this and add some of them in detail here. For now though here're a few ideas: