In-Camera VFX for Indie filmmakers using Unreal Engine

An intro using just an iPhone, a Camera, and Unreal Engine

Watch the video

Unreal and ICVFX (In-Camera VFX) with Scott Kirvan : Getting into Virtual Production without an LED Volume in UE5

Presented by VFX/VP Supervisor, Scott Kirvan, May 16, 2023, at the Seattle Unreal Engine Meetup.

00:00 Intro to virtual production
02:24 ICVFX ideas for independent filmmakers
05:25 Required equipment review
06:50 The “real world” scale and position discussion
08:50 Measuring the “real world”
12:50 Getting into Unreal. (UE_QuickLaunch) - https://github.com/ScottKirvan/UE_Qui…
14:20 (ScooterUtils Edit plugin) - https://github.com/ScottKirvan/Scoote…
15:00 Create and save a default level
16:10 Build the screen from a basic plane inside Unreal
17:28 Create UV material
25:15 Cinecam & LiveLink
27:00 Unreal Remote 2 on iPhone/Tracker
29:40 nDisplay Config
30:40 Add Cluster Node
31:25 Add Static Mesh to nDisplay
32:51 Add ICFVX (inner frustum) Camera to nDisplay
33:46 LiveLink Preset
34:40 Switchboard
38:12 Launching nDisplay
40:30 Camera overheats + general tech fail! haha :-)
42:55 Tech Test sample: Magic Box
43:30 Tech Test sample: Meerkat, nDisplay, multiple monitors

Further details on the project: https://www.scottkirvan.com/ICVFX/
Unreal Seattle’s Discord: https://discord.gg/aPdE2QqqeZ
Seattle’s Unreal Engine Community page: https://communities.unrealengine.com/…
Filmed on-location by Natt Thangvijit. Edited by Natt & Scott.
Equipment/software used: Asus Nitro 5 laptop w/ NVIDIA 3080, Sigma fp camera, Unreal 5.2.

Required Unreal Project Plugins:

Add’l plugins/tools referenced:

Process Outline:

  1. Collect IRL (In Real Life) Data.
    • Display
      • measurements (X, Y, Height to base of screen – all in cm)
      • resolution (we used Full HD, 1920x1080px)
    • Camera
      • film back measurements (we had a full-frame DSLR video camera)
      • lens length (45mm fixed lens in our case)
      • addn’l lens stats (f-stop, etc. - which we weren’t going to be using in the demo)
      • position relative to display (we set out camera up 1 meter away from the screen, aligned to the bottom of the screen)
    • Computer - ip address (use ipconfig in a windows command prompt to find your current IP)
  2. Reconstruct our IRL display screen in Unreal.
    • create a plane actor, duplicate it’s static mesh into your content folder - delete the actor
    • drag the new SM plane into your scene, rotate it, align it’s uv’s, then scale it
      • we created a quick test material to visualize our UVs here
  3. Add CineCamera Actor and set it up to match your IRL camera.
    • set the lens length, filmback, disable focus so we’re not dealing with DOF
  4. Set up Livelink & connect the iPhone using Unreal Remote 2, and save the LiveLink Preset.
    • open livelink window
    • open UnrealRemote2 on iPhone - set the IP address, connect.
    • find the iphone in LiveLink, create a preset, save the preset, set the preset as default in Project Settings
    • add a livelink transform component to your cinecam, and
  5. Configure nDisplay - add VP (Viewport) node, add static mesh screen, and add camera.
  6. Launch listener.
  7. Launch, Configure, and Connect Switchboard.
  8. From Switchboard, run your Unreal project in nDisplay mode.
  9. Make your movie and get famous.

Video copyright (c) 2023 Natt Thangvijit & Scott Kirvan

Required Equipment

Resources