The process of composing, or placing actors in front of a backdrop that doesn’t actually exist, is as old as filmmaking itself – and it has always been tedious. Netflix has a new technique based on machine learning to do some of the hard work, but it requires lighting the actors in a garish magenta.
For decades, the simplest compositing method was chroma keying, where actors stand in front of a brightly colored background (originally blue, later green) that’s easy to spot and can be replaced with anything from a weather map to a fight with Thanos. The foreground is meant to be “matted” and the background is a transparent “alpha” channel that is manipulated along with the red, green, and blue channels.
It’s simple and cheap, but it also has a few downsides, including problems with transparent objects, fine details like hair, and of course anything that’s a similar color to the background. However, it’s usually so good that attempts to replace it with more sophisticated and expensive methods (like a light field camera) have stalled.
However, Netflix researchers are trying a combination of old and new that could allow for easy, flawless compositing — at the expense of hellish on-set lighting.
As detailed in a recent article, their “Magenta Green Screen” delivers stunning results by essentially sandwiching the actors in a light sandwich. Light green behind (actively lit, no background); A mix of red and blue in the front for a dramatic color contrast.
Actors lit in magenta in front of a green screen. Photo credit: Netflix
The resulting on-set look is likely to put off even the most seasoned post-production artist. Typically, you want your actors to be brightly lit with a reasonably natural light. While they may need a little tweaking here and there, their on-camera appearance is relatively normal. However, illuminating them with only red and blue light completely distorts this appearance, since normal light naturally does not block out a large part of its spectrum.
But the technique is also clever in that it simplifies the process of separating the two by making the foreground just red/blue and the background just green. A regular camera that would normally capture these colors will capture red, blue and alpha instead. As a result, the resulting mattes are extremely precise and lack the artifacts caused by having to separate a full spectrum input from a limited spectrum key background.
Of course, they just seem to have replaced one difficulty with another: the compositing process is now easy, but it’s difficult to recover the green channel on the magenta-lit subjects.
This needs to be done in a systematic and adaptive way, as subjects and compositions vary, but a “naïve” linear approach to injecting green results in a washed-out, yellowish look. How can it be automated? AI to the rescue!
The team trained a machine learning model using their own training data, essentially “sample” shots of similar scenes but with normal lighting. The convolutional neural network obtains patches of the full-spectrum image for comparison with the magenta images and develops a process to quickly restore the missing green channel in a smarter way than a simple algorithm.
A simple algorithm yields poor results (above), while a more sophisticated ML model produces colors very close to ground truth. Photo credit: Netflix
So color can be recovered surprisingly well in post-production (it’s “virtually indistinguishable” from some in-camera ground truth) – but there’s still the problem of the actors and set being lit in that horrible way Need to become. Many actors are already complaining about how unnatural it is to work in front of a green screen — imagine doing it while it’s lit in a harsh, inhuman light.
However, the paper addresses the possibility of temporally multiplexing the illumination, essentially by turning the magenta/green illumination on and off multiple times per second. This is distracting (even dangerous) when you’re doing it at 24 times per second (that’s the frame rate at which most movies and TV shows are shot), but when you turn on the light faster – 144 times per second – it appears ” almost constant”.
However, this requires complex synchronization with the camera, which is only allowed to capture light in the brief moments when the scene is magenta. And they also have to account for missing frames for motion…
As you can see, this is still very experimental. But it’s also an interesting way to take a new, high-tech approach to an age-old media production problem. That wouldn’t have been possible five years ago, and while it may or may not be adopted on set, it’s definitely worth trying.