The holographic effects in “Social Networking”, the third video in our Adobe Open Screen Project series, involved two scenes that required superimposing graphic objects over live-action footage. The plan was to insert a semi-transparent newspaper over the bendable screen in the cafe scene, and a 3D animated video game racetrack onto a table in the final scene. Both instances were filmed with 3D camera movements and therefore the 3D assets had to motion match to the video. To do that we tracked the footage using SynthEyes.
SynthEyes (Andersson Technologies) is a motion matching standalone software that won’t break your piggy bank. The application, which will run you around $400, takes imported video footage, or image sequences, and analyzes the camera movement. Even though the program tracks 2D pans and camera tilts (2 1/2D), its most valuable function is tracking 3D camera motion to reconstruct the environment in the live action footage. The application can export its evaluation into various filetypes, two most common being a Lightwave scene for Cinema 4D, and a native After Effects file with a Camera layer in the composition.
The initial SynthEyes screen will display a side panel with a giant green button that says “AUTO” that you can’t miss. Hitting that button will–yes you guessed it–automatically run the SynthEyes 2D and 3D tracker. If you don’t want to generate both at once you can click the button below it that says “Run Auto-tracker”, which will generate 2D reference points. Likewise, the “Solve” button will track 3D points.
After SynthEyes calculates the movements in a footage, it decorates the picture with a pretty constellation of yellow, green, and blue/green thingies. The thingies indicate tracking points of reference, which SynthEyes collectively refers to as “blips”. The yellow X’s are 3D markers and green diamonds 2D markers. In addition, you can specify your own tracking points which will make a green box within a green box. After having these blips you must select three of them as your coordinate system points–I call them the stars of Bethlehem. These chosen three will set up your geometry planes, so choose wisely. Think about how you decide which friend to call when you’re in a jam: select ones that are most stable, reliable. Also choose points preferably close to the area where you’re inserting the graphic. The three points of references should be somewhat spread apart and represent different planes.
Different SynthEyes “blips”
For our cafe scene, we chose our three points to set up our coordinate system. When we scrubbed through the footage, there were a few bouncy markers that indicate an unreliable points of reference. Those jitterbugs are common and they will interfere with the smoothness of your matched motions. However, if you have enough other blips to rely on, you can find these bouncy blips and simply rid of them from your life. Delete. Goodbye.
Track markers in Cafe scene:
However, if you do not have enough blips leftover to make a successful track, you can try to “Refine”. After deleting the offending markers you’ll want to open the Solve panel by clicking the calculate icon. Choose “Refine”. The markers will readjust that should give you a better track.
Because we needed to replicate the bend of the newspaper screen our actor is holding, we took advantage of SynthEyes’s curve tracking. To recreate a curved 3D shape in the footage first click “New Curve” and trace a path around the object. It will act as a 2D spline in SynthEyes. Keyframe it through the video, adjusting the path and position so it always stays on the object. Then to create the 3D aspect of the version open up Quad view and hit “New Flex”. Flex is what SynthEyes calls a 3D curved shape. Draw the shape shape again in another view while keeping your eye on the shape in the remaining points of view. It’s going to be a trial and error, so you’ll likely have to constantly readjust your shapes in all four panels to get it right.
We exported the matched motion to Cinema 4D and used a newspaper image that matched the curve, angles, and movement of the stand-in prop in the footage.
Placing the graphic object in Cinema 4D using matched motion
The second scene was a little more challenging to work with in SynthEyes. There is a lot of activity in the holographic game scene, thanks to the moving four people in the shot and the more complicated camera movement. The first attempt at tracking was unsuccessful as SynthEyes kept picking up on the people’s movements to use as geometry references. It resulted in a margin of error of 5.125 horizontal pixels. The recommended margin of error is .3 or less. Oops. Therefore, we decided to mask out the people in the scene, shield the eyes of SynthEyes from those confusing moving humans. The masks had to be adjusted frame by frame, since the actors’ positions would change, but the results were muuuch better. This time, we were left with a margin of error that is .5, slightly above the recommended number but it still worked.
Before masking. Notice how most of the markers are not sticking to the footage:
After people are masked out. Blips are sticking to the walls and background:
Like the other scene, we exported the motion matching to Cinema 4D where we applied it to our 3D racetrack:
We then exported into After Effects the animated track that included its own camera movement as a footage. The movements of the 3D animation matched the footage and the results are superb. Looked just like a real holographic game. (Is that an oxymoron?) Overall, if you’re unwilling to shell out the whopping $4 Gs for Boujou, we recommend SynthEyes. It just takes some patience to learn how to tweak the settings. So I guess if you’re a short-tempered maniac maybe we don’t recommend it to you.
« From Pickle to Beast, Part 2 Adobe OSP Social Networking Breakdown »
Thanks for the write-up! SynthEyes is a great product, especially for the price.
thanks for posting this. I’ve recently begun playing around with Syntheyes and this is a great resource.