pOrtal! – a technical making of.

During last week, I spent most of my days working in one assignment instead of making real progress with my reel. Why the heck did I do that? Well, I have to go back to last Sunday, while I did the rigid bodies assignment, linked on the previous post. During render time, while Maya wrote out all my frames and passes, an idea came to me. From the Set Extension class, I realized that this could be a very mindfucking assignment and I really wanted to play with portals. Have things crossing walls, or looking forward and seeing myself from up top, weird stuff like that. I couldn’t focus in class and didn’t take any of the plates we shot at the studio downstairs.

Then, the portal thing finally clicked, and I thought it would be super cool if I could go through it, using a phone app to choose my destination. I scribbled some notes of what had to be perfect, ideas of how to shoot it and work it out, avoiding as much complications as I could, from the beginning. For starters, I would love a very wide angle, first person look, with the phone on the screen, so I could show the interaction between the app and the real world, several different destinations, and finally, the one I’d cross to.

I let the idea grow for one more day and got to work on Tuesday, because I got really inspired to see if 1) I could get it done, 2) I could get it done IN TIME (which meant before tonight). I finished yesterday, and you can see the result below. After the video, I’m going really crazy and try to explain how/what I did during the process, because these breakdowns aren’t nearly enough to explain the mindbending I went through.

First of all, I thought about having a greenscreen and markers on the phone and replacing that in post, but I already had too much work on, so it was easier and faster to design and animate everything that would happen in the app in After Effects and just play it while recording my plates. In this process I had to find out android’s favorite video formats, screen resolution, how to output it from After Effects and, later on, add a guide soundtrack so I knew what animation was going to happen and when, in order to time my actions on the plate.

This took my Tuesday afternoon. When I got home I grabbed the camera and took a couple test shots of the corridor – where all the action takes place – because I wanted to go really wide angle, and my only available option was Canon’s 8-15mm fisheye lens. It has an amazing field of view, but the downside is that everything comes out fisheye-distorted, which means it’s impossible to camera project anything – which is key to my matchmoving and all the environments changing at the back.

With these distorted shots I tested Nuke’s LensDistortion node, which gave a kind-of-ok result, but really messing up with the corners of the frame. Also, it was impossible to bring it back to the original image – couldn’t figure out why. It creates a weird circle with black background and everything punched inside. anyway. For my plates, this could work. For the camera projections, not so good. I went then to see if there were any good “defishing” techniques using Photoshop and got great results with their custom presets for the Lens Correction filter. It requires you to install some extra free packages with Adobe Air, but it’s very quick and simple. In there, sometimes the 8-15mm showed up on the list, sometimes not, so I picked the regular 15mm fisheye along with a full frame sensor and the results were very very interesting. With these, I went into Maya and quickly buid some cubes to see if it was camera “projectable”. It worked.

Green light for the shooting then.

I started with the pictures for all the environments. Set the tripod at the same height and level for all the pictures, lowered ISO, closed aperture and selected a very low shutter speed so I could get as much depth detail and as little noise as possible. Not a hard achievement using the 8-15mm. Then I stuck a tape crosshair at my chest at the aproximate tripod height, so things wouldn’t look too weird when combined. I chose environments that would be very simple to build (hallways, which are pretty much cubes) and with lots of depth to them. There are some examples below.

Before shooting the plates I also took some extra pictures to help me building those environments and filling the holes in the projections. Measurements were also very important to speed up the process and make sure everything would match completely. All the corridors have about the same width, so that would also be a good way of spotting any weirdness that could come up.

I shot my first plate a couple times to get the timing right, focus pulls and camera movement. I tried to avoid covering the door with the phone as much as possible – to avoid roto – but that wasn’t so successful. Then I went down to the Sub Basement and shot my second plate using the same technique. I was worried about how to link them together – no idea at all, at the time of shooting – and if they would track properly.

To help me with the timing for the phone animation in the second shot, I have the animation playing for ten seconds and myself counting up. When it reaches ten, the glitches appear and the screen goes black. I needed to do all my transition and regret in around seven seconds, then look down, click the phone just to see it die and turn to the elevator. It was more complex than it looked.

After shooting everything, I was still worried about this fisheye look, so I undistorted all my environments and tried to project them. Got three out of four done in less than an hour. The laundry room seemed to have a lot more detail, it was already past 10pm and I should get some rest.

Day 2 – Wednesday – was a nightmare. Tracking worked well with the Nuke-undistorted plates, but whenever I tried to export the mesh created from the point cloud, it would crash, die, burn and so forth. Tracking took more than one and a half hour! In the meanwhile I tried to figure the logic behind crossing between plates and finally nailed what should happen. I didn’t get it to work on this day, though.

I finished all my projections while the tracking went bonkers, and by the end of the day I decided to ditch Nuke’s undistortion and go with Photoshop’s for the plates as well. I kept the node because I wanted to bring the fisheye look back into the final result, and that was achievable, with some cropping.

With all my projections done in Maya, I brought them into Nuke, along with the re-tracked plate – didn’t even bother with the point cloud this time, just got the geometry through another (fifth, or sixth, by now) camera projection. Aligned it all to the grid and exported as alembic. In Nuke, I placed all my other environments behind the door and animated them according to the app animation. I used cards and a random concrete texture to cover the gaps between them. Defocus with keyframes solved the focus pulling issues and oFlow got me the proper motion blur.

For Day 3 – Thursday – I refined my script, checked the tracking a thousand times and did all the roto work. This day went fast. I ran out of songs to listen while working, so I had to look for new stuff online. Doesn’t happen every day. So I rotoed (?!) the door out, and brought back the phone and my fingers when they went in front of the hole. The work on the door was awful, jumping as crazy, because there isn’t much stability and continuous motion when you are handheld walking while doing everything else at the same time. Each rotos had around 180 keyframes, for 250 frames of footage.

In Day 4 – Friday – my goal was connecting both plates, this was the cherry on top, because it wasn’t mandatory for the assignment, but I really wanted to do it, so I left it to the end.

What happens is: I have two 3d tracked plates, which means two cameras. First camera, inside my place, goes outside the door, which means that, from certain point, there is no more reference of the initial environment. The cut has to happen after this point. From the second plate, I needed to pick a frame not too close to the beginning, so I could transition from the first camera into the second – using a third camera and a keyframed parent constraint and keyframes for focal length because Nuke gave me slightly different numbers (12mm for one and 14mm for the other).

This part took half my morning, then I had to decide whether to fix the door’s roto or to add a portal effect that would benefit from the jaggedness of the mask. The principle behind the portal is the same as heatwaves. Noise that distorts what is around, affecting both foreground and background, changing constanly and waving around. I based it off the door’s alpha with good results. Had to do some keyframing to make it bigger near the end, fix colors and stuff. Then I noticed the portal was cutting into my hand as soon as the phone and fingers masks ended. More roto. Yay! Luckily, it wasn’t that much, and not so much movement either.

As soon as the portal looked good, I went back to the connection challenge.

I needed to decide exactly in which frame I was going back to live footage. Once that was picked, I had to camera project the very previous frame, using this final animated camera, and paint the gaps in the projection so it would fit between going out the door and turning into the other tracked camera. This was by far the most confusing part, in which I camera projected three different images that didn’t work, because I couldn’t figure out how to do it.

When I finally got it to work, all that was left was a crazy amount of painting and perspective to cover the huge amount of corridor that was untextured. Even harder, the painting had to be done UNDER the original texture, so the image wouldn’t jump on the frame it went to footage again. How do I know this? Of course, I painted it wrong the first time. The seamless paint took me way longer – like hours longer, getting to the point of painting single pixels that looked wrong. This was my Friday night.

Day 5 – Saturday – was light, I had to cleanup the markers outside the door, barely visible, and only for a couple frames, and figure out how to do breakdowns for this thing that almost melted my brain. Ended up oversimplifying because there was no way to explain all of this in a few seconds of video. Then, add sound effects, bring back lens distortion and a final grade.