First Attempts

Going into this project I wasn't very comfortable working with the stencil buffer. Actually, I wasn't super comfortable with OpenGL in general. My previous projects in the course got a lot of mileage out of the professor's starter code for the renderer as I wasn't doing much that was interesting graphically, instead focusing on gameplay mechanics. This made portals seem like a pretty difficult choice for a final project, but I knew I would at least learn a good amount.

My first thought was to use framebuffers to render a camera view into a texture, something I was familiar with doing in game engines in the past. However, I somehow stumbled upon the idea of using stencil buffers, which was preferable for a couple of reasons:

• Rendering to a texture could cause aliasing when close to a portal without high resolution
• Such high resolution textures could be memory heavy, especially with many different sets of portals
• I just wanted to learn to use the stencil buffer after briefly interacting with it in Unreal Engine

My mind was fully set when I found a nice article by Thomas Rinsma describing his own implementation of recursive stencil buffer-based portals. I decided that I would start with implementing single portal rendering with functionality, and then adapt Thomas's method in my codebase.

Getting one portal to work was pretty straightforward. I gave the portal (and its destination) a local camera which would update position each frame, and I rendered the portal meshes into the stencil buffer, followed by their cameras' views into the depth/color buffer, and then finally the player camera as usual. For teleportation I basically checked if the player crossed the portal's XZ plane while within the mesh's bounding box, using some simple matrix math with the help of Sebastian Lague.

With that, single portal rendering was done, and I could move on to implementing Thomas's solution for recursion.

This wasn't trivial, as I was still getting into the recursive mindset- for example, I started off with a Portal struct which had a Camera member, but I started to realize that recursion would only work if view matrices were calculated at each depth rather than once per frame, since the same portal would effectively need to be viewed from multiple perspectives.

Even with that fixed, though, I found that my adaptation of Thomas's solution didn't work nicely with multiple sets of portals. In particular, portals were not able to correctly occlude each other.

I realized that I needed to store depth information from each portal at the same recursive layer in order to allow later-drawn portals to be occluded if necessary. The current implementation sometimes called glClear(GL_DEPTH_BUFFER_BIT), which didn't support conditional operation based on stencil buffer value. Therefore I opted to replace these calls with drawing a fullscreen triangle at max depth, which was compatible with the stencil test.

The rest of my implementation simply follows from making that adjustment work, so let's now take a look at the algorithm.

Recursive Implementation

The algorithm is a recursive function outlined as follows:

For each active, visible portal:

1. Enable stencil write only, and depth/stecil test.
2. Set the stencil test to pass where stencil value equals current recursion level, and increment the buffer by 1 on depth and stencil pass.
3. Draw the portal into the stencil buffer, incrementing it, and set glStencilFunc(GL_EQUAL, recursion_lvl + 1, 0xFF), allowing drawing only within this new portal.
4. Clear the depth and color buffer within the newly drawn portal. To do this we enable color and depth writing, disable stencil writing, and drawing a fullscreen triangle with glDepthFunc(GL_ALWAYS) and glDepthRange(1,1) and output color equal to our glClearColor. Then we reset glDepthFunc(GL_LESS) and glDepthRange(0,1).
5. Calculate a new camera transform, representing where the player would be looking from if they were already teleported.
6. If we are at max recursion level, simply draw all objects into color and depth within the portal view. Otherwise, recurse, passing our new camera transform as the view point.
7. Disable color writing, enable stencil and depth writing. Set depth to always pass and stencil to pass inside the new portal (glStencilFunc(GL_EQUAL, recursion_lvl + 1, 0xFF)).
8. Set glStencilOp to decrement on stencil and depth test pass, and draw the portal into stencil and depth buffers. This decrements our stencil buffer back to what it was before we drew the portal the first time, and sets up depth to let our portal occlude and be occluded.
9. Reset the depth comparison function to GL_LESS.

Then, just once per recursive function:

10. Set stencil test to pass when the buffer is greater than or equal to current recursion level (preventing us from drawing outside whatever portal's view is being drawn this call).
11. Disable stencil write, and draw all the normal scene objects into depth/color normally.

The actual implementation can be seen in the source code, specifically Scene::Draw() in Scene.cpp.

This approach is basically the same as Thomas's, using the stencil buffer value to represent depth levels, or how many portals we see into when drawing. We draw the scene by traversing portal views depth-first, incrementing the stencil buffer on the way up and decrementing once a specific portal view has finished drawing.

As mentioned, the main differences lie in handling the depth buffer, as this implementation takes care to never clear the entire thing in order to let portals occlude each other correctly (see step 4).

Clipping Planes

If you do look through the source, you may notice some stuff about clip planes that I didn't mention in the algorithm. One purpose is to avoid drawing objects behind portals as discussed in the single portal section, but there's another important purpose- making teleportation seamless.

Seamless teleportation was extremely important to the game we were developing, and one thing Sebastian Lague mentioned was the possibility of seeing behind a portal incorrectly up close, if the player angled their camera such that the near plane intersected with the portal plane.

This was easily fixed by using a cuboid for the portal rather than a flat plane and drawing on both front and back faces. However, a flickering issue persisted.

Our teleportation worked by checking for the player crossing the middle of the portal bounding box, then teleporting them to the same relative position in the destination portal as they had in the source portal. But a player crossing the middle of a portal can still be within the bounding box, meaning they would be looking at the inside of the destination portal once teleported, which would show the view from the source portal.

My solution to this was to detect which side of the portal the player was viewing from, and clip the portal such that it only drew the opposite side. This way, if a player crossed the middle of a source portal (looking through the source portal, showing the destination's view), they would not see any part of the destination portal after teleporting, avoiding the problem.