How do graphic engine developers debug their 3D graphics code?

Question

I'm currently trying to create an amateur game engine, and I often run into very basic bugs (eg I change the node hierarchy code and suddenly nothing is displayed). Some of the methods I will use include to track down these bugs include:

• Use my IDE's debugger and lots of printf calls to look at what's in memory.

• Make lots of tests with different parameters, placing the camera in different positions, to get a better feel of what the code is doing.

• Just brute force my way through by re-reading each line of code until I find the one that's misbehaving.

These methods all seem sluggish and inefficient to me. The problem is, that 3D rendering involves a lot of unintuitive operations with not much to visualize except the final "product"; when I'm running into a bug (especially "nothing is displayed" types of bugs), I often feel like I'm fumbling around blindly.

---

So.

Let's say you're a developer working on Unity, Godot, or some other 3D engine, and you just changed some steps in your rendering pipeline, and now, for whatever reason, the wireframe renderer displays incoherent results when entity batching is activated. The "wires" are crisscrossing all over the screen instead of representing the edges of the batched entity's triangles.

How do you track down and fix that bug? Do you have a specific workflow? Do you use some kind of GPU analysis tool? Or do you just fumble around blindly?

In other words, what methods, tools and best practices can engine developers (not engine users) use to analyze the data and behavior of their 3D graphics code (not just its performance) to efficiently address defects and bugs?

Answer 1

I'm programming mostly in OpenGL. In this case, the very first thing I do is setting up KHR_debug extension. It provides a way to subscribe to messages generated by GL drivers - from errors to warnings to detailed information about what the driver is doing. Especially open source Linux drivers (Mesa) generate lots of useful messages.

When it comes to tools, my first choice is usually ApiTrace. It allows you record a trace of all rendering API calls and then replay it and inspect state at any point in time (this includes contents of buffers, render targets, etc.). It also allows you to share a trace with other people, which is useful when working in a team. Then, there are other tools such as RenderDoc, NVIDIA Tegra Graphics Debugger, Xcode's graphics debugger and some others. Whichever you chose, it will improve your work a lot.

Another thing is testing. One thing is building a collection of simple scenes excercising single feature of the engine (such as different material settings, different animation techniques, etc.). When a bug is introduced into the codebase, this helps debugging it (since you have simple to analyse test cases readily available) and verifying that a fix doesn't accidentally break something else. I guess it's quite common thing to do; Unity3D developers do it, for example.

You could extend this and automate it by creating test harness, which would build and run all test on regular basis, and verify that no regressions happened. If you have lots of test scenarios, this might be feasible, although obviously requires additional investments into infrastructure.

Then there are more "standard" unit tests. One area I've applied them is node-based material system, which can target different rendering techniques (forward, deferred, whatever) and APIs. I approached it by writing unit tests which generate shader code from material and other settings, compiler it and issues a simple draw calls with given inputs (pixel shader inputs). Outputs are written to a render target and fetched for comparison with expected values. IMHO this is a nice thing to do when you have complex material system, esp. with shader code generation.

Finally, more "traditional" approaches can be applied to areas which can be separated from rendering. Things like spatial data structures, asset loading and similar can be unit/integration tested, as applicable in specific case.

Edit: Here's a post from Unity3D developer describing how they test graphics code.

Answer 2

I usually prefer RenderDoc to debug a frame. Gives you enough information on particular frame was rendered right from Vertex to Pixel. There are other equally useful tools available like Nvidia Nsight

Finally, there is always color coding which can be useful when debugging shaders visually. Decomposing shader outputs into separate colors can be of help too. Not sure though if your case requires this.

Hope this helps.

Answer 3

I work mainly on macOS where the OpenGL tools are almost non-existent. (I'm starting to get into Metal where the tools are better.) So I feel your pain.

The first thing I do is make frequent, small commits of my code to source control. This isn't always possible, but usually it is. If that could disrupt others, work on a branch. That way, if something goes horribly wrong, I can do a quick diff of the current code against HEAD or trunk, and isolate the problem to only the code in the diffs.

The next thing I do is try to analyze what the problem is. In your example, where geometry that was previously working is suddenly drawing the wrong shapes, I'd think about the situations that can cause that:

1. Sending a bogus pointer to glBufferData()
2. Starting on the wrong byte of my buffer by setting an attribute pointer/offset incorrectly

There are numerous causes of general symptoms like the dreaded black screen. There are a number of techniques that are the graphics equivalent of printf(). I start breaking down what's working and what's not. I start with really broad strokes. Is my drawing going to the right FBO? Set the clear color of the FBO to something garish like RGB=(1, 0.5, 0) or (0, 1, 1) and see if the screen turns orange or magenta. You might pick a different debug color for each FBO you normally use so you can tell at a glance which is going to screen on a given render. If that works, I check things like clipping planes - have I clipped all the geometry out? Check lighting - do I have lights on? Am I sending the correct lighting info to the shaders? Are textures correct? It can help to have some test textures with things like grids on them, or if you're dealing with cube maps, textures with face names on them. If you have shader issues, you can do things like simply output the texture coordinates as red and green (and blue if it's a 3D texture) at each point to make sure they're valid. Same with normals - normalize them and output the x, y, and z in the r, g, and b channels.

As you do more of this you'll learn tricks like these for displaying the data and verifying your assumptions. Being able to roll back quickly and easily is enormously helpful. And of course, don't forget about glGetError(). The small numbers of errors it returns is frustrating, but it can often point you in the direction of where a problem is.

If you can get a tool that shows you what's different from the default OpenGL state, it can also direct where to spend your effort. On macOS there's OpenGL Profiler. You can set a breakpoint on any OpenGL function and examine the entire state machine. It will highlight differences from the defaults, or differences from the last time it stopped, or both.