Best practices for VRAM management in dynamic environments (e.g. games)

Question

While making a GPU Marching Cubes implementation a while back, I ran into the issue of OpenGL holding on to memory I had asked it to release. There are a lot of threads on this over at Stack Overflow, and the gist seems to be that you simply can't rely on the API to release VRAM in a timely manner. In my app, I was trying to be smart, releasing all temp buffers used in mesh creation once I was done with them, and allocating an exact-sized vertex buffer each time the mesh changed. This was counterproductive, and rapidly led to out-of-memory crashes.

In my case, since it was just a toy app for a uni project, I solved it by allocating worst-case sized buffers and leaving them allocated, recycling buffer space wherever possible for different passes during the mesh extraction process. This was also a bit of an edge-case, since buffers were pretty huge (sometimes hundreds of MB) so out of memory errors appeared quickly.

This is fine for a strictly limited use case like mine, but my question is, how is VRAM handled in environments like games, where large numbers of meshes and textures need to be loaded and unloaded dynamically? I'm thinking particularly of open-world games like GTA where the world is constantly being streamed-in as the player moves around, but even in more linear games, loading a new level would need a lot of unloading and reloading assets.

Are assets generally loaded to sections of large fixed buffers and space usage tracked somehow? Or if using a large number of smallish buffers, can the APIs keep up with a few of them being created and destroyed regularly? Can anyone with experience in engine design shed some light on how this is done in real world usage?

Answer 1

For consoles (and D3D12/Metal/Vulkan), you know everything you need to know to develop an allocation strategy. You know:

1. How much memory, of each kind, is available.

2. How big each kind of resource will be. Textures of a particular format and size take up X bytes, require Y alignment, etc.

Given such information, you can devise an allocation strategy for data uploading. Open-world games use streaming of fixed-sized blocks of data. Each sector of the game contains resources that take up X number of bytes, and therefore, artists have X number of bytes of storage to work with. It's their job to fit all of the stuff needed for that region of the game into that much space. There will also be stream blocks for larger collections of regions, or other data that gets shared between all game regions.

For other games, some of them load everything at level-load time. Some do hard-loads in the middle of "levels" (see many Source-engine games). Others stream things more dynamically, but even there, artists are restricted to using resources that total up to X number of bytes.

On consoles, such sizes can be known a priori, but with Vulkan/Metal/D3D12, these have to be queried for a particular piece of hardware. So in the latter case, you have to be more reactive to what's available, but all of the information is available to the user.

For other APIs/platforms (OpenGL/D3D pre-12), things are rather more complicated. Consoles and the lower-level APIs give you direct access to memory allocations, which gives you more flexibility. You can use a piece of memory for vertex data at one point, then put an image into it later on. Memory is divorced from its current use, so it's much easier for an application to choose how to recycle memory.

For GL/D3D-pre-12, things are difficult. A buffer object is a buffer object, and the memory currently associated with it cannot later be used as a texture. Well, it can, but you cannot explicitly ask that to be done. The most you can do is deallocate the buffer, allocate the image, and hope that the buffer's memory was recycled.

For cases where you're doing dynamic loading in fixed-sized blocks, you basically have to give your artists more limitations. Instead of limiting the artists by memory, you have to spell out the limitations by object So you'd say that a stream block can have "X numbers of textures of Y formats and Z size, and W bytes of vertex data". Artists can't sacrifice 4 textures to allow one texture to be 4 times the size, or sacrifice a texture to get more vertex data, or sacrifice some vertex data to get more texture space.

That is what you have to do if you don't trust the implementation's memory allocator to effectively recycle resources. For hard-load cases, it's generally OK to just delete all of the objects and expect the implementation to recycle effectively. But even here, it's a good idea to make sure that you don't try to allocate all of the available storage; leave the implementation a healthy percentage for fragmentation and so forth.

Of course, these kinds of things are a big part of the reason why IHVs have application-specific optimizations: they know up-front which resources were allocated for which purpose, and the driver makers can design allocation schemes that work with these assumptions.

Which is why Vulkan et. al exist: so that application developers don't have to rely on IHVs to work around problems in the API.