Recently I have been having a rather confusing performance drop with loading in layers of a 2D texture array, but only in one case from what I can tell, leaving me to speculate on why it is happening as I have never seen anything like this before.
Firstly to explain some context for this problem, I implemented a simple version of a virtual texture atlas based on some papers I read by NVIDIA and idTech which in short loads textures (which have been split into fixed size tiles) into various levels of the 2D texture array based on which layers are considered "free" in a CPU-side list of free indices. Sampling in the fragment shader is then preformed by accessing a page table stored as a persistently coherently mapped SSBO (for CPU-side updating) which associates offsets in the UVs with layers of the texture array.
Due to the fact that the texture when created is uninitialized, every layer is considered free, so texture tiles to be loaded will slowly fill up the uninitialized layers (preferring them over previously freed layers). Once all of the uninitialized layers have been used, additional tiles will then begin overwriting previously freed layers, if any exist when needed. Also note that I am using
glCompressedTexSubImage3D to load in each layer and that I load in mipmap levels as well manually per layer with this call as well.
This system works fine and was tested rather extensively until I began to notice the problem which this question is about. In short, after all of the uninitialized layers of the texture array were used up, the next 5-15 (I don't know exactly how many, but it's just a few) texture tiles which are loaded suffer massive performance drops. After that, everything returns to normal and the problem never returns (from what I can tell). Also note that this is while some currently drawn textures were also in use at the lower end of the texture array which is right before where the first free tiles begin (due to the first in first out nature of the free list).
Upon examining the problem further, I noticed through using the NVIDIA Nsight graphics profiling tool that the call after
glCompressedTexSubImage3D in the cases where performance was impacted blocks on the CPU for very long periods of time, usually 10-40ms per mipmap level (in this case it was
glGetError, but any other call following it had the same effect after disabling error checking to ensure that was not the problem).
Since the API was blocking, I assumed that it was some sort of synchronization issue with the GPU perhaps due to the locality of the newly loaded tiles with respect to tiles currently being drawn as it may have considered them "in use". That hypothesis did not add up in my mind due to how long the calls were blocking for as each one was the length of multiple frames at 16ms as well as how this problem never appears to return even after multiple times through this same process, though I'm not ignoring that possibility.
Another odd thing I found was that during the time the next call was blocking, many tiny GPU commands were being executed according to NVIDIA Nsight, which I usually only see under any sort of drawing activity.
At this point I'm pretty much stumped and don't know what could possibly be causing this, so any help would be greatly appreciated. Additional details are provided below (and more can be requested if needed):
Texture Array (there are 3 of them):
- Extents: 256x256x512, 9 mipmap levels
GeForce GTX 960M
NVIDIA Nsight Results:
Wide view of 3 long frames where the slow texture loading happens (the line down the screen is centered on the first texture loading call):
Zoomed in view near the very start of the first slow call to
glCompressedTexSubImage3D (which cannot be seen as it only blocks for a few microseconds), note the many tiny green lines below the blocking
glGetError calls representing GPU commands (as again the line down the screen is centered on the first texture loading call):