I'm in the process of writing a Ray Tracer using DirectCompute / HLSL. First, eye rays are generated (one per pixel). Then, rays are traced, shaded and reflected in a loop. Also, shadow rays for each light source are generated and tested for occlusion. As a scene structure, I'm using a KD Tree with Ropes$^1$.
I have analyzed the shader with Joshua Barczak's Pyramid and it seems that I have a major problem with register pressure, mainly vectors. On a Fiji GCN architecture, the shader uses 85 SGPRs and 81 VGPRs, limiting the number of simultaneous wavefronts per SIMD to 3 (VPGR).
Furthermore, I have implemented a simple counter in my shader, that atomically increases a value when a thread starts, decreases it when a thread ends and keeps track of the maximum number of shaders that were concurrently running. I've managed to get rid of some data and bring it up from 8192 to 9216, resulting in a correlated speedup of ~%13. When I use an empty dummy shader, I get 16384 concurrent threads.
I've tried to get rid of excessive variables, especially vectors, by keeping their lifetime as short as possible, and also tried storing some variables in groupshared memory and reading / writing directly from there, all without any change in register count whatsoever.
Are there any practical tips on how to take pressure of the registers? Is this even as much of a problem as I think it is?
I have also thought about splitting my Ray Tracer into various kernels. This should, the way I understand it, take pressure of the registers significantly, but also takes quite some coding effort on my part. Does this sound like an idea worth trying?
$^1$ Popov et al.: Stackless KD-Tree Traversal for High Performance GPU Ray Tracing (2007).