# Direct3D Global Atomic Issue

Recently I ran into an odd issue with some of my DirectX11 DirectCompute code when trying it on an AMD GPU for the first time where as on a NVIDIA one it had worked fine all through development. The fact that this only appeared on AMD either means that NVIDIA GPUs are more resistant to the issue or perhaps something about them such as a different wavefront size is not triggering the issue, or that it's not even an problem with my code and instead is a bug somewhere along the pipeline (driver, GPU itself, etc) for AMD things. After further testing however, it did appear to work on another more modern AMD card (the Vega 56), though this may be because it was running on Windows 7, or because the GCN 5 architecture (or its drivers) somehow fixed this issue.

A summary of the GPUs tested is as follows:

• NVIDIA GeForce GTX 960, Feature Set 11.0/11.1, Windows 10 (Ok)
• NVIDIA GeForce GTX 1060, Feature Set 11.1, Windows 10 (Ok)
• Radeon RX 580, Feature Set 11.0/11.1, Windows 10 (Fail)
• Radeon RX Vega 56, Feature Set 11.0, Windows 7 (Ok)

The general setup of the issue can be represented by the following pseudocode which I turned into a minimal test application to confirms the problem indeed exists to ensure nothing else in the other much larger codebase was causing the issues:

if (condition) {
globalArray[i].first = value
} else {
globalArray[i].second = value
}

DeviceMemoryBarrier()

InterlockedExchange(globalArray[i].flag, true, oldFlag)

if (!oldFlag) {
return
}

if (condition) {
new_value = globalArray[i].second
} else {
new_value = globalArray[i].first
}


With this setup, an example of its functionality would be best represented by 2 threads executing this code (the important bit being that they may not be within the same work group, more on this later), one with condition set to true and one with it set to false. Next, each thread sets the value they possess into a global buffer (both threads have the same index i) in the first and second slots of a structure. The interlocked exchange operation which follows attempts to set the flag (also stored in global memory) to true, allowing only one of the two threads to proceed (one will see that the flag was not set beforehand, meaning it was the first to set it and return). Finally, the remaining thread can now make the assumption that both first and second values have been set (as the other now terminated thread has already set the atomic along with the global memory barrier which should ensure this), allowing it to read the value written by the other thread to new_value (based on the condition once again).

From a more higher level perspective, this code simply allows two arbitrary threads to combine their information into one which continues on, all through global memory.

The issue I observed however was that very rarely (only seen in about 1 in a million cases), new_value would be set to something uninitialized, implying that the data it read from in the global array was. This however is confusing as the fact that the atomic flag is set should ensure that the value has been written to the proper location by the other thread. I thought perhaps that the atomic operation finished before the full write to the first/second value did for whatever reason but since there is a DeviceMemoryBarrier after the value write but before the atomic exchange, it should be waiting on the write to finish before signaling the atomic.

This behavior leads me to believe that it is something bizarre which is happening so rarely due to unlikely circumstances, such as threads from different wavefronts running into eachother at the exact same time as mentioned previously which should in theory work fine, but something such as caching preventing the writes from being seen properly. This further is strengthened by my testing which purposely collides threads together from random wavefronts, causing the issue to arise most frequently when the number of values to process is around 100,000 whereas counts too small have no effect and counts well into the millions do not either, likely as fewer threads are colliding at the proper exact time due to it being much more unlikely (even still, it sometimes takes 100+ iterations of the same test repeated to invoke the issue with 100,000 values). Most of that is simply speculation though since I do not know too much about the specifics of how GPU memory hierarchies function. Another possibility is that this kind of behavior simply isn't guaranteed by DirectCompute and I am using it incorrectly, though DirectX's documentation is very lacking on actual in-depth descriptions of functionality for intrinsics like InterlockedExchange, so I am not sure what to think.

Does anyone know why this might happen and what I can do to fix it? If it'd help, I can also make the proof of concept test application I made available to show a more specific example of the problem occurring (since the pseudocode did leave out a lot of "irrelevant" details which may actually be significant).

### Update:

Upon more investigation, I decided to implement this same setup in OpenCL 1.2 to see if the issue persisted there as well. To my surprise, it did not and after much testing I seem to have narrowed down the potential cause. From what I can see, the mem_fence(CLK_GLOBAL_MEM_FENCE) operation in OpenCL appears to generate slightly different ISA compared to DeviceMemoryBarrier(), specifically both are the same (as far as I can tell, though I am not an expert at reading GPU ISA) except for the presence of a BUFFER_WBINVL1_VOL instruction which seems to be consistent with how such a construct is described here.

Removing the memory fence from the working OpenCL implementation causes it to exhibit the same issue as the DirectCompute implementation which leads me to believe the presence of this instruction is somehow responsible for its success, which does make sense as something relating to cache does seem able to be responsible for such odd and rare behavior. This seems to me that it is an issue with how DXBC is converted to the AMD ISA if such an instruction is indeed required, but that is also just speculation. I will post the full ISA of the test implementations here soon-ish as well in case I have misinterpreted what is going on.