Compute shader only 5 times faster on the GPU than on the CPU

Question

I'm running following compute shader in Vulkan. worldImage is device local memory that holds the world. resultImage is reference to the memory of the swapchain image:

#version 450

layout (binding = 0, rgba8) uniform writeonly image2D resultImage;
layout (binding = 1, r16ui) uniform readonly uimage3D worldImage;

layout(push_constant) uniform PushConstants {
    vec3 rayOrigin;
    float viewGridTopLeftCornerX;

    vec3 viewGridRight;
    float viewGridTopLeftCornerY;

    vec3 viewGridDown;
    float viewGridTopLeftCornerZ;

    float viewGridDensity;
} push;


void main() {
    vec3 viewGridTopLeftCorner = vec3(push.viewGridTopLeftCornerX, push.viewGridTopLeftCornerY, push.viewGridTopLeftCornerZ);
    vec3 samplePoint = viewGridTopLeftCorner + push.viewGridDensity * (gl_GlobalInvocationID.x * push.viewGridRight + gl_GlobalInvocationID.y * push.viewGridDown);

    vec3 ray = normalize(samplePoint - push.rayOrigin);
    vec3 inverseRay = 1 / ray;

    ivec3 voxelCoords = ivec3(floor(push.rayOrigin.x), floor(push.rayOrigin.y), floor(push.rayOrigin.z));
    ivec3 rayOrientation = ivec3(sign(ray));
    ivec3 rayPositivity = (1 + rayOrientation) >> 1;
    vec3 withinVoxelCoords = push.rayOrigin - voxelCoords;

    uint blockID = 0;
    int smallestComponentIndex;
    do {
        // Calculate how far each of the neighbouring voxels are
        vec3 distanceFactor = (rayPositivity - withinVoxelCoords) * inverseRay;

        // Pinpoint the closest voxel of the three candidates
        smallestComponentIndex = distanceFactor.x < distanceFactor.y
        ? (distanceFactor.x < distanceFactor.z ? 0 : 2)
        : (distanceFactor.y < distanceFactor.z ? 1 : 2);

        // Move to that voxel
        voxelCoords[smallestComponentIndex] += rayOrientation[smallestComponentIndex];

        // Advance the ray the distance to the closest voxel in all dimensions
        withinVoxelCoords += ray * distanceFactor[smallestComponentIndex];

        // The axis towards the next voxel will now have value -1 or 1. Next line resets it to 0.
        // It can be imageined as going jumping the side of the previus voxel to the side of the next one.
        withinVoxelCoords[smallestComponentIndex] = 1 - rayPositivity[smallestComponentIndex];

        blockID = imageLoad(worldImage, voxelCoords).r;
    } while (blockID == 0);

    switch (blockID) {
        case 1:
            vec3 normal = vec3(0, 0, 0);
            normal[smallestComponentIndex] = rayOrientation[smallestComponentIndex];
            imageStore(resultImage, ivec2(gl_GlobalInvocationID.xy), vec4(0.0, 1.0 * dot(ray, normal), 0.0, 1.0));
            break;
        case 2:
            imageStore(resultImage, ivec2(gl_GlobalInvocationID.xy), vec4(1.0, 0.0, 0.0, 1.0));
            break;
        default:
            imageStore(resultImage, ivec2(gl_GlobalInvocationID.xy), vec4(0.0, 0.0, 1.0, 1.0));
    }
}

The code traces one ray per pixel until it hits a block (value 1) or the edge (value 2, every edge block has the value 2).

If the parameters are:

• 32x32x32 world with player standing in the middle (so ray tracing itself goes through half of the voxels at worst)
• 1280x720 resolution (gl_GlobalInvocationID x and y)

It takes about 50ms to render one frame on GTX 1050 or about 5ms on RTX 2080 Ti.

The main loop acquires the image, submits the command buffer which transitions the image to proper (general) layout, binds the pipeline and descriptor sets, pushes the push_constants, dispatches the compute shader and transitions image to present layout again. This is what happens in those 50ms/5ms.

I ran similar code on a single CPU core (i5 4760K) and it takes about 250ms. This would be equivalent to running just the shader code though.

I can't believe that something like this would be only five times faster on the GPU than on the CPU.

I have no experience in graphics, so I can't really tell. Is this really the time it takes for this, or could I expect (hopefully a much) better performance, once I fix whatever is bogging this down?

Could it be the memory access on the GPU? I don't know what else it could be.