Convert RGB image to YUV420 using a hlsl shader

Question

I am new to shaders and I need to convert efficiently an RGB image to YUV420.

I can convert the RGB image to YUV420 with a simple c++ program that uses the

long length = m_width * m_height;

long yuv420FrameLength = (float)length * 1.5;
unsigned char *yuvdest = (unsigned char *)malloc(yuv420FrameLength * sizeof(unsigned char));

int r, g, b, y, u, v, ypos, upos, vpos;

for (int j = 0; j < m_height; ++j) {
    for (int i = 0; i < m_width; ++i) {
        r = (int)m_pRGBImage[(j * m_width + i) * 3 + 0];
        g = (int)m_pRGBImage[(j * m_width + i) * 3 + 1];
        b = (int)m_pRGBImage[(j * m_width + i) * 3 + 2];

        y = ((66 * r + 129 * g + 25 * b + 128) >> 8) + 16;
        u = ((-38 * r - 74 * g + 112 * b + 128) >> 8) + 128;
        v = ((112 * r - 94 * g - 18 * b + 128) >> 8) + 128;
        ypos = j * m_width + i;
        upos = (j / 2) * (m_width / 2) + (i / 2) + length;
        vpos = (j / 2) * (m_width / 2) + (i / 2) + length + (length / 4);

        yuvdest[ypos] = y;
        yuvdest[upos] = u;
        yuvdest[vpos] = v;
    }
}
std::ofstream myfile;
myfile.open(pPath, std::ios_base::trunc);
myfile << yuvdest;
myfile.close();

I can transform the pixels and create the YUV with the following shader code (I did not test it, if I'm wrong, please tell me).

sampler2D sampler0            : register(s0);

struct VS_INPUT
{
  float4 Position   : POSITION;
  float2 Tex0       : TEXCOORD0;
};

struct VS_OUTPUT
{
  float4 Position   : POSITION;
  float2 Tex0       : TEXCOORD0;
};

struct PS_OUTPUT
{
  float4 Color   : COLOR0;
};

PS_OUTPUT psmain(in VS_OUTPUT In)
{
  PS_OUTPUT Out;
  Out.Color = tex2D(sampler0, In.Tex0);
  int r = Out.Color.x;
  int g = Out.Color.y;
  int b = Out.Color.z;
  float y = ((66 * r + 129 * g + 25 * b + 128) >> 8) + 16;
  float u = ((-38 * r - 74 * g + 112 * b + 128) >> 8) + 128;
  float v = ((112 * r - 94 * g - 18 * b + 128) >> 8) + 128;;
  return Out;
};

VS_OUTPUT vsmain(in VS_INPUT In)
{
  VS_OUTPUT Out;
  Out.Position = In.Position;
  Out.Tex0 = In.Tex0;
  return Out;
};

The thing that is missing to me is how to arrange the bytes formation correctly as the C++ code:

ypos = j * m_width + i;
upos = (j / 2) * (m_width / 2) + (i / 2) + length;
vpos = (j / 2) * (m_width / 2) + (i / 2) + length + (length / 4);

Answer 1

The problem here is that you're not storing the YUV values of a pixel at the same place in the result image. What you seem to be doing is first storing the full size Y image and after that the quarter-sized U image and then the quarter-sized V image after that (also with a strange gap of half the image size between the Y and the UV data).

This doesn't work when trying to map it to a standard pixel shader approach, since a pixel shader outputs the value of exactly 1 pixel in the output image, but you actually would need to write 3 different scattered pixels in the output image, at least as your image data is currently laid out in memory.

So there's a few ways you could get around that:

• You could output the YUV values directly into a full-sized YUV image (with the Y, u, and v values of a single pixel stored together in the RGB channels of the output) and then further compress this into the disparate memory layout you need, either on the CPU or in another pixel shader that just decides what channels to read from the Yuv image based on the pixel position inside of the output image.

• You could combine the above two shaders into one, doing either the Y, the U or the V computation, based on which quarter of the output image you're in.

• You can switch to a different layout for your YUV data, e.g. storing the Yuv values of a 2x2 pixel block of the input image in one compressed pixel of the output image. Then it might actually be useful to write into an unsigned integer image and do the converstion to bytes and the compression yourself in the shader.

• You can use your C++ algorithm and write the data in a scattered fashion, but not using a pixel shader rather than a compute shader and writing into a generalized buffer/image instead (or combining the two and writing into a generalized buffer/image rather than the frambuffer directly from the pixel shader).

However, I left the above explanations decidedly vague and general for further studying, since there are a few more direct practical mistakes in your actual shader code (e.g. Out.Color should contain your output values, you're also computing with float, so they're likely in the [0,1] range and you shouldn't do all these integer operations on them), which indicate that you might first want to delve a little deeper into how shaders and HLSL work.