I am grabbing video from a web cam using Microsoft Media Foundation. The image data is in YUY2 format. The pixels are packed in a 4:2:2 ratio. The color format is YCbCr. The color data gets repeated 2x. Ex the packed pixel data is:

y0 u y1 v

Gets unpacked to 2 pixels:

y0 u v and y1 u v

I want to use this data as an OpenGL texture. The question is: is there a way to pass this type of texture to glTexImage2D as-is? Or do I need to unpack it first? I would like you keep the YCbCr color and convert to RGBA in a shader.

(I know I can use OpenCV to get video - I've already done that. I am trying a more low level solution mostly as a learning experience).

  • $\begingroup$ Normally you would use YUV semi-planar for this, which is a seperate single channel Y texture and a packed UV channel. You could also unpack inside a fragment shader by testing if the Y texel you are reading is on a odd or even column. So say you read Y0,U for the first pixel (Assuming a 2 channel texture), you test if Y0 is on a even-column, which means you need to read the texel to right to get the missing V. For odd-columns you would read the texel to the left to get the missing U. $\endgroup$ – PaulHK Jul 21 '16 at 4:36
  • $\begingroup$ Just did a little googling, you want NV12 pixel format in Microsoft Media Foundation if you want the semi-planar format. Should be the fastest format for doing colour space conversion on the GPU. $\endgroup$ – PaulHK Jul 21 '16 at 9:04
  • $\begingroup$ @PaulHK The camera only supports YUY2 and MJPG formats. I tried unpacking in the shader - the results were close to correct but there were vertical lines. I assume this is because I am incorrectly fetching the next pixel: texture(tex, vec2(Texcoord.x + 1, Texcoord.y)); I'm guessing this is due to scaling. $\endgroup$ – Johnny Mopp Jul 22 '16 at 19:57
  • $\begingroup$ It might be simpler to deinterleave it into 2 buffers on the CPU ( Y & CrCb ), then the GPU can do CSC. $\endgroup$ – PaulHK Jul 25 '16 at 2:25

The question is: is there a way to pass this type of texture to glTexImage2D as-is?

Yes; you can just pass it as-is, but (looking at the docs) it is not supported directly. You'll need a custom shader to interpret it.

Depending on your needs, expanding the pixels first (or even doing the full conversion CPU-side) may be worthwhile. At a guess, I'd say that expanding the pixels (as you copy from the camera's driver buffer into a staging buffer for upload to the GPU) is a pretty consistently good idea, unless you're extremely bandwidth-saturated. I'd add the color-space conversion for easier debugging, or if the image doesn't update frequently.

  • 1
    $\begingroup$ Yes, I've decided it's best to unpack the data on the CPU (a task that lends itself well to multi-threading), then do color conversion in a shader. $\endgroup$ – Johnny Mopp Jul 27 '16 at 14:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.