4
$\begingroup$

I am grabbing video from a web cam using Microsoft Media Foundation. The image data is in YUY2 format. The pixels are packed in a 4:2:2 ratio. The color format is YCbCr. The color data gets repeated 2x. Ex the packed pixel data is:

y0 u y1 v

Gets unpacked to 2 pixels:

y0 u v and y1 u v

I want to use this data as an OpenGL texture. The question is: is there a way to pass this type of texture to glTexImage2D as-is? Or do I need to unpack it first? I would like you keep the YCbCr color and convert to RGBA in a shader.

(I know I can use OpenCV to get video - I've already done that. I am trying a more low level solution mostly as a learning experience).

Improve this question
$\endgroup$
4
  • $\begingroup$ Normally you would use YUV semi-planar for this, which is a seperate single channel Y texture and a packed UV channel. You could also unpack inside a fragment shader by testing if the Y texel you are reading is on a odd or even column. So say you read Y0,U for the first pixel (Assuming a 2 channel texture), you test if Y0 is on a even-column, which means you need to read the texel to right to get the missing V. For odd-columns you would read the texel to the left to get the missing U. $\endgroup$
    – PaulHK
    Jul 21, 2016 at 4:36
  • $\begingroup$ Just did a little googling, you want NV12 pixel format in Microsoft Media Foundation if you want the semi-planar format. Should be the fastest format for doing colour space conversion on the GPU. $\endgroup$
    – PaulHK
    Jul 21, 2016 at 9:04
  • $\begingroup$ @PaulHK The camera only supports YUY2 and MJPG formats. I tried unpacking in the shader - the results were close to correct but there were vertical lines. I assume this is because I am incorrectly fetching the next pixel: texture(tex, vec2(Texcoord.x + 1, Texcoord.y)); I'm guessing this is due to scaling. $\endgroup$
    – 001
    Jul 22, 2016 at 19:57
  • $\begingroup$ It might be simpler to deinterleave it into 2 buffers on the CPU ( Y & CrCb ), then the GPU can do CSC. $\endgroup$
    – PaulHK
    Jul 25, 2016 at 2:25

1 Answer 1

Reset to default
3
$\begingroup$

The question is: is there a way to pass this type of texture to glTexImage2D as-is?

Yes; you can just pass it as-is, but (looking at the docs) it is not supported directly. You'll need a custom shader to interpret it.

Depending on your needs, expanding the pixels first (or even doing the full conversion CPU-side) may be worthwhile. At a guess, I'd say that expanding the pixels (as you copy from the camera's driver buffer into a staging buffer for upload to the GPU) is a pretty consistently good idea, unless you're extremely bandwidth-saturated. I'd add the color-space conversion for easier debugging, or if the image doesn't update frequently.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Yes, I've decided it's best to unpack the data on the CPU (a task that lends itself well to multi-threading), then do color conversion in a shader. $\endgroup$
    – 001
    Jul 27, 2016 at 14:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.