0
$\begingroup$

Are there any way to optimize this flow:

  1. The frame captured from camera (NV21 byte array)
  2. Convert this frame to I420 format
  3. Upload Y data to y_tex; U data to u_tex; V data to v_tex
  4. Run the following fragment shader:
...
 float greenValue(vec2 coord)
 {
  return texture2D(y_tex, coord).r 
    - 0.344 * (texture2D(u_tex, coord).r - 0.5) 
    - 0.714 * (texture2D(v_tex, coord).r - 0.5);
 }
 // some code .... if (current pixel is skin color)
 vec2 blurCoordinates[20];
 blurCoordinates[0] = interp_tc.xy + singleStepOffset * vec2(0.0, -10.0);
 blurCoordinates[1] = interp_tc.xy + singleStepOffset * vec2(0.0, 10.0);
 blurCoordinates[2] = interp_tc.xy + singleStepOffset * vec2(-10.0, 0.0);
 blurCoordinates[3] = interp_tc.xy + singleStepOffset * vec2(10.0, 0.0);
 blurCoordinates[4] = interp_tc.xy + singleStepOffset * vec2(5.0, -8.0);
 blurCoordinates[5] = interp_tc.xy + singleStepOffset * vec2(5.0, 8.0);
 blurCoordinates[6] = interp_tc.xy + singleStepOffset * vec2(-5.0, 8.0);
 blurCoordinates[7] = interp_tc.xy + singleStepOffset * vec2(-5.0, -8.0);
 blurCoordinates[8] = interp_tc.xy + singleStepOffset * vec2(8.0, -5.0);
 blurCoordinates[9] = interp_tc.xy + singleStepOffset * vec2(8.0, 5.0);
 blurCoordinates[10] = interp_tc.xy + singleStepOffset * vec2(-8.0, 5.0);
 blurCoordinates[11] = interp_tc.xy + singleStepOffset * vec2(-8.0, -5.0);
 blurCoordinates[12] = interp_tc.xy + singleStepOffset * vec2(0.0, -6.0);
 blurCoordinates[13] = interp_tc.xy + singleStepOffset * vec2(0.0, 6.0);
 blurCoordinates[14] = interp_tc.xy + singleStepOffset * vec2(6.0, 0.0);
 blurCoordinates[15] = interp_tc.xy + singleStepOffset * vec2(-6.0, 0.0);
 blurCoordinates[16] = interp_tc.xy + singleStepOffset * vec2(-4.0, -4.0);
 blurCoordinates[17] = interp_tc.xy + singleStepOffset * vec2(-4.0, 4.0);
 blurCoordinates[18] = interp_tc.xy + singleStepOffset * vec2(4.0, -4.0);
 blurCoordinates[19] = interp_tc.xy + singleStepOffset * vec2(4.0, 4.0);
 // some code ....
 float sampleColor = centralColor.g * 20.0;
 sampleColor += greenValue(blurCoordinates[0]);
 sampleColor += greenValue(blurCoordinates[1]);
 sampleColor += greenValue(blurCoordinates[2]);
 sampleColor += greenValue(blurCoordinates[3]);
 sampleColor += greenValue(blurCoordinates[4]);
 sampleColor += greenValue(blurCoordinates[5]);
 sampleColor += greenValue(blurCoordinates[6]);
 sampleColor += greenValue(blurCoordinates[7]);
 sampleColor += greenValue(blurCoordinates[8]);
 sampleColor += greenValue(blurCoordinates[9]);
 sampleColor += greenValue(blurCoordinates[10]);
 sampleColor += greenValue(blurCoordinates[11]);
 sampleColor += greenValue(blurCoordinates[12]) * 2.0;
 sampleColor += greenValue(blurCoordinates[13]) * 2.0;
 sampleColor += greenValue(blurCoordinates[14]) * 2.0;
 sampleColor += greenValue(blurCoordinates[15]) * 2.0;
 sampleColor += greenValue(blurCoordinates[16]) * 2.0;
 sampleColor += greenValue(blurCoordinates[17]) * 2.0;
 sampleColor += greenValue(blurCoordinates[18]) * 2.0;
 sampleColor += greenValue(blurCoordinates[19]) * 2.0;

Problem:

I tested on weak device (galaxy A6) and capture with HD resolution frame (1280x720). It take ~100ms/frame. The main reason of this bottle neck is the large number of texture2D() call. Each frame make ~60,000,000 call texture2D().

Idea:

When I replace (u_tex, v_tex) by y_tex for test, it's very fast. So, I think performance maybe improved by use only one texture instead of three textures (y_tex, u_tex, v_tex).

Question:

Does my approach is feasible? Are there any way to optimize this shader or improve entire this flow?

$\endgroup$
6
  • $\begingroup$ Can you give us more detail about how your shader works, and how neighboring pixels are interacting with each other? Why are you using these blur coordinates and not raster (n x n) coordinates? $\endgroup$
    – Thomas
    Feb 7 at 11:34
  • $\begingroup$ Storing all three values to one texture will improve your performance, because texture lookups are very slow. So Yes, this is feasible $\endgroup$
    – Thomas
    Feb 7 at 11:42
  • $\begingroup$ Thanks @Thomas, this code in an old project and wrote by an ex-employees. So I don't understand entire algorithm in shader! This is full shader: pqviet.notion.site/6552db82a8e6472e8aad8ff018f250a6 $\endgroup$
    – Viète
    Feb 8 at 2:48
  • $\begingroup$ @Thomas, I think the source of shader got from this: github.com/MasayukiSuda/GPUVideo-android/issues/76 $\endgroup$
    – Viète
    Feb 8 at 3:16
  • $\begingroup$ @Thomas, Because many of the texels fetched by one pixel will also be used by neighbouring pixels, so can I pre-compute and store all result of greenValue(blurCoordinate[i]) to some where like SSBO then reuse the result in SSBO? $\endgroup$
    – Viète
    Feb 8 at 4:30

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.