6
$\begingroup$

I have been trying to understand display and graphics. I feel that it's hard to make a logical image of the terms. I understand this:

  1. Frame buffer is a dedicated memory that has the image/video/3dimage or video to be shown, in pixel format.

  2. Overlay is some kind of hardware that is used with the frame buffer.

  3. Surfaceflinger is a composition manager. It takes rendered data from the GPU, and composites them in one surface. That means, it uses the underlying hardware and driver to do all this composition.

  4. Composition means blending, mixing of all the surfaces.

  5. Overlay, plane, framebuffer and sprite? What does all this do, or how to combine them and understand?

Any book, links will be appreciated. A lot of books go to graphics way, instead of taking one particular SoC and explain display first then graphics in detail.

$\endgroup$
8
$\begingroup$

As you've understood, the framebuffer is an array in memory that holds all the pixels to display on the screen. On a desktop PC, it's probably special memory on the graphics card, but in a SoC with one memory shared by GPU and CPU, it's probably a normal memory allocation that the display controller uses DMA to read from.

The display controller is a piece of hardware that reads pixels from the frame buffer and displays them on the physical screen.

Many display controllers support the use of an overlay, which is like a very simple window. Sometimes they're fixed-size, but sometimes they're masked off. In this case, then there might be a secondary framebuffer that covers the whole screen. When the display controller reads a pixel of a particular colour from the primary framebuffer, then instead of displaying that pixel, it substitutes the corresponding pixel from the secondary framebuffer. It's like the chroma key or green screen technique used in movie effects and weather forecasts.

That's one particular configuration: sometimes the secondary framebuffer is much smaller than the primary framebuffer, and a particular register in the display controller controls the dimensions and offset.

The overlay technique is commonly used for displaying video streams. The frames of video might be coming from camera hardware or from video decoder hardware. Using an overlay means that this hardware and the GPU that's producing the rest of the screen (such as camera or playback GUI) aren't trying to access the same memory at once. The overlay might also be in a different pixel format to the primary framebuffer.

The overlay technique only lets you do this simple substitution, though (and maybe some scaling), not anything complicated. You can't make the frame of video spin around and shrink to an icon when you're transitioning between different applications. You can't alpha-blend the video with the primary framebuffer. Also, the number of overlays is limited by the hardware: usually one, but some controllers support two.

Composition in this context refers to something the window manager does. It takes the frames from each application on the screen (the system GUI is also an application), and produces the final frame. On a smartphone, this usually means just taking the current foreground app and the system GUI (notification bar and navigation bar) and drawing them next to each other. But when there are transitions between apps, or if you use the "recent tasks" button, the situation is more complicated. The compositor also has to handle choreographing applications so the screen gets updated at 60 Hz even if applications are drawing at different rates.

Surface Flinger is Android's compositor. It uses OpenGL ES to do the things described in the previous paragraph. It's also in charge of creating a framebuffer for each application to render into. (These framebuffers show up as textures inside Surface Flinger, using EGLImage.)

Android also detects when there are simple cases that the display controller on the phone can handle as an overlay, so that it can switch to the faster, more energy-efficient path of using the overlay instead of using Surface Flinger's full GLES-based composition.

$\endgroup$
  • $\begingroup$ Is there any image or picture with all of these components working together? $\endgroup$ – mrigendra Jun 23 '17 at 4:02
  • $\begingroup$ @mrigendra Every frame displayed on your Android phone shows these components working together! Seriously, if you mean a diagram, I don't know of one. $\endgroup$ – Dan Hulme Jun 23 '17 at 8:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.