2
$\begingroup$

I've been reading a bit about the graphics pipeline for processing on a graphics card, and I'm interested to know in what kind of format data is passed from the CPU to the GPU when rendering 3D graphics, for example for a video game.

I understood that the CPU would calculate what's actually going on inside the game each frame, for example it might have some internal representation of the different characters on the screen and maybe calculate some collision detection between them, and then it might move them around a bit and output the change to the GPU. Then the GPU must receive some information corresponding to many different triangles in different locations, and it has rasterize them and add textures and then combine them all together into a HD pixel array.

So what would be a kind of schematic overview of the form of the data passed to the GPU each frame? I imagine something like a great big array of sets of three points which respond to the vertices of triangles, with associated textures or pointers to textures or something? I expect there would need to be additional information passed each frame as well as the actual triangles and textures, and I'm interested to know what this would be also.

Also, is there some kind of standard format for this, or does it depend on the hardware?

$\endgroup$

2 Answers 2

3
$\begingroup$

How much data is passed from the CPU to the GPU per frame depends on the engine you are using and its needs. Usually, you want to keep it minimal to avoid that the GPU needs to wait for data because the CPU can't deliver it fast enough. I can not guarantee you, that the following is always the same for all engines and graphic APIs since I have only written OpenGL code myself so far, but to my knowledge, it is quite similar.

What you usually do is that you send all data that doesn't change to the GPU at the beginning. Objects (meshes) and textures are typical candidates for this. Animated meshes like characters can also be loaded in advance by providing a default position.

The object data consists of vertices. They are control points that store the position of each point and an arbitrary set of additional information. A typical additional piece of information might be a color, surface normal, or texture coordinate (this does not specify which texture). Note, that there is no information at this point, how those vertices are connected. This is why I put "meshes" into parenthesis in the previous section.

Now you have a bunch of vertex groups and textures and it is time to render our scene. We assume, that all work that is not related to rendering is done by the CPU. Most of it can also be done by the GPU, but lets keep it simple.

The CPU calculates where every object is positioned in the current frame, including the camera. It also calculates the skeletal state of our animated characters. Now, to render a single object, we need to provide the following information to the GPU:

  • the object data we want to use
  • the textures we want to use
  • connectivity of the vertices
  • the shader program we want to use
  • the data the shader program requires to do its job (usually some transformation matrices and vectors)

The first two are just IDs we got from our API/engine when we uploaded the data. You should know for all your objects, which mesh and texture IDs they use. The third point, connectivity, actually tells the GPU how the single vertices are connected to each other. In OpenGL, there are several ways to specify this. A lot of them assume a specific order in the data, for example that 3 vertices in a row represent a single triangle. But there are also explicit ways by providing an index list to the GPU (This is usually also done at startup).

Now that the GPU knows which data to use and how it is connected, it needs to know what to do with it. This is done by selecting a shader program.

Shaders are small programs that run on the GPU. They have several stages that turn your mesh into actual pixels on your screen (see OpenGL rendering pipeline). You usually compile several of them at the beginning when uploading all the other data. Those programs generally also need some additional information to run and you need to pass that to the GPU. What that data is, strongly depends on the effects you want to use, but something you need very often are the transformations needed to place a vertex in relation to the camera and the projection you want to use (perspective, parallel). Now the shader program can calculate each vertex position on the screen, create triangles from it and color each of its pixel as desired (using the textures you specified). You do this for every object in your scene (that is not culled).

To sum it up:

For a single object in a basic OpenGL engine, you pass the IDs of its vertex data, textures and shader program to the GPU, which are just numbers. Then you pass all data needed by the shader program. This is mostly a bunch of floats for the transformations. Finally, you also tell the GPU how the data is connected for your current "draw call". There is also not much data transfer involved here, since you either have your data ordered in a specific way and need to tell the GPU just which method to use or you have uploaded a connectivity list upfront and tell the GPU the ID of the index list it should use.

Please note, that the described procedure is based on the approach that is described in most tutorials. Modern engines use several optimization techniques to balance CPU and GPU processing power so that they are always busy. This might require more involved data transfer. Also note, that the data transfer itself has bandwidth limitations that need to be considered.

$\endgroup$
2
  • $\begingroup$ Thanks for a great answer. Do you have a link to an explanation of some different ways to specify connectivity? $\endgroup$
    – Jojo
    Aug 10, 2021 at 12:41
  • $\begingroup$ You can check the API description of glDrawArrays. It gives you a list of possible inputs like GL_TRIANGLES or GL_TRIANGLE_STRIP. However, it is not explained there how the data needs to be ordered and how the API uses it. Maybe you should read some simple tutorials like this one and some of the following chapters. It should give you a good idea how the things that I described work. You don't need to code anything. Just read it. ;) $\endgroup$
    – wychmaster
    Aug 10, 2021 at 13:17
1
$\begingroup$

The short answer to this question is: everything

On any give frame almost any kind of data my be needed to render that new and wonderful object that just popped into view. It's shader needs to be loaded and compiled, its textures, buffer data, specializations, push constants, if you can dream it up for a render call then it needs to find its way to the GPU on a per frame basis.

This is the reason GPU's have rich and diverse paths for loading data, along with dedicated hardware and code for handling all the red tape that goes along with it.

What is its format: Again the answer to this question is: everything GPU's handle specialized data types that we don't see on the CPU while doing c/c++/whatever programming. Normalized 8 bit floating point values for example.

For technology demo's and simple scene's much of that data can be preloaded, but for any program that is going to do meaningful work, expect any kind of data to be loaded on a per frame basis.

Don't imagine great big arrays, imagine rich and diverse data. Tutorials will focus on the great big arrays, but they are just the tip of the iceberg.

Finally, don't image the CPU as driving everything. Thanks to compute shaders the GPU is taking more control of the action.

$\endgroup$
8
  • $\begingroup$ I understood from the other answer that textures would be passed to the VRAM at the start rather than each frame. Are you saying they're passed each frame? Or just when a new texture is needed it's passed that frame? $\endgroup$
    – Jojo
    Aug 10, 2021 at 12:44
  • $\begingroup$ Whenever possible the majority of textures are sent early on, but in a large program, the number and size of texture data will far surpass the available memory and texture data will need to be uploaded dynamically. (and textures that are no longer in needed will be removed or overwritten) $\endgroup$
    – pmw1234
    Aug 10, 2021 at 13:30
  • $\begingroup$ But you won't need to transfer that data every single frame. Loading and freeing data can usually be done less frequently, for example when hitting sector boundaries. $\endgroup$
    – wychmaster
    Aug 10, 2021 at 14:45
  • $\begingroup$ Now we are splitting hairs, keeping the hardware transfer units busy and avoiding bottlenecks is an interesting subject. One solution to long start up times is to load only the lower mip levels since they can be brought in exponentially faster, then continuing to load higher mip levels interactively will require a load every frame. This extends nicely to distance objects that get only the low mip levels loaded and higher mip levels are loaded only as the camera passes a certain distance from the object. Using this system textures are continually being loaded. $\endgroup$
    – pmw1234
    Aug 10, 2021 at 15:10
  • $\begingroup$ My answer was in support of the other answer here, with a slightly different take on the subject, the 40 thousand foot view so to speak. $\endgroup$
    – pmw1234
    Aug 10, 2021 at 15:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.