2
$\begingroup$

This question is motivated by looking at two numbers for different GPU's: (1) the memory size and (2) the memory bandwidth.

My understanding is that memory bandwidth means, the amount of data that can be copied from the system RAM to the GPU RAM (or vice versa) per second.

But looking at typical GPU's, the memory bandwitdh per second is much larger than the memory size: e.g. the Nvidia A100 has memory size 40 or 80 GB, and the memory bandwitdh is 1555 GB/s. This suggests that the GPU can copy its entire memory contents over to system RAM or back within about 1/20th or 1/40th of a second. Maybe this is true, but it just sounds extremely fast.

Improve this question
$\endgroup$
3
  • 3
    $\begingroup$ memory bandwidth is how fast the GPU processor can access its own memory $\endgroup$
    – Esther
    May 26, 2022 at 14:35
  • $\begingroup$ So what exactly does it measure? does it measure the amount of data that can be transferred between the GPU RAM and the registers in the streaming multiprocessors? $\endgroup$
    – user56834
    May 26, 2022 at 15:07
  • 1
    $\begingroup$ It's probably something like: the theoretical max rate at which data can be transferred from the GPU RAM into the GPU's L2 cache (or vice versa). This would set an upper limit on how quickly a rendering pass can run that needs to read a large amount of texture data (more than can fit into L2). BTW, the bandwidth between GPU RAM and system RAM (for discrete GPU setups) is limited by the PCIe link bandwidth, which for PCIe Gen4 x16 is ~32 GB/s theoretical max (in practice somewhat less). $\endgroup$
    – Nathan Reed
    May 26, 2022 at 21:52

2 Answers 2

Reset to default
3
$\begingroup$

Before we begin, it's important to understand that raw spec numbers are typically not meant for use by actual graphics programmers. They are marketing tools to sell products. Sure, they have to be "accurate" (somewhat), but that does not mean that they are meaningful.

As for what bandwidth actually means, consider two parking lots. They can hold a number of cars. Connecting them is a road. This road has a number of lanes. In terms of meaningful "bandwidth," that is the answer to the question, "how many cars can I get from one lot to the other?" In our analogy, parking lots are pools of memory; the road is the connection between them.

So if someone tells you they have a 16 lane highway, does that mean something? Not really, not unless you know what parking lots that highway connects. The same goes for bandwidth; a raw number means nothing without knowing the source and destination memory for that bandwidth.

Marketers sometimes just add bandwidths from completely different memory pools together, as though the sum of lanes from 4 unconnected highways has some actual meaning.

So bandwidth is a measure of the rate of flow between pools of memory. It is the amount of data that can flow from one pool to another in a unit of time. These pools can be general GPU RAM, storage within a compute unit, various caches, etc. Many modern GPUs have memory architectures that connect many of these pools onto a single bus (effectively linking all parking lots by a single set of roads).

The other thing to remember is that pools of memory are not always permanent. Caches, compute storage, and the like are volatile; they get overwritten constantly by design. They exist because accessing bulk storage is slow, so operations that need to access memory quickly need small pools of faster memory closer to their use of this data.

If the bandwidth from GPU memory to a texture cache is 1'555GB/sec, this means that, within a 60fps frame, the total amount of storage that all shaders can access via texture fetches is 25.9GB. You may note that this is much smaller than the 40GB of storage such a GPU may have.

GPU bandwidths are so large per second because GPUs have to use that bandwidth many times per second. Once you look at them per frame, they're not so large.

Also, all bandwidth from one pool to another is purely theoretical. If nobody is reading from memory at any particular cycle, that cycle's worth of bandwidth goes unused. It can't be stored and use later when there's more memory pressure. You either use it or you lose it. And since GPU workloads tend to read memory in spurts, the usable bandwidth is often lower than the numerically available bandwidth.

Improve this answer
$\endgroup$
3
  • $\begingroup$ So do you agree with Nathan reed that "memory bandwidth" is the bandwidth between gpu memory and gpu L2 cache? $\endgroup$
    – user56834
    May 27, 2022 at 7:11
  • 3
    $\begingroup$ @user56834: No. "Memory bandwidth" as a term only makes sense in relation to the connection between two pools of memory. GPUs have lots of pools of memory. So the term "memory bandwidth" with no additional context is at best underspecified and should only be thought of as a general marketing buzzword, not a useful construct. They may have added together many actual bandwidth numbers to compute it. You don't know. $\endgroup$
    – Nicol Bolas
    May 27, 2022 at 13:20
  • $\begingroup$ @user56834: Also, GPUs have many caches. $\endgroup$
    – Nicol Bolas
    May 27, 2022 at 13:20
0
$\begingroup$

Memory Bandwidth is the theoretical maximum amount of data that the bus can handle at any given time, playing a determining role in how quickly a GPU can access and utilize its frame buffer.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.