Before we begin, it's important to understand that raw spec numbers are typically not meant for use by actual graphics programmers. They are marketing tools to sell products. Sure, they have to be "accurate" (somewhat), but that does not mean that they are meaningful.
As for what bandwidth actually means, consider two parking lots. They can hold a number of cars. Connecting them is a road. This road has a number of lanes. In terms of meaningful "bandwidth," that is the answer to the question, "how many cars can I get from one lot to the other?" In our analogy, parking lots are pools of memory; the road is the connection between them.
So if someone tells you they have a 16 lane highway, does that mean something? Not really, not unless you know what parking lots that highway connects. The same goes for bandwidth; a raw number means nothing without knowing the source and destination memory for that bandwidth.
Marketers sometimes just add bandwidths from completely different memory pools together, as though the sum of lanes from 4 unconnected highways has some actual meaning.
So bandwidth is a measure of the rate of flow between pools of memory. It is the amount of data that can flow from one pool to another in a unit of time. These pools can be general GPU RAM, storage within a compute unit, various caches, etc. Many modern GPUs have memory architectures that connect many of these pools onto a single bus (effectively linking all parking lots by a single set of roads).
The other thing to remember is that pools of memory are not always permanent. Caches, compute storage, and the like are volatile; they get overwritten constantly by design. They exist because accessing bulk storage is slow, so operations that need to access memory quickly need small pools of faster memory closer to their use of this data.
If the bandwidth from GPU memory to a texture cache is 1'555GB/sec, this means that, within a 60fps frame, the total amount of storage that all shaders can access via texture fetches is 25.9GB. You may note that this is much smaller than the 40GB of storage such a GPU may have.
GPU bandwidths are so large per second because GPUs have to use that bandwidth many times per second. Once you look at them per frame, they're not so large.
Also, all bandwidth from one pool to another is purely theoretical. If nobody is reading from memory at any particular cycle, that cycle's worth of bandwidth goes unused. It can't be stored and use later when there's more memory pressure. You either use it or you lose it. And since GPU workloads tend to read memory in spurts, the usable bandwidth is often lower than the numerically available bandwidth.