# FLOP - exact definition

I am interested in how much a GPU can actually calculate. When learning about rendering, it seems like a good idea as well.

My Question: WHAT EXACTLY is a Floating Point Operation?

I know what a floating point number is.

Now, when reading other questions and articles about it, they always say, any simple operation, + - * / , would be a floating point operation, which makes sense in terms of the word itself

BUT: A multiplication is (obviously) often "far" more (calculating) than an addition, isn't it?! (Apart from the sheer size of the calculation)

Following, just a simple example for demonstration:

5+5= 5+5

5*5= 5+5+5+5+5

So does the term "flop" actually just describe additions and subtractions or is a flop in the end just a very rough definition? Or is the GPU somehow equally fast at all of those calculations?

Now, when reading other questions and articles about it, they always say, any simple operation, + - * / , would be a floating-point operation, which makes sense in terms of the word itself

This is basically what is meant when talking about floating-point operations although I would exclude the division since it takes much longer than the other operations (see below).

BUT: A multiplication is (obviously) often "far" more (calculating) than an addition, isn't it?! (Apart from the sheer size of the calculation)

Most modern processors can usually calculate additions and multiplications equally fast. Even though the numbers are for CPUs and not for GPUs, you can take a look at the corresponding commands in the Intel intrinsics guide. A GPU shouldn't behave too different in this matter. Here are some examples:

As you can see, the numbers of the Skylake processor are all identical except for the division.

In case you wonder what those numbers mean: Latency is the number of cycles that are necessary to calculate the result and throughput is the inverse number of operations you can start per cycle. So for addition, subtraction and multiplication, a skylake processor can start the calculation of 2 operations per cycle and get the results 4 cycles later. This means that you can actually have 8 of those operations in progress if they do not depend on each other.

Knowing this it is easy to see, that divisions take much longer to compute. This is why you should always calculate the inverse and multiply it instead if you need to divide a bunch of numbers by the same value.

BUT: A multiplication is (obviously) often "far" more (calculating) than an addition, isn't it?! (Apart from the sheer size of the calculation)

Following, just a simple example for demonstration:

5+5= 5+5

5*5= 5+5+5+5+5

Well, this would be very inefficient. If you are curious how a computer actually performs a multiplication efficiently, take a look into this question

• Thank you for the in-depth answer. I though that a PC would maybe calculate multiplications differently, but wasn't sure, that's also good to know. A very helpful answer:) Commented Jan 29, 2021 at 12:18
• " take a look into this question" Ahhh, but that leaves out the joys of "modified booth encoding" and "carry save adders" etc :-) Commented Jan 29, 2021 at 14:00
• And if you want to be impressed take a look at this monster:hpe.com/us/en/newsroom/press-release/2020/03/… Commented Jan 29, 2021 at 15:11

Just to add to whychmaster's reply, in my experience, on a GPU, a floating-point operation, when used in quoting FLOPS (floating-point operations per second) benchmark figures will refer to Addition (equiv subtraction) and Multiplication.

The other standard IEEE operators, division + sqrt, are more computationally expensive and so unlikely to be included in the figure. (Note that, there is often HW support in GPUs for accelerating operations like 1/x, 1/sqrt(x) etc, perhaps to make them as (or nearly as) fast as muls & adds)

Having said that, it is very likely that a GPU will actually implement a Fused Multipy-Add operation, FMAD, i.e. do A * B + C in "one step" as this is

1. more accurate than doing separate multiply and add instructions
2. is faster than separate ops
3. is about the same silicon cost
4. is a very common building block of CG operations (e.g dot products, convolutions)

I suspect that in quoted figures, though, an FMAD would be considered 2 operations.

I spent many years doing hardware work creating CPU's for the "big iron" market. We considered a FLOP to be any instruction that requires the "floating point execution unit" inside the CPU to perform the calculation that another instruction has a dependency on. This unit used to be considered one of the most complex, slowest, and one of the most power hungry units inside the CPU. It could result in long stalls, lots of NOPS being inserted into the pipeline, lots of heat being generated, and the electric bill to go sky high.

Much research later, the floating point units are much less power hungry, and can often be compared to the performance of a good integer execution unit. (last I heard research was combining the two into a flexible integer/floating unit)

This all lead to measuring how many FLOPS a CPU (and nowadays the GPU) can perform in any given second, and more generally an entire system with many CPU's and GPU's working together. This measurement is still relevant because GPU's have turned number crunching into a fine art and while throughput is often dependent on memory architecture the raw number of FLOPs still tends to be a good measurement but can be very workflow dependent.