FLOP - exact definition

Question

I am interested in how much a GPU can actually calculate. When learning about rendering, it seems like a good idea as well.

My Question: WHAT EXACTLY is a Floating Point Operation?

I know what a floating point number is.

Now, when reading other questions and articles about it, they always say, any simple operation, + - * / , would be a floating point operation, which makes sense in terms of the word itself

BUT: A multiplication is (obviously) often "far" more (calculating) than an addition, isn't it?! (Apart from the sheer size of the calculation)

Following, just a simple example for demonstration:

5+5= 5+5

5*5= 5+5+5+5+5

So does the term "flop" actually just describe additions and subtractions or is a flop in the end just a very rough definition? Or is the GPU somehow equally fast at all of those calculations?

Thanks in advance.

Answer 1

Now, when reading other questions and articles about it, they always say, any simple operation, + - * / , would be a floating-point operation, which makes sense in terms of the word itself

This is basically what is meant when talking about floating-point operations although I would exclude the division since it takes much longer than the other operations (see below).

BUT: A multiplication is (obviously) often "far" more (calculating) than an addition, isn't it?! (Apart from the sheer size of the calculation)

Most modern processors can usually calculate additions and multiplications equally fast. Even though the numbers are for CPUs and not for GPUs, you can take a look at the corresponding commands in the Intel intrinsics guide. A GPU shouldn't behave too different in this matter. Here are some examples:

• addition
• subtraction
• multiplication
• division

As you can see, the numbers of the Skylake processor are all identical except for the division.

In case you wonder what those numbers mean: Latency is the number of cycles that are necessary to calculate the result and throughput is the inverse number of operations you can start per cycle. So for addition, subtraction and multiplication, a skylake processor can start the calculation of 2 operations per cycle and get the results 4 cycles later. This means that you can actually have 8 of those operations in progress if they do not depend on each other.

Knowing this it is easy to see, that divisions take much longer to compute. This is why you should always calculate the inverse and multiply it instead if you need to divide a bunch of numbers by the same value.

BUT: A multiplication is (obviously) often "far" more (calculating) than an addition, isn't it?! (Apart from the sheer size of the calculation)

Following, just a simple example for demonstration:

5+5= 5+5

5*5= 5+5+5+5+5

Well, this would be very inefficient. If you are curious how a computer actually performs a multiplication efficiently, take a look into this question

Answer 2

Just to add to whychmaster's reply, in my experience, on a GPU, a floating-point operation, when used in quoting FLOPS (floating-point operations per second) benchmark figures will refer to Addition (equiv subtraction) and Multiplication.

The other standard IEEE operators, division + sqrt, are more computationally expensive and so unlikely to be included in the figure. (Note that, there is often HW support in GPUs for accelerating operations like 1/x, 1/sqrt(x) etc, perhaps to make them as (or nearly as) fast as muls & adds)

Having said that, it is very likely that a GPU will actually implement a Fused Multipy-Add operation, FMAD, i.e. do A * B + C in "one step" as this is

1. more accurate than doing separate multiply and add instructions
2. is faster than separate ops
3. is about the same silicon cost
4. is a very common building block of CG operations (e.g dot products, convolutions)

I suspect that in quoted figures, though, an FMAD would be considered 2 operations.

Answer 3

I spent many years doing hardware work creating CPU's for the "big iron" market. We considered a FLOP to be any instruction that requires the "floating point execution unit" inside the CPU to perform the calculation that another instruction has a dependency on. This unit used to be considered one of the most complex, slowest, and one of the most power hungry units inside the CPU. It could result in long stalls, lots of NOPS being inserted into the pipeline, lots of heat being generated, and the electric bill to go sky high.

Much research later, the floating point units are much less power hungry, and can often be compared to the performance of a good integer execution unit. (last I heard research was combining the two into a flexible integer/floating unit)

This all lead to measuring how many FLOPS a CPU (and nowadays the GPU) can perform in any given second, and more generally an entire system with many CPU's and GPU's working together. This measurement is still relevant because GPU's have turned number crunching into a fine art and while throughput is often dependent on memory architecture the raw number of FLOPs still tends to be a good measurement but can be very workflow dependent.