Dealing with wasted space when using SIMD on CPU for vector maths

Question

I've been getting interested in SIMD programming on CPU using SSE recently, but I'm a complete noob on the subject. I found this article describing how to make an efficient float3 type using the recent __vectorcall convention, and it made me wonder how best to deal with the wasted space inherent in using a 128-bit type to represent vectors smaller than 4.

For a typical use case, you might want to transform a bunch of positions and normals on the CPU in order to do a batched draw call. When transmitting the data over to the GPU, though, ideally you'd want this as tightly packed arrays of float3, either interleaved or in separate buffers.

It seems like there are two options here:

A - store the data tightly packed, and unpack and repack to and from SSE-friendly format when you need to manipulate it, taking the CPU performance hit, or

B - store the data loosely packed on 16-byte boundaries, SSE it directly, and take the bandwidth and VRAM hit when sending to GPU.

Which of these options would be preferred in a real-world 3D engine, or under what circumstances would you prefer one to the other?

Answer 1

There's a third option you've neglected. Because of how graphics APIs work, typically you have to copy the data between the CPU and GPU anyway - even if you're on a mobile SoC where they share the same physical memory. This copy is the ideal time to take the hit of changing format, since you have to touch all of the memory anyway. So the copy in user memory, owned by the CPU, can be stored padded for SIMD alignment. Then, when you need to send that data to GPU memory, instead of doing a memcpy, just read in the 4-vectors and write out 3-vectors. It's less memory bandwidth than a normal copy (you're reading just as much, but writing less) and negligible CPU work.

If you've got that fourth float doing nothing, you might as well use it and make your co-ordinates homogeneous instead of just xyz. Depending on what compute you're doing on the CPU, that might make some things easier (transforms are the obvious example, but also reducing divisions if you were renormalising your normals a lot before).

This is similar but not identical to a trick I used when implementing OpenGL texture upload. On ARM, most Neon instructions operate at memory bandwidth rate, so if you're already doing a memcpy, it's free to do a pipeline-worth of instructions at the same time. You can use those free instructions to do pixel format conversion and reordering to Z-order or U-order.

Answer 2

While the existing answer adress your question quite well, I'd like to add some general advice, even if you might already be aware of that. Rather than just wasting that fourth component on the GPU, you might as well use it for additional data.

It might be an obvious suggestion, but a 4th unused 32-bit word can store a variety of useful data:

• 8-bit RGBA colour components.
• 2D texture coordinates in either 16-bit half-precision floating point or normalized fixed-point format, depending on your precision and range requirements.
• packed normal data using some of the excellent normal compression formats discussed in this survey of unit vector representations.

This would let you use packed 128-bit SSE copy instructions for uploading data (especially since mapped buffer memory is usally well aligned) after just a bit of (easily SSE-optimizable) packing. Of course the viablity of this highly depends on your data. But if you have e.g. 5 floats for position and texCoord or 6 for position and normal, trying one of these schemes might be a neat consideration.

And using 4-component vectors for your vertices on the GPU can have additional advantages beyond just SSEability on the CPU.

• It might help your GPU optimize on-chip memory reads/writes if your data is well aligned (although, of course it has to be measured individually if there's a practical advantage).
• But even disregarding performance it can make development a little more convenient, too. Speaking from OpenGL experience, using 3-component vertex data in shader storage buffers (i.e. for advanced memory access and/or compute shaders) is a bit of a hassle because of GLSL's data layout. A buffer filled with 4-component vectors makes for generally cleaner and more streamlined code.

Answer 3

The conversion from option A is likely faster than you think. See, for example, Intel's version which claims to do the round trip at 1.5 cycles per vector with AVX.

There are also some operations which don't require the conversion at all. For example, if you want to compute a bounding box, you can do it directly with the packed data in chunks of 4 vectors and fix things up once you've gone through the entire array.

And finally a word of warning: a vector class as presented in the article is not how you write high performance code nor how you generally should use SIMD.