Vulkan Queue (Family), Command (pool) and Hardware usage

Question

I have some questions about the GPU usage of Queues / Queue families, command buffer / pool. I am reading the Book: Vulkan Programming Guide: The Official Guide to Learning Vulkan. What I don't get is how the usage of the GPU is based on the listed items:

How I understand how it works: There are Queue families (my hardware has 3 families) which can handle different things. There I should use the family with the minimum support with which my program is able to work. So in case not doing graphics, I should use a family without graphics support. That is fine! Using the same resources between queues within the same family has no negative effect on performance, except synchronization.

Question 1: Does each Queue family has the full excess of the GPU? Meaning when using just one Queue, does the GPU can work with 100% usage?

Question 2: Does it make sense to use 2 Queues from different families (graphics and transfer) in case the task is to render a scenario (queue graphics) and transfer the finished image to the CPU to save it as a file (queue transfer)? I am asking that because my graphics queue family also has a transfer bit. So maybe it is better to use just this one graphics queue.

Question 3: The book says that the command buffers belong to a command pool. What is the difference when using 1 pool with several buffers instead of using several pools with each having 1 buffer? Is it only because of the barriers? Or do I have performance issues or access problems when using several pools? It would be nice if someone could write a bit about the pools and buffers (I miss that in the book).

Question 4: I don't think there is a general answer for this question, but maybe some rule of thumb? When using Vulkan, multi-threading should be used, to not block the CPU. As far as I understood, I would use one thread per command pool, because the barriers (smallest synchronization tools) work per pool. Does that make sense? Or is it better to use one thread per Queue? Or maybe per Queue family? I also thought about having one thread per queue, which is controlling one thread per command pool, to keep the synchronization simple (each queue thread is like the master of each command pool thread). But here I am confused because the command pool belongs to a queue family (not a queue itself). So two queues from the same family could use the same command pool.

Please correct me if something I wrote is wrong. I am new to Vulkan but comming from OpenGL / DirectX 11. I also would like to receive some links to better explanations regarding the Queues and command buffer / pools hierarchy and maybe something about the threading for Vulkan applications. When answering the questions, please give an example scenario if possible (to see which difference it makes). Thanks a lot!

Answer 1

Queues

The Vulkan Queues aren't as segregated as that book makes them sound. If you look at the spec very carefully you will notice that there must be at least one queue that supports Graphics, transfer and compute. So you will find a combined Graphics/Transfer/Compute queue on every machine. Some machines will only have this one queue. Then there are specialized Queues such as those dedicated to Compute and dedicated transfer queues.

Some GPU's have a dedicated hardware unit that only supports transfers. That unit really has no access to the rest of the GPU, so to answer question 1: no not every queue can access 100% of the GPU.

But those dedicated pieces of hardware also give us programmers the power to split up work which answers question 2: Taking full advantage of the hardware requires splitting the work up among queues. The better a piece of software can do that, the more performance we are likely to get.

Command Buffers

Question 3: Command buffers are not thread-safe, writing to a buffer with the vkxCmd* functions causes memory allocations. Those memory allocations go back to the pool. The pool then manages memory for all its Command buffers. So writing to several command buffers from a single pool will cause contention on memory and can lead to a pool using much more memory than it needs. Using multiple pools allows that memory to be released periodically(by resetting the pool), it also allows multiple threads to manage their pools/buffers more effectively. There are a few other details here but memory contention and thread safety are the major reason to split up buffers between pools.

Question 4: Splitting threads up between Command pools is the way to go, splitting threads between queues can also be a big win but there is less hardware out there that can take advantage of it. For high-end hardware, you want threads that can manage async compute, dedicated transfers, and multiple command pools along with the graphics queue.