TL;DR: There's some room for optimization in Vulkan drivers, but not nearly to the degree that there is in GL / DX11 drivers.
how much do drivers affect performance?
It depends a great deal on how the application is written and the hardware. Specifically it depends on whether you're using features where there are large opportunities for optimization on the driver side.
For example, if an application is written to take full advantage of renderpass subpasses and properly specifies the subpass dependencies and properly sets up transient attachments for intermediate work, then the difference between an extremely naive Vulkan driver and a well written one could easily be an order of magnitude. But at the same time, this is basically the difference between naive client code and well written code both running on a well written driver.
people on the internet talk a lot about NVIDIA drivers being more finely tuned and optimized (especially for the most popular games)
This isn't just a matter of driver tuning. It's arguably a form of cheating. It's pretty well known that in the case of certain AAA games, nVidia will actually swap out functionality from the game with baked functionality from it's own drivers, doing things like swapping out shaders that were bundled in the game with their own customized shaders, or toggling internal flags that conventional developers have no access to.
The problem is that DX11 and GL are both very old APIs with lots of legacy functionality. Their very design in some places is at odds with modern GPU hardware, especially in the area of bundling individual commands together before actually sending them to be executed. This is why all modern APIs like Metal, Vulkan and DX12 have the concept of a command buffer or batch. So that the developer can actually be involved in the process of bundling lots of small bind and draw commands together in a logical way. Older API's like OpenGL have to guess at where the command buffer boundaries should be and (one assumes) drivers tend to have a variety of strategies for doing this depending on how the client application is calling functions.
This is where the driver level optimizations come in, where nVidia or AMD can look at how a given game is calling functions and decide that strategy A is better than the strategy B (the one that driver version X is using), and so they release driver X + 1, and it forces the use of strategy A, or even a new strategy C that they came up with for this specific game.
The end result is that OpenGL (and I assume DX < 12) drivers end up being monstrously complex, not just because of the enormous amount of legacy functionality they have to support, but also because of this per-game baggage that they accumulate (although I suspect that such per-game baggage has some kind of finite lifetime.... no one is going to be benchmarking a RTX 2080 with Battlefield 1942).
