Yes, you do need to transform the fetch direction into the space of the cubemap. If you could somehow figure out the fetch direction in the vertex shader, then you could do the transformation there instead, but that would produce worse lighting.
It also may be worth optimizing for a smaller number of interpolants between the vertex and pixel shaders (rather ...
This is not a full answer, I would just like to share the knowledge I obtained by studying two of the papers mentioned in the question: Steerable Importance Sampling and Practical Product Importance Sampling for Direct Illumination.
Steerable Importance Sampling
In this paper they propose a method for sampling the product of the clamped cosine component ...
Disclaimer: I have no idea what is the state of the art in the environmental map sampling. In fact, I have very little knowledge about this topic. So this will not be complete answer but I will formulate the problem mathematically and analyze it. I do this mainly for myself, so I make it clear for my self but I hope that OP and others will find it useful.
There's two main things it seems like you've not understood correctly.
IBL is for indoor scenes and outdoor scenes
In the early days when IBL was hot, the Uffizi light-probe was probably the most popular for researchers. IBL works really well for indoor scenes with strong light coming from multiple directions, such as churches with stained glass. That kind ...
I've actually managed to find problem thanks to @PaulHK, thanks!
I decided to pass which face I am currently working on and setting then coordinates manually per face. It is awful but it works rather good and since it is prefiltered and not done in runtime I found it good enough:
[branch] if (g_upVectorVal.z == 1.0f)
input.position.z = -1.0f;