I am implementing my own simple ray tracing and I came across some sort of confusion. When defining the verticies in the world I multiply them by the modelView transformation system (the matrix similar to the one generated by glm::lookAt function) in order to transform them to the camera coordinate. However, the verticies are invisible in the world when I do this transformation. My question is: are we supposed to transform the veriticies in Ray tracing with the model view transformation matrix as we do in OpenGL ?
The space at which you transform your vertices is completely up to you, because it depends on what algorithms and kind of effects that you are trying to achieve.
As of my personal experience, I usually shoot rays in world space because eventually we all need some sort of "world-space" acceleration data structure, such as a space-partition tree, that gathers all objects into a single space. As this data structure is probably going to be pre-computed for static objects, world space seems to be the most preferred one, as the vertices representation never change when you move the camera around.
When it comes the time to actually test the ray-object intersection, I transform the ray direction onto object space by using the inverse of the model matrix. This makes it possible for all intersection tests to be implemented as if the object were axis-aligned. Thus, you can easily use AABBs (axis-aligned bounding boxes) for coarse intersection testing, even when your models are rotated in world space. This is also a good optimisation for dynamic objects, as the AABB does not change in object space.