There is no universal solution here since there are many different small variations that can be applied to get a specific result.
One of the things that trips people up when computing ray's in the vertex shader is how a vector's length changes when it is interpolated. This results in vectors that have slightly different lengths arriving in the fragment shader. This is the same problem that happens with lighting vectors. With lighting the solution is to re-normalize the vector in the fragment shader. With ray casting the vector length needs to be recomputed to match the needs of the technique. This means we can effectively ignore ray length in the vertex shader. But the code must be careful to compute the proper length in the fragment shader.
One good technique is to compute the ray direction in the vertex shader and pass it to the fragment shader. The ray origin can then be computed in the fragment shader in a straight forward manner. Where ray direction is computed by finding the point of intersection with the camera space starting plane in 2D (not necessarily the projection near plane) and just plugging the distance from the camera to the plane in as the 'z' value. The vector length will be odd but its going to be changed with interpolation anyway, then transform it into the appropriate space for interpolation.
Ray origin can be computed in the fragment shader after rescaling (or computing a scaling factor) for the interpolated ray. A scaling factor can be computed with something along the lines of
scale = plane_distance * (1/camera_vector_length)
And then the Ray origin computed with:
ray_origin = interpolated_ray_direction * scale + transformed_camera_postion
It can be better to just pass a camera space 2D "ray" and another ray transformed into the same space that the ray will be marched/cast in so that there is no need to do any transformations of the ray in the fragment shader and is the reason I wrote "camera_vector_length" above. The 2D ray is really just interpolating a point across the start plane then the plane distance is plugged in as the 'z' value in the fragment shader. (saving some interpolation hardware) Also, the transformed_camera_position above, is the camera position transformed into whatever space the ray marching will be done in.
Another nice technique is to compute the ray end point, then compute a time t that is used to interpolate between the start and end points. Where t is a step size that can be set in the shader, or passed in as a parameter, computed on the fly, etc. The interpolated point is then used to sample the data. This drops ray direction out of the marching loops altogether.
Yet another reason to use t is that a random offset can be easily added to it which helps "jitter" the sampling points breaking up regular artifacts that crop up often with ray casting. The goal hear is to move the rays slightly from frame to frame rather then within a single frame.
Using these approach's results in shaders that look considerably different from the standard "Rd = hitpos - camerapos" but are more efficient and produce better results.
// out cs_ray == camera space ray
// out t_ray == transformed ray created using transform matrix
// such as the camera to world matrix, or camera to shadow space matrix
void ComputeRayFromClipSpace( in float2 vp, out float2 cs_ray, out float3 t_ray )
{
// Inputs to shader
float pd; // distance to camera space plane, ie Ray Origin plane
float s; // aspect ratio
float g; // focal distance
// Compute point on camera space plane
float3 hitpoint = float3( vp*pd*s/g, vp*pd*1.0/g, pd);
cs_ray = hitpoint.xy;
t_ray = hitpoint * transform_matrix;
}
This is setup for a "full screen" effect. The inputs would usually be generated from a single clip space triangle that covers the entire vieport. More info here. It is also using info from a perspective frustum but similar info could be generated for a orthographic frustum. The main point here is to compute the points on the plane that intersect with the frustum using whatever method that is appropriate.
This function is leaving some performance on the table for clarity, the constants and transform computations.
Then in the fragment shader:
float3 camera_ray = float3( cameraRayIn, pd ); // pd == plane distance
float length = sqrt( dot( camera_ray, camera_ray));
float scale = pd * (1.0/length);
float3 scaled_ray = t_ray_in * scale;
The scaled_ray can be used to compute the start point scaled_ray+camera_pos where camera_pos is the position of the camera in whatever space we are working in. The ray can then be rescaled (such as normalized or scaled to the required step distance) or used to compute a ray end point.
The best sources I know of for this technique are talks and books describing ray casting effects used in games like fged2.
