convert right handed matrix into left handed forward/up/right vectors

Question

I have an openGL style matrix (column major, right handed ) that needs to be converted into two vectors (forward and up) I've managed to find, in a weird way, a solution that matches the the coordinates rotation, but if I move the camera there is a mismatch that increases if I'm moving the cam from the world center.

What I do is taking an identity matrix and invert the 1.f values to -1.f (apart from the w axis) and then I multiply it by the cam matrix and then again to the identity matrix like this.

new_base_matrix = (weird identity) * cam_matrix * (weird identity).inverted();

then I multiply the new_base_matrix by the camera eye (cam_matrix w axis), the forward vector (0,0,1) and the up vector (0,1,0).

here is a bit of code:

Matrix4 input_cam__matrix = cam->matrix();
Matrix4 inverted_axis_m = {             -1.f, 0, 0, 0,
                                        0, -1.f, 0, 0,
                                        0, 0, -1.f, 0,
                                        0, 0, 0, 1.f };
input_cam__matrix = inverted_axis_m * input_cam__matrix * inverted_axis_m;

//apply rotation scale and pos using the new base
Vector4 pos = input_cam__matrix * Vector4(input_cam__matrix.translation().x, input_cam__matrix.translation().y, input_cam__matrix.translation().z);
Vector4 dir = input_cam__matrix * Vector4(0, 0, 1.f, 0) ;
Vector4 up = input_cam__matrix * Vector4(0, 1.f, 0, 0);


//final vector for the left handed camera
float cam_pos[] =  { pos.x,  pos.y, pos.z };
float cam_view[] = { dir.x , dir.y, dir.z };
float cam_up[]   = {  up.x,   up.y,  up.z };

float fov = 2 * atan(cam->film_height() / (2 * cam->focal_length()))  * (180 / M_PI);

I don't know what sign the system that uses the up/forward vectors uses, but I think is +Y on the Y axis. My system, which is the input_camera_matrix is +X+Y+Z .

Answer 1

I was applying the transform twice, here the solution:

Matrix4 input_cam__matrix = cam->matrix();
Vector3 translation = input_cam__matrix.translation();
input_cam__matrix.rotationOnly();
Matrix4 inverted_axis_m = {  -1.f, 0, 0, 0,
                                        0, -1.f, 0, 0,
                                        0, 0, -1.f, 0,
                                        0, 0, 0, 1.f 
};
input_cam__matrix = inverted_axis_m * input_cam__matrix * inverted_axis_m;

Vector3 pos = -translation;
Vector4 dir = input_cam__matrix * Vector4(0, 0, 1.f, 1.f) ;
Vector4 up = input_cam__matrix  * Vector4(0, 1.f, 0, 1.f);

The forward and up vectors only get the rotation (I removed the position data from the cam_matrix) and then applied the position to the eye vector.

Everything is inverted, position and rotation. I found this unique