OpenGL : How to rotate an object around viewing space x-axis

I have an object(cow).

This cow was translated and rotated(with arbitrary axis) in the modeling space for several times.

And I can't know how much cow is translated or rotated.

Only I can obtain is this cow's current matrix and it's inverse.

In this circumstance, how can I rotate the cow around the x-axis in the viewing space?

(center of rotation is at the center of modeling space)

A rotation around the x-axis is achieved by this matrix $$\mathtt{T}_\mathrm{rot X} = \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0& \cos \alpha & -\sin \alpha& 0 \\ 0 & \sin \alpha & \cos \alpha & 0 \\ 0 & 0 & 0 &1 \\ \end{bmatrix}$$

In order to rotate around the x-axis in the viewing space, one needs to think about when to apply the above transformation in the chain of transformations that is applied to the 3D model vertices $\mathbf{X}_{\mathrm{3d}}$ in the employed rendering pipeline.

A typical transformation chain is:

$$\mathbf{X}_{\mathrm{view}} = \mathtt{T}_{\mathrm{projection}} \,\mathtt{T}_{\mathrm{camera\ transform}}\, \mathtt{T}_{\mathrm{mesh\ transform}}\,\mathbf{X}_{\mathrm{3d}}$$

Let's split the camera tranformation into its rotational and translational part: $$\mathbf{X}_{\mathrm{view}} = \mathtt{T}_{\mathrm{projection}} \,\underbrace{\mathtt{T}_{\mathrm{cam\ trans}}\,\mathtt{T}_{\mathrm{cam\ rot}}}_{\mathtt{T}_{\mathrm{camera\ transform}}}\, \mathtt{T}_{\mathrm{mesh\ transform}}\,\mathbf{X}_{\mathrm{3d}}$$

Then we need to apply the x-rotation in-between the camera rotation and translation, because we want to have the rotation of the camera applied beforehand, but not the camera's translation:

$$\mathbf{X}_{\mathrm{view}} = \mathtt{T}_{\mathrm{projection}} \,\underbrace{\mathtt{T}_{\mathrm{cam\ trans}}\,\mathtt{T}_\mathrm{rot X}\,\mathtt{T}_{\mathrm{cam\ rot}}}_{\mathtt{T}_{\mathrm{new\ camera\ transform}}}\, \mathtt{T}_{\mathrm{mesh\ transform}}\,\mathbf{X}_{\mathrm{3d}}$$

This would achieve the desired rotation, but typically we do not want to change the camera transformation but the the mesh transformation. This is achieved by $$\mathbf{X}_{\mathrm{view}} = \mathtt{T}_{\mathrm{projection}} \,\mathtt{T}_{\mathrm{cam\ trans}}\,\mathtt{T}_\mathrm{rot X}\,\mathtt{T}_{\mathrm{cam\ rot}}\, \mathtt{T}_{\mathrm{mesh\ transform}}\,\mathbf{X}_{\mathrm{3d}}\\ \Leftrightarrow \mathbf{X}_{\mathrm{view}} = \mathtt{T}_{\mathrm{projection}} \,\mathtt{T}_{\mathrm{cam\ trans}}\,(\mathtt{T}_{\mathrm{cam\ rot}}\,\mathtt{T}_{\mathrm{cam\ rot}}^{-1})\,\mathtt{T}_\mathrm{rot X}\,\mathtt{T}_{\mathrm{cam\ rot}}\, \mathtt{T}_{\mathrm{mesh\ transform}}\,\mathbf{X}_{\mathrm{3d}}\\ \Leftrightarrow \mathbf{X}_{\mathrm{view}} = \mathtt{T}_{\mathrm{projection}} \,\mathtt{T}_{\mathrm{camera\ transform}}\,\underbrace{\mathtt{T}_{\mathrm{cam\ rot}}^{-1}\,\mathtt{T}_\mathrm{rot X}\,\mathtt{T}_{\mathrm{cam\ rot}}\, \mathtt{T}_{\mathrm{mesh\ transform}}}_{\mathtt{T}_{\mathrm{new\ mesh\ transform}}}\,\mathbf{X}_{\mathrm{3d}}$$ Therefore, the short answer to your question is:

Left-multiply the current transformation matrix of your model with $\mathtt{T}_{\mathrm{cam\ rot}}^{-1}\,\mathtt{T}_\mathrm{rot X}\,\mathtt{T}_{\mathrm{cam\ rot}}$. I have generated an "cow" example scene in the GSN Composer that can be used for interactive testing in a WebGL-capable browser:

Example: CowQuestion | GSN Composer

To get the axis of rotation you could multiply the x-axis by the inverse of the view matrix:

$b = V^{-1} e_1$

Where $b$ is the axis of rotation in world space, $V$ is the view matrix and $e_1$ is the x-axis unit vector.

Now, to make the object to spin around the axis of rotation $b$, we have to translate the object back to the origin, spin it, and then translate it back to its position in world space.

Suppose you have a translation matrix $T$ that send the origin point to the object's location. The translated rotation (spin) of the object $M$ would be:

$M = T R T^{-1}$

Where $R$ is the rotation matrix around the $b$ axis. The matrix $R$ could be obtained using the Rodrigues formula or from a Quaternion or through some other methods out there.

Finally you need to multiply the matrix $M$ to the existing transformation matrix of the object $W$. $W' = M W$ and use $W'$ for transforming the object.