Transformations, and especially rotations, may have different visual effects, depending on the order they are applied to the target object. For example, using a rotation $M_R$ and a translation $M_T$ to the object's vertices $v_i$, $M_R \cdot M_T \cdot v_i$ will usually1 result in a different pose than $M_T \cdot M_R \cdot v_i$ does.
90 degree rotation and translation along the x axis applied to the box. Left: translation first ($M_R \cdot M_T \cdot v_i$). Right: rotation first ($M_T \cdot M_R \cdot v_i$).
The key here is to keep in mind to which coordinate system the transformation is applied. Assuming that your to be rotated object initially is in the origin without transformations applied, the first rotation will always yield the desired effect. This is because you're applying the first transformation to the object.
But once your object is already transformed (rotated, translated, whatever) by $M_1$, technically you can already apply the next transformation $M_2$ in two places. Namely,
- $M_1 \cdot M_2 \cdot v_i$ or
- $M_2 \cdot M_1 \cdot v_i$
You describe that
so if I rotate the object around the x axis by 180° and then move my mouse left, the object will rotate to the right instead of left (since the y axis is now inverted).
This is, because the second rotation is applied to the object itself, which is case 1 of the above. But you want to have the second case, so you need to apply the rotations the other way around. That way, the new rotation will be applied to the already rotated object, i.e. the outer coordinate system.
Ideally, as long as the user holds the mouse button down you adjust that transformation $M_2$. Calculate the distance the mouse moved from the point the user pressed the button, recalculate the whole rotation ($M_2$) and drop whatever you calculated in the time between the mouse-down event and the previous frame. So for the ongoing rotation, pretend that the user moved the mouse instantly from the start to the current position.
Once the mouse button is released, you can finalize the transformation and multiply it into $M_1$, which is the identity matrix in the beginning. (Assuming you always want to rotate around the world's center.)
1 There are some special cases where the result is the same, e.g. if the translation direction matches the rotation axis.