Every 3D rotation has an associated space that can be generated by 3 basis vectors, which can be seen as the axis of the coordenate system of this space. These vectors are also the 3 columns of the transformation matrix. In order to decrease the dof of the rotation, you should provide one or more of these basis vectors and solve for the rest of the matrix. The same principle applies to the translation vector, you should provide constants for one or more distances of the translation. Of course, if you restrict the transformation too much you will end with an unsolvable exact system or a transformation that take the first triangle too far away from the second one, if you are using a non-linear solver.