Is it topological correct to combine triangles with quads to form a mesh. If not, why?
Topologically correct is a very vague.
I believe you think of Polygon-Meshes when you say meshes, which is to represent the surface of the object by a patch of polygons, in most cases triangles or quads. There are other options to model meshes, one idea would be to use signed distance functions as for example in constructive solid geometry.
So back to the question. Is it correct to mix the type of polygon? Yes. There is, mathematically and theoretically nothing stopping you from doing it.
The only real 'correctness' criteria that comes to my mind is the one of the mesh being a manifold. This is in (overly) simplficated the property that the mesh has a defined outside and inside and no strange places where you can't decide. (This happens for example if the mesh intersects with itself, but it get's freaky). And having different types of primitive does not violate the manifoldness of the mesh.
But most algorithms can actually deal with non-manifolds. There is no problem in rasterizing or raycasting a non-manifold as these algorithms process each primitive separately.
The more interesting question here is: Why DONT we mix the type of primitive?
And here the answer is very simple: Efficiency. You want to design an algorithm that handles your mesh as efficient as possible, and so you assume it's only one type of primitive. And at some point in time, because triangle-meshes are really handy, hardware and hardware-api began to adpot mainly to processing triangles. And now they are good at that and even other primitive types get split into triangles since splitting and rendering triangles is still faster than rendering the polygons directly.
So in short: You can mix polygon types, but if you want to be fast, use only triangles since they are simple and widely used.
None of today's graphics interfaces support quads by default in the first place.
This means that quads, when actually set up for rendering, are just pairs of triangles.