The sentence says it: radiosity precomputes an "image" for all potential viewpoints at the same time, i.e. it doesn't focus on just the rays that hit a particular observer. Hence there are naturally many more rays to consider, as you are actually rendering a multitude of views simultaneously.
Whether the surfaces are specular or diffuse isn't really relevant in regard to this statement.
To make the approach viable, radiosity performs very gross rendering, as if using large "beams"beams instead of thin rays.