Path tracing : fireflies/hot pixels when directly sampling lights

Question

I am implementing a pathtracer with both indirect and direct light sampling at each intersection (I believe the "direct light sampling" part is also referred to as "next event estimation"). I am trying to compute both direct-diffuse and direct-specular contributions of the light sources.

For that, I use a normalized Phong BRDF (formula taken from here), in which the contribution of each light source at an intersection is defined as (pseudo-code) :

directLighting = diffuse() + specular() / (intersection.location - light.location).lengthSquared();


diffuse() {
    normalization = 1.0f / PI;
    return material.metal ? Color::BLACK :
                            lightDotN * lightIntensity * lightColor * material.albedo * normalization;
}


specular() {
    if (observerDotReflected <= 0.0f) {
        return Color::BLACK;
    }

    normalization = (material.smoothness + 1.0f) / (2.0f * PI);
    return (observerDotReflected ^ material.smoothness) * lightIntensity * lightColor *
           (material.metal ? material.albedo : material.specularity) * normalization;
}

This works well as long as I have pretty rough objects in my scene (below, a rough metallic sphere with smoothness = 25.0f, and a rough diffuse sphere with smoothness = 2.0f).

However, as soon as I introduce a very smooth and reflective object (below, a metallic sphere with smoothness = 10'000.0f), two problems appear : the specular lobe on the sphere is very ugly (we can see the individual points that resulted in non-zero specular contribution), and hot pixels/fireflies appear everywhere in the scene.

I think I understand why these exist : when the smoothness is high, the specular Phong BRDF returns ~0 almost all the time (when observerDotReflected is not very close to 1), but sometimes returns extremely bright values, which cause fireflies at other points in the scene when they are the result of an indirect bounce.

Note that the same effect appears whether I use point lights, or sphere lights (with a non-zero volume that I randomly sample).

Is there a conventional way to solve this problem ? I have seen this answer where the author indicates that for mirror-like specular surfaces (i.e. smoothness -> infinity), you should not directly sample specular lighting, but rather rely on indirect bounces intersecting light objects. Is this the only way to go about direct specular lighting ? Are there other solutions to this variance problem ?