I'm currently preparing to implement variance shadow mapping based on this article. However, one point it makes concerns me:
Rendering only casters (and not receivers) into the variance shadow map is incorrect! For the interpolation to work properly, the receiving surface must be represented in the depth distribution. If it is not, shadow penumbrae will be improperly sized and fail to fade smoothly from light to dark.
This means that the naive method of rendering an object that does not cast a shadow, but can recieve shadows from other objects - simply skip rendering it into the shadow map - will not work. How else can this be accomplished?