It sounds to me like you're slightly mixing up several concepts: how lighting due to point-light sources (Phong model) and lighting due to other objects in the scene (recursively raytraced reflection or transmission) combine with each other, as compared to how specular lighting combines with diffuse lighting and/or transmission (using the Fresnel equations).
A useful way to think about specular vs diffuse is that specular happens at the surface of the material while diffuse, transmissive, and refractive properties are associated with the bulk (the interior) of the material. So you can think of it as two layers composited together. Specular is on top: it reflects a portion of the incoming light, defined by the Fresnel equations, and any remaining light proceeds into the material to light the bulk.
The bulk will have some combination of scattering and absorption properties. Low scattering makes a transmissive material and high scattering makes a diffuse material (the diffuse color is due to scattering on a scale too small to see). Some materials have an intermediate degree of scattering too, but we can ignore that for simplicity and just treat it as a boolean: opaque (Lambert diffuse) or transmissive.
Meanwhile, each of these potential material components (specular, diffuse, and transmissive) has to deal with light arriving both from point light sources and from the rest of the scene. Physically, all light sources are just emissive objects that are part of the scene (area lights) but again for simplicity we often approximate by using point lights.
So, putting it all together, when lighting a surface:
- For each point light
- Calculate Fresnel factor based on light vector
- Accumulate Phong specular weighted by Fresnel reflectance
- [if opaque] Accumulate Lambert diffuse weighted by Fresnel transmittance
- [if transparent] Nothing; light goes through and isn't scattered into the camera.
- For reflecting the rest of the scene
- Calculate Fresnel factor based on view vector
- Cast a reflection ray and accumulate it weighted by Fresnel reflectance
- [if opaque] Nothing. (Ideally, accumulate Lambert diffuse due to all objects visible in the normal hemisphere of the surface, i.e. indirect diffuse. However, for a basic recursive raytracer this isn't feasible; it requires path tracing, photon mapping, or suchlike.)
- [if transparent] Cast a transmission ray and accumulate it weighted by Fresnel transmittance
Also, a note about your material properties: physically, not all of those properties are actually independent of each other. Ks, reflectivity, and refractive index are all the same thing. Ks is the same thing as reflectivity (literally the same value) because both control specular reflections; Ks is for point lights and reflectivity is for reflected environment rays, but it's the same physical mechanism for both. And Ks is determined in turn by the Fresnel equations based on refractive index (which can vary with wavelength).
So, strictly speaking, you should pick one of those variables and let it determine the others. For opaque materials where you don't really care about refractive index, you might let users set Ks (at normal incidence) and backsolve the Fresnel equations to get the appropriate refractive index from that. For transparent materials like glass you can let users set the refractive index and then let the Fresnel equations do their thing.
BTW, to get some physics background that will hopefully help clarify all this, I highly recommend Naty Hoffman's "Physics and Math of Shading" talk from the SIGGRAPH 2015 Physically-Based Shading Course. It gives a quick and not-too-mathematical intro to the physical basis of things like diffuse and specular lighting, BRDFs and so on.