There are a few confusing statements in your question. However, I'll still assume you're largely aware of how the stencil buffer and the stencil test actually work. I'll also talk about OpenGL, based on your tags, but the general principles should apply to any similar API.
First of all, you say
...stencil buffer can only be filled in by some hardcoded rules during geometry drawing (like depth buffer)...
But for the depth buffer that isn't actually true. You can very well decide what depth value you actually want to write by just writing anything to gl_FragDepth in your shader.
Now you're right that in unextended GL you can't just write arbitrary values to the stencil buffer like this. However, if your platform supports the GL_ARB_shader_stencil_export extension, you can do exactly that by writing to gl_FragStencilRefARB. Or more precisely, you write the reference value that is used in the stencil test. So if you have a stencil op of GL_REPLACE, you can basically just write the value per fragment.
However, even if that extension isn't present, you can achieve similar things for simple scenarios by merely deciding which fragments to write at all. You can just globally specificy whichever stencil reference value you want to write for your fragments and then in the fragment shader simply discard any fragments for which you don't want to write anything. Of course this doesn't let you arbitrarily choose the value you want to write per fragment, but when using stencil testing you normally don't really need any such kind of fine-grained control over its bits (afterall, it's usually got just 256 different values anyway).
As to your other question:
...is it possible to use stencil buffer separate of depth buffer in this case, don't use depth buffer at all?
The stencil buffer really doesn't have anything to do with the depth buffer (at least functionally). Yes, the stencil op lets you choose different actions depending if the fragment passed the depth test or not. But noone requires you to use the depth test at all. If you don't need depth testing, just don't do it. Or if you don't need stencil testing and the depth buffer already works for you, then use that.
There is a bit of a practical consideration that usually the actual framebuffer is laid out so that the depth and stencil buffer share their storage in a combined depth-stencil buffer, because for the common case of a 24-bit depth buffer and an 8-bit stencil buffer this goes well together. It's what the window system usually does for you and what you should strive for when binding storage to framebuffer objects, too. However, that is merely the internal data layout and doesn't require you to always use depth and stencil testing together. It does, however, mean, that sparing away the stencil buffer usually doesn't save you any actual memory (at least in case of a common 24-bit depth format).
As a side note, though, from your question I have a bit of hunch that you might not even need the stencil buffer to begin with. Maybe you weren't aware that you can just discard fragments in the fragment shader or decide yourself which depth value you want to write and maybe your possible multi-pass algorithm (since that's usually how stencil testing is used) can be done in a single pass or just differently altogether. If you'd need to write compilated stencil values from your shader, it could be a sign that the stencil buffer might not actually be the tool you're looking for.
