Yes, it's totally possible, and in fact it's preferable to keep buffers in memory if you will be accessing them over the life of your program, rather than deallocating and reallocating them again.
I can't speak for Unity's way of allocating persistent buffers - I work with Unity and do a lot of shader writing in Cg, but most of our stuff is on mobile so compute shaders aren't an option. However, from a recent project I did using OpenGL, a buffer will generally hang around until you tell it not to, and even then, it is not in any way guaranteed to release the memory in a timely manner.
Allocating a buffer in modern OpenGL is done by calling glCreateBuffers and passing it a pointer to a GLuint array which will store your buffer 'names', then calling glBufferData or glBufferStorage to actually allocate and populate the VRAM for that buffer. Calling glBufferData a second time on the same buffer name will allocate new storage for that buffer, and the original storage will be 'orphaned'. It's kind of like reallocating an object in Java, the variable name will no longer point to that object, but other parts of the system may still have references to it (e.g. if it's still being used for a texture read somewhere), so it will not be eligible for cleanup until all references to it are gone.
Calling glDeleteBuffers with the buffer name has a similar effect but without allocating new storage. In both cases, the storage will not be deleted until absolutely all references to it within the system are gone. Even then, it is more like Java's garbage collection than C++'s delete, the system may well hold on to the memory for quite some time in order to reuse it for any future allocations you may make.
The upshot of all this is it's often better to allocate buffers once when you launch your app and cycle different data through them as needed. In OpenGL, a buffer will persist for the lifetime of the application unless you explicitly delete or reallocate it. Unity's rendering system has its own quirks as I'm sure you're aware, but in principle it should behave similarly.