As I understand, the back-buffer is used for rendering the image while the front-buffer is drawn on the screen. So if you have only one image to be drawn on the screen, then you only need to swap the buffers one time because in the second swap the image is already drawn and there is no point to draw the same image again. However, when I tried to swap buffers no more than 2 times the graphics slow down a lot and move the window from one position to another takes 10-20 seconds. My question is... Why? How double-buffers works then?
Yes, the back buffer is for the GPU to write the frame that's in-progress, while the display controller sends the front buffer to the attached display. Swapping buffers doesn't really copy anything these days. The display controller has a register which tells it the address of the frame buffer to display. Swapping buffers simply puts the address of the old back buffer in this register, making it the front buffer. You should only swap buffers when the frame is complete (i.e. you've finished drawing all the objects) and you want it to be displayed. This frame will continue to be displayed until you swap buffers again.
In some configurations (e.g. in EGL it depends on what swap interval you set) swapping buffers will make the CPU wait (block) until the GPU has finished rendering that frame. In others, the CPU can continue to queue up draw calls for further frames.