I've heard a lot of people working on VR talk about scanline racing and that it's supposed to help improve latency for motion-to-photon. However, it isn't clear to me how this can be done with OpenGL. Could someone explain how scanline racing works, and how it can be implemented on modern GPUs.
$\begingroup$
$\endgroup$
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
4
When your GPU displays a new frame on the screen, it transfers the image over the HDMI cable (or whatever kind) in a process called "scanout". The pixels are sent out in linear order, usually left-to-right and top-to-bottom. The process is timed so that it takes most of the duration of a refresh interval to do this. For instance, at 60Hz, one frame is ~17 ms. Each scanout will take probably around 15-16 ms, with 1-2 ms of vblank in between (the exact values vary according to the display and video mode).
Traditionally, rendering is double-buffered, which means there are two buffers stored in GPU memory: one that is currently being scanned out ("front buffer"), and one that is being rendered to ("back buffer"). Each frame, the two are swapped. The GPU never renders to the same buffer that's being scanned out, which prevents artifacts due to potentially seeing parts of an incomplete frame. However, a side effect of this is increased latency, since each frame may sit around in the buffer for several ms before it starts being scanned out.
VR is very latency-sensitive, so this isn't desirable. An alternative approach is to render directly to the front buffer, but time things out very carefully so that you have rendered each line of the image shortly before the scanout gets there. That's called "scanline racing" or "racing the beam" (the "beam" harkening back to the CRT days of yore). This more or less requires that you render the image in scanline order, i.e. the same order that the pixels get scanned out. It doesn't literally have to be rendered one line at a time—it could be rendered in thin strips a few pixels high, but it does have to be done in order, as you can't go back and edit pixels that have already been scanned out.
There are a lot of disadvantages to this approach; it has very stringent performance requirements, has to be timed very carefully against vsync, and it greatly complicates the rendering process. But in principle it can shave milliseconds off your latency, which why VR folks are interested in it.
answered Aug 5, 2015 at 9:04
-
1$\begingroup$ So my question is how do we do this on modern GPUs? I don't think there's any way to query the scanout, and it seems to me you can't really submit per-scanline draw calls. Even if you could -- what guarantees do you have that your draws are going to get there before the scanout? $\endgroup$– MokoshaAug 5, 2015 at 16:10
-
1$\begingroup$ @Mokosha Correct, there's no way to query the scanout directly AFAIK. At best, you can figure out when vsync is (via some OS signal) and estimate where the scanout is by timing relative to that (knowing details of the video mode). For rendering, you can experiment to find out how long it usually takes between glFlush and when the rendering is done, and make some guesses based on that. Ultimately, you have to build in some slack in your timing in case of error (e.g. stay 2-3 ms ahead of scanout), and accept that there will be probably be occasional artifacts. $\endgroup$ Aug 5, 2015 at 20:39
-
$\begingroup$ The effect of increased latency is due to vsync, which causes the front and backbuffer swaps to synchronize with the vblank of the monitor. Double buffering itself doesn't cause this issue by itself and it is useful to minimize flickering because a pixel can only change once in the front buffer. $\endgroup$– MauriceAug 6, 2015 at 0:29
-
$\begingroup$ I've come up with an accurate way to predict rasters without a scan line query, see answer below. $\endgroup$ Mar 27, 2018 at 14:24
$\begingroup$
$\endgroup$
The great thing is we can finally predict scan-line exact raster accuracy with no access to a per-scanline query:
https://www.youtube.com/watch?v=OZ7Loh830Ec
I have come up with the exact microsecond-accurate formulas as a VSYNC offset, to predict the position of a tearline. Tearlines during VSYNC OFF are always raster-exact, so you can steer them out of visibility during strip-level "simulated front-buffer rendering" via repeated VSYNC OFF buffer swapping.
Pay attention to forum thread -- there is some open source code being continually added -- https://forums.blurbusters.com/viewtopic.php?f=10&p=32002
$\begingroup$
$\endgroup$
If it's of interest, the Dreamcast had a "racing the beam" rendering mode, whereby it was able to dedicate a relatively small fraction of memory to framebuffer pixels (e.g. 64 scan lines) and it would render rows of 32 in turn, synchronised to the display update. This however, was only used to save memory. I doubt anyone was generating "modified" geometry for latter parts of the display.
