1
$\begingroup$

According to the following assumptions, I think Cloud based VR would be the future:

1) Real world scene rendering needs ray tracing or other complex computing graphics technologies. It needs a lot of computing power.

2) Single GPU can't afford enough GFLOPS for that. And the powerful GPU is costly and hot. Since the moore's law is ending soon, the process of GPU will also end evolution, that means GPU can't get much faster than needed.

3) Mobility is the king. But mobile GPU speed will be limited by the power and process. It won't get much faster too.

4) Cloud gaming would be promise for that. The network bandwidth and latency would be the biggest issues. Since Google Fiber has been deployed, I don't think network is the issue in the future.

5) Rendering a real world needs unlimited computing power, but human eye resolution is limited. Limited resolution means limited network bandwidth consumption. For example, 1Gbps is enough to transmite 8K/16K H265 streaming video in stereo format.

So, I believe Cloud based VR (Rendering in Cloud and stream to the client) would be the future. Am I correct?


added more detail @20160406

I did some research on it. The motion-to-photon has to be less than 20ms. Maybe only 5ms is left for round trip network. So that the Cloud DC has to be very close to the client. In case optical fiber access, only two hops ( switch and BRAS ) to local router and then to local Cloud DC. (Transmission speed in optical is 200K KM/s, so 100KM radius would take only 0.5ms. That means Cloud DC would cover 30K square KM area.)If no network congestion with good QoS policy, the network latency would be very low. At the mean time, use distributed GPU clusters to render the VR content and then stream it to the client in parallel to reduce the time further. Let's count the time consumptions during each stages. 1.5ms for head tracking and local cpu processing. 1.5ms for network transmission to Cloud. 1.5 ms for distributed GPU cluster rendering and encoding. 5ms for network transmission to client. 1ms for decoding with special decoder since current video decoder designed for video playing not for VR. 5ms for transmission to monitor (equal 200Hz monitor, or very high bandwidth transmission as display port 1.4). The total time is 1.5+1.5+1.5+5+1+5 = 15.5ms. Of course, if using Foveated Rendering technology, addition 3ms for eye tracking delay, but the network bandwidth and latency will be reduced.

$\endgroup$
  • $\begingroup$ For VR you need super low latency, any type of network hop will be too long. $\endgroup$ – ratchet freak Apr 5 '16 at 10:18
  • $\begingroup$ Only if the cloud is very close to the vr gear like, within tens of meters. But given that everything and their dad is going to be a computer that is certainly possible. $\endgroup$ – joojaa Apr 5 '16 at 12:59
  • $\begingroup$ @ratchetfreak, I added more detail info above. That explains what I'm thinking $\endgroup$ – Hao Zhang Apr 6 '16 at 0:59
  • $\begingroup$ @HaoZhang well if your calculations that the total time to display a frame to the user from the cloud is 15.5ms then things should work out. In order to have a game running at 60FPS you need to display a frame on the screen every 16.6ms. Are you sure its going to take 15.5ms to display both frames(one for each screen of the VR system)? $\endgroup$ – Uri Popov Apr 6 '16 at 7:18
  • $\begingroup$ I give it 5ms for transmission to OLED display of 2K*2@200Hz, that would need at least 40Gbps interface and cable, as display port 1.4. (en.wikipedia.org/wiki/DisplayPort) supports. Yes it would be a little costly but it is achievable. $\endgroup$ – Hao Zhang Apr 6 '16 at 8:45
8
$\begingroup$

As other people have mentioned, due to network latency issues, I think that full rendering on a cloud server and streaming video to a client device is unlikely to be workable. Even if the latency can be kept low enough in a best-case scenario, it would only work for a very limited number of users with very high-quality internet connections. (Don't forget that e.g. in the USA, there are still tens of millions of people on dialup or DSL, with no access to cable internet, let alone fiber. It will be decades still before fiber is widespread.) Also, note that many cloud gaming services do exist, but that according to players, input latency is noticeably worse on these services than playing locally on a PC.

However, all that being said, rendering entirely on a cloud server is not the only way to approach the problem. The most latency-sensitive aspect of VR is head tracking. Oculus's "timewarp" technology shows that you can quickly and cheaply update a previously rendered image to respond to head rotation, as a 2D image operation: if the user turns their head to the right, you shift the image to the left, and so on. With a depth buffer, you can do positional timewarp as well, although this creates various artifacts with parallax because it's based on a single-layered 2D image.

Timewarp is certainly cheap enough to run on a mobile device. So, one approach is to render frames in the cloud, stream the video to the client, then timewarp the video on the client device to obtain very low-latency head tracking. This wouldn't help the latency of other inputs such as keyboard/mouse/controller inputs, but those don't induce nausea when they're delayed, the way delayed head-tracking does.

There are also other possibilities along the lines of splitting rendering between the cloud and the client. For example, object-space lighting and shading could be done on the cloud and streamed to the client as textures; then the client only needs to render simple textured geometry. This would work better than positional timewarp for getting correct parallax when you move your head around, though you'd still have some artifacts with specular shading and reflections. There are a lot of possibilities for offloading rendering to the cloud in VR-friendly ways.

$\endgroup$
  • $\begingroup$ Hi Nathan, nice to meet you here. I have read some of your presentations on Nvidia Gameworks VR. "Timewarp" can reduce the latency but with image quality loss. And I think, if the head rotated very fast, it's very hard to warp the 2D image to cheat the eyes. Motion blurring may be help in this case. Local client or remote server could send an unclear image to the display in very low latency. You mentioned Cloud could do some offloadings on VR. That's a cool. What's the requirement for the network bandwidth and latency? Does it still need high bandwidth and low latency? Thanks. $\endgroup$ – Hao Zhang Apr 7 '16 at 6:08
  • $\begingroup$ You are right, current internet access are mostly based on dialup or DSL. But, in the near future, e.g. 5-10 years later, could something be changed? Google has deployed 1Gbps fiber in some cities and I heard Google also tried to introduce 10Gbps to home. $\endgroup$ – Hao Zhang Apr 7 '16 at 6:48
  • $\begingroup$ haha, Nathan, I got some information from your blog: (reedbeta.com/blog/2014/04/03/vr-and-multi-gpu/#more-591), "That being said, what if the high-spec dGPUs were far away from you? Cloud-based rendering is a topic that’s been getting interest lately, and the same kind of latency reduction strategies could be applicable there. Imagine having your VR headset driven by the little GPU in your phone, but streaming down source frames or shading cache updates from a dGPU in the cloud (or your home PC)! " $\endgroup$ – Hao Zhang Apr 7 '16 at 7:23
3
$\begingroup$

Strictly speaking I think that the latency coming from transferring data over the network will not be low enough for VR. The device will need to send all the head tracking data and the user input over the network to the cloud based rendering farm. Then that data and input would need to be processed, rendered and the video streamed back to the user. Can you imagine playing a game and the the actual game stopping for buffering?

$\endgroup$
  • $\begingroup$ I added more detail info above. That explains what I'm thinking $\endgroup$ – Hao Zhang Apr 6 '16 at 1:00
2
$\begingroup$

I always feel gaming, particularly AAA gaming (e.g., FPS), is and will be the last application for cloud rendering, regardless being VR or not. VR simply makes the problem/requirement harder.

Because of the tight loop of interactivity, latency is the the blocking factor as others have pointed out -- and we know that unlike bandwidth, latency is a much harder problem to solve practically in a commercial world. Though technically solvable (say deploying a group of game servers in every county, which is well within the Geo-boundary range of your back-of-envolop calculation above; also your matchmaking process needs to do the right player grouping), it really needs justifiable business case.

Having said that, some non-AAA titles could well fall into cloud side rendering.

$\endgroup$
0
$\begingroup$

Network latency will not be low enough VR in near future. Mobile VR is a key driver for Cloud Rendering. However, the mobile network latency is much higher. Split rendering with future micro-cloud based Gamelets architecture (http://ieeexplore.ieee.org/document/6799051/) is one possible solution for Cloud Rendering for VR.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.