Cloud based VR would be the future?

Question

According to the following assumptions, I think Cloud based VR would be the future:

1) Real world scene rendering needs ray tracing or other complex computing graphics technologies. It needs a lot of computing power.

2) Single GPU can't afford enough GFLOPS for that. And the powerful GPU is costly and hot. Since the moore's law is ending soon, the process of GPU will also end evolution, that means GPU can't get much faster than needed.

3) Mobility is the king. But mobile GPU speed will be limited by the power and process. It won't get much faster too.

4) Cloud gaming would be promise for that. The network bandwidth and latency would be the biggest issues. Since Google Fiber has been deployed, I don't think network is the issue in the future.

5) Rendering a real world needs unlimited computing power, but human eye resolution is limited. Limited resolution means limited network bandwidth consumption. For example, 1Gbps is enough to transmite 8K/16K H265 streaming video in stereo format.

So, I believe Cloud based VR (Rendering in Cloud and stream to the client) would be the future. Am I correct?

---

added more detail @20160406

I did some research on it. The motion-to-photon has to be less than 20ms. Maybe only 5ms is left for round trip network. So that the Cloud DC has to be very close to the client. In case optical fiber access, only two hops ( switch and BRAS ) to local router and then to local Cloud DC. (Transmission speed in optical is 200K KM/s, so 100KM radius would take only 0.5ms. That means Cloud DC would cover 30K square KM area.)If no network congestion with good QoS policy, the network latency would be very low. At the mean time, use distributed GPU clusters to render the VR content and then stream it to the client in parallel to reduce the time further. Let's count the time consumptions during each stages. 1.5ms for head tracking and local cpu processing. 1.5ms for network transmission to Cloud. 1.5 ms for distributed GPU cluster rendering and encoding. 5ms for network transmission to client. 1ms for decoding with special decoder since current video decoder designed for video playing not for VR. 5ms for transmission to monitor (equal 200Hz monitor, or very high bandwidth transmission as display port 1.4). The total time is 1.5+1.5+1.5+5+1+5 = 15.5ms. Of course, if using Foveated Rendering technology, addition 3ms for eye tracking delay, but the network bandwidth and latency will be reduced.

Answer 1

As other people have mentioned, due to network latency issues, I think that full rendering on a cloud server and streaming video to a client device is unlikely to be workable. Even if the latency can be kept low enough in a best-case scenario, it would only work for a very limited number of users with very high-quality internet connections. (Don't forget that e.g. in the USA, there are still tens of millions of people on dialup or DSL, with no access to cable internet, let alone fiber. It will be decades still before fiber is widespread.) Also, note that many cloud gaming services do exist, but that according to players, input latency is noticeably worse on these services than playing locally on a PC.

However, all that being said, rendering entirely on a cloud server is not the only way to approach the problem. The most latency-sensitive aspect of VR is head tracking. Oculus's "timewarp" technology shows that you can quickly and cheaply update a previously rendered image to respond to head rotation, as a 2D image operation: if the user turns their head to the right, you shift the image to the left, and so on. With a depth buffer, you can do positional timewarp as well, although this creates various artifacts with parallax because it's based on a single-layered 2D image.

Timewarp is certainly cheap enough to run on a mobile device. So, one approach is to render frames in the cloud, stream the video to the client, then timewarp the video on the client device to obtain very low-latency head tracking. This wouldn't help the latency of other inputs such as keyboard/mouse/controller inputs, but those don't induce nausea when they're delayed, the way delayed head-tracking does.

There are also other possibilities along the lines of splitting rendering between the cloud and the client. For example, object-space lighting and shading could be done on the cloud and streamed to the client as textures; then the client only needs to render simple textured geometry. This would work better than positional timewarp for getting correct parallax when you move your head around, though you'd still have some artifacts with specular shading and reflections. There are a lot of possibilities for offloading rendering to the cloud in VR-friendly ways.

Answer 2

Strictly speaking I think that the latency coming from transferring data over the network will not be low enough for VR. The device will need to send all the head tracking data and the user input over the network to the cloud based rendering farm. Then that data and input would need to be processed, rendered and the video streamed back to the user. Can you imagine playing a game and the the actual game stopping for buffering?

Answer 3

I always feel gaming, particularly AAA gaming (e.g., FPS), is and will be the last application for cloud rendering, regardless being VR or not. VR simply makes the problem/requirement harder.

Because of the tight loop of interactivity, latency is the the blocking factor as others have pointed out -- and we know that unlike bandwidth, latency is a much harder problem to solve practically in a commercial world. Though technically solvable (say deploying a group of game servers in every county, which is well within the Geo-boundary range of your back-of-envolop calculation above; also your matchmaking process needs to do the right player grouping), it really needs justifiable business case.

Having said that, some non-AAA titles could well fall into cloud side rendering.

Answer 4

Network latency will not be low enough VR in near future. Mobile VR is a key driver for Cloud Rendering. However, the mobile network latency is much higher. Split rendering with future micro-cloud based Gamelets architecture (http://ieeexplore.ieee.org/document/6799051/) is one possible solution for Cloud Rendering for VR.

Answer 5

I do this already.

TL;DR: The latency between services decelerate the enthusiasm.

I have a steam-vr-client supporting a wide range of Helmets and the vr-client is connecting to a local server (Currently connectors for Java exsits only) and then further cloud-services.

As most games have two threads who must be synchronized

1. Frontend: The visualization of the gamelogic
2. Backend: The gamelogic, model, rules....
3. (Optional) Multiplayer Thread: Synchronizing gamestate between contributors.

In VR there will be a growing latency between Frontend and Backend the more work is done in the Clouded-Backend. So we have to estimate what contribution to the VR is a lightweight-transported information and do the rest in the cloud.

1. Frontend (locally): The low-level visualisation. Calculating shadows and "depth of field blur", but not calculating ambient occlusion, not calculating raytracing, not calculating Bumpmapping.
2. Backend (locally): Collecting results from the cloud, collect Bumpmaps, transport Mesh, UV-Coodinates, Normal-Mappings, Bumpmappings, Raytrace-Results and send it to Frontend.
3. Cloud (web/internet). Calculating raytrace, generate Bumpmaps, Calculate Mesh resolution, calculate UV-Coordinates and Normals.

The key to seperate work for different cloud services is to understand what work can be done packed autark.

My project: https://github.com/enexusde/VRServer