In the past twenty years display technology has moved on considerably
with improvements such as higher resolution, faster frame rates and
high-dynamic range colour. In the same period graphics processing
units (GPUs or graphics cards) have become significantly faster with
broader functionality. However, we argue that current implementations
of the traditional graphics pipeline, which is based on the
rasterisation and z-buffering, are unsuited to emerging displays.
In particular, the types of near-eye displays used for augmented
reality and virtual reality provide new challenges. Since their
contexts of use are very different, it is not even clear what
properties are ideal. Should the displays be multi-focal or
vari-focal? How fast do display update rates need to be to support
registered augmented reality systems? How do we exploit the properties
of the human visual system to render more efficiently?
The traditional computer graphics pipeline puts emphasis on generating
full images at the display size and rate, where that display might be
1920x1080 at 60Hz or higher. For a near-eye display, it is clear that
such a pipeline is nowhere near suitable: already one consumer HMD is
demanding 2480x1080 at 90Hz, roughly double the bandwidth that a common
desktop display requires. At these rates, there will still be problems
with visual acuity and latency (e.g. there is an inherent 11ms display
lag). Future HMDs are touted with resolutions of 8K pixels across, but
this will require a very significant increase in graphics compute and
thus power consumption. Not only is this expensive, it could limit the
form factor of the device and thus user-acceptance of the technology.
There is very exciting work going on in display technologies at the
moment. For example, Varjo (a project partner) are building a HMD that
has a moving display insert that tracks the eye; Facebook have built a
demonstration display that uses a focal surface to generate multiple
focal depths in the same image.
In comparison, the graphics pipeline is taken mostly as a given.
Recently, the proposers, along with a small group of colleagues in the
field, have started to the challenge the status quo. We don't propose
to ditch the highly optimised compute units in graphics cards, but
rather to study frameworks within they can be exploited more readily.
We believe that by reformulating the graphics pipeline and paying
attention to the very specific needs of near-eye displays, we can
radically reduce the power required from GPUs, and thus make near-eye
display more usable.
We will focus on three connected challenges that we have labelled the
latency, redundancy and bandwidth challenges. First, we will target
extremely low latency displays. We will develop systems that achieve
>1000 fps visual output, with latency under 1ms and study how these
impact visual response. Second, we will explore stronger decoupling of
frame-based rendering from display. We note that in near-eye displays
most pixels are wasted, and thus we target novel spatial and temporal
algorithms that reduce redundancy. Third, to exploit redundancy more
generally, we need to use it to reduce bandwidth between graphics card
and display. Taking inspiration from the concept of surface light
fields, our concept of ambient fields will render to buffers that are
expected to be valid for re-rendering to the user for 10-100ms.
In summary, we believe that the current graphics pipeline and its
associated implementation in GPUs is unsuited to drive near-eye
displays. We want near-eye displays to be low-cost, power efficient
and highly acceptable to users. To achieve this, we propose new
algorithms that can use GPU capabilities more effectively. We target
reducing redundant compute to enable lower latency and lower bandwidth
requirements through the graphics pipeline.
|