“Ray tracing is setting off a new wave of more realistic graphics for the new generation of PCs and game consoles. This is a technology that is expected to be introduced into mobile devices, with the help of new and efficient dedicated hardware and design techniques. This is a real game changer. It will bring the world’s largest group of game players to developers, provide mobile device manufacturers with a new way of differentiation, and achieve a new immersive mobile user experience.
Kristof Beets, Vice President of Technology Foresight, Imagination Technologies
When was the last time you played a video game on a computer or game console? If it is one of the most popular games, you may have noticed how realistic the graphics are. Today, the graphics in the most cutting-edge PCs and game consoles are approaching the real pictures we see in movies.
When creating photorealistic real-time computer-generated graphics, one of the most important elements is the lighting of the scene. When simulating real lights and shadows, the traditional 3D graphics rendering method-rasterization-is not the most effective technique. This is because the working principle of rasterization is to mesh virtual polygons into 3D models and then decompose them into pixels that must be individually colored. This process requires extremely heavy work and computing resources, and usually requires complex development techniques.
Film animators have known this for a long time, so they use ray tracing technology to create effects such as global illumination, shadows, and reflections. Ray tracing is a simple and accurate method of simulating the behavior of light. It can make the scene look more realistic with less work. This technology is more in line with the principle of human eyes seeing the world. Light is reflected by objects in the environment, and light is absorbed/reflected based on the material of the object.
Ray tracing can achieve more realistic effects, and developers can use it to create a more immersive experience for gamers. At the same time, using ray tracing is easier to create these effects than using traditional rasters, so developers can free up time to deal with other aspects of the game.
However, the disadvantage of using ray tracing technology in games is that to get a truly immersive experience, ray tracing must be done in real time. This means that the image must be rendered within a few seconds, which makes it expensive in terms of processing calculations and resources.
Nowadays, the hardware on the market can achieve real-time ray tracing effects in game consoles and PC games, but it is selectively only used to illuminate the most critical objects in the scene. In some cases, developers can decide how to use it. . Considering performance constraints, developers can decide whether to use it to generate reflections or shadows, or use ray tracing entirely-they can choose to turn it off to optimize the frame rate. Game performance is the bottom line.
Compared with previous generations of technology, the current hardware with ray tracing improves performance by providing more ray tracing processing, but it requires a larger chip area and higher power consumption. Real-time ray tracing is currently extremely challenging in power-constrained devices, so this technology has not yet made its mark on the device screens of mobile gamers. However, according to relevant data, mobile games will soon occupy 52% of the global game market (NewZoo, 2021), and the experience of mobile games has become a direction that cannot be ignored in the future.
Graphics in mobile devices
Even without the help of ray tracing technology, the quality of mobile graphics is still improving. As graphics become more and more realistic, the ability to generate graphics using traditional technologies becomes more and more complex, which is pushing performance, bandwidth, and power consumption requirements to the limits of mobile processors.
Shadow generation is a good example. Shadow generation is traditionally done through cascaded shadow maps (CSM, Cascaded shadow maps). This process requires a lot of geometry processing, a lot of buffer allocation, processing of multiple rendering targets at high resolution, and expensive shaders. operate. In terms of processing cycle, power consumption and bandwidth, the cost is huge. Moreover, after performing all these operations, the result is still not as easy as using ray tracing, and the result created is so realistic.
Using ray tracing technology, light is sent from a single pixel to the light source. If the light hits an object, the area will be in shadow. One light per pixel makes this a simple, inexpensive and straightforward process, especially when dedicated hardware is in place. Many of the resolution issues, artifacts, and other challenges encountered by traditional rasterization (such as the offset required to avoid floating geometry) do not exist in ray tracing. As long as dedicated hardware can be implemented efficiently, creating lighting and shadow effects is a trivial challenge.
Today, the industry is at a crossover point, approximate operations using shadow mapping and other traditional techniques are becoming very expensive, and real-time ray tracing is becoming a more efficient choice. In addition, as Moore’s Law draws to a close, we can no longer rely on programmable hardware and still exhibit exponential growth every two years. Therefore, to continue to accelerate the advancement of ray tracing technology, especially for mobile platforms with limited power supplies, the industry must seek efficient and designed fixed-function accelerator solutions.
The key to continuously creating better graphics in mobile devices lies in optimizing technical efficiency for hardware and developers.
Current state of ray tracing hardware
Real-time ray tracing technology has long been the holy grail of 3D graphics. Ray tracing can be performed at different levels of performance and efficiency. In order to clarify this point, Imagination established the Ray Tracing Level System (RTLS), which determines 6 levels of ray tracing, from level 0 to level 5. The initial hardware accelerated ray tracing work-the RTLS level we call “level 0”, is not optimal. These solutions have limited functionality and require customized hardware and application programming interfaces (APIs). Of course, these solutions do not attract developers. Then, the market developed to a “level 1” solution, which uses software technology based on traditional GPUs. These solutions provide greater flexibility, but are still a long way from ideal.
Most of the ray tracing technologies we see on the market today (for example for PCs and game consoles) are what we call “level 2” RTLS solutions. This solution provides dedicated hardware for the most basic and extensive operations of ray tracing: a triangle/box intersection tester. Putting this operation in fixed-function hardware can achieve higher energy efficiency, but it is still not good enough for mobile devices. Because multiple levels of ray tracing processing will still be run on the shader on the GPU, these processing stages are not friendly to the GPU’s parallel execution engine. This also leads to lower traditional graphics performance and arithmetic logic unit (ALU) pipeline efficiency, which limits overall throughput.
The most complex ray tracing solution on the market today is at level 3. These solutions implement more ray tracing functions in dedicated hardware, reduce the burden on shaders, and improve efficiency. At this level, by traversing the Boundary Volume Hierarchy (BVH)-the main data structure of ray tracing, the complete ray intersection processing is implemented in dedicated hardware. This improves the efficiency of ray tracing for more complex scenes and better transfers the ray tracing function, reducing the impact on traditional graphics performance.
However, these solutions still lack a necessary component that makes ray tracing possible in mobile devices: consistent ordering. Since the light tends to be scattered in many different directions, if the consistency problem is not solved, the many parallel advantages brought by the GPU will usually be lost. This will result in lower bandwidth utilization, complex data access patterns, and reduced shader pipeline processing efficiency. Although these solutions may claim to achieve processing power of super gigabit light per second, their efficiency is usually low, which is related to the low utilization of GPU processing resources or memory access restrictions, which is caused by light scattering in the entire scene Caused by incoherent memory access patterns.
To realize real-time ray tracing in mobile devices, the key is efficiency. We need to take advantage of the inherent parallelism of GPUs and develop smart algorithms that optimize hardware. This requires smarter hardware solutions.
Intelligent ray tracing hardware
The industry must go beyond Level 3 hardware to achieve real-time ray tracing in mobile devices. At Imagination, we have provided 3D graphics technology to more than 10 billion mobile devices. We know how to provide stunning graphics in efficient hardware. In 2016, Imagination’s ray tracing development board was much more complex than the solutions on the market today. Now, we have brought a 4-level RTLS ray tracing solution to the market.
In the 4-level solution, the ray traversal of the boundary volume level (BVH) is completed in dedicated hardware like the 3-level, and more importantly, the light coherence sorting is also the same. In this process, we group the rays in the same direction, allowing large-scale processing to take full advantage of the parallel computing power of the GPU.
For a long time, we have been taking advantage of the inherent parallelism of GPUs, using techniques such as block-based rendering to improve efficiency through spatial layout and block sorting. Today, this is the generally accepted practice. Now, we bring the same idea to ray tracing. In this way, we have increased the overall utilization of the wide ALU and significantly improved the test efficiency. The impact on traditional graphics performance is minimal, because we almost completely transfer ray tracing processing to dedicated hardware, so that shaders can be used for other non-ray tracing graphics processing.
Our Level 4 RTLS solution represents the most advanced ray tracing solution available today, and devices using this solution will soon be available on the market. With the help of Imagination Level 4 ray tracing IP (IMG CXT GPU IP), the company can build up to 9TFLOPS of FP32 lighting performance and ray tracing performance of more than 7.2G ray/s, while providing up to 2.5 times the power of the Level 2 and Level 3 solutions. Consumption efficiency.
When we launched IMG CXT, we did not stand still. We are already demonstrating how to implement fully programmable ray tracing technology using a 5-level ray tracing solution.
Developer optimization technology
Even the most efficient hardware requires technical assistance to provide the best graphics quality on mobile platforms. The optimization of mobile devices is even more important, because game consoles or PCs bear the problem of excessive availability of bandwidth and power budget to some extent. Mobile phones do not have this kind of brute force tolerance. Everything must work under a tiny form factor and run on the energy provided by a tiny battery. The characteristics of mobile phones mean that the hardware must be efficiently driven by the game engine to ensure that the mobile phone will not overheat and reduce the frame rate. This means that one of the keys to creating hardware for mobile devices is to optimally manage the impact of battery heating.
The first thing we tell developers is that less is better: not every pixel of the scene needs ray tracing. We recommend that developers use their ray tracing budget to achieve maximum visual impact and gaming experience value. Also, it is too complicated to test the light of every triangle. By building an effective hierarchy, developers can improve hardware efficiency when determining the triangle to test. Building the best acceleration structure is usually best done offline.
Developers should also avoid all brute force algorithms. Effects like soft shadows can be achieved with smarter sampler modes, combined with spatial filters, instead of emitting a lot of light per pixel. In addition, most ray tracing effects can be rendered at a lower resolution, for example, processed at a quarter resolution and then scaled up. Then, you can use the temporal and spatial characteristics of the image to use denoising to reduce the ray tracing noise caused by sparse sampling. Many of these stages can be combined and run efficiently on traditional GPU computing hardware. It can also use dedicated neural network acceleration engines, which already exist on almost every mobile phone.
Developers can also use many other technologies to optimize mobile devices, such as motion detectors for global illumination, careful use of application programming interfaces (APIs), etc. In the final analysis, the combination of efficient ray tracing hardware and intelligent development and optimization will be the key to real-time ray tracing of mobile devices.
Game changer for mobile devices
Ray tracing is setting off a new wave of more realistic graphics for the new generation of PCs and game consoles. This is a technology that is expected to be introduced into mobile devices, with the help of new and efficient dedicated hardware and design techniques. This is a real game changer. It will bring the world’s largest group of game players to developers, provide mobile device manufacturers with a new way of differentiation, and achieve a new immersive mobile user experience.