In the ever-evolving world of gaming, graphics quality is a paramount concern for both gamers and developers. To address the demand for improved visuals while maintaining optimal performance, two prominent companies have introduced technologies to improve graphics: AMD’s FidelityFX Super Resolution 2.0 (FSR) and Nvidia’s Deep Learning Super Sampling (DLSS). This article will explore the philosophy behind the technology and the difference between these techniques.
As hardware, particularly GPUs, continues to improve in their technical capability, so too does the graphics quality of games. However, the progress in rendering technologies and the GPUs within our PCs and consoles does not always synchronize seamlessly.
Over the past few years, since the introduction of Nvidia’s Turing GPUs to the market, Raytracing has gained widespread popularity. For the first time, it became feasible to implement raytracing in interactive environments, such as games. In our context, when referring to real-time raytracing, we are alluding to mathematically accurate (although precision can be debated) lighting that closely mimics the behaviour of light in the real world.
While image upscaling and image reconstruction are distinct concepts from raytracing, the impetus for these image reconstruction techniques primarily arose from the emergence of raytracing in gaming. This led to the engineers asking themselves a simple question: Why not render the graphics at a lower resolution, then execute all the light bounces and interactions on this smaller frame, and subsequently upscale it to a higher resolution for the target display?
This elementary idea is the foundation for our exploration of FSR and DLSS, along with their various iterations. Below, we present essential insights into each of these technologies.
AMD FidelityFX Super Resolution 1.0
AMD’s initial implementation of FSR 1.0 made its debut on 21 June 2021. What sets this first iteration apart is its unique approach as a spatial upscaler. FSR 1.0 operates by taking a low-resolution current frame and upscaling it to the target resolution. Notably, unlike FSR 2.0 and alternative techniques, FSR 1.0 is applied to frames that have already undergone an anti-aliasing pass. Consequently, it functions relatively late in the rendering pipeline, performing its tasks just before rendering elements like the HUD.
The simplicity of FSR 1.0 lies in its independence from external data sources beyond the current anti-aliased frame. Conceptually, it follows a two-pass process. First, data from the frame undergoes an up-sampling pass, and subsequently, a sharpening pass is executed to extract pixel detail within the upscaled image.
FSR offers a range of quality modes, spanning from ultra quality to performance, each with distinct internal resolutions and scale factors. For example, the ultra-quality mode employs an input frame that’s 77% of the target frame size. This means that activating ultra quality for an intended 1080P output involves upscaling from a 1477×831 image.
Furthermore, FSR features various scaling modes that function similarly by adapting the input frame size accordingly. These scaling modes are not fixed and can be dynamically adjusted, particularly when a specific target frame rate is required. In such cases, the scaling modes adapt on the fly to achieve the desired frame rate.
As it currently stands, there are few compelling reasons to utilise FSR 1.0, unless, for some unusual reason, the particular game lacks support for the newer model. Despite potential performance improvements, it’s important to note that FSR 1.0 tends to produce noticeable issues such as edge crawling, shimmering, and other undesirable artefacts.
As a result of how FSR 1.0 functions, it doesn’t rely on any special maths units or hardware to operate and is thus usable on a number of GPUs.
AMD FidelityFX Super Resolution 2.x
FSR2 and its various iterations, including the latest 2.2 version, diverge significantly from FSR 1.0 in several key aspects.
Perhaps the most obvious difference lies in its temporal approach, in stark contrast to the spatial method of FSR 1.0. Consequently, FSR 2.0 has greater computational demands. Moreover, its integration into the rendering pipeline occurs at an earlier point and effectively replaces the game’s built-in Temporal Anti-Aliasing (TAA).
FSR 2.0 also distinguishes itself by using more data points. This expanded data intake allows FSR 2.0 to conduct a more comprehensive analysis of the scene, resulting in the production of a significantly superior image. Beyond frame data, FSR 2.0 takes into account information from the depth buffer, motion vectors, and colour buffer to produce the output frame. This introduces some complexity, but the benefits to image quality make it a worthwhile trade-off when compared to FSR 1.0.
Another notable change in FSR 2.0 concerns its predefined scale factors. Due to its access to a more extensive dataset of the scene, the highest quality mode in FSR 2.0 operates with a smaller source image as input. For instance, while the highest quality mode in FSR 1.0 required a 1477×831 image to render a final 1920×1080 image, FSR 2.0 produces a superior output using a 1280×720 source. This adjustment applies across all modes, resulting in smaller input frames compared to what FSR 1.0 necessitated.
FSR 2.0 has garnered widespread adoption due to its superior image quality and performance, surpassing its predecessor by a significant margin. FSR 2.0 is what you will find in most games today. FSR 2.0, much like its predecessor, doesn’t require dedicated hardware units but relies heavily on functionality that’s available across all modern-day GPUs from every vendor.
Nvidia Deep Learning Super Sampling (DLSS) 1.0
The initial incarnation of DLSS is a spatial image upscaling technique comprising two stages, both reliant on neural networks. The first phase involves an image enhancement network, which utilises the current frame and motion vectors to execute edge enhancement and spatial anti-aliasing. The subsequent stage encompasses an image upscaling process that employs a single raw, low-resolution frame to expand it to the desired output resolution. These neural networks undergo training on Nvidia’s supercomputers, tailored to each game’s specifics. This training involves generating a “perfect frame” through traditional super sampling, along with incorporating motion vectors.
Unfortunately, this initial iteration proved underwhelming, yielding a relatively soft image susceptible to artefacts. Similar to FSR 1.0, these issues were attributed to the limited scope of data, as it relied on a single frame for input. Consequently, the neural network excelled primarily in scenarios closely resembling its training data.
DLSS 1.0 swiftly gave way to DLSS 2.0, resulting in limited support for the former version in only a handful of games, including Anthem, Battlefield V, Final Fantasy XV, Shadow of the Tomb Raider, and a select few others.
Nvidia Deep Learning Super Sampling (DLSS) 2.0
DLSS 2.0, unlike the spatial nature of its predecessor, is a temporal anti-aliasing upsampling (TAAU) solution. It sets itself apart in that it leverages data from previous frames to address fine details and mitigate aliasing. The information it collects encompasses a raw, low-resolution input frame, motion vectors, depth buffers, and brightness data from these previous frames as well. It thus deviates from DLSS 1.0 in that it’s not trying to generate new data from a low-resolution source exclusively but incorporates previous frame data as well.
In addition, DLSS 2.0 adopts an unsupervised learning model within its neural network, capable of identifying and rectifying temporal artefacts, in contrast to the manual programmed heuristics employed by DLSS 1.0. Consequently, DLSS 2.0 excels in detail resolution and exhibits a superior ability to reduce temporal artefacts. This enables DLSS 2.0 to occasionally yield a sharper image compared to rendering at higher resolutions or even native resolutions using conventional. However, it’s important to note that no temporal solution is flawless, and while some artefacts may still be observable, they are typical of a much smaller magnitude.
A pivotal aspect of DLSS 2.0 is the transformation in how the AI is trained. After extensive data ingestion by the neural network, patterns emerge that facilitate the identification of typical errors across a diverse array of scenes. As a result, DLSS 2.0 exhibits adaptability across numerous games without necessitating specific retraining for each individual title. This adaptability is a key reason why modders are capable of implementing DLSS 2.0 into multiple games, even if the developer has not officially incorporated it.
Currently, DLSS 2.0 stands out as the most widely available image reconstruction technique in use today.
NVIDIA Deep Learning Super Sampling (DLSS) 3.0 | 3.5
DLSS 3 largely maintains the foundation established by DLSS 2.0, but with a noteworthy addition: the integration of an optical-flow frame generation technique. In simpler terms, optical flow is the computer’s method for discerning motion by tracking the movement of individual points within an image from one frame to the next. This technique aids the GPU in comprehending and analysing movement within the rendered scene.
The DLSS frame generation algorithm employs two rendered frames from the rendering pipeline and employs optical flow to create a new frame that smoothly transitions between them. As is customary with Nvidia, this technology relies on specialised hardware, specifically the new optical flow accelerators within the RTX 4000 series of GPUs. While it is theoretically possible to implement this on Ampere and potentially Turing-based GPUs, the outcome is unlikely to meet satisfactory standards in terms of image quality, performance, or both.
This leads us to DLSS 3.5 and its ray reconstruction capabilities. One drawback commonly associated with denoisers is the potential loss of original lighting detail within the scene. DLSS 3.5’s ray reconstruction tackles this issue by employing a single AI model trained on a considerably larger dataset. This model preserves lighting data and thus enhances the scene. It achieves this by replacing multiple denoising algorithms with a singular, more effective approach, resulting in improved performance and superior image quality.
DLSS 3.5’s ray reconstruction functionality is compatible with all Nvidia RTX GPUs, spanning from the RTX 2,000 series to the latest iterations. Importantly, it does not introduce any additional performance overhead while providing image enhancements, and in certain scenarios, it even leads to an improved frame rate.
DLSS 3 Frame Generation is already integrated into several games, and it appears that DLSS 3.5’s ray reconstruction will follow suit. These technologies—one enhancing performance and the other improving image quality—are pivotal in Nvidia’s marketing, which establishes the merits of the GeForce RTX 4000 series of graphics cards.
The Takeaway
In conclusion, we’ve discussed the image reconstruction techniques currently available, which most users are likely to encounter. While AMD has announced FSR3 and recently showcased a demo at Gamescom 2023, it has not yet been implemented in any games, and detailed information on its functionality remains limited.
That being said, if you are using an Nvidia GPU, the majority of the image reconstruction techniques we’ve discussed are accessible, depending on the age of your GPU. However, if you are using an AMD GPU, your primary option is FSR. Nevertheless, this limitation does not necessarily have negative implications because 9th-generation consoles like the Xbox Series X are already utilising FSR in their various forms. This implies ongoing development and enhancements to the technology, even though it may not currently match Nvidia’s DLSS feature set precisely.
In the future, when AMD FSR3 becomes available, we will provide coverage of both FSR 3 and Intel’s XeSS technology as the only two technologies that have cross-vendor support.