Interview zu FSR 2.0 - Englische Version

AMD's eagerly awaited temporal upsampling method FSR 2.0 recently had it's first apperance in Deathloop and shortly after that the Farming Simulatior 2022 and leaves a sublime impression considering the quality delivered in these first applications. We also received an invitation to interview Nicolas Thibieroz from AMD. As Director of Game Engineering, Nick is not only responsible for the development of FSR and the entire FidelityFX program, but also leads AMD's developer relations team, which is the link between AMD's hardware and effects developers and the game developers who want to use AMDs hardware and effects. But let's have Nick himself have his say. This is the english version of the interview, including some stylistic corrections and added remarks AMD suggested. You can find our first review of FSR 2.0 here (German).

Nicolas Thibieroz: Let's start with exactly what I do. I am the Worldwide Game Engineering Director at AMD which means my team gets to work with game developers worldwide to make their games look better and run faster. This is the responsibility of the Developer Technology ("DevTech") team but my organization also includes a research and development arm. This is the team that put together FSR 1.0, FSR 2.0, as well as other GPUOpen software, such as hybrid raytracing and FidelityFX Effects like FidelityFX Ambient Occlusion, FidelityFX CAS, FidelityFX Screen Space Reflections and more. It's a great job and it's quite fun to invent new technologies and get to see what all the game developers are up to in terms of new tech in their games. I'm very happy to be here to answer any questions you have on FSR 2.0, thank you for having me!)

PCGH: This is also what we like about our jobs.

Nicolas Thibieroz We love PCGH, by the way! I think you guys do a very good job, I'm very impressed by how quickly you produce benchmarks from games, you tend to be one of the one of the first technical press to release performance and quality analysis of new titles and their technologies... So yeah, I'm a big fan of what you guys do!

PCGH: Thank you. We tested FSR 2.0 in Deathloop - obviously, there are a lot of advancements regarding FSR 2.0 compared to 1.0, conventional upscaling and even the TAA in the game. But could you maybe describe what you think are the major benefits of FSR 2.0 are and how these might influence gaming in the longer run?

Nicolas Thibieroz: FSR 1.0 was released almost a year ago now. We feel we have a very good solution: very easy to integrate, very cheap solution in terms of performance cost, and cross-platform support. While we're happy with that solution we never intended to stop there. We knew that even though we had very good trade-offs in terms of all performance/quality , image quality was something we wanted to keep focusing on. And particularly, one thing we wanted to do is achieve better than native image quality.

At AMD we have multiple teams working on image upscaling research, and we set out a long time ago to actually develop a temporal-based upscaling solution. Temporal upscaling means that the upscaling you're doing to your image not only purely relies on a single source resolution frame, but also it uses additional inputs and previous frames to reconstruct a high quality upscale image. In addition to source resolution color buffer we also use a source resolution depth buffer, as well as the source resolution motion vector buffer. Motion vectors are the key trick in that algorithm as they tell you where pixels in the current frame used to be in the previous frame. This data essentially allows us to do supersampling over frames. And that's how we get much better quality compared to a pure spatial solution

Quality was definitely the number one item we wanted to improve for the FSR 2.0 solution and we're very happy with where we ended up. At the same time we wanted to maintain that broad compatibility with multiple platforms. At AMD we strongly believe in open source: being able to support multiple platforms so that multiple game developers, users benefit was also key to our goals.

At the same time it is critical that good performance was achieved. Upscaling is great, but if it takes a lot of GPU cycles to get there then obviously you start going into diminishing returns.

PCGH: Yeah, if upsampling is too expensive, it kind of becomes moot.

Nicolas Thibieroz Exactly right. And now, having said that, we have to be cognizant that a temporal solution -and in particular FSR 2.0- is quite complex. It's a very advanced algorithm with multiple passes needed to achieve the quality that it gets. It is definitely higher cost compared to FSR 1.0, but we spent a lot of time optimizing that algorithm. We even ended up doing multiple permutations based on different hardware architecture to extract best performance across not only across different vendors, but even across different GPU generations. I Overall I think we've achieved outstanding performance for the complexity of algorithm, meaning that a wide variety of GPUs and platforms get a good amount of performance scaling.

PCGH: Oh, nice - so FSR 2.0 uses different optimizations/paths of your algorithm for different hardware, interesting. Why is there only support for DX12? What about Vulkan and DX11?

Nicolas Thibieroz Yeah. So let me explain to you what will happen. Absolutely. During development, we wanted to focus on a single API for simplicity. Because during lead research and developments it's a bit early to try supporting multiple API's at the same time. You can do that once you have the product, if you want to, once you get to finalization stage. So yes, during development, we definitely focused on DX 12, I am happy to tell you that we will have Vulkan support at launch as well. Now a DX 11 is a bit more involved in terms of support. But what I can tell you is that any game developer, who are interested to support FSR 2.0 on DX 11, should contact us. We will probably not release it publicly. But we may have something or a version that we can share with developers that need to have it. So we're pretty covered, I think in terms of API support.

PCGH: How does AMD want to compel developers to use FSR 2.0 on a broader basis?

Nicolas Thibieroz: Do you mean encourage? It's a good question. When I heard the question, I thought "compelled" was a rather strong word as it could mean you're forcing people to do things. And that's not what we do at AMD: we don't shove technologies down the throat of developers (I won't comment on whether this happens with other GPU vendors). At the end of the day our job is to help developers integrate great technologies into their games. But for this to happen, we obviously need to make sure that our tech stands on its own merits. We have to provide technology that developers genuinely want to use, we have to provide technologies that actually solve developers' problems. That is the best way to achieve mass adoption, which is what you were hinting at earlier. If we solve developers' problems, AMD wins, developers win, gamers win, everybody wins. It's a simple as that and that's what I consider the job of my team to be. If you provide technology that is compelling to use, optimized, and allow developers to support multiple architectures and platforms to make the games better then it's a no brainer and adoption will naturally follow. In particular, we knew there was a gap to fill in terms of the availability of a high quality and cross platform temporal upscaling solution and we're very pleased with the response from game studios so far.

PCGH: Is FSR 2.0 also going to work on the consoles? Are there any recent endeavors by console developers using FSR 2.0 you are free to tell us about?

Nicolas Thibieroz: we can talk about console a bit. Obviously, there are things that I think you'd be better off talking to Microsoft and Sony about. But for instance, we've already announced that though, as far as Microsoft is concerned, that FSR 2.0 will be provided in the Microsoft GDK, the developer kit, as well, for any Xbox-developer to use. I would encourage you to speak to console game developers if you want more information on this as to whether FSR 2.0 will be available on other consoles. The technology is open source. And again, the full source of the API is provided. So it really is up to the developer if they want to port it to other platforms. There's nothing preventing FSR 2.0 from being ported to other platforms if they want to.

PCGH: Why did AMD choose to release FSR 2.0 as open source? Obviously, this potentially helps the adoption of FSR 2.0. But is there maybe another philosophy behind this decision?

Nicolas Thibieroz: You're right, it is definitely a philosophy. I mean, I'm assuming you're familiar with AMD. And at AMD, there's a long legacy of open source software, right? We generally believe, like I said earlier, that if the game developer wins, everybody wins. Like developers and their publishers get bigger, get better games but FSR 2.0 is a feature that also gives better performance to AMD and gamers win because all this tech results in better looking games that are also faster and so on. That open source nature of FSR 2.0 is, I think, key to the GPUOpen philosophy that we started quite a few years ago now. And in particular, to the FidelityFX brand, which is basically the brand of effects that we are shipping all our effects under.

For me open source is the best way to secure adoption. Why? Because if you have a black box (i.e. a library for which the source code is not provided), then you don't know what code is running on your target hardware, you don't really know or you can't really port it But it's not just that. Open source is also providing developers with confidence about the code actually running on their game, like the last thing, they want actually is some issue that they can't debug, right? Because something breaks basically in the black box library that they actually have. And also, for me, open source is the best vehicle to encourage innovation. If you have a black box, I mean, yes, it could be great technology, that you're integrating into the black box. But with open source, you're allowing developers to peek at the code, understand the code. And potentially make it better. Potentially optimize it for visual quality or for performance, and obviously help them with porting to any other platforms as well. So for me, that's important. And in the game industry, there's this long philosophy of sharing knowledge, right? And, you know, at the end, we see GPUOpen and the FidelityFX brand really as key enablers of this philosophy, and we want to play our part, you know. So that's why, GPUOpen or most of the content of GPUOpen and FSR 2.0 is open source. And that's why I prefer to use open source. That decision was particularly important to me and my staff, because we all strongly believe in open source. So again, I'm really happy that that's where we ended up with this.

PCGH: As a developer, how much work is required to use FSR 2.0 compared to FSR 1.0 - how does that compare to other modern techniques? You're also working on an FSR-2.0-API, or so we've heard - how is that going to work? Do you just kind of plug it into your game engine, basically like some kind of middleware, and that's it?

Nicolas Thibieroz: Yeah, these are good questions. So let me start from the start. So because FSR 2.0 is a temporal based algorithm, to produce the higher quality image, I think it's fair to say that it's more advanced than FSR 1.0, and therefore a bit more involved in terms of integration. So as an example, for instance, it requires more inputs. I think I mentioned earlier, like color inputs, as well as depth buffer and motion vectors, and even like some optional inputs for even better image quality that are required, or at least, that could be provided for a better result.

It also requires the game to support a temporal loop for rendering. So basically, the ability for the current buffer that is produced to be fed back to the pipeline, as well as the concept of decoupling display resolution from source resolution. Now, most games already support these features in one way or another. And if they do, then integration of FSR 2.0 will be significantly simpler. In particular, if they already support some form of temporal upscaling solution, then yes, the integration will be definitely, definitely simpler. So those cases, these are the best case scenario, pretty easy integrations. There still may be like a few special cases to take care of maybe when dealing with transparency, and mirrors and your these kind of things. But overall, this would be an easier integration.

For those games that don't already support decoupled resolutions and temporal feedback loops, then, obviously, that infrastructure work has to be put in place before for FSR 2.0 to be integrated. And then we're talking maybe a week, two weeks of work, and so on. So really, it can vary. So some games work pretty much right out of the box, with a straight integration. Actually Farming Simulator 2022 was an example of that. That was released yesterday, they integrated the API, they did very minor changes, very minor tweaks, and the result is really good. Right, some of the games were more involved. So it really depends on a game-to-game basis what kind of complexity that they have to deal with, to cater to special cases and so on.

In the case of Unreal Engine titles, like UE4 and eventually UE5 the integration is even simpler, because we have or we will have a plugin. All you have to do then is to basically integrate the plugin in your game, and maybe play with a few CVARs which are console variables to tweak the results based on, the artists preferences, and then integration, should, if you do good a job, provide very good results. But to your question earlier about the API: Yes, we do have an API provided. Like with FSR 1.0 it was so simple that we just provided the shader. And then people can just put it in, implement it into the game and off you go. You had good upscaling results. In the case of FSR 2.0 because we're talking about multiple paths in the algorithm, we decided to go with an API instead. We still provide a full source of the API, if a developer wants to modify the API itself. But typically, we expect most developers to simply call API points to basically create a context and dispatch. The dispatch to the actual FSR 2.0-calls, and maybe call a few, what we call optional functions, right, to maybe get helper functions to maybe, you know, deal with calculating the source position size from target resolution, these kind of things. (The API supports a few optional "helper" functions e.g. to calculate source dimensions). And then, the API will allow the integration to be successful, essentially. We even provide what we call callback functions. So those callback functions are for developers who really want to tweak, special parts of the algorithm, that, for instance, if you have to deal with special texture formats, that may not be supported by default in the API. So in that case, you would implement a callback for those functions, and automatically, then it will allow the integration from the API to work, without having to really modify the API itself. Again, if you want to, you can go all the way deep into the API to modify it. But we try to make it as easy as possible for that, hence, the use of helper functions and callback functions.
(laughs) Did that make any sense to you?

PCGH: Yes it actually has. That actually doesn't sound that complicated. Can you give us some other details? Like how many frames does FSR 2.0 actually use for its temporal reconstruction? I suppose, it depends?

Nicolas Thibieroz: All right. The answer is: it depends (laughs). The answer is actually a bit involved: I can't give you a straight number because how many frames of temporal history actually contributes to each frame depends on a number of factors. It depends on what quality mode is being used i.e. Quality, Balanced, Performance and Ultra Performance if supported. It depends on the amplitude of motion vectors. If the motion vector is null (meaning this part of the frame has not moved compared to the previous frame) then obviously we can include more frames of history compared to more "dynamic" motion vectors. It also depends on whether occlusion or disocclusion has happened in the frame - as in new geometry appearing in the frame, or being obscured by other geometry. Overall, I would estimate that between 18 and 72 frames of history are being used by FSR 2.0 to produce an upscaled image. How many exactly will depend on those factors, but this is the bracket I'm happy to share with you.

PCGH: Wow, nice. That seems to be quite a lot.

Nicolas Thibieroz: Well, but that also depends, right? If you have a very high frame rate, that's only a few milliseconds. The only way developers can affect the actual history contributions is if they want to reset FSR 2.0 frame accumulation history. This may happen in cases when the current frame data no longer has any correlation with the previous frame such as when you do a "camera cut".

PCGH: Cool. That dynamic upsampling also sounds pretty interesting, particular for console-developers I would think. Isn't that rather complicated? Isn't it difficult to temporarily reconstruct an image, if your source resolution changes all the time? Or so I would imagine.

Nicolas Thibieroz: I was actually surprised you asked this very good question, because it's a fair concern: if your source resolution changes on a regular basis, then you may have issues in terms of correlating actual motion vectors and their history data due to frame size differences. It turns out the FSR 2.0 algorithm is surprisingly resilient to input resolution size changes. But also in practice when you do Dynamic Resolution Scaling (DRS) frame size changes tend to be gradual: with a target frame rate set the source resolution size will typically vary smoothly to try to keep under the performance budget specified. Thus no large input size variations should occur, helping the DRS feature of FSR 2.0 produce compelling results. I will also say DRS is not only for console: PC can benefit as well and the guys that at Arkane Studios have done a good job integrating this feature in Deathloop.

PCGH: Yeah, and it does look very impressive in Deathloop. FSR 2.0, I have to say, it's a treat. I mean, it's the first game and so you can probably rather expect more improvements in the future. We did notice some small issues, I guess. One being particles, they get somewhat pronounced, highlighted kind of. And some of them tend to ghost a little bit. But I guess, there's always some side effects when using temporal upsampling compared to native resolution.

Nicolas Thibieroz: I think it's fair to say that no temporal upscaling solution is perfect. At the end of the day you're dealing with super sampling across frames, you're trying to make sense of the current image from previous frames' data where pixels may have been obscured, or maybe there is a whole bunch of smoke or other particles in front of the geometry you're trying to upscale etc... It is a complex problem to try to solve and currently there is no perfect solution. FSR 2.0 provides a very good result overall but it's fair to say there are limitations.

FSR 2.0 proposes some quality/performance trade-offs with the modes supported like Quality, Balanced, Performance, and even Ultra Performance mode for a maximum performance gain. If a user is comfortable with leaving a bit of image quality under the table, you can use Performance modes. And if you'd rather go for highest quality, then obviously, you go for Quality mode. So you know, there are trade-offs that will allow the user to select what they want.

I think I can also say that FSR 2.0 has been the topic of ongoing research for quite a while at AMD and it continues to be. I can't comment on future releases - as usual - I'm sure as a press member you're used to that (laughs). But I can tell you that FSR is here to stay and that we are going to keep working on improvements in the foreseeable future. It's not unreasonable to expect further improvements in terms of quality, performance, usability features, and so on. And I'm hopefully looking forward to talking to you more about those when we are ready to share more details about the future.

PCGH: That would be great. I was about to ask what you are going to do now, with FSR 2.0 being launched. But you are not actually going to have too much spare time, right?

Nicolas Thibieroz: Managing a pretty large team of DevTech engineers, already take a lot out of my time! But FSR 2.0 as you can imagine has been a big focus for us so no, there is no spare time (laugh). And certainly, our time is occupied with not only preparing the actual open source release of FSR 2.0 that will happen before the end of the first half of 2022, but also assisting our existing partners to integrate FSR 2.0 in their games.

Even after FSR 2.0 is released publicly we will try to help developers integrate it as we want to make sure that the tech is represented in the best way possible. A focus for us is really getting that quality and that performance where it needs to be at with all our partner titles. And I'm very pleased to work with so many talented game developers out there who we've been asking us about the tech.

PCGH: Yeah, I thought as much. Well, that's about it. I guess. That concludes my questions. Thank you very much for your time and your detailed explanations.

Interview with AMD regarding FSR 2.0 - Conclusion

In the about half-hour-long interview, Nicolas Thibieroz talked about some interesting features and revealed some exciting details. We wouldn't have guessed that FSR 2.0 can process up to 72 frames, depending on the image content and situation, for example. The information regarding APIs, implementations and details about the support on (next-gen-) consoles was also very interesting. That includes not only that Microsoft respectively Xbox supports FSR 2.0 by default via the software development kit (SDK), but also that Nick obviously cannot - or may not - say much about Sony and the Playstation.

It is also somewhat obvious that FSR 2.0 is not the end of the line - at least if Nick and his team have it their way. It will also be interesting to see how fast AMD's temporal upsampling will be adopted via it's API and also the integration into the Unreal Engine, as well as how and in which form FSR 2.0 might change in the future due to the open source approach, or maybe even be adapted to be used on other platforms or in perhaps more exotic applications. We remain curious and will gladly come back to Nick's offer to tell us more about AMD's temporal reconstruction technique FSR 2.0 or FidelityFX effects in general in a future interview.

Artikel teilen

Interview zu FSR 2.0 mit Nicolas Thibieroz, AMDs Director of Game Engineering Hier finden Sie das für unsere Übersetzung genutzte, englische Transkript des Interviews. Bitte beachten Sie, dass es sich um eine vergleichsweise grobe textliche Abschrift des originalen Video-Interviews handelt.

Per E-Mail versenden

22

1. Seite 1 Interview zu FSR 2.0 - Deutsche Übersetzung
2. Seite 2 Interview zu FSR 2.0 - Englische Version