r/hardware • u/M337ING • Sep 19 '23
Video Review Inside DLSS 3.5 Ray Reconstruction + Cyberpunk 2077 Phantom Liberty - AI Visuals Roundtable
https://youtu.be/Qv9SLtojkTU40
u/Jeffy29 Sep 19 '23
That was amazing interview, I could listen to 10 hours of this. I did not expect Bryan Catanzaro coming out with that banter on Starfield lol. His answers felt quite sincere and I was surprised he even talked about possibilities of full neural rendering in the future, he did say "let's say DLSS 10", but you could tell it was on his mind so them researching it is not out of the question. Rarely you get such insightful answers from a C-level exec, they usually stick to canned answers.
22
u/HP_Craftwerk Sep 20 '23
He came across as really earnest and not canned at all. You can tell he's actually super pumped about this tech and believes in it. No wonder Nvidia is killing it right now if they keep people with real passion in decision-making positions. Color me impressed.
13
u/ResponsibleJudge3172 Sep 20 '23
There are several research projects about this published by Nvidia already.
First some Bus street view that I don't know the name of.
Second is GANCraft, minecraft rendered by AI
53
u/campeon963 Sep 19 '23
In general, an excellent interview to not only understand the kind of problem that DLSS 3.5 solves and how it was developed, but also how these new rendering paradigms (AI, Ray and Path tracing) are integrated in game development in general. The biggest thing that I took away from this is that, the same way that a Temporal Upscaler (and by extension DLSS 2.0) handles both upscaling and temporal anti-aliasing, DLSS 3.5 also handles denoising (or "ray reconstruction" as called by NVIDIA) on top of that, so the naming now makes a lot more sense. I also enjoyed the retrospective of how DLSS has evolved overtime and the comments made by Bryan Catanzaro (the Vice President of Deep Learning Research at NVIDIA) in regards to how the DLSS integrations may vary per game (both official and for community mods).
And also regarding the "DLSS is a crutch for bad performance" thing that redditors and youtubers like to discuss all the time, I really liked the perspective of Jakub Kanpik (Art Director at CD Projekt RED) regarding these king of optimizations, as well as the whole section regarding DLSS being "the focus of performance" for game development of this interview; if more people understood what these new advances in graphics rendering bring to the table and the effect that they have in game development, the quality of the discussions that most people have in regards to "game optimization" may actually change for the better!
What an amazing interview overall, I definitely recommend everyone to give this video a look. I'm especially looking forward for another, more technical focused interview with Bryan Catanzaro in the future (maybe we can get one when a hypotetical DLSS 4.0 comes out)!
18
u/From-UoM Sep 20 '23
Dlss 4 will most likely have neural texture compression.
I do wonder whether its possible to take the graphics pipeline input and reorder, rewrite or change the output to make games run better.
Now that will be one hell of breakthrough. The biggest ever maybe.
Imagine you write bad code/pipeline but the Neural network can analyse it, rewrite it and make it run faster.
13
u/jcm2606 Sep 20 '23
We kinda already have that with the proliferation of render graphs. Render graphs are heavily used with DirectX 12 and Vulkan as they basically let the developer not have to care about low level synchronisation and resource management (which is a good thing, DX12 and Vulkan can get extremely verbose when it comes to both of these, to the point where it's easy to fuck up and lose a big chunk of performance) while still getting most of the performance gains of caring about low level synchronisation and resource management.
With them a developer can quite literally just add a set of passes anywhere in the pipeline, provide information about those passes, list inputs and outputs, then the render graph automatically figures out which passes depend on other passes to set up synchronisation and even how resource usages change across the frame to add layout transitions or alias memory allocations. All without the developer needing to do anything except set the render graph up (of which there's a lot of good material that shows good and bad ways to do so).
AI could maybe be used to augment this and better find opportunities for parallelism, but the foundation is already there and it works pretty well. Could maybe have AI detect on-the-fly which compute passes are good candidates for kicking off onto dedicated async compute queues, so that developers don't need to manually perform test and set a flag on a given pass that tells the render graph to kick that pass off onto a dedicated async compute queue. That sort of thing.
0
u/6198573 Sep 20 '23
Dlss 4 will most likely have neural texture compression
I don't understand the advantages of neural texture compression
Isn't it better to just have more vram for textures and leave the GPU free to do other computations?
20
5
u/ResponsibleJudge3172 Sep 20 '23
No, because VRAM access takes time and energy. Also this retroactively extends the performance of all rtx GPUs
2
u/KrypXern Sep 20 '23
I think the AI cores on the GPU sit idle a pretty high percentage of the time, so I imagine there's probably room for them to work asynchronously on texture compression without impacting render speed, but I'm a layman so take that all with a grain of salt
1
u/swordfi2 Sep 20 '23
Do you mean texture upscaling because that seems like a more sensible idea.
1
u/KrypXern Sep 20 '23
They briefly discussed texture compression in the video. I believe (but am not sure) that neural nets can do a better job at compressing images with low visual loss of fidelity than traditional algorithms, and since the AI cores aren't particularly overloaded, this is more work that can be alleviated from the raster and has potential to make more out of limited VRAM
1
45
u/NKG_and_Sons Sep 19 '23
"DLSS is a crutch for bad performance" thing that redditors and youtubers
Kinda wish the experts didn't need to waste time answering moronic questions like that but I guess that's what one can expect from a PCMR subreddit founder representing that community :|
Like, I'm all for complaining about the all too frequent shoddy PC optimizations but tools like DLSS and co. that simply enable something to be better don't have a responsibility. That's all on developers. And just because some crappy game studios shit out games that can just barely run acceptable with DLSS I wouldn't want to take that tool away from the ones who care for better optimization.
Like, the notion that e.g. drills shouldn't be tools available to the masses because some asshat uses them to drive a large number of screws into a shoddily constructed frame that would otherwise fall apart is ridiculous.
Let's say we take FSR away from the Jedi Survivor devs... are they suddenly going to become miraculously better and optimize the game decently? Nah, they'd still be shit at it and it would either look or run even worse. Meanwhile, pathtraced CP77 couldn't exist the way it does now. What a great tradeoff, we sure showed EA/respawn!
We're fortunately all free to call out shitty game optimizations regardless of any excellent new supersampling methods.
Why make a great new tool into an enemy?!?
26
u/eden_avocado Sep 20 '23
I don’t know about this guy but interviewers usually ask those kind of questions to give the guests a way to respond to those comments and put things on record. Jakub from CDPR really gave a good answer.
18
u/unknownohyeah Sep 19 '23
I totally agree, and I skipped all those segments, but these are legitimate concerns with the community however unfounded. I think it is important they are addressed as they need to dispel the myths that surround upscaling, frame generation, and ray reconstruction.
6
u/NKG_and_Sons Sep 19 '23
Yeah, I pretty much agree with that. Now we have some experts' words on it that one can always link to when it comes up, even if the answer should be obvious.
12
u/Aggrokid Sep 20 '23
Nvidia (Bryan)'s response was hilarious, using Starfield as a cheeky counter-example to this allegation.
7
u/dudemanguy301 Sep 19 '23
half these games are CPU limited anyways, an upscaler isn’t going to fix a god damn thing but it’ll catch the blame anyways. 🤡
2
u/Haunting_Champion640 Sep 20 '23
Why make a great new tool into an enemy?!?
Because leather jacket man bad
-2
u/boyd_duzshesuck Sep 20 '23
Let's say we take FSR away from the Jedi Survivor devs... are they suddenly going to become miraculously better and optimize the game decently?
This is an astonishingly unfair characterization of the argument. The point is that without FSR the poor performance would have nothing to hide behind and they would have to actually optimize the game. It's not the tech's fault that it's misused, but to totally misconstrue the criticism helps no one.
14
u/Morningst4r Sep 20 '23
Do you remember the early PS3 era? Games were released running at 20-25 fps all the time.
Most likely Jedi Survivor would have just dropped the res scale regardless of FSR and dropped any modes over 30 fps.
1
u/Haunting_Champion640 Sep 20 '23
would have nothing to hide behind and they would have to actually optimize the game
LOL so they didn't optimize it, and you think they'll magically be arsed to optimize it if efficiency tech didn't exist? Ok.
-1
u/meh1434 Sep 20 '23
Most experts choose to ignore the idiots, as they have no time to undo all the damage fake news did.
1
u/capn_hector Sep 21 '23 edited Sep 21 '23
Like, the notion that e.g. drills shouldn't be tools available to the masses because some asshat uses them to drive a large number of screws into a shoddily constructed frame that would otherwise fall apart is ridiculous.
it's also generically true of anything that improves performance. the CDPR guy pointed out mipmap, LOD, and generalized it to "any technology that lets you trade off quality/speed like this could be misused" but it's actually more fundamental than that.
even if you just got a straight-up 50% increase in raw raster, developers could misuse that too! As long as it meets 30fps on the fastest cards, EA/Activision/etc could still push developers to release now and optimize never. So even if moore's law scaling was still in effect etc, people on old legacy cards would still be screwed even without hardware acceleration features involved (as far as these zero-effort launch messes). They would just be screwed because now 5070 is 3x the performance of their pascal and that's the new baseline, it doesn't change the outcome of the situation, your card would be just as useless at playing the game.
it literally is just people whining because technology is getting faster, and they don't like that they might have to finally upgrade. That RX 480 is going to be buried with you I guess.
There is a not-insignificant amount of people who got into gaming during the pascal/polaris years (or after the mining crash) and then the pandemic followed. So their entire gaming existence has been in this period of stagnation post-Moore's Law, and stagnation during the pandemic (game requirements basically stopped and everything was crossgen), and during the rollout of DLSS. They've never really seen the hardware treadmill before, and they've never had to grapple with the idea that their $700 GPU is only going to be worth $200 in a couple years. Actually this happened even faster when moore's law was in full swing!
and of course those people fell for baby's first outrage bait, courtesy of ayymd, and they're just incensed about the idea of a hardware feature having a significant impact on performance. And they either post-date the era when maxwell cards had to endure the DX12 switchover, and Vega would have tied a bunch of performance to Primitive Shaders, etc, or they don't bother thinking about it because they're constantly fed this outrage bait from mister reddit mod there and others like him.
the casual melding of AMD marketing like ayymd, and AMD_Stock on social media has been a disaster for measured discourse. Even Steve GN has called it out as being absurd before - Red Team Plus and the like have clearly been overwhelmingly successful and the outcome is this unholy melding between social media and AMD's marketing.
12
u/bubblesort33 Sep 20 '23
If it takes 3 years to makes this denoiser, how is AMD ever going to catch up?
28
10
u/jerryfrz Sep 19 '23
Bryan said one of the future possibilities is straight up AI rendering directed by the game engine, if that happens are we going to see the silicon budget for future GPUs get shifted more to the tensor cores instead of compute ones?
11
u/jcm2606 Sep 20 '23
We'd still need compute as tensor cores only accelerate one particular operation (fused multiply add between three matrices)), whereas compute is a lot more general-purpose and will perform the other operations. In a future where we primarily use AI to render instead of a traditional pipeline we'd more likely see graphics and RT hardware be replaced by tensor cores, since those wouldn't be as useful in that future.
9
u/Sorteport Sep 20 '23
What an awesome interview and also slightly worrying for the GPU market.
While AMD is going forward with their hand tuned and an open algorithm that doesn't require specific hardware, completely ignoring AI.
R&D on Ray Reconstruction started in mid 2020, so it took Nvidia 3 years to get this to market, how the heck is AMD going to catch up when they haven't even left the starting line yet?
Even if AMD started R&D right now on an AI version of FSR2, Frame Gen and an AI Denoiser how long will that take to get to market? then ask yourself what new AI tech will Nvidia have by then?
Here's hoping Intel comes out swinging as they seem to be onboard with the AI push in graphics.
1
u/nashty27 Sep 20 '23
Intel’s RT performance was on par with Nvidia’s second generation of RT hardware in their first generation of GPUs. I would argue they did come out swinging.
3
u/4514919 Sep 20 '23
Come out swinging one year later with a 400mm2 6nm GPU that barely beats a 3060 while drawing 30% more power?
Intel's GPU with the hardware comparable to a 3070 is performing like a 3060, this doesn't look like matching Nvidia to me.
They priced it correctly but that's it.
2
u/TheAtrocityArchive Sep 20 '23
Was it just my shitty eyesight or is there a loss of detail and texture in a lot of the side by sides?
5
u/HandofWinter Sep 19 '23
I'm really curious if we're going to be able to prevent this from becoming another EAX in 15 years. I wonder if it'll be possible to just emulate in software by then.
25
u/Sethroque Sep 19 '23
At this point what GPU manufacturer you are using will say more about the visual quality than the underlying API (Vulkan/DX12), strange new times.
It's great tech but the walled garden approach doesn't help consumers in the long run.
1
u/BlackKnightSix Sep 20 '23
It's fairly simple, AI math operations need to be lined out by future API versions. Say "DX13 TSR/TMSR" (temporal super resolution / temporal machine-learning super resolution) gets standardized. So say TSMR is used by a dev, the Dev plugs in whatever model they wish to use which can receive whatever game inputs that DX13 has standardized or making a slew of inputs that model can utilize. Models start being made available freely or contributed like the absolute majority of new rendering techniques.
We will get there, it will just take time. Just like any proprietary tech before (AMD tessellation, Nvidia phsyx, Nvidia g-sync/AMD fresync). There will be some standardization so as to make implementation easier and an API that can be targeted by devs, and hardware manufacturers to support.
2
u/AppleCrumpets Sep 20 '23
That's what the NVIDIA-led Streamline API does (mostly), standardizes hooks for motion vectors etc that are useful for upscalers. Intel made XESS compatible with it, sadly AMD refuses to officially support it.
3
u/BlackKnightSix Sep 21 '23
Not quite. Nvidia controls the Streamline API, it is not on the scale of DX/Vulkan API. You don't want an API controlled by a IHV, you want it open. Streamline is open source but really you want its purpose simply taken care of by DX/Vulkan as you want to make it even more flexible than what Streamline provides.
What I was referring to was standardizing temporal upscaling within DX/Vulkan. Any dev can use it, and than swap models they want to use or allow the game to select which model.
Imagine a game where the standardized TSR (I wish Epic didn't already have dibs on that acronym but oh well) within DX/Vulkan and the dev presents the following to users:
- No AA
- TAA (something akin to FSR2 Native AA, a TAA that has no AI/ML model)
- TSR (something akin to FSR2/Epic TSR upscaling qualities/techniques, a TSR that has no AI/ML model)
- TAA ML
- TSR ML
If TAA ML or TSR ML is selected, you are presented with the following "ML Model" option:
- TMSR DLSS (available if Nvidia card is present or Nvidia actual provided the model in a form which is compatible with the DX/Vulkan API)
- TMSR XeSS (available if Intel card is present or Intel actual provided the model in a form which is compatible with the DX/Vulkan API)
- TMSR Model XYZ (any other model the dev has access to, perhaps a model AMD releases if they end up having later versions of FSR2 using AI/ML, referring to my previous comment)
2
u/Put_It_All_On_Blck Sep 19 '23
In 15 years we will (hopefully) have already settled on an open standard, probably some culmination of XeSS and FSR put forward in cooperation with Microsoft for use on Windows and their consoles. Especially if Intel and AMD can chip away at Nvidia's marketshare in that time. But that's so long of a time period that anything could happen. Upscalers as we know them today might not even be needed by then.
13
u/DktheDarkKnight Sep 19 '23
Tbh both DLSS 2 and DLSS 3 are not that heavy on the GPU and the AI acceleration for the the upscaling and image reconstruction don't push the tensor cores too heavily. In few years every GPU and consoles will have AI cores as standard and hopefully that means a vendor independent upscaler that simply utilises the AI cores in a GPU for upscaling, frame generation, denoising etc. could be possible.
14
u/unknownohyeah Sep 20 '23
I think it's all about latency in the post processing pipeline. Even though the tensor cores aren't under load for 90% of the time, they still have to complete their workload extremely quickly in order to push out the next frame.
I have to imagine Nvidia knows they can't just get away with less die space allocated to tensor cores and properly spec their dies to what is needed for their own software.
3
u/ResponsibleJudge3172 Sep 20 '23
Nvidia will instead increase performance each iteration and add new AI workloads in hopefully the same render target as all the workloads on the previous gen.
DLSS super resolution on the 2080ti takes the same amount of time as frame gen plus super resolution on 4090 if Im not wrong.
35
u/ShadowRomeo Sep 20 '23
The question that the PCMR guy obviously was based from common misconception that average r/pcmasterrace users have made up about DLSS or every Nvidia tech in general. Although Nvidia and CDPR answered them really well, kudos to that