Nvidia's Jen-Hsun Huang reflects on how AI already creates pixels and entire frames, before saying that 'games will be generated with AI'

In a Q&A session at this year's Computex event, Nvidia CEO Jen-Hsun Huang was asked whether AI will be used to generate games' graphics directly, helping the traditional rasterizing method. After pointing out that neural graphics were already in use, through the likes of frame generation, Huang went on to state that AI would go on to infuse games and PCs, creating high-resolution objects, textures, and characters.

"We already use the idea of neural graphics," said Jen-Hsun Huang in the Q&A session. "We can achieve very high-quality ray tracing, half tracing 100% of the time and still achieve excellent performance. We also generate frames between frames, not interpolation but frame generation. And so not only that, we generate pixels, we also generate frames."

Most PC gamers, especially those with an Nvidia graphics card in their rig, will know that AI is currently leveraged in games via DLSS—initially just an upscaling system but now comprising a frame generation system and a neural denoiser for cleaning up ray-traced images. In the case of DLSS Super Resolution, the upscaling isn't done by AI. That's handled by a normal shader routine but the resulting image is then scanned and corrected by a neural network.

In the case of DLSS Frame Generation, two previously rendered images, along with some other information from the rendering pipeline, get fed into a different neural network. This one has been trained on how motion affects images and the result is an entirely new frame, that gets inserted in between the other two frames.

It doesn't have to be a completed frame that can be upscaled or generated in this way, as textures are also a 2D grid of pixels. In theory, any data array could be processed by AI and then improved, making it higher in resolution. And that's exactly what Huang was referring to—taking meshes and textures, low in detail, and using AI to create better versions of them.

Huang agreed, saying "The future will even generate textures and generate objects and so that the objects will be lower resolution...textures can be low quality and generate higher quality. We can even compress it. So we're gonna get richer and richer games in the future."

By doing so, less VRAM is needed to store all the assets, less bandwidth is required to transfer them, and less time is required to create them in the first place. In short, developers can speed up the process of making a game and gamers potentially get more performance.

Nvidia's CEO went on to describe a scenario where you log on to your favourite multiplayer game and play a few rounds with some friends. At the moment, if you don't have enough people for a round, many games will drop a bot into the slot, but anyone who's played against bots in Counter-Strike 2 will know that they are seriously limited in scope. But what if you had an AI taking on that role instead? What if it acted so human-like, you wouldn't be able to tell the difference?

This is the future of gaming that Jen-Hsun Huang envisions: "Inside the games, all the characters will be there. Right? It's like having you go into A into A into a battle with six colleagues. And you know all those six colleagues, maybe two of them are real people. The other four are AIs, and they can play with you for a long time. So they actually remember you."

For some people, having everything AI-generated might be quite concerning or, at the very least, tantamount to being a 'fake' game. But consider this: upscaling has gone from being glitchy, weird, and very optional to something that's often completely transparent and automatically enabled by the average PC gamer. And that's just in five years.

It's not hard to imagine that in another five years, game developers will have access to an API that lets them generate high-resolution assets via the GPU, all without the player ever noticing it take place. Would anyone complain about this if it simply just worked?

Is an AI-powered NPC any different to a standard PC-controlled bot? Is an AI-improved character model any different to a mesh that's been improved via tesselation and geometry shaders? Would it matter that a neural network could be used to constantly check your PC and game, as well as how you play it and tweak everything behind the scenes to keep it all running as best as it could?

Or as Jen-Hsun Huang put it: "AI infused with all aspects of gaming and PCs."

We're far off the point where any games are powered this way, let alone all of them, so it is perhaps nothing to be even remotely concerned about right now. However, some might say that if the entire gaming PC ecosystem becomes dependent on AI, then consumers will become beholden to the hardware manufacturer(s) who have the capability of doing this the best.

Computex 2024

Catch up with Computex 2024: We're on the ground at Taiwan's biggest tech show to see what Nvidia, AMD, Intel, Asus, Gigabyte, MSI and more have to show.

Graphics cards are now one of the most expensive parts of any gaming PC and the cost of the most powerful ones is so high, that you could buy an entire mid-range rig for the same amount of money. If the best graphics or best performance in games is only possible on the most AI-capable graphics card, would the market become dominated by that particular vendor?

Of course, this is already the case, as Nvidia holds the majority share of the discrete GPU market and by pushing the use of ray tracing, upscaling, and frame generation in games, that's the direction it's all gone in, regardless of whether anyone wanted it to happen.

Let's just hope that when we're at the stage where a significant portion of a game is being generated by an AI in real-time, the gaming PC market has enough competition in it to keep prices under control. I'm really keen to see what the future of graphics and gaming is going to be like but I'd rather not be sitting on the sidelines, watching a select few enjoy such treats.