ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

All the inane chatter goes in here. If you're curious about whether we will support a game, post HERE not in General Discussion :)

Moderator: ScummVM Team

Bluddy
ScummVM Developer
Posts: 144
Joined: Sun Nov 18, 2007 8:43 am

Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

Post by Bluddy »

So, now that AI upscaling has come to the mainstream with NVIDIA DLSS 2.0, and can run in real-time and enable incredible rendering... I think even if ScummVM doesn't do it, some open source project needs to figure it out.
Kasenshlogga
Posts: 28
Joined: Thu Jul 18, 2019 1:31 pm

Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

Post by Kasenshlogga »

Bluddy wrote: Wed Jul 15, 2020 12:46 am So, now that AI upscaling has come to the mainstream with NVIDIA DLSS 2.0, and can run in real-time and enable incredible rendering... I think even if ScummVM doesn't do it, some open source project needs to figure it out.
Would DLSS even work on 2D content? The way I understand it is they train a model by letting the algorithm look at the game's textures and 3D assets from various angles and distances. 2D content is always pre-baked so it would need a pre-trained general purpose model to generate more detail from the image.
invwar
Posts: 20
Joined: Mon Jul 28, 2008 11:35 am

Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

Post by invwar »

If I understand DLSS and FidelityFX (AMD) correct, its all based on presets made on the high resolution, so instead of really rendering e.g. the 4K, the cards know roughly how the graphic should look like based on the preset and make it look like it is the actual resolution. I highly doubt that this system would work with 2D games as here is no high resolution source which could be analyzed and then used as preset.
At least NVIDIA and AMD now supports integer scaling, which definitely improve full screen on low resolution games. But its just normal scaling, making the picture at least sharp and not blurred, but 640x480 still remains as this.

I don't know either the limitations of ScummVM, but I would expect that theoretically it would be possible to do as Laserschwert suggested. Like replacing the backgrounds and other assets. I would really like to know from a programmer what would happen, if the backgrounds get swapped by higher resolutions.
But what definitely should be feasible is replacing the old FMVs by better quality ones. For the start, a ESRGAN upscaled Movie could be again downscaled in higher fidelity to the normal resolution. As its just a video, I don't expect this would break anything but would increase the quality drastically (back then compression was just not good enough). If this works fine, we could try to use higher resolutions. Of course, it would massively help, if someone would make it possible to support avi (or other container) support for videos like the enhanced audio tracks. So we just drop the video in the folder and this is played.
User avatar
Laserschwert
Posts: 280
Joined: Mon Mar 06, 2006 11:48 pm

Re: ESRGAN: AI Neural Net Upscaling

Post by Laserschwert »

sev wrote: Sun Mar 15, 2020 10:30 am To all "non-programmers" here. ESRGAN will not come to ScummVM. The reason is simple: it is not real time, and is unable to produce 60 frames per second on modern computers. There are attempts to work around that by lowering the quality of the output, and algorithms like RTSRGAN produce 15 images per second on a single 2.3GHz CPU.

Thus, no, this topic is futile.


Eugene
Even though realtime upscaling with ESRGAN isn't the suggestion here, your proposed 60 frames-per-second are hardly necessary for adventure games. No classic adventure game runs at 60fps, plus most times the backgrounds are mostly static, so those need to be updated just once on load.

By the way, by now I have finished my remaster of the CMI intro cutscene:
https://youtu.be/3KRddnLgFZg

And here's a comparison with the original:
https://youtu.be/QfkJLDD2jCI

While ESRGAN was the primary tool for this, a good amount of manual cleanup went in as well.
User avatar
MusicallyInspired
Posts: 1061
Joined: Fri Mar 02, 2007 8:03 am
Location: Manitoba, Canada
Contact:

Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

Post by MusicallyInspired »

Nice!
Bluddy
ScummVM Developer
Posts: 144
Joined: Sun Nov 18, 2007 8:43 am

Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

Post by Bluddy »

Beautiful job! I'd love to have this instead of the intro we have. Is there any way to integrate it?

It's funny how picky one can get. Like, it's a shame the ESRGAN can't get rid of all of the compression artifacts.
Kasenshlogga
Posts: 28
Joined: Thu Jul 18, 2019 1:31 pm

Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

Post by Kasenshlogga »

Bluddy wrote: Tue Jul 28, 2020 6:42 pm Beautiful job! I'd love to have this instead of the intro we have. Is there any way to integrate it?

It's funny how picky one can get. Like, it's a shame the ESRGAN can't get rid of all of the compression artifacts.
You tell us, ScummVM dev. :wink:
Jacob_
Posts: 3
Joined: Tue Dec 24, 2019 3:36 am

Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

Post by Jacob_ »

Good to see there's still interest in this. Made some more progress on Riven, I've gotten the videos upscaled with a custom EDVR model + DAIN. The compression artifacts were so bad that processing each frame individually with ESRGAN didn't work well. https://streamable.com/czimjh

There's a problem though, playing 1080p 30fps video in ScummVM gets really slow for me, even on a fast PC (notice how the dome slows down as you approach). Question for the devs: any way to make it run faster? MinGW build, videos are QT/MJPEG codec but that can be changed if another one would be more optimized.
User avatar
Praetorian
ScummVM Developer
Posts: 263
Joined: Tue May 08, 2007 8:54 am
Location: Greece
Contact:

Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

Post by Praetorian »

Jacob_ wrote: Thu Jul 30, 2020 2:31 am There's a problem though, playing 1080p 30fps video in ScummVM gets really slow for me, even on a fast PC (notice how the dome slows down as you approach). Question for the devs: any way to make it run faster? MinGW build, videos are QT/MJPEG codec but that can be changed if another one would be more optimized.
Is this your own custom build of ScummVM? Are you using the "--disable-debug --enable-release" options for configure prior to building (make)? Maybe that could help
User avatar
MusicallyInspired
Posts: 1061
Joined: Fri Mar 02, 2007 8:03 am
Location: Manitoba, Canada
Contact:

Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

Post by MusicallyInspired »

Wow! That video is much smoother! Looks incredible!
Jacob_
Posts: 3
Joined: Tue Dec 24, 2019 3:36 am

Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

Post by Jacob_ »

Praetorian wrote: Thu Jul 30, 2020 6:20 am
Jacob_ wrote: Thu Jul 30, 2020 2:31 am There's a problem though, playing 1080p 30fps video in ScummVM gets really slow for me, even on a fast PC (notice how the dome slows down as you approach). Question for the devs: any way to make it run faster? MinGW build, videos are QT/MJPEG codec but that can be changed if another one would be more optimized.
Is this your own custom build of ScummVM? Are you using the "--disable-debug --enable-release" options for configure prior to building (make)? Maybe that could help
Yeah this is for a custom build that can load and display higher resolution images and video. Using those options fixes the slowness, thanks!
Post Reply