*UPDATES!* ESRGAN AI Upscaling (HiRes packs now possible!)

All the inane chatter goes in here. If you're curious about whether we will support a game, post HERE not in General Discussion :)

Moderator: ScummVM Team

Post Reply
Bluddy
ScummVM Developer
Posts: 145
Joined: Sun Nov 18, 2007 8:43 am

Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

Post by Bluddy »

So, now that AI upscaling has come to the mainstream with NVIDIA DLSS 2.0, and can run in real-time and enable incredible rendering... I think even if ScummVM doesn't do it, some open source project needs to figure it out.
Kasenshlogga
Posts: 63
Joined: Thu Jul 18, 2019 1:31 pm

Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

Post by Kasenshlogga »

Bluddy wrote: Wed Jul 15, 2020 12:46 am So, now that AI upscaling has come to the mainstream with NVIDIA DLSS 2.0, and can run in real-time and enable incredible rendering... I think even if ScummVM doesn't do it, some open source project needs to figure it out.
Would DLSS even work on 2D content? The way I understand it is they train a model by letting the algorithm look at the game's textures and 3D assets from various angles and distances. 2D content is always pre-baked so it would need a pre-trained general purpose model to generate more detail from the image.
invwar
Posts: 67
Joined: Mon Jul 28, 2008 11:35 am

Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

Post by invwar »

If I understand DLSS and FidelityFX (AMD) correct, its all based on presets made on the high resolution, so instead of really rendering e.g. the 4K, the cards know roughly how the graphic should look like based on the preset and make it look like it is the actual resolution. I highly doubt that this system would work with 2D games as here is no high resolution source which could be analyzed and then used as preset.
At least NVIDIA and AMD now supports integer scaling, which definitely improve full screen on low resolution games. But its just normal scaling, making the picture at least sharp and not blurred, but 640x480 still remains as this.

I don't know either the limitations of ScummVM, but I would expect that theoretically it would be possible to do as Laserschwert suggested. Like replacing the backgrounds and other assets. I would really like to know from a programmer what would happen, if the backgrounds get swapped by higher resolutions.
But what definitely should be feasible is replacing the old FMVs by better quality ones. For the start, a ESRGAN upscaled Movie could be again downscaled in higher fidelity to the normal resolution. As its just a video, I don't expect this would break anything but would increase the quality drastically (back then compression was just not good enough). If this works fine, we could try to use higher resolutions. Of course, it would massively help, if someone would make it possible to support avi (or other container) support for videos like the enhanced audio tracks. So we just drop the video in the folder and this is played.
User avatar
Laserschwert
Posts: 280
Joined: Mon Mar 06, 2006 11:48 pm

Re: ESRGAN: AI Neural Net Upscaling

Post by Laserschwert »

sev wrote: Sun Mar 15, 2020 10:30 am To all "non-programmers" here. ESRGAN will not come to ScummVM. The reason is simple: it is not real time, and is unable to produce 60 frames per second on modern computers. There are attempts to work around that by lowering the quality of the output, and algorithms like RTSRGAN produce 15 images per second on a single 2.3GHz CPU.

Thus, no, this topic is futile.


Eugene
Even though realtime upscaling with ESRGAN isn't the suggestion here, your proposed 60 frames-per-second are hardly necessary for adventure games. No classic adventure game runs at 60fps, plus most times the backgrounds are mostly static, so those need to be updated just once on load.

By the way, by now I have finished my remaster of the CMI intro cutscene:
https://youtu.be/3KRddnLgFZg

And here's a comparison with the original:
https://youtu.be/QfkJLDD2jCI

While ESRGAN was the primary tool for this, a good amount of manual cleanup went in as well.
User avatar
MusicallyInspired
Posts: 1136
Joined: Fri Mar 02, 2007 8:03 am
Location: Manitoba, Canada
Contact:

Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

Post by MusicallyInspired »

Nice!
Bluddy
ScummVM Developer
Posts: 145
Joined: Sun Nov 18, 2007 8:43 am

Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

Post by Bluddy »

Beautiful job! I'd love to have this instead of the intro we have. Is there any way to integrate it?

It's funny how picky one can get. Like, it's a shame the ESRGAN can't get rid of all of the compression artifacts.
Kasenshlogga
Posts: 63
Joined: Thu Jul 18, 2019 1:31 pm

Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

Post by Kasenshlogga »

Bluddy wrote: Tue Jul 28, 2020 6:42 pm Beautiful job! I'd love to have this instead of the intro we have. Is there any way to integrate it?

It's funny how picky one can get. Like, it's a shame the ESRGAN can't get rid of all of the compression artifacts.
You tell us, ScummVM dev. :wink:
Jacob_
Posts: 3
Joined: Tue Dec 24, 2019 3:36 am

Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

Post by Jacob_ »

Good to see there's still interest in this. Made some more progress on Riven, I've gotten the videos upscaled with a custom EDVR model + DAIN. The compression artifacts were so bad that processing each frame individually with ESRGAN didn't work well. https://streamable.com/czimjh

There's a problem though, playing 1080p 30fps video in ScummVM gets really slow for me, even on a fast PC (notice how the dome slows down as you approach). Question for the devs: any way to make it run faster? MinGW build, videos are QT/MJPEG codec but that can be changed if another one would be more optimized.
User avatar
Praetorian
ScummVM Developer
Posts: 787
Joined: Tue May 08, 2007 8:54 am
Location: Greece
Contact:

Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

Post by Praetorian »

Jacob_ wrote: Thu Jul 30, 2020 2:31 am There's a problem though, playing 1080p 30fps video in ScummVM gets really slow for me, even on a fast PC (notice how the dome slows down as you approach). Question for the devs: any way to make it run faster? MinGW build, videos are QT/MJPEG codec but that can be changed if another one would be more optimized.
Is this your own custom build of ScummVM? Are you using the "--disable-debug --enable-release" options for configure prior to building (make)? Maybe that could help
User avatar
MusicallyInspired
Posts: 1136
Joined: Fri Mar 02, 2007 8:03 am
Location: Manitoba, Canada
Contact:

Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

Post by MusicallyInspired »

Wow! That video is much smoother! Looks incredible!
Jacob_
Posts: 3
Joined: Tue Dec 24, 2019 3:36 am

Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

Post by Jacob_ »

Praetorian wrote: Thu Jul 30, 2020 6:20 am
Jacob_ wrote: Thu Jul 30, 2020 2:31 am There's a problem though, playing 1080p 30fps video in ScummVM gets really slow for me, even on a fast PC (notice how the dome slows down as you approach). Question for the devs: any way to make it run faster? MinGW build, videos are QT/MJPEG codec but that can be changed if another one would be more optimized.
Is this your own custom build of ScummVM? Are you using the "--disable-debug --enable-release" options for configure prior to building (make)? Maybe that could help
Yeah this is for a custom build that can load and display higher resolution images and video. Using those options fixes the slowness, thanks!
Blakes7
Posts: 39
Joined: Sun Nov 03, 2019 9:44 am

Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

Post by Blakes7 »

Jim Leonard upscaled Gabriel Knight 2 video from 308x167 playing back at 10fps, and uploaded to YouTube at 720p60
https://www.youtube.com/watch?v=6HJndgXMKYY

This was done back in 2018 so who knows whats possible now.
Kasenshlogga
Posts: 63
Joined: Thu Jul 18, 2019 1:31 pm

Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

Post by Kasenshlogga »

Blakes7 wrote: Tue Dec 29, 2020 5:16 am Jim Leonard upscaled Gabriel Knight 2 video from 308x167 playing back at 10fps, and uploaded to YouTube at 720p60
https://www.youtube.com/watch?v=6HJndgXMKYY

This was done back in 2018 so who knows whats possible now.
That's not bad, I didn't even know this was doable back in 2018. With the tools we have available today we might already have everything to remaster the entire game. Once ScummVM supports it, that is.
MiLO83
Posts: 1
Joined: Sun Jan 17, 2021 6:14 am

Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

Post by MiLO83 »

https://youtu.be/pmtRH1hcG50

https://youtu.be/bQ3JFdbWCJ0

https://github.com/MiLO83/scummvmx (Modified SCI Engine)
https://github.com/MiLO83/SCICompanionX (Modified to Batch Dump SCI Graphics [Views] as individual Cels required for mod). SCICompanionX currently dumps backgrounds as .bmp, these need to be Converted to .png. (Remember to somehow Enlarge all graphics exactly 4x.) [ESRGAN for instance.. :)]

Not all SCI Games work, this is a W.i.P Alpha demonstration.

- MiLO83
User avatar
MusicallyInspired
Posts: 1136
Joined: Fri Mar 02, 2007 8:03 am
Location: Manitoba, Canada
Contact:

Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

Post by MusicallyInspired »

You are my new favourite person! ScummVM's first High Resolution Pack capability! Well done.
Post Reply