*UPDATES!* ESRGAN AI Upscaling (HiRes packs now possible!)

All the inane chatter goes in here. If you're curious about whether we will support a game, post HERE not in General Discussion :)

Moderator: ScummVM Team

Post Reply
User avatar
Laserschwert
Posts: 280
Joined: Mon Mar 06, 2006 11:48 pm

Re: ESRGAN: AI Neural Net Upscaling

Post by Laserschwert »

Right now all of this is not more than tossing around ideas anyway. I'm no developer, so there needs to be a dev who's actually interested in working on stuff like this.

This includes swapping out the original backgrounds either with cleaned up ones (at the same resolution, but a higher color depth) or even upscaled ones at a higher resolutions. I don't know if Scumm games can be made resolution independent... cursor and click coordinates would probably be trivially easy to "scale", but stuff like walk-boxes or other layers not so much. Being able to swap out (or "override") assets that get loaded would make modding these games a lot easier. Still, I figured replacing the videos would require considerably less work than all that (which is still a lot of work, I guess :lol: ).
Kasenshlogga
Posts: 63
Joined: Thu Jul 18, 2019 1:31 pm

Re: ESRGAN: AI Neural Net Upscaling

Post by Kasenshlogga »

Laserschwert wrote: Wed Jul 17, 2019 5:54 pm Right now all of this is not more than tossing around ideas anyway. I'm no developer, so there needs to be a dev who's actually interested in working on stuff like this.

This includes swapping out the original backgrounds either with cleaned up ones (at the same resolution, but a higher color depth) or even upscaled ones at a higher resolutions. I don't know if Scumm games can be made resolution independent... cursor and click coordinates would probably be trivially easy to "scale", but stuff like walk-boxes or other layers not so much. Being able to swap out (or "override") assets that get loaded would make modding these games a lot easier. Still, I figured replacing the videos would require considerably less work than all that (which is still a lot of work, I guess :lol: ).
I would actually pay money if somebody ever worked on such a feature for ScummVM. :mrgreen:
User avatar
CyberAxe
Posts: 25
Joined: Thu Apr 20, 2006 9:20 pm
Location: Scotland
Contact:

Re: ESRGAN: AI Neural Net Upscaling

Post by CyberAxe »

I really want to see support for this and the topic seams dead, I'm considering implementing the necessary support myself (And while i'm at it a shader engine [though lower priority] which would probably be compatible with the Arch Slang format as i'd created a similar engine years ago).

If there are other people willing to do the graphic side we should be in good standing, i could do the graphics myself but have limited time so would take a lot longer.

I'm also thinking we have 3 versions of the graphical updates

an ESRGAN pack which is all games upscaled via ESRGAN to 1080/4K (i'd be interested to see ESRGAN multiple pass results to get the lower res games to 4K)

an original pack which is all the original art paintings converted at high resolution using original palettes

and a hybrid pack which is original where possible and ESRGAN for everything else

Personally I'd love to see Gabriel Knight ESRGAN'd up as I've always wanted a high res version and have no desire to play the remake
User avatar
Praetorian
ScummVM Developer
Posts: 786
Joined: Tue May 08, 2007 8:54 am
Location: Greece
Contact:

Re: ESRGAN: AI Neural Net Upscaling

Post by Praetorian »

CyberAxe wrote: Tue Mar 03, 2020 10:52 am I really want to see support for this and the topic seams dead, I'm considering implementing the necessary support myself (And while i'm at it a shader engine [though lower priority] which would probably be compatible with the Arch Slang format as i'd created a similar engine years ago).

If there are other people willing to do the graphic side we should be in good standing, i could do the graphics myself but have limited time so would take a lot longer.

I'm also thinking we have 3 versions of the graphical updates

an ESRGAN pack which is all games upscaled via ESRGAN to 1080/4K (i'd be interested to see ESRGAN multiple pass results to get the lower res games to 4K)

an original pack which is all the original art paintings converted at high resolution using original palettes

and a hybrid pack which is original where possible and ESRGAN for everything else

Personally I'd love to see Gabriel Knight ESRGAN'd up as I've always wanted a high res version and have no desire to play the remake
There's been some very recent discussion on the ScummVM discord server (#offtopic channel) about supporting such upscale like Laserschwert's work for Monkey Island 3.
https://twitter.com/Laserschwert_/statu ... 0986010625
https://twitter.com/Laserschwert_/statu ... 7822681088

ScummVM Discord server link:
https://discord.gg/ReGSkVs
User avatar
MusicallyInspired
Posts: 1136
Joined: Fri Mar 02, 2007 8:03 am
Location: Manitoba, Canada
Contact:

Re: ESRGAN: AI Neural Net Upscaling

Post by MusicallyInspired »

I'd love to see something come of this. Since I lost my computer I haven't been able to upscale without my GPU. But I've recently figured out how to do it via CPU instead but it takes way longer. Been doing some experiments with the plethora of upscaling models out there now intended for different types of games and art and the results are interesting. Still gotta try out training new models though.

Upscaling raw images seems to not be the best way to go as there are usually some visual errors. But doing some pre and post edits would work nicely. Manga109, for example, doesn't like hard pixel staircase edges for contrasting colours and preserves the staircasing effect so it's necessary to manually use a blur tool so that when upscaling the result looks smoother. Different things like that.
Bluddy
ScummVM Developer
Posts: 145
Joined: Sun Nov 18, 2007 8:43 am

Re: ESRGAN: AI Neural Net Upscaling

Post by Bluddy »

Note that as lovely as you can make the backgrounds, ESRGAN doesn't work on the small, pixelated characters. This means the best you can do is to have some filtering on the main characters, and the results will look very jarring. The alternative is to redo all the character sprites by hand.
User avatar
MusicallyInspired
Posts: 1136
Joined: Fri Mar 02, 2007 8:03 am
Location: Manitoba, Canada
Contact:

Re: ESRGAN: AI Neural Net Upscaling

Post by MusicallyInspired »

Yeah. But even so.
Kasenshlogga
Posts: 63
Joined: Thu Jul 18, 2019 1:31 pm

Re: ESRGAN: AI Neural Net Upscaling

Post by Kasenshlogga »

Bluddy wrote: Fri Mar 13, 2020 1:34 pm Note that as lovely as you can make the backgrounds, ESRGAN doesn't work on the small, pixelated characters. This means the best you can do is to have some filtering on the main characters, and the results will look very jarring. The alternative is to redo all the character sprites by hand.
I think even with the character sprites untouched it would be a significant upgrade in most games.
User avatar
sev
ScummVM Lead
Posts: 2273
Joined: Wed Sep 21, 2005 1:06 pm
Contact:

Re: ESRGAN: AI Neural Net Upscaling

Post by sev »

To all "non-programmers" here. ESRGAN will not come to ScummVM. The reason is simple: it is not real time, and is unable to produce 60 frames per second on modern computers. There are attempts to work around that by lowering the quality of the output, and algorithms like RTSRGAN produce 15 images per second on a single 2.3GHz CPU.

Thus, no, this topic is futile.


Eugene
User avatar
MusicallyInspired
Posts: 1136
Joined: Fri Mar 02, 2007 8:03 am
Location: Manitoba, Canada
Contact:

Re: ESRGAN: AI Neural Net Upscaling

Post by MusicallyInspired »

sev wrote: Sun Mar 15, 2020 10:30 am To all "non-programmers" here. ESRGAN will not come to ScummVM. The reason is simple: it is not real time, and is unable to produce 60 frames per second on modern computers. There are attempts to work around that by lowering the quality of the output, and algorithms like RTSRGAN produce 15 images per second on a single 2.3GHz CPU.

Thus, no, this topic is futile.
Eugene, nobody is saying ESRGAN should come to ScummVM. We're merely talking about the possibility of hi res replacement art assets that can be loaded in on the fly with ESRGAN being used outside the game to prerender images. Like a "high resolution pack." Similar to the high quality music packs that have been made in the past for various ScummVM games. Obviously real-time neural net upscaling won't be a feasible concept for quite a number of years yet.
Kasenshlogga
Posts: 63
Joined: Thu Jul 18, 2019 1:31 pm

Re: ESRGAN: AI Neural Net Upscaling

Post by Kasenshlogga »

MusicallyInspired wrote: Sun Mar 15, 2020 1:06 pm Eugene, nobody is saying ESRGAN should come to ScummVM. We're merely talking about the possibility of hi res replacement art assets that can be loaded in on the fly with ESRGAN being used outside the game to prerender images. Like a "high resolution pack." Similar to the high quality music packs that have been made in the past for various ScummVM games. Obviously real-time neural net upscaling won't be a feasible concept for quite a number of years yet.
Exactly and ResidualVM already supports this, there's a really beautiful The Longest Journey HD mod that utilizes ESRGAN for all it's backgrounds. :mrgreen:
JohnnyW
Posts: 41
Joined: Sun Sep 21, 2014 11:02 am

Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

Post by JohnnyW »

Laserschwert, this is incredible! Just removing the dither alone is beautiful. Is there somewhere I could download the dithered and final versions somewhere (the two different versions)? I know there's an IMGUR link, but the original PNGs would be preferable. Thanks!
Kasenshlogga
Posts: 63
Joined: Thu Jul 18, 2019 1:31 pm

Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

Post by Kasenshlogga »

Tried some AI frame interpolation with DAIN: https://imgur.com/a/HBiCP72

Now if there was a way to use this on individual sprites + AI upscaling that would be something. :mrgreen:
Jacob_
Posts: 3
Joined: Tue Dec 24, 2019 3:36 am

Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

Post by Jacob_ »

I've been working on ESRGAN upscaling for Riven (code). It involves a lot of coding since any references to pixel coordinates or dimensions have to be multiplied by the upscaling factor. And then there's the trouble of dealing with different resolutions since all of the numbers are hard-coded, seems like the only option is going to be setting it to something like 1920x1080 and hoping that works for everyone.

Video upscaling is another problem, and the main reason I haven't released anything yet. I trained a custom model for EDVR (same authors as ESRGAN) and it was much better than I expected at dealing with the compression artifacts, but the output was smooth and not very detailed compared to the images. Hopefully a new, improved technique will be published soon.
User avatar
MusicallyInspired
Posts: 1136
Joined: Fri Mar 02, 2007 8:03 am
Location: Manitoba, Canada
Contact:

Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)

Post by MusicallyInspired »

Beautiful work on Riven!
Post Reply