ESRGAN: AI Neural Net Upscaling

All the inane chatter goes in here. If you're curious about whether we will support a game, post HERE not in General Discussion :)

Moderator: ScummVM Team

User avatar
MusicallyInspired
Posts: 1012
Joined: Fri Mar 02, 2007 8:03 am
Location: Manitoba, Canada
Contact:

ESRGAN: AI Neural Net Upscaling

Post by MusicallyInspired » Wed Jan 09, 2019 9:19 pm

Some might be familiar with this already (Laserschwert?), but I'm just discovering it and boooooy is it fantastic. These are the results of upscaling and nothing more with ESRGAN using a Manga109 dataset-trained model. There are all kinds of models you can use, but this one was the most impressive. It renders 320x200 images beautifully as 4x 1280x800(/960) and almost looks like it could be the original art. In my opinion it, for the most part, puts the Special Editions and Remasters (especially Full Throttle) to shame. It's not perfect but some screens look phenomenal and almost flawless. Check these out:

Image
Image
LucasArts' Special Edition version:
Image
Image
Image
LucasArts' Special Edition version:
Image
Peter Chan's original artwork:
Image

Image
Image
Double Fine's Remastered version:
Image

Grim Fandango:
Image
Image

Here's some Sierra backgrounds (click for larger images):

ImageImage
ImageImage
ImageImage

These three were hi-res scans of the original artwork taken from the KQ6 CD-ROM bonus files and were originally 512x356, rendered up to 2048x1460!:
ImageImage
Image

Here's tons more Sierra images
Even more from Sierra, LucasArts, and other games
Thread I discovered this from - Has many examples people have made from Grim Fandango to Final Fantasy VII to Metroid Prime to Hexen....

It runs in Python. Here are some links to set it up yourself. It takes several seconds to generate an image with a high-end GPU, so real-time filtering in ScummVM or DOSBox is many years away I'm sure ;). But hey, perhaps ScummVM can in the future create some infrastructure for the patching in of hi-res HD packs for game art resources? *shrug*

Helpful links:
Getting Started
Further instructions + Manga109 Dataset Model
ESRGAN Github

User avatar
Laserschwert
Posts: 268
Joined: Mon Mar 06, 2006 11:48 pm

Re: ESRGAN: AI Neural Net Upscaling

Post by Laserschwert » Thu Jan 10, 2019 12:09 am

Well, I know about AI-supported upscaling (though it wasn't a thing yet when I was still working on my posters), but this is a whole new level. Just found out about this and saw your tweet. Boy, this looks great! The first thing that came to my mind was CMI. I mean, we already have the uncompressed music, now we could in theory upscale all the backgrounds? Could CMI:SE finally be achieved, maybe even by the fan community?

I have no idea in how far ScummVM might be capable of swapping out assets of the supported games, no less with assets at higher resolutions.

And putting your Grim Fandango upscales into the remaster? Even EFMI's backgrounds scale up surprisingly well.

User avatar
MusicallyInspired
Posts: 1012
Joined: Fri Mar 02, 2007 8:03 am
Location: Manitoba, Canada
Contact:

Re: ESRGAN: AI Neural Net Upscaling

Post by MusicallyInspired » Thu Jan 10, 2019 1:10 am

Here's some examples for CMI that I found in that thread.
The only problem with this someone noticed is that the Manga109 dataset was made from JPEG scans of manga so the AI is effectively trained to create JPEG artifacts. But even so, the possibilities....

Image
Image
Image
Image
Image
Image

This one was made with an anime-trained model:
Image

User avatar
Laserschwert
Posts: 268
Joined: Mon Mar 06, 2006 11:48 pm

Re: ESRGAN: AI Neural Net Upscaling

Post by Laserschwert » Thu Jan 10, 2019 9:00 am

Yeah, I saw those as well... incredible stuff.

So, how exactly can you train the algorithm? Could we feed it with all the original MI2 paintings and the resulting VGA backgrounds and off you go?

User avatar
MusicallyInspired
Posts: 1012
Joined: Fri Mar 02, 2007 8:03 am
Location: Manitoba, Canada
Contact:

Re: ESRGAN: AI Neural Net Upscaling

Post by MusicallyInspired » Thu Jan 10, 2019 1:08 pm

I haven't tried testing out training yet, but something like that from what I understand. Though, there are differences in lighting and contrast between the art sources and the game backgrounds. It would probably brighten up everything it touched.

I'm going to try to experiment later today. Do we have a repository of all the MI2 art sources by Peter Chan?

User avatar
Laserschwert
Posts: 268
Joined: Mon Mar 06, 2006 11:48 pm

Re: ESRGAN: AI Neural Net Upscaling

Post by Laserschwert » Thu Jan 10, 2019 4:00 pm

I think Mojo has them all in their gallery, unfortuantely they are down right now.

Until then: https://imgur.com/gallery/GG7oE
Last edited by Laserschwert on Thu Jan 10, 2019 5:18 pm, edited 2 times in total.

User avatar
MusicallyInspired
Posts: 1012
Joined: Fri Mar 02, 2007 8:03 am
Location: Manitoba, Canada
Contact:

Re: ESRGAN: AI Neural Net Upscaling

Post by MusicallyInspired » Thu Jan 10, 2019 4:19 pm

Thanks!

I took most of yesterday and wrote a script that extracts background resources from a Sierra game, converts to PNG, and upscales with ESRGAN. I'm doing all the Sierra games and uploading them to a Google Drive folder. I'll upload more as time goes on: https://drive.google.com/drive/folders/ ... sp=sharing

For PQ4 (and some screens in PQ3) I'm going to try a different model that's more accustomed and trained to restore low resolution live photos. It's coming out more like a painting, and while that's cool, I think it can do these better.

For QFG4, all the backgrounds are in pieces with foreground objects as different picture resources entirely so I'm going to take more time with that one and stitch them all together before upscaling.

User avatar
Laserschwert
Posts: 268
Joined: Mon Mar 06, 2006 11:48 pm

Re: ESRGAN: AI Neural Net Upscaling

Post by Laserschwert » Thu Jan 10, 2019 5:19 pm

Is there any usable tool for removing color dithering? I know ReShade has filters for that, but I'm having quite some trouble using ReShade with ScummVM, plus I'd rather use it on the raw extracted backgrounds. I think dithering is one of the things that still trips the algorithms up, so getting rid of it beforehand would certainly help.

User avatar
MusicallyInspired
Posts: 1012
Joined: Fri Mar 02, 2007 8:03 am
Location: Manitoba, Canada
Contact:

Re: ESRGAN: AI Neural Net Upscaling

Post by MusicallyInspired » Thu Jan 10, 2019 5:41 pm

Indeed. Perhaps we could train a model to transform dithering into smooth shading somehow. there's got to be some kind of algorithm out there that transforms dithering into fine gradients...

User avatar
MusicallyInspired
Posts: 1012
Joined: Fri Mar 02, 2007 8:03 am
Location: Manitoba, Canada
Contact:

Re: ESRGAN: AI Neural Net Upscaling

Post by MusicallyInspired » Fri Jan 11, 2019 3:10 am

This guy trained his own model to upscale Riven screens. He used images of grass, dirt, rocks, trees, water, etc and added dithering similar to Riven's. The result handles the dithering quite nicely (at the slight expense of finer details, he noted), but man is the result gorgeous!

Riven Upscale Album

Image

Mim
Posts: 67
Joined: Sun Dec 11, 2005 4:59 pm

Re: ESRGAN: AI Neural Net Upscaling

Post by Mim » Mon Jan 21, 2019 10:30 am

MusicallyInspired wrote:
Wed Jan 09, 2019 9:19 pm
It takes several seconds to generate an image with a high-end GPU, so real-time filtering in ScummVM or DOSBox is many years away I'm sure ;)
User prarts writes in this thread : Really low res pictures (VGA - like DOS SCUMM games) takes less then 1 second, it`s almost instant - that`s way I was thinking of implementing this in ScummVM "on the fly".

So maybe there is a hope for faster implementation? BTW: Lots of interesting pictures in that thread.
MusicallyInspired wrote:
Wed Jan 09, 2019 9:19 pm
But hey, perhaps ScummVM can in the future create some infrastructure for the patching in of hi-res HD packs for game art resources?
That would be great.

User avatar
MusicallyInspired
Posts: 1012
Joined: Fri Mar 02, 2007 8:03 am
Location: Manitoba, Canada
Contact:

Re: ESRGAN: AI Neural Net Upscaling

Post by MusicallyInspired » Mon Jan 21, 2019 9:59 pm

Mim wrote:
Mon Jan 21, 2019 10:30 am
MusicallyInspired wrote:
Wed Jan 09, 2019 9:19 pm
It takes several seconds to generate an image with a high-end GPU, so real-time filtering in ScummVM or DOSBox is many years away I'm sure ;)
User prarts writes in this thread : Really low res pictures (VGA - like DOS SCUMM games) takes less then 1 second, it`s almost instant - that`s way I was thinking of implementing this in ScummVM "on the fly".

So maybe there is a hope for faster implementation? BTW: Lots of interesting pictures in that thread.
Less than one second is not necessarily fast enough. We need 60 in one second. 30 for barely passable framerate and it would necessitate a really high-end bleeding-edge graphics card to do it continuously for a whole game sitting. It takes me 1-3 seconds to do a 320x200 image with my 970.

Bluddy
ScummVM Developer
Posts: 135
Joined: Sun Nov 18, 2007 8:43 am

Re: ESRGAN: AI Neural Net Upscaling

Post by Bluddy » Wed Jan 30, 2019 5:23 pm

This is a big development in the machine learning world and it's great that it can be used for such practical stuff. These results are truly phenomenal.

Let's figure out a way to integrate pre-made images into ScummVM instead of the ones in the data files, rather than trying to do this on the fly, which requires interfacing with a neural net library etc.

Ze_PilOt
Posts: 2
Joined: Mon Jan 28, 2019 7:44 am

Re: ESRGAN: AI Neural Net Upscaling

Post by Ze_PilOt » Wed Jan 30, 2019 6:53 pm

Bluddy wrote:
Wed Jan 30, 2019 5:23 pm
This is a big development in the machine learning world and it's great that it can be used for such practical stuff. These results are truly phenomenal.

Let's figure out a way to integrate pre-made images into ScummVM instead of the ones in the data files, rather than trying to do this on the fly, which requires interfacing with a neural net library etc.
Indeed. One "easy" way to achieve would be for ScummVM to dump a "cache" folder with all the decoded images (sprites / backgrounds / depth / masks / ....) and load from there (while supporting a different image resolution).

No idea if if it's a difficult job (I don't know ScummVM well enough), but I think it's doable, at least for the PC port.

User avatar
Raziel
ScummVM Porter
Posts: 1039
Joined: Tue Oct 25, 2005 8:27 am
Location: a story that hasn't been written yet

Re: ESRGAN: AI Neural Net Upscaling

Post by Raziel » Wed Jan 30, 2019 7:44 pm

Ze_PilOt wrote:
Wed Jan 30, 2019 6:53 pm
Bluddy wrote:
Wed Jan 30, 2019 5:23 pm
This is a big development in the machine learning world and it's great that it can be used for such practical stuff. These results are truly phenomenal.

Let's figure out a way to integrate pre-made images into ScummVM instead of the ones in the data files, rather than trying to do this on the fly, which requires interfacing with a neural net library etc.
Indeed. One "easy" way to achieve would be for ScummVM to dump a "cache" folder with all the decoded images (sprites / backgrounds / depth / masks / ....) and load from there (while supporting a different image resolution).

No idea if if it's a difficult job (I don't know ScummVM well enough), but I think it's doable, at least for the PC port.
Thinking one step further by dropping the local cache and instead using cloud space to "cache" the data, so that, at one point, every image of every supported game has been uploaded and/or upscaled for anyone (with internet) to use...

That is, of course, depending on the size of said images and speed of the connection and IP problems (Hell, not a good idea then:-/)

Post Reply