ESRGAN: AI Neural Net Upscaling

[MusicallyInspired]

Some might be familiar with this already (Laserschwert?), but I'm just discovering it and boooooy is it fantastic. These are the results of upscaling and nothing more with ESRGAN using a Manga109 dataset-trained model. There are all kinds of models you can use, but this one was the most impressive. It renders 320x200 images beautifully as 4x 1280x800(/960) and almost looks like it could be the original art. In my opinion it, for the most part, puts the Special Editions and Remasters (especially Full Throttle) to shame. It's not perfect but some screens look phenomenal and almost flawless. Check these out:



LucasArts' Special Edition version:



LucasArts' Special Edition version:

Peter Chan's original artwork:




Double Fine's Remastered version:


Grim Fandango:



Here's some Sierra backgrounds (click for larger images):





These three were hi-res scans of the original artwork taken from the KQ6 CD-ROM bonus files and were originally 512x356, rendered up to 2048x1460!:



Here's tons more Sierra images
Even more from Sierra, LucasArts, and other games
Thread I discovered this from - Has many examples people have made from Grim Fandango to Final Fantasy VII to Metroid Prime to Hexen....

It runs in Python. Here are some links to set it up yourself. It takes several seconds to generate an image with a high-end GPU, so real-time filtering in ScummVM or DOSBox is many years away I'm sure . But hey, perhaps ScummVM can in the future create some infrastructure for the patching in of hi-res HD packs for game art resources? *shrug*

Helpful links:
Getting Started
Further instructions + Manga109 Dataset Model
ESRGAN Github

[Laserschwert]

Well, I know about AI-supported upscaling (though it wasn't a thing yet when I was still working on my posters), but this is a whole new level. Just found out about this and saw your tweet. Boy, this looks great! The first thing that came to my mind was CMI. I mean, we already have the uncompressed music, now we could in theory upscale all the backgrounds? Could CMI:SE finally be achieved, maybe even by the fan community?

I have no idea in how far ScummVM might be capable of swapping out assets of the supported games, no less with assets at higher resolutions.

And putting your Grim Fandango upscales into the remaster? Even EFMI's backgrounds scale up surprisingly well.

[MusicallyInspired]

Here's some examples for CMI that I found in that thread.
The only problem with this someone noticed is that the Manga109 dataset was made from JPEG scans of manga so the AI is effectively trained to create JPEG artifacts. But even so, the possibilities....








This one was made with an anime-trained model:

[Laserschwert]

Yeah, I saw those as well... incredible stuff.

So, how exactly can you train the algorithm? Could we feed it with all the original MI2 paintings and the resulting VGA backgrounds and off you go?

[MusicallyInspired]

I haven't tried testing out training yet, but something like that from what I understand. Though, there are differences in lighting and contrast between the art sources and the game backgrounds. It would probably brighten up everything it touched.

I'm going to try to experiment later today. Do we have a repository of all the MI2 art sources by Peter Chan?

[Laserschwert]

I think Mojo has them all in their gallery, unfortuantely they are down right now.

Until then: https://imgur.com/gallery/GG7oE

[MusicallyInspired]

Thanks!

I took most of yesterday and wrote a script that extracts background resources from a Sierra game, converts to PNG, and upscales with ESRGAN. I'm doing all the Sierra games and uploading them to a Google Drive folder. I'll upload more as time goes on: https://drive.google.com/drive/folders/ ... sp=sharing

For PQ4 (and some screens in PQ3) I'm going to try a different model that's more accustomed and trained to restore low resolution live photos. It's coming out more like a painting, and while that's cool, I think it can do these better.

For QFG4, all the backgrounds are in pieces with foreground objects as different picture resources entirely so I'm going to take more time with that one and stitch them all together before upscaling.

[Laserschwert]

Is there any usable tool for removing color dithering? I know ReShade has filters for that, but I'm having quite some trouble using ReShade with ScummVM, plus I'd rather use it on the raw extracted backgrounds. I think dithering is one of the things that still trips the algorithms up, so getting rid of it beforehand would certainly help.

[MusicallyInspired]

Indeed. Perhaps we could train a model to transform dithering into smooth shading somehow. there's got to be some kind of algorithm out there that transforms dithering into fine gradients...

[MusicallyInspired]

This guy trained his own model to upscale Riven screens. He used images of grass, dirt, rocks, trees, water, etc and added dithering similar to Riven's. The result handles the dithering quite nicely (at the slight expense of finer details, he noted), but man is the result gorgeous!

Riven Upscale Album

[Mim]

It takes several seconds to generate an image with a high-end GPU, so real-time filtering in ScummVM or DOSBox is many years away I'm sure

User prarts writes in this thread : Really low res pictures (VGA - like DOS SCUMM games) takes less then 1 second, it`s almost instant - that`s way I was thinking of implementing this in ScummVM "on the fly".

So maybe there is a hope for faster implementation? BTW: Lots of interesting pictures in that thread.

But hey, perhaps ScummVM can in the future create some infrastructure for the patching in of hi-res HD packs for game art resources?

That would be great.

[MusicallyInspired]

It takes several seconds to generate an image with a high-end GPU, so real-time filtering in ScummVM or DOSBox is many years away I'm sure

User prarts writes in this thread : Really low res pictures (VGA - like DOS SCUMM games) takes less then 1 second, it`s almost instant - that`s way I was thinking of implementing this in ScummVM "on the fly".

So maybe there is a hope for faster implementation? BTW: Lots of interesting pictures in that thread.

Less than one second is not necessarily fast enough. We need 60 in one second. 30 for barely passable framerate and it would necessitate a really high-end bleeding-edge graphics card to do it continuously for a whole game sitting. It takes me 1-3 seconds to do a 320x200 image with my 970.

[Bluddy]

This is a big development in the machine learning world and it's great that it can be used for such practical stuff. These results are truly phenomenal.

Let's figure out a way to integrate pre-made images into ScummVM instead of the ones in the data files, rather than trying to do this on the fly, which requires interfacing with a neural net library etc.

[Ze_PilOt]

This is a big development in the machine learning world and it's great that it can be used for such practical stuff. These results are truly phenomenal.

Let's figure out a way to integrate pre-made images into ScummVM instead of the ones in the data files, rather than trying to do this on the fly, which requires interfacing with a neural net library etc.

Indeed. One "easy" way to achieve would be for ScummVM to dump a "cache" folder with all the decoded images (sprites / backgrounds / depth / masks / ....) and load from there (while supporting a different image resolution).

No idea if if it's a difficult job (I don't know ScummVM well enough), but I think it's doable, at least for the PC port.

[Raziel]

This is a big development in the machine learning world and it's great that it can be used for such practical stuff. These results are truly phenomenal.

Let's figure out a way to integrate pre-made images into ScummVM instead of the ones in the data files, rather than trying to do this on the fly, which requires interfacing with a neural net library etc.

Indeed. One "easy" way to achieve would be for ScummVM to dump a "cache" folder with all the decoded images (sprites / backgrounds / depth / masks / ....) and load from there (while supporting a different image resolution).

No idea if if it's a difficult job (I don't know ScummVM well enough), but I think it's doable, at least for the PC port.

Thinking one step further by dropping the local cache and instead using cloud space to "cache" the data, so that, at one point, every image of every supported game has been uploaded and/or upscaled for anyone (with internet) to use...

That is, of course, depending on the size of said images and speed of the connection and IP problems (Hell, not a good idea then:-/)