ESRGAN: AI Neural Net Upscaling

All the inane chatter goes in here. If you're curious about whether we will support a game, post HERE not in General Discussion :)

Moderator: ScummVM Team

Robot_Maker20
Posts: 45
Joined: Tue Nov 07, 2006 1:23 pm

Re: ESRGAN: AI Neural Net Upscaling

Post by Robot_Maker20 » Thu Jan 31, 2019 4:45 am

Can this be applied to preprocess low-res FMV cutscenes? Maybe even ones with line-skipping like the Z-vision games?

Ze_PilOt
Posts: 2
Joined: Mon Jan 28, 2019 7:44 am

Re: ESRGAN: AI Neural Net Upscaling

Post by Ze_PilOt » Thu Jan 31, 2019 12:25 pm

Robot_Maker20 wrote:
Thu Jan 31, 2019 4:45 am
Can this be applied to preprocess low-res FMV cutscenes? Maybe even ones with line-skipping like the Z-vision games?
- The GAN need to be trained with frames where half of the line is missing (it's a question of preparing the training set properly)
- The GAN need to be temporally coherent - That's still an issue but feasible.

So, I would say yes... The main problem would be to handle compression artefacts. You will need to train your GAN with everykind of artefacted videos.
That depend of the compression factor & technique (ie. some FMV are compressing red very much, some a bit less), so no magic bullet for every game I'm afraid.

User avatar
Laserschwert
Posts: 278
Joined: Mon Mar 06, 2006 11:48 pm

Re: ESRGAN: AI Neural Net Upscaling

Post by Laserschwert » Tue May 28, 2019 10:03 am

I took another stab at the CoMI backgrounds and used ESRGAN for dither removal first and upscaling after that. The results are even better now:
https://t.co/fGScl1VzL7

And here's a comparison showing the before -> dither removal -> upscaling: https://t.co/BsC3fW85qg

There's quite a lot of ESRGAN models in this WIKI: https://github.com/alsa64/AI-wiki/wiki/Model-Database

User avatar
MusicallyInspired
Posts: 1033
Joined: Fri Mar 02, 2007 8:03 am
Location: Manitoba, Canada
Contact:

Re: ESRGAN: AI Neural Net Upscaling

Post by MusicallyInspired » Wed May 29, 2019 2:11 am

The scene is really coming along! Sadly, without me. :( My desktop PC mobo died and I'm stuck on my laptop without a great GPU. Those results look fantastic! Who needs a special edition with this capability!.....if ScummVM gets a "texture pack/asset swap" feature, that is. ;)

User avatar
BeleG
Posts: 41
Joined: Mon Feb 16, 2009 12:31 pm

Re: ESRGAN: AI Neural Net Upscaling

Post by BeleG » Sun Jun 09, 2019 5:38 pm

This looks absolutely wonderful. I am sure there are all kinds of technical issues preventing use of the upscaled backgrounds without significant changes to the code, but would it be possible to use the undithered ones in the engine (e.g. no colour palette support issues)? They are already a significant improvement, are 100% faithful to the original material, and are unlikely to clash with the 640x480 animations (which would be a big problem for the upscaled backgrounds).

User avatar
Laserschwert
Posts: 278
Joined: Mon Mar 06, 2006 11:48 pm

Re: ESRGAN: AI Neural Net Upscaling

Post by Laserschwert » Sat Jun 15, 2019 9:53 am

I agree, having the undithered backgrounds in the game would be a great first step. Any ScummVM devs reading this?

User avatar
Laserschwert
Posts: 278
Joined: Mon Mar 06, 2006 11:48 pm

Re: ESRGAN: AI Neural Net Upscaling

Post by Laserschwert » Thu Jun 20, 2019 10:41 am

I've posted many more comparisons of the CMI backgrounds here:

https://www.reddit.com/r/GameUpscale/co ... ey_island/

User avatar
MusicallyInspired
Posts: 1033
Joined: Fri Mar 02, 2007 8:03 am
Location: Manitoba, Canada
Contact:

Re: ESRGAN: AI Neural Net Upscaling

Post by MusicallyInspired » Tue Jun 25, 2019 4:15 am

Man that's good!

User avatar
Laserschwert
Posts: 278
Joined: Mon Mar 06, 2006 11:48 pm

Re: ESRGAN: AI Neural Net Upscaling

Post by Laserschwert » Tue Jul 16, 2019 11:05 pm

I've finally managed training my own models for ESRGAN, and right now I'm working on a model to clean up the compression artifacts in the "Curse of Monkey Island" cutscenes. Here are a few comparison shots: https://imgsli.com/NDYzNQ/28/29 (more comparisons can be selected from the drop-downs)

And here's a first test in motion (cleaned up and upscaled 2x): https://www.youtube.com/watch?v=4sqje_OFfi0

Would it be doable to implement loading and playing of external cutscenes (like for example in MP4 format) in CoMI?

User avatar
criezy
ScummVM Developer
Posts: 631
Joined: Sat Sep 23, 2006 10:41 am
Location: West Sussex, UK

Re: ESRGAN: AI Neural Net Upscaling

Post by criezy » Tue Jul 16, 2019 11:52 pm

Laserschwert wrote:
Tue Jul 16, 2019 11:05 pm
Would it be doable to implement loading and playing of external cutscenes (like for example in MP4 format) in CoMI?
Possibly. I am not really familiar with the SCUMM engine, but generally speaking the engines in ScummVM work at the resolution of the original game (for example for rendering the graphics and handling mouse events) and don't know anything about the resolution at which the backend ends up displaying the game. The backends makes all the graphics scaling, and converts the mouse position between the screen/window coordinates and the game coordinates.

This means that to render something at a resolution higher than the original game resolution, the engine would actually need to work at that higher resolution. For cutscenes it might not be too hard to implement as the engine could switch resolution to use a higher resolution only when playing the cutscene (for example the zvision engine does that when playing Zork GI with the high resolution movies from the DVD version of the game), and if there is no interaction it might not need to convert mouse position back to the lower resolution that the game script would expect.

But since I am not sure how videos are handled in the SCUMM engine, I can't comment on whether this is doable or not in that specific engine.

User avatar
MusicallyInspired
Posts: 1033
Joined: Fri Mar 02, 2007 8:03 am
Location: Manitoba, Canada
Contact:

Re: ESRGAN: AI Neural Net Upscaling

Post by MusicallyInspired » Wed Jul 17, 2019 1:27 am

Beautiful!

User avatar
Laserschwert
Posts: 278
Joined: Mon Mar 06, 2006 11:48 pm

Re: ESRGAN: AI Neural Net Upscaling

Post by Laserschwert » Wed Jul 17, 2019 11:27 am

criezy wrote:
Tue Jul 16, 2019 11:52 pm
But since I am not sure how videos are handled in the SCUMM engine, I can't comment on whether this is doable or not in that specific engine.
Didn't the "Broken Sword" games use specifically re-compressed cutscenes in older ScummVM versions, because the original Smacker format wasn't supported? Maybe that's a lead...

User avatar
criezy
ScummVM Developer
Posts: 631
Joined: Sat Sep 23, 2006 10:41 am
Location: West Sussex, UK

Re: ESRGAN: AI Neural Net Upscaling

Post by criezy » Wed Jul 17, 2019 12:51 pm

Yes, they did (and we can still uses those recompressed mpeg videos), but they were at the same resolution as the original videos and the rest of the game.
And the Broken Sword games also use a completely different engine, so this is not really relevant for the SCUMM games.

User avatar
Laserschwert
Posts: 278
Joined: Mon Mar 06, 2006 11:48 pm

Re: ESRGAN: AI Neural Net Upscaling

Post by Laserschwert » Wed Jul 17, 2019 2:28 pm

I see. Well, technically using MP4 files at the original 640x480 resolution of CoMI would already allow for much better looking cutscenes, because compression wouldn't be an issue.

User avatar
criezy
ScummVM Developer
Posts: 631
Joined: Sat Sep 23, 2006 10:41 am
Location: West Sussex, UK

Re: ESRGAN: AI Neural Net Upscaling

Post by criezy » Wed Jul 17, 2019 5:45 pm

Regarding the specific codec, we don't have anything that uses mpeg4 yet in ScummVM so that would be some additional work to support that. But theora is already being used by a few engines, and would thus require less work for us (we already have a decoder for it in ScummVM).

Post Reply