Possible support for "The Neverhood"?

All the inane chatter goes in here. If you're curious about whether we will support a game, post HERE not in General Discussion :)

Moderator: ScummVM Team

"The Neverhood" for ScummVM?

Yes
37
46%
Yes
37
46%
No
4
5%
Maybe
3
4%
 
Total votes: 81

SonicTheBro
Posts: 1
Joined: Tue Dec 23, 2008 4:10 pm

Possible support for "The Neverhood"?

Post by SonicTheBro »

Just wondering. Here are some pro's and con's:

Pros:
It's an adventure game
Sprite based (I think)

Cons:
Has Video
Has also been for Windows
Not really the same as a SCUMM game

There's probably more, but that's all I could think up.
User avatar
md5
ScummVM Developer
Posts: 2250
Joined: Thu Nov 03, 2005 9:31 pm
Location: Athens, Greece

Post by md5 »

clem
Posts: 2159
Joined: Mon Oct 31, 2005 11:14 am

Post by clem »

And forum rule 1a.
User avatar
iPwnzorz
Posts: 300
Joined: Sat Jan 06, 2007 3:55 pm
Location: Hampshire, England

Post by iPwnzorz »

But the description for The Junkyard...
User avatar
sev
ScummVM Lead
Posts: 2272
Joined: Wed Sep 21, 2005 1:06 pm
Contact:

Re: Possible support for "The Neverhood"?

Post by sev »

SonicTheBro wrote:Just wondering. Here are some pro's and con's:

Pros:
It's an adventure game
Sprite based (I think)
Wow! Finally there is someone who wants to develop an engine for Neverhood! Cool! I voted "yes"!!


Eugene
User avatar
Longcat
Posts: 1061
Joined: Sat Sep 23, 2006 3:15 pm

Re: Possible support for "The Neverhood"?

Post by Longcat »

sev wrote:
SonicTheBro wrote:Just wondering. Here are some pro's and con's:

Pros:
It's an adventure game
Sprite based (I think)
Wow! Finally there is someone who wants to develop an engine for Neverhood! Cool! I voted "yes"!!


Eugene
Not sarcastic at all:P

I recently played The Neverhood actually, and incidently, it ran just fine in win xp. But portability sure would be nice:)
User avatar
iPwnzorz
Posts: 300
Joined: Sat Jan 06, 2007 3:55 pm
Location: Hampshire, England

Re: Possible support for "The Neverhood"?

Post by iPwnzorz »

sev wrote:
SonicTheBro wrote:Just wondering. Here are some pro's and con's:

Pros:
It's an adventure game
Sprite based (I think)
Wow! Finally there is someone who wants to develop an engine for Neverhood! Cool! I voted "yes"!!


Eugene
Don't be like that. Isn't it easier to say "Sorry, but ..."

You know you don't have the FAQ very clear on the frontpage, it's one tiny little link hidden in the menu, so no surprise no one reads it.
User avatar
noize
Posts: 126
Joined: Mon Oct 31, 2005 3:08 pm

Re: Possible support for "The Neverhood"?

Post by noize »

iPwnzorz wrote:You know you don't have the FAQ very clear on the frontpage, it's one tiny little link hidden in the menu, so no surprise no one reads it.
Every forum subject has a sticky called "Forum rules". One of these rules is reading the faq before posting.

I also voted yes. Neverhood support would be great of course.
Onkel Hotte
Posts: 3
Joined: Fri Aug 28, 2009 10:01 pm

Post by Onkel Hotte »

(Sorry for the thread necromancy.)
Cons:
- Doesn't seem to be script based, so would require the original source code or a re-implementation.
Having said that:

Code: Select all

import struct, array, wave, sys, os
from ctypes import *

# Enable headless pygame
 
# set SDL to use the dummy NULL video driver, 
#   so it doesn't need a windowing system.
os.environ["SDL_VIDEODRIVER"] = "dummy"
 
if True:
    # Some platforms might need to init the display for some parts of pygame.
    import pygame.display
    pygame.display.init()
    screen = pygame.display.set_mode((1,1))

import pygame
from pygame.locals import *

# Some magic to make the blast decompression function available from inside python

class BUFFER(Structure):
    _fields_ = [("data", c_void_p), ("len", c_uint)]

# typedef unsigned (*blast_in)(void *how, unsigned char **buf);
BLASTINFUNC = CFUNCTYPE(c_uint, c_void_p, POINTER(c_void_p))

def py_blast_in_func(a, b):
    buf = cast(a, POINTER(BUFFER))
    b[0] = buf.contents.data
    return buf.contents.len

blast_in_func = BLASTINFUNC(py_blast_in_func)
    
# typedef int (*blast_out)(void *how, unsigned char *buf, unsigned len);
BLASTOUTFUNC = CFUNCTYPE(c_int, c_void_p, c_void_p, c_uint)

def py_blast_out_func(a, b, c):
    buf = cast(a, POINTER(BUFFER))
    memmove(buf.contents.data + buf.contents.len, b, c)
    buf.contents.len += c
    return 0
    
blast_out_func = BLASTOUTFUNC(py_blast_out_func)

if __name__ == "__main__":
    # make an output directory
    if not os.access("data", os.F_OK):
        os.mkdir("data")

    # You need to make a dynamic library of zlib/contrib/blast.c, on MacOS that is
    # gcc -dynamiclib -o blast.dylib -dylib blast.c

    lib_blast = CDLL("blast.dylib")
    blast = lib_blast.blast
    blast.restype = c_int
    blast.argtypes = [BLASTINFUNC, c_void_p, BLASTOUTFUNC, c_void_p]

    f = open("nevdemo_full/NEVDEMO.BLB", "rb")
    
    # Header
    &#40;magic, id, unknown, dataSize, fileSize, file_number&#41; = struct.unpack&#40;'<4sBBHLL', f.read&#40;16&#41;&#41;
    
    if magic != "\x40\x49\x00\x02"&#58;
        print "wrong id, not a blb file"
        exit&#40;&#41;
    
    # file ID
    id_table = &#91;&#93;
    for i in xrange&#40;file_number&#41;&#58;
        &#40;id,&#41; = struct.unpack&#40;'<L', f.read&#40;4&#41;&#41;
        id_table.append&#40;id&#41;
    
    # dir entries
    file_table = &#91;&#93;
    for i in xrange&#40;file_number&#41;&#58;
        file_entry = struct.unpack&#40;'<BBHLLLL', f.read&#40;20&#41;&#41;
        file_table.append&#40;file_entry&#41;

    # offset shift table &#40;for music and sfx files&#41;
    shift_table = f.read&#40;dataSize&#41;

    # initial pal
    pal = &#91;&#40;x,x,x&#41; for x in xrange&#40;256&#41;&#93;
    
    # export files into raw
    file_type = &#123; 7&#58; "sound", 8&#58; "music", 10&#58; "video", 2&#58;"image", 4&#58;"animation" &#125;
    file_ext = &#123; 7&#58; "wav", 8&#58; "wav", 10&#58; "smk", 2&#58;"tga", 4&#58;"tga" &#125;
    for i in xrange&#40;file_number&#41;&#58;
        &#40;type, mode, index, id, start, in_len, out_len&#41; = file_table&#91;i&#93;
        if mode != 101&#58;
            f.seek&#40;start&#41;
            
            data = f.read&#40;in_len&#41;
            if mode == 3&#58;
                buffer = create_string_buffer&#40;out_len&#41;
                in_buf = BUFFER&#40;cast&#40;data, c_void_p&#41;, in_len&#41;
                out_buf = BUFFER&#40;cast&#40;buffer, c_void_p&#41;, 0&#41;
                blast&#40;blast_in_func, byref&#40;in_buf&#41;, blast_out_func, byref&#40;out_buf&#41;&#41;            
            
                data = string_at&#40;buffer, out_len&#41;
            
            if type in &#91;7,8&#93;&#58;
                # sound/music
                if index > len&#40;shift_table&#41;&#58;
                    shift = 255
                else&#58;
                    shift = ord&#40;shift_table&#91;index&#93;&#41;
            
                w = wave.open&#40;"data/%s-%i.%s" % &#40;file_type&#91;type&#93;, i, file_ext&#91;type&#93;&#41;, "wb"&#41;
                w.setnchannels&#40; 1 &#41;
                w.setsampwidth&#40; 2 &#41;
                w.setframerate&#40; 22050 &#41;
                
                if shift < 16&#58;
                    buffer = array.array&#40; "h" &#41;

                    curValue = 0
                    for val in data&#58;
                        val = &#40;ord&#40;val&#41; ^ 128&#41; - 128
                        curValue += val
                        try&#58;
                            buffer.append&#40;curValue << shift&#41;
                        except OverflowError&#58;
                            try&#58;
                                # This obviously means that something is going wrong with the decompression or with the scaling
                                buffer.append&#40;curValue&#41;
                            except OverflowError&#58;
                                buffer.append&#40;0&#41;

                    # I'm not sure if this is required, it doesn't seem to be on MacOS. Maybe WAV does the endian-swap internally, or stores a flag in the wav header?
                    if sys.byteorder == "little"&#58;
                        buffer.byteswap&#40;&#41;
                    w.writeframes&#40; buffer.tostring&#40;&#41; &#41;
                    del buffer
                else&#58;
                    if &#40;len&#40;data&#41; & 1&#41; != 0&#58;
                        data = data + "\0"
                    w.writeframes&#40; data &#41;
                w.close&#40;&#41;

            elif type == 2&#58;
                #  image
                &#40;format, width, height&#41; = struct.unpack&#40;'<HHH', data&#91;0&#58;6&#93;&#41;

                # Format flags
                # 0x1 - compressed
                # 0x4 - offset
                # 0x8 - palette

                pos = 6
                if format & 0x8&#58;
                    pal = &#91;&#40;ord&#40;x&#91;0&#93;&#41;, ord&#40;x&#91;1&#93;&#41;, ord&#40;x&#91;2&#93;&#41;&#41; for x in &#91;data&#91;pos + idx * 4&#58; pos + 3 + idx * 4&#93; for idx in xrange&#40;256&#41;&#93;&#93;
                    pos += 1024
                    # I assume the palette is decided by the room an item appears in. But reusing the last palette makes things at least visible
                
                if format & 0x4&#58;
                    # some kind of position?
                    offset = struct.unpack&#40;'<HH', data&#91;pos&#58;pos+4&#93;&#41;
                    # print offset
                    pos += 4

                if format & 0x1&#58;
                    # compressed
                    framebuffer = array.array&#40;"B"&#41;
                    framebuffer.extend&#40;&#91;0 for x in xrange&#40;width * height&#41;&#93;&#41;
                    ypos = 0
                    while True&#58;
                        header = struct.unpack&#40;'<HH', data&#91;pos&#58;pos+4&#93;&#41;
                        pos += 4
                        if header == &#40;0,0&#41;&#58;
                            break
                        &#40;row_count, items_per_row_count&#41; = header
                        for row_index in xrange&#40;row_count&#41;&#58;
                            for item_index in xrange&#40;items_per_row_count&#41;&#58;
                                &#40;xpos, &#41; = struct.unpack&#40;'<H', data&#91;pos&#58;pos+2&#93;&#41;
                                pos += 2
                                &#40;fragment_len,&#41; = struct.unpack&#40;'<H', data&#91;pos&#58;pos+2&#93;&#41;
                                pos += 2
                                for b in xrange&#40;fragment_len&#41;&#58;
                                    framebuffer&#91;ypos + xpos + b&#93; = ord&#40;data&#91;pos + b&#93;&#41;
                                pos += fragment_len
                            ypos += width
                            
                    surface = pygame.image.fromstring&#40;framebuffer.tostring&#40;&#41;, &#40;width, height&#41;, "P"&#41;
                    surface.set_palette&#40;pal&#41;
                    pygame.image.save&#40;surface, "data/%s-%i.%s" % &#40;file_type&#91;type&#93;, i, file_ext&#91;type&#93;&#41;&#41;
                    del framebuffer
                else&#58;
                    # uncompressed
                    surface = pygame.image.fromstring&#40;data&#91;pos&#58;pos + width * height&#93;, &#40;width, height&#41;, "P"&#41;
                    surface.set_palette&#40;pal&#41;
                    pygame.image.save&#40;surface, "data/%s-%i.%s" % &#40;file_type&#91;type&#93;, i, file_ext&#91;type&#93;&#41;&#41;

            elif type == 4&#58;
                header = struct.unpack&#40;'<HHIIII', data&#91;0&#58;20&#93;&#41;
                # print "%x" % i, header

                palette_offset = header&#91;3&#93;
                pal = &#91;&#40;ord&#40;x&#91;0&#93;&#41;, ord&#40;x&#91;1&#93;&#41;, ord&#40;x&#91;2&#93;&#41;&#41; for x in &#91;data&#91;palette_offset + idx * 4&#58; palette_offset + 3 + idx * 4&#93; for idx in xrange&#40;256&#41;&#93;&#93;
                
                pos = 20
                if header&#91;0&#93; == 2&#58;
                    unknown_header = struct.unpack&#40;'<HHHH', data&#91;pos&#58;pos+8&#93;&#41;
                    # print unknown_header
                    pos += 8
            
                frame_count = header&#91;5&#93;
                frame_list = &#91;&#93;
                for frame_index in xrange&#40;frame_count&#41;&#58;
                    frame = struct.unpack&#40;'<hhhhhhhhhhhhhhI', data&#91;pos + frame_index * 32&#58;pos + 32 + frame_index * 32&#93;&#41;
                    # print frame
                    frame_list.append&#40;frame&#41;

                data_offset = header&#91;2&#93;
                for frame_index in xrange&#40;frame_count&#41;&#58;
                    frame = frame_list&#91;frame_index&#93;
                    width = frame&#91;5&#93;
                    height = frame&#91;6&#93;
                    pos = data_offset + frame&#91;14&#93;

                    framebuffer = array.array&#40;"B"&#41;
                    framebuffer.extend&#40;&#91;0 for x in xrange&#40;width * height&#41;&#93;&#41;
                    ypos = 0
                    while True&#58;
                        header = struct.unpack&#40;'<HH', data&#91;pos&#58;pos+4&#93;&#41;
                        pos += 4
                        if header == &#40;0,0&#41;&#58;
                            break
                        &#40;row_count, items_per_row_count&#41; = header
                        for row_index in xrange&#40;row_count&#41;&#58;
                            for item_index in xrange&#40;items_per_row_count&#41;&#58;
                                &#40;xpos, &#41; = struct.unpack&#40;'<H', data&#91;pos&#58;pos+2&#93;&#41;
                                pos += 2
                                &#40;fragment_len,&#41; = struct.unpack&#40;'<H', data&#91;pos&#58;pos+2&#93;&#41;
                                pos += 2
                                for b in xrange&#40;fragment_len&#41;&#58;
                                    framebuffer&#91;ypos + xpos + b&#93; = ord&#40;data&#91;pos + b&#93;&#41;
                                pos += fragment_len
                            ypos += width
                            
                    surface = pygame.image.fromstring&#40;framebuffer.tostring&#40;&#41;, &#40;width, height&#41;, "P"&#41;
                    surface.set_palette&#40;pal&#41;
                    pygame.image.save&#40;surface, "data/%s-%i-%i.%s" % &#40;file_type&#91;type&#93;, i, frame_index, file_ext&#91;type&#93;&#41;&#41;
                    del framebuffer
            else&#58;
                of = open&#40;"data/%s-%i.%s" % &#40;file_type&#91;type&#93;, i, file_ext&#91;type&#93;&#41;, "wb"&#41;
                of.write&#40;data&#41;
                of.close&#40;&#41;
           del data    

    f.close&#40;&#41;
This is a bit of python code to extract the BLB archive. It gives you music, sound, images and animations. I only tested the free demo that you can download here:
http://download.cnet.com/The-Neverhood- ... 03341.html
I found the demo decompresses fine with The Unarchiver on MacOS. I assume on Linux you can use Wine, and on Windows you can install it directly.
(I don't have the full game myself. And I don't need a PM of where to find it.)
Most credit go to Valery V. Anisimovsky for finding the archive format, compression method (PKWARE) and ADPCM audio compression.
I found that you can decompress PKWARE stream files with blast, which is in the contrib/blast directory of zlib, and the image and animation format. You need to compile blast into a dynamic library for your platform, so it can be loaded from the python script.
Unfortunately I don't have the time to develop an engine myself. If you're lucky I get the smacker files decoded sometime.
(Update: ffmpeg supports the smacker files in Neverhood.)
(Update: added a few "del" statements to buffers.)
(Update: fixed extension for smacker files.)
Last edited by Onkel Hotte on Sat Aug 29, 2009 7:33 am, edited 2 times in total.
User avatar
Laserschwert
Posts: 280
Joined: Mon Mar 06, 2006 11:48 pm

Post by Laserschwert »

Sorry, I have to ask: Onkel "Kalki" Hotte? :D
Onkel Hotte
Posts: 3
Joined: Fri Aug 28, 2009 10:01 pm

Post by Onkel Hotte »

Tiny update: The smacker files didn't work in VLC when I tried it, but they work fine with ffmpeg, so they shouldn't be a problem.
The A.BLB in the demo just contains a single image with the Font.
Laserschwert wrote:Sorry, I have to ask: Onkel "Kalki" Hotte? :D
Yes and no. Did that help?
User avatar
Laserschwert
Posts: 280
Joined: Mon Mar 06, 2006 11:48 pm

Post by Laserschwert »

Onkel Hotte wrote:Did that help?
Yes and no.
Jon God
Posts: 35
Joined: Sat Oct 27, 2007 9:13 pm

Post by Jon God »

I know, this is bumping an ancient thread, but I didn't think my itty bitty question needed a new topic.

My question is just: Is anyone working on The Neverhood support in ScummVM? I got mixed impressions from a few people, so I was curious.
User avatar
sev
ScummVM Lead
Posts: 2272
Joined: Wed Sep 21, 2005 1:06 pm
Contact:

Post by sev »

Yes.


Eugene
User avatar
ezekiel000
Posts: 443
Joined: Mon Aug 25, 2008 5:17 pm
Location: Surrey, England

Post by ezekiel000 »

There is a reimplementation outside of scummvm here:
http://github.com/Blaizer/Neverhood/commits/master
Locked