SCI Compression Tool

[KuroShiro]

Don't worry, I know how to use a hex editor. The version of the game is also 1.000. I don't think there was another one for the CD version.

Actually, it seems to be in the middle of a bunch of data for me as well. The only time "SOL" appears in the file is at the beginning though. I'm starting to think that my resource.aud file might have just gotten messed up somewhere, the disc image has been archived and moved between various harddrives for quite a few years. The in-game speech all works fine though, so I'm not sure what exactly the problem with it is.

[KuroShiro]

Just wanted to post a quick update -- for some reason, after building from the current HEAD, everything now compresses fine with both the CLI and the GUI. Nice work, whatever you changed!

The issue I was having before with the GUI was that it is not capable of creating directories, and I was trying to make a new one for the compressed file. Also, for some reason it seems that the GUI compresses files roughly 2x as much as the CLI.

[Kyahx]

Are there any nightly builds of the tools available anywhere?

I would like to compress my SCI games, but I'm not having any luck building the tools for OSX myself

[KuroShiro]

Are there any nightly builds of the tools available anywhere?

I would like to compress my SCI games, but I'm not having any luck building the tools for OSX myself

Yes, but not for Mac that I know of -- you'll have to build them yourself. To do so, in addition to all the libraries listed on the compiling ScummVM page, you will also need to install wxWidgets. Other than that, if you can compile ScummVM, you should be able to build the tools. Someone with more experience building on OSX could correct me.

[agf]

Digging up this old thread because I have a related question.

The audio file from KQ5 from the KQ collection is in a different format -- it's not RESOURCE.[AUD|SFX], it's AUDIO001.002 -- but it still has a .MAP file associated with it. It's not recognized by the compression utility.

Is this simply unimplemented or is it a format that doesn't lend itself to compression as well as the RESOURCE.AUD format, or is it already compressed in some rudimentary way that renders additional compression less useful?

If it's just unimplemented I might take a crack at it myself.

Thanks.

[md5]

Digging up this old thread because I have a related question.

The audio file from KQ5 from the KQ collection is in a different format -- it's not RESOURCE.[AUD|SFX], it's AUDIO001.002 -- but it still has a .MAP file associated with it. It's not recognized by the compression utility.

Is this simply unimplemented or is it a format that doesn't lend itself to compression as well as the RESOURCE.AUD format, or is it already compressed in some rudimentary way that renders additional compression less useful?

If it's just unimplemented I might take a crack at it myself.

Thanks.

The audio in KQ5CD is in an earlier format, but the compression tool does support it, AFAIK. The digital audio file decoding code can be found in the SCI engine in sci/resource_audio.cpp. Check loadResource(), loadFromAudioVolumeSCI1(), loadFromAudioVolumeSCI11(), readAudioMapSCI1() and readAudioMapSCI11()

[agf]

The audio in KQ5CD is in an earlier format, but the compression tool does support it, AFAIK. The digital audio file decoding code can be found in the SCI engine in sci/resource_audio.cpp. Check loadResource(), loadFromAudioVolumeSCI1(), loadFromAudioVolumeSCI11(), readAudioMapSCI1() and readAudioMapSCI11()

Thanks for the quick response.

Code: Select all

./scummvm-tools-cli.exe --tool compress_sci --vorbis -o AUDIO001.00g 
../../../games/sierra/King\'s\ Quest\ Collection/KQ5/AUDIO001.002
Fatal Error : Input file doesn't seem to be a valid sci audio resource file

The same command works fine for KQ6's RESOURCE.AUD. It fails with a mingw compiled current git checkout for the KQ5 AUDIO001.002. Renaming the file has no effect.

I tried the GUI (1.2.0), and after selecting the file it suggests either compress_agos or encode_dxa, which are both obviously incorrect. I tried compress_agos anyway and it quickly gives a wrong format error (as expected) once it starts. Renaming the file to RESOURCE.AUD causes the GUI to select compress_kyra, which errors out with an assertion failed once it starts.

EDIT: compress_sci isn't listed in the 1.2.0 tools readme, was it perhaps added after that?

EDIT2: looking at the compress_sci source, this error is thrown if it doesn't find a recognized header at the beginning of the file -- maybe there is no header in the SCI1 audio file.

EDIT3: So, according to resource_audio, there is no header in SCI1 audio resources, and the map file looks like this:

Code: Select all

AUDIOnnn.MAP contains 10-byte entries:
Early format:
w 5 bits resource type and 11 bits resource number
dw 7 bits volume number and 25 bits offset
dw size
Later format:
w nEntry
dw offset+volume (as in resource.map)
dw size
ending with 10 0xFFs

Doesn't appear to be supported.

EDIT Final:

From somewhere else in compress_sci.c:

Code: Select all

	// First find out how many samples are in this file
	//  We only support SOL audio and WAVE
	//  We can't support games that use "raw" audio (would have to walk through audio map for those, which would
	//   complicate this code - all talkie games that use this (kq5) don't use much space anyway, so not supporting
	//   those isn't a big issue

[agf]

I wrote a python script to convert the KQ5 AUDIO001.002 to a format compress_sci will work on. I just use the AUDIO001.MAP file to run each resource through python's wave module then write them out to one file.

compress_sci dropped the file size (using vorbis default settings) from 93,229KB to 35,700KB -- this seems like a worthwhile improvement to me.

Next step is to change scummvm/engines/sci/resource_audio.cpp to recognize this format for KQ5.

EDIT: The error I get right now is "could not translate offset to compressed offset in audio volume!"

EDIT2: Realized what I missed. I didn't actually write the new offsets to a new map file.

[agf]

OK, I'm using KQ5 now with compressed audio.

The python script below rewrites the AUDIO001.002 from raw sounds to WAVE sounds, so compress_sci understands it.

It also rewrites AUDIO001.MAP to match the new WAVE formatted file, which compress_sci needs to create an offset translation table.

The files created by this program work fine with ScummVM without compression as well, and the AUDIO001.002 file is actually slightly smaller than the original because the raw sounds were padded to 2048 bytes.

The script should work on any normal python2 install (2.5+)

Code: Select all

#!/usr/bin/env python2
from __future__ import with_statement
from struct import unpack, pack
from StringIO import StringIO
from wave import open as wave_open

# dir with KQ5 installed
kq5_dir = "C:\\Users\\adam\\Games\\sierra\\King's Quest Collection\\KQ5"
# original AUDIO001.002
kq5_aud = "AUDIO001.002.bak"
# original AUDIO001.MAP
kq5_map = "AUDIO001.MAP.bak"
# new AUDIO001.002
aud_new = "AUDIO001.002.new"
# new AUDIO001.MAP
map_new = "AUDIO001.MAP.new"

def parse_map():
    with open('\\'.join((kq5_dir, kq5_map)), 'rb') as file: map = file.read()
    return tuple((fields[0], fields[1] >> 28, fields[1] & 0x0fffffff, fields[2])
                    for fields in &#40;unpack&#40;'<HII', map&#91;n&#58;n+10&#93;&#41;
                        for n in range&#40;0, len&#40;map&#41; - 10, 10&#41;&#41;&#41;

def convert_to_wav&#40;&#41;&#58;
    res_map = parse_map&#40;&#41;
    with open&#40;'\\'.join&#40;&#40;kq5_dir, kq5_aud&#41;&#41;, 'rb'&#41; as aud_in&#58;
        with open&#40;'\\'.join&#40;&#40;kq5dir, aud_new&#41;&#41;, 'wb'&#41; as aud_out&#58;
            with open&#40;'\\'.join&#40;&#40;kq5dir, map_new&#41;&#41;, 'wb'&#41; as new_map&#58;
                for num, volume, offset, size in res_map&#58;
                    aud_in.seek&#40;offset&#41;
                    temp = StringIO&#40;''&#41;
                    wav = wave_open&#40;temp, 'wb'&#41;
                    wav.setparams&#40;&#40;1, 1, 11025, 0, 'NONE', 'not compressed'&#41;&#41;
                    wav.writeframes&#40;aud_in.read&#40;size&#41;&#41;
                    new_offset = aud_out.tell&#40;&#41;
                    aud_out.write&#40;temp.getvalue&#40;&#41;&#41;
                    new_map.write&#40;pack&#40;'<HII', num,
                                       &#40;&#40;volume << 28&#41; & 0xffffffff&#41; + new_offset,
                                       aud_out.tell&#40;&#41; - new_offset&#41;&#41;
                new_map.write&#40;pack&#40;'<HII', 0xffff, 0xffffffff, 0xffffffff&#41;&#41;

if __name__ == '__main__'&#58; convert_to_wav&#40;&#41;

[md5]

Thanks for your work
This functionality should be included in the compression tool eventually, but until then your script is very useful indeed!

[agf]

I've posted a patch to the tracker to add this functionality to compress_sci.

https://sourceforge.net/tracker/?func=d ... tid=418822

EDIT: And I posted a bug report about an issue I spotted while writing the patch.

https://sourceforge.net/tracker/?func=d ... tid=418820

[NoiZje]

The CD version of Jones in the Fast Lane also uses the AUDIO001.MAP AUDIO001.002 file structure.

[agf]

The CD version of Jones in the Fast Lane also uses the AUDIO001.MAP AUDIO001.002 file structure.

I don't have this game, but you could try changing the

Code: Select all

if (_input.size() == 0x05C9B000) {

line in the patch to

Code: Select all

if (_input.size() == 0x05C9B000 or _input.size() == 0xjones_file_size_in_hex) {

with the size of your AUDIO001.002 file and see if it works.

It's dependent on the sample format (8bit mono 11025 with each sample padded to multiples of 2048 bytes) as well as the AUDIO001.MAP format.