RFE: support “implode” ZIP compression

[Hirogen2]

Hi,


I stumbled over a zip file that is compressed with the “Implode” method. ZDoom does not yet know how to unpack these, but /usr/bin/unzip has no issue. It would be nice if it understood this one too besides Deflate, bzip2, lzma.
http://www.doomworld.com/idgames/index.php?id=12999

[Graf Zahl]

You got usable code? That's the main reason it's not in.

[Gez]

There's a BSD-licensed C code for that in explode.c from Info ZIP's UnZip 6.0.

[Graf Zahl]

Sadly that code shares a lot of properties with zlib, i.e. it's close to unreadable. I hope I can figure out how to use it...

[randi]

I found some much easier to understand code from GunZip.c64.

[Graf Zahl]

Thank god!

I would have given up on the Info_Zip code. The function call interface of it is nearly impenetrable.

[Graf Zahl]

Since that linked code also contains decompression routines for the other old methods Shrink and Reduce, wouldn't it make sense to make ZDoom's Zip reader feature complete? I have to admit though that I haven't found and Doom relates Zip which uses these 2 methods yet.

[randi]

Okay, I added Shrink support. (Good thing the LZW patent has expired!)

Reduce, however, that code indicates has only been partially tested, and since PKZIP 1.1 doesn't support Reduce (though it does support Shrink and Implode), I would prefer to have a real-world Reduced file to test with before adding it.

[Graf Zahl]

Skulltag sure won't like this:

/* SetCodeLengths() and DecodeSFValue() are from 7-Zip, which is LGPL. */



(not that I care...)

[Graf Zahl]

FYI, I just checked the entire levels/ directory of /idgames.

There's quite a few files which use Implode as their compression algorithm and a handful that use Shrink - mostly for text files but even for some WADs. However, there is not one single file in there using one of the 'reduce' methods so maybe they really aren't worth bothering.

[Enjay]

FYI, I just checked the entire levels/ directory of /idgames.

I've seen you say things like that before when looking for DECORATE issues or something. My question is, how do you manage to check every one of those thousands of files? Presumably it's not a manual check.

[Graf Zahl]

First I downloaded the entire archive with an FTP manager and then I wrote a tool that loads a zip, prints information about each entry, then wrote a small batch that ran it over all files in the archive and then I loaded the output into a text editor and searched for the information I was interested in.

There were exactly 4 Zips which contained WADs that were Shrunk, approx. 50-100 (I didn't count) that contained Imploded WADs, none that uses BZip2, LZMA or Reduce for anything. There were also 20 or so Zips which contained Shrunk text files.

[Blzut3]

Skulltag sure won't like this:

/* SetCodeLengths() and DecodeSFValue() are from 7-Zip, which is LGPL. */



(not that I care...)

I thought 7-Zip was public domain? (Or does the current version not have those functions.) Also, contrary to what you may think, we have nothing against releasing free open source software. In fact we are releasing our own GPL server browser since we don't have the source to IDESE.

I'm curious as to if the zdoom.pk3 compression program has been updated to use these formats or if they are actually worse than LZMA, bzip2, and DEFLATE in most cases?

[Graf Zahl]

The Zipdir program has not been updated. These are old and rather inefficient compression algorithms that are no longer actively used. Deflate is just better in almost all situations. They are merely supported so that old Zips from /idgames can be loaded.

Regarding the LGPL, don't forget its precise wording! You can not use this code inside a closed source binary without jumping through hoops to comply with it because the end user must be able to recreate the binary with a changed version of the LGPL code. Or you may just stick to the first revision of explode.cpp...

[randi]

Skulltag sure won't like this:

Where do you see that?

I'm curious ... if they are actually worse than LZMA, bzip2, and DEFLATE in most cases?

Shrink is the same LZW compression used in GIFs. Just as PNG beats GIF, so to do any of the LZ77 variants ZIP supports beat Shrink.

Implode is basically a less "refined" version of Deflate. Like Deflate, it's LZ77, but its smaller window and use of mixed Shannon-Fano and fixed-length bit codes instead of strictly Shannon-Fano/Huffman codes for everything means that it will probably never beat Deflate except for contrived cases designed to let it win.