DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

[Kroc]

DOOM-Crusher shrinks your PK3, WAD, PNG & JPG files, recursively -- if your PK3 contains a WAD that contains a PNG, DOOM-Crusher will optimize it.

How effective is it? On Blade of Agony (boa_c1.pk3), DOOM-Crusher saved 69 MB, though results vary from WAD to WAD you can expect anywhere between 5% to 30% improvement.

DOOM-Crusher icon

icon.png (6.57 KiB) Viewed 7260 times

Instructions: https://github.com/Kroc/DOOM-Crusher
Download: https://github.com/Kroc/DOOM-Crusher/releases

Enjoy!

[AFADoomer]

Nice! Automating all of these tools under one roof is very useful...

It shaved ~10MB off of my Star trek mod's contents, but seems to still be running almost 24 hrs later... Does it loop or something?

[Kroc]

PNGs take the most time by far, so it depends on the number of them and the size. Above 100 K PNGs start taking a while and the 1 MB+ ones can even take an hour on their own. You can tell doom-crusher to ignore the PNG files using the command line -- `doom-crusher.bat /NOPNG mywad.pk3` and you'd be able to better tell if the process as a whole is completing, but if you can send me your mod I'll be happy to test it myself and look for problems.

[Kroc]

v1.0.1 released fixing WADs being processed twice in a row, wasting a lot of time.

[Wiw]

Fascinating! How does it work?

[Enjay]

I don't know what impact squishing individual lumps has on game loading times but it is very much my impression that using high-compression modes when making a PK3 archive increases loading times. Indeed, on a friend's slow computer, PK3s repackaged to just use "store" (i.e. no compression at all) seemed to load and perform much better than even regular PK3s, much less highly compressed archives.

As someone with a relatively slow internet connection, I'm all for squishing down the size of distribution files. However, if doing so means the files take longer to load in ZDoom, I'd much rather have the one-off download take slightly longer and take up a bit more disk space than take longer than necessary every time the mod is loaded into ZDoom.

[Kroc]

@enjay

There will always be a noticable difference between no-compression and any compression, the CPU is skipping a whole bunch of work -- but, there is no difference between something reasonably compressed and something compressed more tightly; this is a matter of what method the compressor used to examine the content. The same content can be compressed literally an infinite number of ways, so optimizers just try *many* more ways than usual to see if any come out smaller.

Using DOOM-Crusher will not slow down resource loading on already compressed resources (such as PNG, PK3) -- and what's more PNG files are always compressed in some form, so DOOM-Crusher will always speed them up: the speed of the CPU is tens of thousands of times faster than the storage, hundreds of thousands of times faster than mechanical drives and USB.

I.e. if you want ultimate speed: use WAD & DOOM image formats, not PNG/JPG; but just because you can do that, that isn't a reason not to use DOOM-Crusher. DOOM-Crusher includes WADPTR which can 'de-duplicate' nodes and lumps in a WAD speeding these up too, without the use of compression.

The reason I made DOOM-Crusher is because I'm also making PortaDOOM and whilst shaving a few hundred K off of a resource might not be worthwhile, as soon as you get 100s of WADs in one place it starts making a notable difference. There's no way you can say that shrinking Blade of Agony by 69 MB is not going to make a difference

@wiw

It's a batch file that brings together a number of existing tools, see the page here for a description of these: https://github.com/Kroc/DOOM-Crusher/bl ... /README.md

[Enjay]

OK, I'm running it on one of my mods right now to see how things turn out. I guess it will take a while. I'll report back though.

One thing does concern me from the docs though. I'm not sure how the map sidedef packing is done but if it is the kind of sidedef packing where linedefs which share identical siddefs are given the same sidedef references I have experienced problems with that in the past (although, being perfectly honest, I forget what the problems were).

As a matter of interest, where are the temporary files extracted to?

[edit]Just got this:


I'm going to be away from my computer while it's running. Does it store the output from the cmd window anywhere in case other errors flag up?

[Kroc]

So the batch script runs a handful of PNG utilities one after the other and relies upon the executable to return an error state ("%ERRORLEVEL%"); if an error state is returned the error mark is displayed " ! error <exe>", but if the executable flat out segfaults then *no* state (success or otherwise) is returned and the batch file output will get confused. It'll proceed with the next step anyway so it's nothing to worry about as the original file will not have been overwritten.

I looked at logging just before release but there's a ton of problems with this that need some thought. The various executables output different kinds of information and unfortunately some of them spit out a ton of very unhelpful information -- I forget each one, but if logged you'll get thirty-thousand lines of "1%", "2%" ... "100%".

I'll work out some solution but it'll take a while. I'm aiming to increase the reliability and reporting for crushing entire folders full of files as I need to do this for PortaDOOM and that could take *days* of processing to do.

[Enjay]

Aaargh! I was scrolling around to see if there were any other errors and I hit PgUp instead of the scroll wheel so, obviously, the pause was cancelled, the bat file finished and the cmd window shut. I did notice that some maps in the PK3 had given an error with WADSRC though. One map (a titlemap) was trashed by the process and one model lost its skin (the tank2.png in the error I posted before). It seems as if the original file was overwritten, or rather, deleted after all.

I don't know how long it took because I went out for a bit but the end result is:
Before: 98,275,964 bytes
After: 92,683,881 bytes
So that's just about a 5.7% reduction in PK3 size.

[Kroc]

It'd be great if you could send me that PK3 and I'd be able to ascertain the nature of these failures and increase the robustness of DOOM-Crusher!

[Enjay]

Unless I've edited it since uploading (and I don't think I have) it should be the same as this one: http://forum.zdoom.org/viewtopic.php?f=19&t=46718

(Unless you mean the one that was the end result of DOOM-Crusher)

[Kroc]

That's great, thanks. I'll use this for testing and producing some kind of log that can narrow down the faults. I'm implementing file-hashing cache so that files that have been (successfully) optimized in the past can be skipped.

[Fused]

Hi,

I tried this out on my mod to see how much it affects it.
Firstly I had this weird moment where it inflates image size rather than compressing it (http://image.prntscr.com/image/148b2ce8 ... ca79b4.png).
Secondly when I tried to test it, I ended up crashing for some reason. I attached the crashlog in case you want to check it out. It only happends after using the program.

[NightFright]

Tested this with a few PWADs. Here are the results:

Ancient Aliens (aaliens.wad) ............... 132 MB --> 132 MB (100%)
Bloodstain (bstain.wad) .................... 31.0 MB --> 25.6 MB (82.6%)
Chex: Galactic Conquest (chexgc.wad) ... 26.8 MB --> 26.0 MB (97%)
Echelon (echelon.wad) ...................... 26.5 MB --> 24.6 MB (92.8%)
Mutiny (mutiny.wad) ........................ 15.4 MB --> 13.0 MB (84.4%)
No End In Sight (neis.wad) ................. 18.7 MB --> 16.1 MB (86.1%)
Pirate DOOM! (pirates.wad) ................ 40.4 MB --> 36.0 MB (89.2%)

As expected, results depend on the wad file and what's inside. It's not too overwhelming, but if you want to optimize file size to the max, I guess it cannot hurt to use this.