DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Any utility that assists in the creation of mods, assets, etc, go here.

DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Postby Kroc » Wed Dec 14, 2016 4:29 pm

DOOM-Crusher shrinks your PK3, WAD, PNG & JPG files, recursively -- if your PK3 contains a WAD that contains a PNG, DOOM-Crusher will optimize it.

How effective is it? On Blade of Agony (boa_c1.pk3), DOOM-Crusher saved 69 MB, though results vary from WAD to WAD you can expect anywhere between 5% to 30% improvement.

icon.png
DOOM-Crusher icon
icon.png (6.57 KiB) Viewed 4213 times


Instructions: https://github.com/Kroc/DOOM-Crusher
Download: https://github.com/Kroc/DOOM-Crusher/releases

Enjoy!
Last edited by Kroc on Sat Dec 17, 2016 3:51 pm, edited 1 time in total.
User avatar
Kroc
PortaDOOM, DOOM-Crusher
 
Joined: 02 Oct 2016
Github ID: Kroc
Operating System: Windows 10/8.1/8 64-bit
Graphics Processor: Intel (Modern GZDoom)

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Postby AFADoomer » Thu Dec 15, 2016 6:43 pm

Nice! Automating all of these tools under one roof is very useful...

It shaved ~10MB off of my Star trek mod's contents, but seems to still be running almost 24 hrs later... Does it loop or something?
User avatar
AFADoomer
 
Joined: 15 Jul 2003

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Postby Kroc » Thu Dec 15, 2016 7:22 pm

PNGs take the most time by far, so it depends on the number of them and the size. Above 100 K PNGs start taking a while and the 1 MB+ ones can even take an hour on their own. You can tell doom-crusher to ignore the PNG files using the command line -- `doom-crusher.bat /NOPNG mywad.pk3` and you'd be able to better tell if the process as a whole is completing, but if you can send me your mod I'll be happy to test it myself and look for problems.
User avatar
Kroc
PortaDOOM, DOOM-Crusher
 
Joined: 02 Oct 2016
Github ID: Kroc
Operating System: Windows 10/8.1/8 64-bit
Graphics Processor: Intel (Modern GZDoom)

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Postby Kroc » Sat Dec 17, 2016 7:23 am

v1.0.1 released fixing WADs being processed twice in a row, wasting a lot of time.
User avatar
Kroc
PortaDOOM, DOOM-Crusher
 
Joined: 02 Oct 2016
Github ID: Kroc
Operating System: Windows 10/8.1/8 64-bit
Graphics Processor: Intel (Modern GZDoom)

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Postby Wiw » Sat Dec 17, 2016 2:39 pm

Fascinating! How does it work?
User avatar
Wiw
Frequently puts foot in mouth
 
Joined: 11 Jun 2015
Location: Everywhere and nowhere.

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Postby Enjay » Sat Dec 17, 2016 3:11 pm

I don't know what impact squishing individual lumps has on game loading times but it is very much my impression that using high-compression modes when making a PK3 archive increases loading times. Indeed, on a friend's slow computer, PK3s repackaged to just use "store" (i.e. no compression at all) seemed to load and perform much better than even regular PK3s, much less highly compressed archives.

As someone with a relatively slow internet connection, I'm all for squishing down the size of distribution files. However, if doing so means the files take longer to load in ZDoom, I'd much rather have the one-off download take slightly longer and take up a bit more disk space than take longer than necessary every time the mod is loaded into ZDoom.
User avatar
Enjay
Everyone is a moon, and has a dark side which he never shows to anybody. Twain
 
 
 
Joined: 15 Jul 2003
Location: Scotland

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Postby Kroc » Sat Dec 17, 2016 3:47 pm

@enjay

There will always be a noticable difference between no-compression and any compression, the CPU is skipping a whole bunch of work -- but, there is no difference between something reasonably compressed and something compressed more tightly; this is a matter of what method the compressor used to examine the content. The same content can be compressed literally an infinite number of ways, so optimizers just try *many* more ways than usual to see if any come out smaller.

Using DOOM-Crusher will not slow down resource loading on already compressed resources (such as PNG, PK3) -- and what's more PNG files are always compressed in some form, so DOOM-Crusher will always speed them up: the speed of the CPU is tens of thousands of times faster than the storage, hundreds of thousands of times faster than mechanical drives and USB.

I.e. if you want ultimate speed: use WAD & DOOM image formats, not PNG/JPG; but just because you can do that, that isn't a reason not to use DOOM-Crusher. DOOM-Crusher includes WADPTR which can 'de-duplicate' nodes and lumps in a WAD speeding these up too, without the use of compression.

The reason I made DOOM-Crusher is because I'm also making PortaDOOM and whilst shaving a few hundred K off of a resource might not be worthwhile, as soon as you get 100s of WADs in one place it starts making a notable difference. There's no way you can say that shrinking Blade of Agony by 69 MB is not going to make a difference :)

@wiw

It's a batch file that brings together a number of existing tools, see the page here for a description of these: https://github.com/Kroc/DOOM-Crusher/bl ... /README.md
User avatar
Kroc
PortaDOOM, DOOM-Crusher
 
Joined: 02 Oct 2016
Github ID: Kroc
Operating System: Windows 10/8.1/8 64-bit
Graphics Processor: Intel (Modern GZDoom)

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Postby Enjay » Sun Dec 18, 2016 1:39 pm

OK, I'm running it on one of my mods right now to see how things turn out. I guess it will take a while. I'll report back though.

One thing does concern me from the docs though. I'm not sure how the map sidedef packing is done but if it is the kind of sidedef packing where linedefs which share identical siddefs are given the same sidedef references I have experienced problems with that in the past (although, being perfectly honest, I forget what the problems were).

As a matter of interest, where are the temporary files extracted to?

[edit]Just got this:
Image

I'm going to be away from my computer while it's running. Does it store the output from the cmd window anywhere in case other errors flag up?
User avatar
Enjay
Everyone is a moon, and has a dark side which he never shows to anybody. Twain
 
 
 
Joined: 15 Jul 2003
Location: Scotland

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Postby Kroc » Sun Dec 18, 2016 5:00 pm

So the batch script runs a handful of PNG utilities one after the other and relies upon the executable to return an error state ("%ERRORLEVEL%"); if an error state is returned the error mark is displayed " ! error <exe>", but if the executable flat out segfaults then *no* state (success or otherwise) is returned and the batch file output will get confused. It'll proceed with the next step anyway so it's nothing to worry about as the original file will not have been overwritten.

I looked at logging just before release but there's a ton of problems with this that need some thought. The various executables output different kinds of information and unfortunately some of them spit out a ton of very unhelpful information -- I forget each one, but if logged you'll get thirty-thousand lines of "1%", "2%" ... "100%".

I'll work out some solution but it'll take a while. I'm aiming to increase the reliability and reporting for crushing entire folders full of files as I need to do this for PortaDOOM and that could take *days* of processing to do.
User avatar
Kroc
PortaDOOM, DOOM-Crusher
 
Joined: 02 Oct 2016
Github ID: Kroc
Operating System: Windows 10/8.1/8 64-bit
Graphics Processor: Intel (Modern GZDoom)

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Postby Enjay » Sun Dec 18, 2016 6:37 pm

Aaargh! I was scrolling around to see if there were any other errors and I hit PgUp instead of the scroll wheel so, obviously, the pause was cancelled, the bat file finished and the cmd window shut. I did notice that some maps in the PK3 had given an error with WADSRC though. One map (a titlemap) was trashed by the process and one model lost its skin (the tank2.png in the error I posted before). It seems as if the original file was overwritten, or rather, deleted after all.

I don't know how long it took because I went out for a bit but the end result is:
Before: 98,275,964 bytes
After: 92,683,881 bytes
So that's just about a 5.7% reduction in PK3 size.
User avatar
Enjay
Everyone is a moon, and has a dark side which he never shows to anybody. Twain
 
 
 
Joined: 15 Jul 2003
Location: Scotland

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Postby Kroc » Sun Dec 18, 2016 7:50 pm

It'd be great if you could send me that PK3 and I'd be able to ascertain the nature of these failures and increase the robustness of DOOM-Crusher!
User avatar
Kroc
PortaDOOM, DOOM-Crusher
 
Joined: 02 Oct 2016
Github ID: Kroc
Operating System: Windows 10/8.1/8 64-bit
Graphics Processor: Intel (Modern GZDoom)

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Postby Enjay » Mon Dec 19, 2016 12:50 pm

Unless I've edited it since uploading (and I don't think I have) it should be the same as this one: viewtopic.php?f=19&t=46718

(Unless you mean the one that was the end result of DOOM-Crusher)
User avatar
Enjay
Everyone is a moon, and has a dark side which he never shows to anybody. Twain
 
 
 
Joined: 15 Jul 2003
Location: Scotland

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Postby Kroc » Tue Dec 20, 2016 12:14 pm

That's great, thanks. I'll use this for testing and producing some kind of log that can narrow down the faults. I'm implementing file-hashing cache so that files that have been (successfully) optimized in the past can be skipped.
User avatar
Kroc
PortaDOOM, DOOM-Crusher
 
Joined: 02 Oct 2016
Github ID: Kroc
Operating System: Windows 10/8.1/8 64-bit
Graphics Processor: Intel (Modern GZDoom)

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Postby Fused » Wed Dec 21, 2016 3:13 pm

Hi,

I tried this out on my mod to see how much it affects it.
Firstly I had this weird moment where it inflates image size rather than compressing it (http://image.prntscr.com/image/148b2ce8 ... ca79b4.png).
Secondly when I tried to test it, I ended up crashing for some reason. I attached the crashlog in case you want to check it out. It only happends after using the program.
Attachments
CrashReport.zip
(24.02 KiB) Downloaded 61 times
User avatar
Fused
 
Joined: 02 Jul 2014
Discord: FusedQyou#0357

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Postby NightFright » Thu Dec 22, 2016 7:01 am

Tested this with a few PWADs. Here are the results:

Ancient Aliens (aaliens.wad) ............... 132 MB --> 132 MB (100%)
Bloodstain (bstain.wad) .................... 31.0 MB --> 25.6 MB (82.6%)
Chex: Galactic Conquest (chexgc.wad) ... 26.8 MB --> 26.0 MB (97%)
Echelon (echelon.wad) ...................... 26.5 MB --> 24.6 MB (92.8%)
Mutiny (mutiny.wad) ........................ 15.4 MB --> 13.0 MB (84.4%)
No End In Sight (neis.wad) ................. 18.7 MB --> 16.1 MB (86.1%)
Pirate DOOM! (pirates.wad) ................ 40.4 MB --> 36.0 MB (89.2%)

As expected, results depend on the wad file and what's inside. It's not too overwhelming, but if you want to optimize file size to the max, I guess it cannot hurt to use this.
User avatar
NightFright
 
Joined: 02 May 2008
Location: Germany

Next

Return to Editors / Asset Manipulation

Who is online

Users browsing this forum: No registered users and 2 guests