DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Any utility that assists in the creation of mods, assets, etc, go here. For example: Ultimate Doom Builder, Slade, WadSmoosh, Oblige, etc.
Forum rules
The Projects forums are ONLY for YOUR PROJECTS! If you are asking questions about a project, either find that project's thread, or start a thread in the General section instead.

Got a cool project idea but nothing else? Put it in the project ideas thread instead!

Projects for any Doom-based engine (especially 3DGE) are perfectly acceptable here too.

Please read the full rules for more details.
User avatar
Kroc
Posts: 116
Joined: Sun Oct 02, 2016 11:37 am
Graphics Processor: Intel (Modern GZDoom)
Contact:

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Post by Kroc »

New version released: v1.1
  • Skip files that have already been crushed before (a cache of file hashes is kept in `bin\hashes.txt`) -- incrementally crush your files without having to waste time on files that had been crushed before!
  • WADPTR's sidedef packing is disabled to prevent glitches in maps
  • Added `/NOPK3` and `/NOWAD` options to ignore these types
  • Added `/ZSTORE` option to repack the PK3s without compression: Whilst the PK3 file will be larger than before, it will boot faster. If you are compressing a number of PK3s together, then using /ZSTORE on them might drastically improve the final size of .7Z and .RAR archives when using a very large dictionary size (256 MB or more)
  • Fixes for file size reduction reporting
The added file caching to skip previously crushed files should make this release much more practical for use in a development project. Whilst the gains might be minimal in some cases, you won't have to think about what files you've crushed or not since last time.

Instructions: https://github.com/Kroc/DOOM-Crusher
Download: https://github.com/Kroc/DOOM-Crusher/releases

I'm working towards adding logging so that it's safer to throw entire directories full of PK3/WADs at DOOM-Crusher and you can be assured that nothing breaks afterwards. :)
Gez
 
 
Posts: 17833
Joined: Fri Jul 06, 2007 3:22 pm

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Post by Gez »

Enjay wrote:One thing does concern me from the docs though. I'm not sure how the map sidedef packing is done but if it is the kind of sidedef packing where linedefs which share identical siddefs are given the same sidedef references I have experienced problems with that in the past (although, being perfectly honest, I forget what the problems were).
A side effect of sidedef packing IIRC is that if you have scrolling effect on a line, it will affect the other lines that use the same sidedef (and that's not necessarily desired). And still IIRC, it can even be exploited deliberately by using several lines scrolling the same to get a faster scrolling effect, because each line will cumulatively apply.
User avatar
Kroc
Posts: 116
Joined: Sun Oct 02, 2016 11:37 am
Graphics Processor: Intel (Modern GZDoom)
Contact:

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Post by Kroc »

I disabled sidedef packing in this latest release due to the potential of it causing glitches in maps; only image and deflate optimisations are done.
User avatar
Kroc
Posts: 116
Joined: Sun Oct 02, 2016 11:37 am
Graphics Processor: Intel (Modern GZDoom)
Contact:

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Post by Kroc »

New version released: v1.2

First, my apologies for v1.1 being completely broken; I hadn't touched the code for a while and forgot how terribly fragile batch scripts can be.
  • Fixed options not applying!
  • Many major fixes and improvements to caching:
    • If skipping PNG/JPG files, allow a PK3/WAD to be added to the cache if does not contain any such files
    • Separate hashes into different buckets for file-type, this resolves the issue `/ZSTORE` not repacking PK3s that had been maximally compressed and cached
    • A "cache" folder is now used. Your existing "hashes.txt" will be invalid due to critical bugs in v1.1, sorry :(
  • Fixes and improvements to filesize and percentage reporting
  • Added basic logging (a replica of what appears on screen), see "bin\log.txt", more advanced logging is planned which will capture full error details
  • Signigicantly faster WAD processing (lumpmod modified to identify PNG/JPG lumps, with thanks to _mental_)
  • Faster PK3 processing
Instructions: https://github.com/Kroc/DOOM-Crusher
Download: https://github.com/Kroc/DOOM-Crusher/releases
User avatar
Graf Zahl
Lead GZDoom+Raze Developer
Lead GZDoom+Raze Developer
Posts: 49056
Joined: Sat Jul 19, 2003 10:19 am
Location: Germany

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Post by Graf Zahl »

Gez wrote: A side effect of sidedef packing IIRC is that if you have scrolling effect on a line, it will affect the other lines that use the same sidedef (and that's not necessarily desired). And still IIRC, it can even be exploited deliberately by using several lines scrolling the same to get a faster scrolling effect, because each line will cumulatively apply.

In ZDoom sidedef packing is irrelevant because the engine requires unique sidedefs in the game and unpacks them again.
For other engines a good sidedef compressor ignores sidedefs with attached specials.
User avatar
Kroc
Posts: 116
Joined: Sun Oct 02, 2016 11:37 am
Graphics Processor: Intel (Modern GZDoom)
Contact:

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Post by Kroc »

More fixes.

DOOM-Crusher v1.2.1:

Known issue: Above approximately 30 MB the percentage calculations are completely wrong; this is a limit of Windows' Batch processor and I'll fix it in the next release or so -- it has no effect on the actual functionality of DOOM-Crusher.
  • Fixed hash check failing
  • If JPG/PNG/WAD optimisation encounters an error, a parent WAD/PK3 will not be added to the cache. This is so that it will be retried in the future until there are no more errors
  • Command line options given displayed in the header
Instructions: https://github.com/Kroc/DOOM-Crusher
Download: https://github.com/Kroc/DOOM-Crusher/releases
MrDowntempo
Posts: 38
Joined: Wed Mar 01, 2017 1:58 pm

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Post by MrDowntempo »

Is this safe to run on the official WADs or the Smoosh.wad made by Wadsmoosher?
User avatar
NightFright
Spotlight Team
Posts: 1343
Joined: Fri May 02, 2008 12:29 pm
Location: Germany

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Post by NightFright »

Never ever edit original retail files in any way for any reason. It screams sacrilege all over the place. Anyway, you cannot tell me you need the couple of MBs this may possibly "save" - if it's even that much.
User avatar
Kroc
Posts: 116
Joined: Sun Oct 02, 2016 11:37 am
Graphics Processor: Intel (Modern GZDoom)
Contact:

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Post by Kroc »

DOOM-Crusher will not modify IWADs -- it checks the file header -- and yes, it will work with what WADSmoosher produces. DOOM-Crusher is intended for people developing WADs so that they can distribute the smallest possible WAD and do so as they develop without having to repeat the crushing they did before (DOOM-Crusher skips files that it has crushed before). You won't save much on most WADs outside of ones that have thousands of PNGs that were exported lazily from Photoshop.
VecterStroke
Posts: 51
Joined: Mon Dec 11, 2017 12:28 am

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Post by VecterStroke »

I tried using version 1.2.1, but it immediately exited with the line unpacking ... done. Could it be the operating system?
VecterStroke
Posts: 51
Joined: Mon Dec 11, 2017 12:28 am

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Post by VecterStroke »

Never Mind. I used the latest git release, and changed the line ending in the main bat file. Everything works fine now
MrDowntempo
Posts: 38
Joined: Wed Mar 01, 2017 1:58 pm

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Post by MrDowntempo »

I can't find any documentation that explains where to put PNGOUT.EXE. Can I just drop it in the bin subdir?
(edit) N/M, I figured it out. It goes in bin/pngout/
User avatar
Kroc
Posts: 116
Joined: Sun Oct 02, 2016 11:37 am
Graphics Processor: Intel (Modern GZDoom)
Contact:

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Post by Kroc »

v2.1 released!

Instructions: https://github.com/Kroc/DOOM-Crusher
Download: https://github.com/Kroc/DOOM-Crusher/releases

A massive overhaul of the project; the batch code is significantly more reliable when handling unusual file-names and edge-cases. It's been run over 20 GB of DOOM WADs and I'm confident it's capable enough for developers to use regularly now. Remember -- the benefit of using DOOM-Crusher is that it doesn't crush files it's crushed before!
User avatar
ShockwaveS08
Posts: 193
Joined: Thu Jul 07, 2016 7:29 pm
Preferred Pronouns: He/Him
Operating System Version (Optional): Android 10
Location: Manhattan, IL
Contact:

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Post by ShockwaveS08 »

I can confirm that DOOM-Crusher is working properly now, with the new v2.1.1 update. Crushing down Brutal Doom v21 (RC9) even as we speak, in the hopes of ekeing out an additional performance boost on my phone.
User avatar
Kroc
Posts: 116
Joined: Sun Oct 02, 2016 11:37 am
Graphics Processor: Intel (Modern GZDoom)
Contact:

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Post by Kroc »

I doubt it'll give you a performance boost unfortunately, this only impact WAD loading. Since the images are pushed to the GPU the size there will be the same regardless. I am working on features to allow for actual resizing of textures and converting to JPG to drastically reduce the size of WADs (with loss of quality).

However, if it's Brutal DOOM you want to play, you'll want BDLite: viewtopic.php?t=62203
Brutal DOOM is slow because of its bad code and this cleans it all up into something much more performant.
Post Reply

Return to “Creation, Conversion, and Editing”