DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Software projects like source ports (3DGE, Eternity, etc), launchers like ZDL, and other useful utilities belong in this forum.
Forum rules
The Projects forums are ONLY for YOUR PROJECTS! If you are asking questions about a project, either find that project's thread, or start a thread in the General section instead.

Got a cool project idea but nothing else? Put it in the project ideas thread instead!

Projects for any Doom-based engine (especially 3DGE) are perfectly acceptable here too.

Please read the full rules for more details.

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Postby Kroc » Fri Jan 13, 2017 1:50 pm

New version released: v1.1

  • Skip files that have already been crushed before (a cache of file hashes is kept in `bin\hashes.txt`) -- incrementally crush your files without having to waste time on files that had been crushed before!
  • WADPTR's sidedef packing is disabled to prevent glitches in maps
  • Added `/NOPK3` and `/NOWAD` options to ignore these types
  • Added `/ZSTORE` option to repack the PK3s without compression: Whilst the PK3 file will be larger than before, it will boot faster. If you are compressing a number of PK3s together, then using /ZSTORE on them might drastically improve the final size of .7Z and .RAR archives when using a very large dictionary size (256 MB or more)
  • Fixes for file size reduction reporting
The added file caching to skip previously crushed files should make this release much more practical for use in a development project. Whilst the gains might be minimal in some cases, you won't have to think about what files you've crushed or not since last time.

Instructions: https://github.com/Kroc/DOOM-Crusher
Download: https://github.com/Kroc/DOOM-Crusher/releases

I'm working towards adding logging so that it's safer to throw entire directories full of PK3/WADs at DOOM-Crusher and you can be assured that nothing breaks afterwards. :)
Kroc
 
Joined: 02 Oct 2016

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Postby Gez » Fri Jan 13, 2017 6:28 pm

Enjay wrote:One thing does concern me from the docs though. I'm not sure how the map sidedef packing is done but if it is the kind of sidedef packing where linedefs which share identical siddefs are given the same sidedef references I have experienced problems with that in the past (although, being perfectly honest, I forget what the problems were).

A side effect of sidedef packing IIRC is that if you have scrolling effect on a line, it will affect the other lines that use the same sidedef (and that's not necessarily desired). And still IIRC, it can even be exploited deliberately by using several lines scrolling the same to get a faster scrolling effect, because each line will cumulatively apply.
Gez
 
 
 
Joined: 06 Jul 2007

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Postby Kroc » Fri Jan 13, 2017 8:02 pm

I disabled sidedef packing in this latest release due to the potential of it causing glitches in maps; only image and deflate optimisations are done.
Kroc
 
Joined: 02 Oct 2016

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Postby Kroc » Sat Jan 21, 2017 4:48 pm

New version released: v1.2

First, my apologies for v1.1 being completely broken; I hadn't touched the code for a while and forgot how terribly fragile batch scripts can be.

  • Fixed options not applying!
  • Many major fixes and improvements to caching:
    • If skipping PNG/JPG files, allow a PK3/WAD to be added to the cache if does not contain any such files
    • Separate hashes into different buckets for file-type, this resolves the issue `/ZSTORE` not repacking PK3s that had been maximally compressed and cached
    • A "cache" folder is now used. Your existing "hashes.txt" will be invalid due to critical bugs in v1.1, sorry :(
  • Fixes and improvements to filesize and percentage reporting
  • Added basic logging (a replica of what appears on screen), see "bin\log.txt", more advanced logging is planned which will capture full error details
  • Signigicantly faster WAD processing (lumpmod modified to identify PNG/JPG lumps, with thanks to _mental_)
  • Faster PK3 processing

Instructions: https://github.com/Kroc/DOOM-Crusher
Download: https://github.com/Kroc/DOOM-Crusher/releases
Kroc
 
Joined: 02 Oct 2016

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Postby Graf Zahl » Sat Jan 21, 2017 6:14 pm

Gez wrote:A side effect of sidedef packing IIRC is that if you have scrolling effect on a line, it will affect the other lines that use the same sidedef (and that's not necessarily desired). And still IIRC, it can even be exploited deliberately by using several lines scrolling the same to get a faster scrolling effect, because each line will cumulatively apply.



In ZDoom sidedef packing is irrelevant because the engine requires unique sidedefs in the game and unpacks them again.
For other engines a good sidedef compressor ignores sidedefs with attached specials.
User avatar
Graf Zahl
Lead GZDoom Developer
 
Joined: 19 Jul 2003
Location: Germany

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Postby Kroc » Fri Jan 27, 2017 5:26 pm

More fixes.

DOOM-Crusher v1.2.1:

Known issue: Above approximately 30 MB the percentage calculations are completely wrong; this is a limit of Windows' Batch processor and I'll fix it in the next release or so -- it has no effect on the actual functionality of DOOM-Crusher.

  • Fixed hash check failing
  • If JPG/PNG/WAD optimisation encounters an error, a parent WAD/PK3 will not be added to the cache. This is so that it will be retried in the future until there are no more errors
  • Command line options given displayed in the header
Instructions: https://github.com/Kroc/DOOM-Crusher
Download: https://github.com/Kroc/DOOM-Crusher/releases
Kroc
 
Joined: 02 Oct 2016

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Postby MrDowntempo » Wed Feb 21, 2018 10:26 am

Is this safe to run on the official WADs or the Smoosh.wad made by Wadsmoosher?
MrDowntempo
 
Joined: 01 Mar 2017

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Postby NightFright » Fri Mar 30, 2018 2:31 pm

Never ever edit original retail files in any way for any reason. It screams sacrilege all over the place. Anyway, you cannot tell me you need the couple of MBs this may possibly "save" - if it's even that much.
User avatar
NightFright
 
Joined: 02 May 2008
Location: Germany

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Postby Kroc » Sat Mar 31, 2018 7:30 am

DOOM-Crusher will not modify IWADs -- it checks the file header -- and yes, it will work with what WADSmoosher produces. DOOM-Crusher is intended for people developing WADs so that they can distribute the smallest possible WAD and do so as they develop without having to repeat the crushing they did before (DOOM-Crusher skips files that it has crushed before). You won't save much on most WADs outside of ones that have thousands of PNGs that were exported lazily from Photoshop.
Kroc
 
Joined: 02 Oct 2016

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Postby VecterStroke » Sun Jul 08, 2018 6:51 pm

I tried using version 1.2.1, but it immediately exited with the line unpacking ... done. Could it be the operating system?
VecterStroke
 
Joined: 11 Dec 2017

Re: DOOM-Crusher : Crush your PK3/WAD/PNG/JPG files!

Postby VecterStroke » Sun Jul 08, 2018 7:12 pm

Never Mind. I used the latest git release, and changed the line ending in the main bat file. Everything works fine now
VecterStroke
 
Joined: 11 Dec 2017

Previous

Return to Software and Ports

Who is online

Users browsing this forum: Blzut3, wildweasel and 2 guests