Can't load PK3 archives with more than 64k files

Is there something that doesn't work right in the latest GZDoom? Post about it here.

Moderator: GZDoom Developers

Forum rules
Please construct and post a simple demo whenever possible for all bug reports. Please provide links to everything.

If you can include a wad demonstrating the problem, please do so. Bug reports that include fully-constructed demos have a much better chance of being investigated in a timely manner than those that don't.

Please make a new topic for every bug. Don't combine multiple bugs into a single topic. Thanks!

Can't load PK3 archives with more than 64k files

Postby Marisa Kirisame » Wed Oct 21, 2020 3:22 pm

I've recently come across this issue by sheer ridiculousness of how my mod has grown in filecount due to the number of unicode fonts it contains. In total, there are about 102432 files right now. This issue does not happen with a PK7 or loading the folder directly. When loading a PK3, even though external programs count the files properly and have no issue whatsoever, gzdoom instead says that there are 65520 lumps and then the mod doesn't appear to be loaded at all. I assume there's some sort of built-in limit in gzdoom's zip reader or something.
User avatar
Marisa Kirisame
ZScript Crimester
 
 
 
Joined: 08 Feb 2008
Location: Vigo, Galicia
Discord: 霧雨魔理沙#1666
Twitch ID: MarisaDOOM
Github ID: OrdinaryMagician
Operating System: Other Linux 64-bit
OS Test Version: No (Using Stable Public Version)
Graphics Processor: nVidia with Vulkan support

Re: Can't load PK3 archives with more than 64k files

Postby Graf Zahl » Wed Oct 21, 2020 3:29 pm

The original Zip format has a 16 bit value for the number of entries in its main directory. Anything more is an extension, and for all intents and purposes a new, unsupported file format, even if it comes with the same name.
User avatar
Graf Zahl
Lead GZDoom+Raze Developer
Lead GZDoom+Raze Developer
 
Joined: 19 Jul 2003
Location: Germany

Re: Can't load PK3 archives with more than 64k files

Postby Marisa Kirisame » Wed Oct 21, 2020 4:28 pm

Hm, that's unfortunate. I'll restrict myself to PK7 instead for this case, then.
User avatar
Marisa Kirisame
ZScript Crimester
 
 
 
Joined: 08 Feb 2008
Location: Vigo, Galicia
Discord: 霧雨魔理沙#1666
Twitch ID: MarisaDOOM
Github ID: OrdinaryMagician
Operating System: Other Linux 64-bit
OS Test Version: No (Using Stable Public Version)
Graphics Processor: nVidia with Vulkan support

Re: Can't load PK3 archives with more than 64k files

Postby Nash » Wed Oct 21, 2020 6:17 pm

There's no way to make a Unicode font format that's a single file, like the classic font lumps?
User avatar
Nash
Twitter/Facebook/Youtube: nashmuhandes
 
 
 
Joined: 27 Oct 2003
Location: Kuala Lumpur, Malaysia
Twitch ID: nashmuhandes
Github ID: nashmuhandes

Re: Can't load PK3 archives with more than 64k files

Postby Graf Zahl » Thu Oct 22, 2020 12:00 am

No. Those classic file formats all have a 256 character limit and overall many fields that are too short. Classic 90's design.
The directory solution is also a lot easier to handle for the engine. If someone comes up with a working font format that can easily handle larger character sets, I'd add support for it, but this has to come from the toolchain side.
User avatar
Graf Zahl
Lead GZDoom+Raze Developer
Lead GZDoom+Raze Developer
 
Joined: 19 Jul 2003
Location: Germany

Re: Can't load PK3 archives with more than 64k files

Postby Rachael » Thu Oct 22, 2020 12:04 am

Graf Zahl wrote: If someone comes up with a working font format that can easily handle larger character sets, I'd add support for it, but this has to come from the toolchain side.

How about a simple .wad archive with all the graphics lumps labeled by their unicode codepoint value?
User avatar
Rachael
Admin
 
Joined: 13 Jan 2004
Discord: Rachael#3767
Twitch ID: madamerachelle
Github ID: madame-rachelle
Operating System: Windows 10/8.1/8/201x 64-bit
OS Test Version: No (Using Stable Public Version)
Graphics Processor: nVidia with Vulkan support

Re: Can't load PK3 archives with more than 64k files

Postby Graf Zahl » Thu Oct 22, 2020 12:19 am

That could work.
Rewgarding extended Zips, I've been reading up on it. The format is a bit hacky but definitely supportable - but it may take a bit of time to get it in. I'm rather busy at work right now and won't be able to do anything major on both Raze and GZDoom outside the weekends.
User avatar
Graf Zahl
Lead GZDoom+Raze Developer
Lead GZDoom+Raze Developer
 
Joined: 19 Jul 2003
Location: Germany

Re: Can't load PK3 archives with more than 64k files

Postby phantombeta » Thu Oct 22, 2020 1:38 am

Graf Zahl wrote:The directory solution is also a lot easier to handle for the engine. If someone comes up with a working font format that can easily handle larger character sets, I'd add support for it, but this has to come from the toolchain side.

Maybe the BMFont format? It's an existing bitmap font format which, as far as I know, is even used by some AAA games. It has existing tools for conversion from vector fonts, so toolchain-wise it should be pretty good.
The rendering itself is pretty simple to implement, and the plain text format shouldn't be (too?) hard to implement... A nice thing is that it even supports kerning pairs, which, as far as I know, none of the formats supported by GZDoom do. It's also designed to render from glyph atlases, which lets you reduce the number of drawcalls needed.

(One additional thing that would be nice to look into if BMFont support got added is Signed Distance Field fonts. They upscale and downscale really well)
User avatar
phantombeta
Tired of being treated like trash by control freaks
 
Joined: 02 May 2013

Re: Can't load PK3 archives with more than 64k files

Postby Graf Zahl » Thu Oct 22, 2020 1:51 am

BMF is already supported, but it would not surprise me if the implementation is a bit outdated by now - as it predates Unicode support by many years.
User avatar
Graf Zahl
Lead GZDoom+Raze Developer
Lead GZDoom+Raze Developer
 
Joined: 19 Jul 2003
Location: Germany

Re: Can't load PK3 archives with more than 64k files

Postby SanyaWaffles » Thu Oct 22, 2020 1:58 am

are these two the same thing? I'd welcome BMFont as PB suggested, because the current modern way to make fonts is very tedious for me.
User avatar
SanyaWaffles
Never trust floor sandwiches
 
Joined: 25 Apr 2013
Location: Eastern Ohio
Discord: SanyaWaffles#5095
Twitch ID: sanyawaffles
Operating System: Windows 10/8.1/8/201x 64-bit
OS Test Version: No (Using Stable Public Version)
Graphics Processor: nVidia with Vulkan support

Re: Can't load PK3 archives with more than 64k files

Postby phantombeta » Thu Oct 22, 2020 2:11 am

Graf Zahl wrote:BMF is already supported, but it would not surprise me if the implementation is a bit outdated by now - as it predates Unicode support by many years.

If you mean this BMF, then these are actually completely different formats.
User avatar
phantombeta
Tired of being treated like trash by control freaks
 
Joined: 02 May 2013

Re: Can't load PK3 archives with more than 64k files

Postby Graf Zahl » Thu Oct 22, 2020 2:12 am

People should be more careful choosing their names... :mrgreen:
User avatar
Graf Zahl
Lead GZDoom+Raze Developer
Lead GZDoom+Raze Developer
 
Joined: 19 Jul 2003
Location: Germany

Re: Can't load PK3 archives with more than 64k files

Postby Apeirogon » Thu Oct 22, 2020 10:52 am

> there are about 102432 files
> gzdoom instead says that there are 65520 lumps
> original Zip format has a 16 bit value for the number of entries in its main directory
2 ^ 16 = 65536
65536 - 65520 = 16
Where 16 additional files?
User avatar
Apeirogon
I have a strange sense of humour
 
Joined: 12 Jun 2017

Re: Can't load PK3 archives with more than 64k files

Postby Graf Zahl » Thu Oct 22, 2020 11:11 am

Simple: 15 directories.
It's 65535 as maximum value, btw.
User avatar
Graf Zahl
Lead GZDoom+Raze Developer
Lead GZDoom+Raze Developer
 
Joined: 19 Jul 2003
Location: Germany


Return to Bugs

Who is online

Users browsing this forum: No registered users and 0 guests