Can't load PK3 archives with more than 64k files

Wed Oct 21, 2020 3:22 pm

I've recently come across this issue by sheer ridiculousness of how my mod has grown in filecount due to the number of unicode fonts it contains. In total, there are about 102432 files right now. This issue does not happen with a PK7 or loading the folder directly. When loading a PK3, even though external programs count the files properly and have no issue whatsoever, gzdoom instead says that there are 65520 lumps and then the mod doesn't appear to be loaded at all. I assume there's some sort of built-in limit in gzdoom's zip reader or something.

Re: Can't load PK3 archives with more than 64k files

Wed Oct 21, 2020 3:29 pm

The original Zip format has a 16 bit value for the number of entries in its main directory. Anything more is an extension, and for all intents and purposes a new, unsupported file format, even if it comes with the same name.

Re: Can't load PK3 archives with more than 64k files

Wed Oct 21, 2020 4:28 pm

Hm, that's unfortunate. I'll restrict myself to PK7 instead for this case, then.

Re: Can't load PK3 archives with more than 64k files

Wed Oct 21, 2020 6:17 pm

There's no way to make a Unicode font format that's a single file, like the classic font lumps?

Re: Can't load PK3 archives with more than 64k files

Thu Oct 22, 2020 12:00 am

No. Those classic file formats all have a 256 character limit and overall many fields that are too short. Classic 90's design.
The directory solution is also a lot easier to handle for the engine. If someone comes up with a working font format that can easily handle larger character sets, I'd add support for it, but this has to come from the toolchain side.

Re: Can't load PK3 archives with more than 64k files

Thu Oct 22, 2020 12:04 am

Graf Zahl wrote: If someone comes up with a working font format that can easily handle larger character sets, I'd add support for it, but this has to come from the toolchain side.

How about a simple .wad archive with all the graphics lumps labeled by their unicode codepoint value?

Re: Can't load PK3 archives with more than 64k files

Thu Oct 22, 2020 12:19 am

That could work.
Rewgarding extended Zips, I've been reading up on it. The format is a bit hacky but definitely supportable - but it may take a bit of time to get it in. I'm rather busy at work right now and won't be able to do anything major on both Raze and GZDoom outside the weekends.

Re: Can't load PK3 archives with more than 64k files

Thu Oct 22, 2020 1:38 am

Graf Zahl wrote:The directory solution is also a lot easier to handle for the engine. If someone comes up with a working font format that can easily handle larger character sets, I'd add support for it, but this has to come from the toolchain side.

Maybe the BMFont format? It's an existing bitmap font format which, as far as I know, is even used by some AAA games. It has existing tools for conversion from vector fonts, so toolchain-wise it should be pretty good.
The rendering itself is pretty simple to implement, and the plain text format shouldn't be (too?) hard to implement... A nice thing is that it even supports kerning pairs, which, as far as I know, none of the formats supported by GZDoom do. It's also designed to render from glyph atlases, which lets you reduce the number of drawcalls needed.

(One additional thing that would be nice to look into if BMFont support got added is Signed Distance Field fonts. They upscale and downscale really well)

Re: Can't load PK3 archives with more than 64k files

Thu Oct 22, 2020 1:51 am

BMF is already supported, but it would not surprise me if the implementation is a bit outdated by now - as it predates Unicode support by many years.

Re: Can't load PK3 archives with more than 64k files

Thu Oct 22, 2020 1:58 am

are these two the same thing? I'd welcome BMFont as PB suggested, because the current modern way to make fonts is very tedious for me.

Re: Can't load PK3 archives with more than 64k files

Thu Oct 22, 2020 2:11 am

Graf Zahl wrote:BMF is already supported, but it would not surprise me if the implementation is a bit outdated by now - as it predates Unicode support by many years.

If you mean this BMF, then these are actually completely different formats.

Re: Can't load PK3 archives with more than 64k files

Thu Oct 22, 2020 2:12 am

People should be more careful choosing their names... :mrgreen:

Re: Can't load PK3 archives with more than 64k files

Thu Oct 22, 2020 10:52 am

> there are about 102432 files
> gzdoom instead says that there are 65520 lumps
> original Zip format has a 16 bit value for the number of entries in its main directory
2 ^ 16 = 65536
65536 - 65520 = 16
Where 16 additional files?

Re: Can't load PK3 archives with more than 64k files

Thu Oct 22, 2020 11:11 am

Simple: 15 directories.
It's 65535 as maximum value, btw.