Support of new image formats

[Professor Hastig]

Seeing that WebP went in, JPEG XL went in and out again and AVIF is currently waiting, what's the plans here?
Are there more formats on the roadmap?

The old formats were mostly unproblematic, they can all be supported with small dependencies that can be distributed along with the engine - in the case of JPEG that bundled dependency could even be dropped in favor of the already included stb-image.

But these new formats are a completely different story. They use large codecs that often come with complex build processes and the only way to handle them is using the provided decoder library, which each time adds a complex dependency.
This makes sense for formats that may be entering the mainstream, but I really do not see this happening for any of the current ones, so I am not sure if the current approach is good.

The same applies to videos as well, of course. Again, none of these formats is easy to handle without linking to a large library.
The main question with all this, can we take the existence of these libraries for granted or is there any danger of them disappearing?

[Rachael]

Honestly my opinion: I like that the project upgraded the build system to use a package manager, but only for very specific reasons: 1) I wanted to eventually get rid of the binary blobs sitting in the repo (the precompiled .libs for Windows sitting in the bin/ folder), 2) I wanted the project to easily be able to upgrade its libraries and swap out some of the old libraries sitting in the repo as well (i.e. zlib if it would ever be needed), 3) It also would allow nearly all dependencies to be statically linked and reduce (perhaps eliminate) the need to ship .dll's (or .so's or .dylib's) along with the executable, and 4) Hopefully it will make building on MacOS much less of a massive pain in the ass and not need a library builder project just to make a distributable executable.

It was never a license to just pull in every third-party library under the sun willy-nilly. I have no problem with new formats being added but third party libraries need to be vetted and they need to be vetted hard. The biggest problem with adding new libraries is: 1) Some add massive bloat, 2) Some are poorly programmed and inefficient and poorly maintained, or have lots of errors, 3) Some are just metalibraries that pull in between 10 to 200 different other libraries and provide little functionality uniquely their own, 4) possible licensing and patent issues, and 5) Some combination of those other 4.

If new formats are being supported via library then the merits of the library need to be weighed along with the necessity of supporting it, versus how problematic the library is in consideration of the points above.

Adding third-party code is not free. I will only be in support of adding new libraries to the project if they provide a greater value over the cost of adding the dependency and do not cause licensing issues. I am not strictly against the new formats and code being added - but I am against them being added without consideration to the cost that they add.

[Guest]

So where do WebP and AVIF come in here? And do we really need AV1 video support, just because it's doable? I can see some value with lossy formats that compress better with less artifacts, but how many of these make sense?

[Nash]

what's the plans here? Are there more formats on the roadmap?

There are no plans and there's no roadmap. They're all external contributions arbitrarily sent by one person. There was no conversation, no surveys, no nothing. Nobody asked for new formats. No mod or game developer has ever asked about, or have ran into any problems with the existing formats, any problems so severe that one would say, "can we get better formats".

Also, everything Rachael said.

[Graf Zahl]

WebP was something I had wanted for quite some time, but I'm still undecided on AVIF. The format is quite recent and doesn't really bring anything to the table that WebP doesn't.
Regarding the alternative video formats, I am even more undecided. We need a format with sound stream included, not more that don't.

With something as old as WebP I am also not worried that the library might vanish, so that is fine. All things considered I think it is far more important to overhaul the texture system to process compressed DDS textures natively than adding more formats with high overhead - which brings me to an important issue. It's really time to part ways with OpenGL and offload it to a legacy fork. Then we can hopefully bring GZDoom and vkDoom back together again.

[Graf Zahl]

BTW, I rejected AVIF. The decoder being used for this and AV1 more than doubles the EXE size. That's just pure and utter bloat. WebP only adds a single MB of file size, not 12!
I think this squarely falls under Rachael's case where third party code becomes a liability.

[Darkcrafter]

If that webp would allow the textures being as small if not smaller as jpegs and also offer alpha channel then I would be really glad to use it as lesser file sizes for mods are always better, especially if it doesn't come at a bigger performance cost. For the performance reasons, I denied using 7z compression in favor of bigger zip.

Now for really big high-res tree leaves textures I must use SuperPNG copression with color quantization in order to get much smaller files, perhaps introducition of "jpeg" with alphas could help it.

[Chris]

I'm actually pretty surprised to even see JPG as a texture format. Aside from the cost of decompression, the chroma subsampling and lossy nature of the compression often makes them unpleasant to look at normally, and that's ignoring using them as a sprite, wall, or floor texture that gets stretched and warped, making the loss more apparent. PNG makes a bit more sense since it's lossless, though still needs a non-negligible amount of time to decompress and load, especially for larger images. Formats that are quick to read and dump straight into VRAM with little intermediate processing seem like the most ideal options, keeping the time to load down especially for cases where large images or image sets aren't preloaded.

[Darkcrafter]

I use JPEG for large textures in my Hell Renaissance project, it's been of a huge help. If it would only be possible to have something like JPEG but with Alpha support or even being able to store RGB and Alpha in two separate files for one texture, that would help even more on big transparent textures like leaves but even with current possibilites, I'm really satisfied with SuperPNG compression capabilities.

[axredneck]

... the chroma subsampling ...

You can actually compress images to JPEG without chroma subsampling.

[Chris]

... the chroma subsampling ...

You can actually compress images to JPEG without chroma subsampling.

Which increases the file size, reducing the primary benefit of the format.

There's also the issue that it creates a false sense of savings, with people seeing they can compress their large textures down to a few 10s or 100s of KB, not realizing that it then has to be decompressed to several MB per image when loaded into VRAM. There are some compression formats that can be stored directly in VRAM, which would make more sense to support as that will actually save runtime memory, along with containers designed for GPU texture formats like DDS and KTX ("KTX 2.0 adds support for Basis Universal supercompressed GPU textures. Developed by Binomial, this compression technology produces compact textures that can be efficiently transcoded to a variety of GPU compressed texture formats at run-time." sounds pretty sweet).

[dpJudas]

That's the thing. If GZDoom should get more image formats, it shouldn't be web formats, but rather things meant for game engines. I'm not a fan of DDS because it is just a garbage format that was never really meant to be used for anything than Direc3D sample projects, but something designed to support block compression formats that GPU's like to use. That KTX thing looks interesting, although I haven't evaluated its quality.

However keep in mind that once we are talking about large textures and GZDoom, the engine isn't designed to deal with it. Everything is built on the assumption we can just load it all into memory - so you'd quickly run into a situation anyway where you'd have to implement an actual texture streaming solution. Those introduce additional complexity since you'd want low resolution versions of the textures loaded always so you always have something to render while you stream in the hires assets.

[Rachael]

Not sure how relevant texture compression is going to be in the near to mid future. GPU RAM seems to be exploding at a rate where it's edging out and sometimes even beating standard system RAM amounts. 30xx and 40xx series GPU's already have 12 GB variants which by itself is more than the 8 GB that was standard in systems a decade ago.

[Graf Zahl]

It's mainly important for hires stuff, its greatest advantage is that it saves GPU bandwidth there. For Doom sized textures it is irrelevant because you can see the compression artifacts.

[Rachael]

What I mean is - newer GPU's these days seem to be getting stuffed with RAM, a great deal moreso than they ever did before. Before, 2-3 GB was enough for your graphics card for most things, it's understandable that amount would go up after a time, but these days it's a lot more than they normally would have at the rate they were going. I'm not saying this is really a bad thing - I'm just pointing out that with such things, compressing textures seems to offer far less benefit than it used to, even with bigger projects. At this rate GPU's and regular motherboards will be edging each other out in total RAM capacity, and I am not sure how I feel about that.

Today systems are being sold with 16 GB total system RAM. I can almost guarantee you the higher end cards in NVidia's next gen will offer more GPU RAM than that.