ZDBSP deletes unknown UDMF blocks

[dpJudas]

Excellent! Let's solve it that way.

[Redneckerz]

@Redneckerz: ZDRay is still under development. Nothing is set in stone there yet - hence why I'm discussing with Nash, Boris and Graf where the best place is to put this data.

I know, that's why i said what i said I am just keeping a close eye to this because ZDRay excites me to no end. I didn't write the public documentation stuff if i werent!

I can give you basic feedback of the current plan.

The TEXTMAP lump contain two new actor/thing types: static lights and light probes. ZDRay is a ZDBSP replacement. It performs exactly the duties as ZDBSP, except it also generates an additional LIGHTMAP lump. That lump contains the lightmap texture, the texture coordinate info that maps subsector, sides and 3d floors to locations in the texture, and light probe data for the sprites and 3d models.

The general idea is that UDB will get editing support for placing light probes and static lights. Then instead of calling zdbsp it will call zdray and the resulting wad will be ready to be consumed by GZDoom. That's partly why the documentation of zdray itself isn't particular important - it will just be an overview of the new arguments that weren't there already from zdbsp. This is all behind the scenes plumbing for the mappers. For them this is just a few new tools in UDB and the rest should be as they've always known doom mapping.

I see, so the general gist is a far more tighter integration of ZDRay and UDB then what was there previously - ZDRay being a lightmapper first, a nodesbuilder second to a full on ZDBSP replacement that introduces a new editing step comparable to modern-game development - Light baking. I assume herein that, as UDB is also used by non-ZDoom authors, ZDRay will be optional - Atleast the lightbaking part of it.

I am certain you know of DLight, so a particularly dumb question for me to ask would be if ZDRay will have a specification, similar to how DLight works?

This is all very useful stuff indeed and definitely requires a rewrite of current Wiki documentation regarding ZDRay. Since, like you say, ''nothing is set in stone'' yet, i think it would be wise to atleast change what is currently hosted in ZDoom.org. (If you weren't aware, there are compiled versions of ZDRay and a custom GZDoom build hosted in its own directory).

As Nash pushed new experimental binaries over at discord (With ZDRay now supporting an indirect bounce) of ZDRay and the latest GZDoom branch, i wish to simplify this directory given this is now a current in development project. The article should reflect these updates.

[dpJudas]

I see, so the general gist is a far more tighter integration of ZDRay and UDB then what was there previously - ZDRay being a lightmapper first, a nodesbuilder second to a full on ZDBSP replacement that introduces a new editing step comparable to modern-game development - Light baking. I assume herein that, as UDB is also used by non-ZDoom authors, ZDRay will be optional - Atleast the lightbaking part of it.

As far as I know, UDB supports multiple BSP node generator options. ZDRay will probably just become another option there, next to ZDBSP.

I am certain you know of DLight, so a particularly dumb question for me to ask would be if ZDRay will have a specification, similar to how DLight works?

ZDRay is actually a merger of two code bases: zdbsp and dlight. There is a very basic specification of the input UDMF in the README.md. The LIGHTMAP lump format is not documented yet. Mostly because I want to still be able to change it as I see fit until the feature actually lands in GZDoom. The light probe part of the lump is still being improved upon.

(If you weren't aware, there are compiled versions of ZDRay and a custom GZDoom build hosted in its own directory).

Those are unsupported versions of both zdray and gzdoom as far as I'm concerned. What will probably happen here is that first Nash will build a proof of concept (his quake in 10 days project). That will show internally to us that it works. Then that will become a PR for GZDoom. If that is merged in then there will be a zdray 1.0 released with documentation of the lightmap lump.

I know you're really want to get started on the documentation part of this, but keep in mind I'm the only active C++ developer on this project for both GZDoom and ZDRay. Nash is the one doing active testing of the feature (with some C++ patches, too), creating maps and such required to see if stuff works as intended or need adjustments. The more time I spend on "informing the public" the less I'll actually be working on this feature. Hopefully soon Nash will offer you a demo. His map looks really cool!

As Nash pushed new experimental binaries over at discord (With ZDRay now supporting an indirect bounce) of ZDRay and the latest GZDoom branch, i wish to simplify this directory given this is now a current in development project. The article should reflect these updates.

The indirect bounce thing is actually pretty old. What's new since Nash resurrected the thing (by porting the lightmap branch to current GZD) are the light probes. Then there's also a fix in the GZD branch for lightmaps on walls with 3d floors. Both have to be in perfect sync right now or all bets are off. That's because if I need a change to the lump format I update both at the same time.

In general the ray tracing part is a separate journey that comes later. There are a lot of things that could be done better there. For now, the goal is to make sure the GZDoom side of things is working solidly.

[Graf Zahl]

@Redneckerz:

Please don't jump the gun here. It's still a WIP feature and one important thing is that both the feature set and the way it can be accessed will be subject to change until several people agree that it's ready.
We've had serious problems with this in the past - unfinished features getting documented, people using them and then having to deal with the fallout of having to continue support poorly developed crutches because they were already out in the wild.

Until this gets merged into the mainline, please stay away from documenting it!

[Redneckerz]

As far as I know, UDB supports multiple BSP node generator options. ZDRay will probably just become another option there, next to ZDBSP.

That makes the most sense i reckon.

The LIGHTMAP lump format is not documented yet. Mostly because I want to still be able to change it as I see fit until the feature actually lands in GZDoom. The light probe part of the lump is still being improved upon.

Yes, this is the part i was referring to - the LIGHTMAP lump. Ill hold off things until this stuff is any finalized but whenever the LIGHTMAP lump spec is ready, consider me ready to put it wherever you want it to.

PS: I know of the spec. Its where the current article replies upon, haha.

Those are unsupported versions of both zdray and gzdoom as far as I'm concerned. What will probably happen here is that first Nash will build a proof of concept (his quake in 10 days project). That will show internally to us that it works. Then that will become a PR for GZDoom. If that is merged in then there will be a zdray 1.0 released with documentation of the lightmap lump.

Yes it is an unsupported build, made out of 2018 source code. Atleast it runs the supplied lightmap test just fine (and looking great whilst doing so) and i happen to have earlier, unlit versions of said test which aren't supplied there. Nash just told me the lightmap format changed so the testmap won't work any longer, but at the very least it serves as a nice proof-of-concept.

I also just realized that i wrote the ZDRay part but never bothered to write up DLight.

I know you're really want to get started on the documentation part of this, but keep in mind I'm the only active C++ developer on this project for both GZDoom and ZDRay. Nash is the one doing active testing of the feature (with some C++ patches, too), creating maps and such required to see if stuff works as intended or need adjustments. The more time I spend on "informing the public" the less I'll actually be working on this feature. Hopefully soon Nash will offer you a demo. His map looks really cool!

Never mind me babbling then! Leave the public part to me or Nash and you just do what you do best

The indirect bounce thing is actually pretty old. What's new since Nash resurrected the thing (by porting the lightmap branch to current GZD) are the light probes. Then there's also a fix in the GZD branch for lightmaps on walls with 3d floors. Both have to be in perfect sync right now or all bets are off. That's because if I need a change to the lump format I update both at the same time.

I noticed indeed, although the latest ZDRay build Nash provided does now have an option for indirect bounces - The build that is in the directory from 2018-12-30 does not list it in the console, although it should be there. So i assumed it was not yet enabled.

TLDR variant; Ill hold off stuff till things become more stable and ill only update that what makes a lick of sense relative to this project.

@Redneckerz:
Please don't jump the gun here. It's still a WIP feature and one important thing is that both the feature set and the way it can be accessed will be subject to change until several people agree that it's ready.
We've had serious problems with this in the past - unfinished features getting documented, people using them and then having to deal with the fallout of having to continue support poorly developed crutches because they were already out in the wild.

I am aware of its issues in the past, just happy that it gets a work on now. I won't jump the gun here, just that the development of this inspired me enough to actually document its direct inspiration, DLight. It is far more limited compared to what ZDRay attempts to do, but - in theory - should support more ports, due to relying on GL-nodes/GLBSP. ZDRay circumvents that problem entirely as far as i can understand.

Until this gets merged into the mainline, please stay away from documenting it!

No worries. The only thing i will add are the following:
* The indirect light bounce option, since that already existed back in 2018.
* Current 2021 progress. Nothing special.

[Nash]

I've removed the files from the Discord.

Let me make it clear for all to read: everything about this is heavily WIP and subject to change. There is a reason I'm keeping its usage mostly to my own project for the time being - dpJudas and me will need plenty of time to figure out a crapton of details - from both the engine side and from the user (mapper) side - and the feature won't be ready for quite some time. There is no ETA.

We WILL break shit and we don't care if you used it prematurely and your shit breaks too. Not our problem. :P

[dpJudas]

I am aware of its issues in the past, just happy that it gets a work on now. I won't jump the gun here, just that the development of this inspired me enough to actually document its direct inspiration, DLight. It is far more limited compared to what ZDRay attempts to do, but - in theory - should support more ports, due to relying on GL-nodes/GLBSP. ZDRay circumvents that problem entirely as far as i can understand.

The difference between DLight and ZDRay is complicated. I started out with only forking DLight and quickly ran into the issue that the nodes it outputs forces gzdoom to rebuild the nodes (*). The lightmap and the subsectors are tightly coupled in the sense that the texture coordinates are mapped to the subsector vertices. If a rebuild creates a new BSP with new vertices the light map can't be applied anymore.

The easiest way to solve this was to replace the map loader and BSP generator with zdbsp. Luckily since they had some common ancestry it wasn't completely impossible do this. In the early days of ZDRay it could actually still output the DLight lumps for non-UDMF wads. I dropped the feature because Strife VE was the only source port supporting it (it is even public?), and I needed to change the format of the sprite light anyway. Since I had now broken backwards compatibility I upgraded the lightmap to use 16 bit half-floats instead of 8 bit.

Supporting non-UDMF wads is fairly easy for the LIGHTMAP lump. One line needs to be uncommented for ZDRay to create it. The problem is how to now define the lights. The original Doom format is not as flexible as UDMF. If I remember correct DLight used a separate file with light definition - a super hostile mapper solution. In short: there was no point supporting something other ports had not adopted anyway.

*) At least that's how I remember it. It might also been that it couldn't use UDMF as input. Or both.

[Enjay]

As far as I know, UDB supports multiple BSP node generator options. ZDRay will probably just become another option there, next to ZDBSP.

Yup.

[boris]

Please put this stuff in a different lump inside the map. There is zero guarantee that any of this can be preserved by tools as custom blocks have never been used by any port so tool support in general will be spotty as well.

The main problem with adding a new section type to TEXTMAP is that it'd invalidate all tools not aware of it and also require any tool wishing to support editing this to be extended, because you couldn't even use a generic property pane to handle it.

Then those tools aren't spec compliant. From the UDMF specs:

Compliant parsers will ignore all unknown keywords in global assignments,
block-level assignments, and block headers. Compliant parsers should attempt
to preserve as much of such information as is possible by using a flexible
mapping such as hashing.

Also ZDBSP doesn't seem to care about the namespace at all. So some other port (like Eternity) could add some new block, and ZDBSP would simply remove it without cause.

It's pretty crazy how fragile the UDMF ecosystem is.

[Graf Zahl]

It's pretty crazy how fragile the UDMF ecosystem is.

Par for course it seems. Just look at how many tools delete the grAb chunks in PNGs, although they are marked as "must be kept" according to the spec.

[Enjay]

Just look at how many tools delete the grAb chunks in PNGs, although they are marked as "must be kept" according to the spec.

The vast majority. Almost every graphics program on my computer trashes them - and I have quite a few graphics programs. The one exception is MS paint which usually keeps them (I only say usually because I rarely use it so can't say that it always manages). Interestingly, other MS graphics programs that I also have do not preserve the chunks.

[Redneckerz]

The difference between DLight and ZDRay is complicated. I started out with only forking DLight and quickly ran into the issue that the nodes it outputs forces gzdoom to rebuild the nodes (*). The lightmap and the subsectors are tightly coupled in the sense that the texture coordinates are mapped to the subsector vertices. If a rebuild creates a new BSP with new vertices the light map can't be applied anymore.

DLight originally relied on a precompiled WAD built with GLBSP before it could take effect. GL_PVS was used to speed up the baking times, but it could work without.

Whilst ports with GLBSP support are rather broad, GL_PVS is only used by Vavoom and possibly K8vavoom. Only Doom 64 EX and Strife VE have support for understanding the lightmap lumps DLight creates, so any port that would want to take advantage of DLight would need to write support for those.*

In essence ZDRay is the same thing: It needs a port (GZDoom) to support the lumps it generates. Luck of the draw here is that ZDRay is far more advanced/optimized and uses ZDBSP as an internal, not external nodes builder.

There is still some potential in DLight, mind you, but those lumps need support.

*DLight is however very generic: It is built for Doom, but because of its reliance on other software, it can basically bake maps for any program, provided said program will recognize the lumps. Its probably a bit too homebrew for that given there are far more advanced baking solutions around, but its there atleast.

Yup.

Well that's easy peasy then.

[Nash]

I'm sure dpJudas is familiar with what dlight is and how it works, seeing that he pretty much started with dlight as a base, and then completely replaced it with something else. Moreover, none of us are interested with dlight as it has nothing to do with the lightmap implementation GZDoom is going to use.

That said, a sun definition actor has already been made as per Graf's recommendation, so I think ZDRay and lightmap discussion don't belong in this thread anymore.

(whether anyone wants to do anything about the actual issue in the OP remains to be seen - highly unlikely, going by the first few replies about breaking standard...)

[Gez]

Whilst ports with GLBSP support are rather broad, GL_PVS is only used by Vavoom and possibly K8vavoom.

I think I remember Ketmar saying he yanked that part of the code away while rewriting the light and shadow code? Don't think GL_PVS is still handled by k8vavoom.

[Redneckerz]

I'm sure dpJudas is familiar with what dlight is and how it works, seeing that he pretty much started with dlight as a base, and then completely replaced it with something else. Moreover, none of us are interested with dlight as it has nothing to do with the lightmap implementation GZDoom is going to use.

That said, a sun definition actor has already been made as per Graf's recommendation, so I think ZDRay and lightmap discussion don't belong in this thread anymore.

I was just explaining but i agree, this has run its course.

Whilst ports with GLBSP support are rather broad, GL_PVS is only used by Vavoom and possibly K8vavoom.

I think I remember Ketmar saying he yanked that part of the code away while rewriting the light and shadow code? Don't think GL_PVS is still handled by k8vavoom.

So i didn't misremember. I wasn't clear so i obmitted it in my response, but you are right.*

*I mean i just instantly believe you are right since you are the actual encyclopedia here. I can't discourse against the kind of academic knowledge you have!