ZDoom

This info is soon to be obsolete, in future versions brightmaps will be independent of the shaders.

material texture "TEXNAME"
{
    shader "shaders/SHADERNAME.fp"
    normal "materials/normalmaps/TEXNAME.png"
    specular "materials/specular/TEXNAME.png"
    brightmap "materials/brightmaps/TEXNAME.png"
    Texture masktexture "materials/masks/TEXNAME.png"

    Speed 1
    Define MASKED
    Define AXIS = "x"
    Define DIRECTION = "1.0 -"
    Define COLOR1 = "vec3(0.2, 0.0, 0.0)"
    Define COLOR2 = "vec3(1.0, 0.0, 0.0)"

    glossiness 10.0
    specularlevel 0.5
}

Thanks for adding this info.

This looks like the correct implementation, and it even supports the Texture/Define props needed for generalizations:

Wiki could probably stand to be updated.

[code]material texture "TEXNAME"
{
shader "shaders/SHADERNAME.fp"
normal "materials/normalmaps/TEXNAME.png"
specular "materials/specular/TEXNAME.png"
brightmap "materials/brightmaps/TEXNAME.png"
Texture masktexture "materials/masks/TEXNAME.png"

Speed 1
Define MASKED
Define AXIS = "x"
Define DIRECTION = "1.0 -"
Define COLOR1 = "vec3(0.2, 0.0, 0.0)"
Define COLOR2 = "vec3(1.0, 0.0, 0.0)"

glossiness 10.0
specularlevel 0.5
}[/code]

material texture "TEXNAME"
{
    shader "shaders/SHADERNAME.fp"
    normal "materials/normalmaps/TEXNAME.png"
    specular "materials/specular/TEXNAME.png"
    brightmap "materials/brightmaps/TEXNAME.png"
    glossiness 10.0
    specularlevel 0.5
}

@Xabis
This should be doable with just material block. It supports defining shaders within itself too. Here's a example:
[code]material texture "TEXNAME"
{
shader "shaders/SHADERNAME.fp"
normal "materials/normalmaps/TEXNAME.png"
specular "materials/specular/TEXNAME.png"
brightmap "materials/brightmaps/TEXNAME.png"
glossiness 10.0
specularlevel 0.5
}[/code]

The proper way to do it would be to use the brightmap keyword/option in the material itself now, according to phantombeta.

HardwareShader TEXNAME
{
	Shader "shaders/SHADERLUMP"
	Speed 1.0
	Texture brighttexture "brightmaps/BMNAME.png"
	Define BRIGHTMAP
}

HardwareShader texture TEXNAME
{
	Shader "shaders/SHADERNAME"
	Material specularbrightmap
}
brightmap texture TEXNAME { map "brightmaps/TEXNAME.png" }
material texture TEXNAME
{
	normal materials/normalmaps/TEXNAME.png
	specular materials/specular/TEXNAME.png

	glossiness 100
	specularlevel 40
}

vec4 ProcessTexel()
{
   //this is your shader code here
   return texture(tex, vTexCoord.st);
}

//this is basically a copy of shaders/glsl/func_spec.fp
//note: pbr will need to use whats in shaders/glsl/func_pbr.fp instead, which i didnt add here
Material ProcessMaterial()
{
	Material material;
	material.Base = ProcessTexel();
#ifdef NORMALMAP
	material.Normal = ApplyNormalMap(vTexCoord.st);
#endif
#ifdef SPECULAR
	material.Specular = texture(speculartexture, vTexCoord.st).rgb;
	material.Glossiness = uSpecularMaterial.x;
	material.SpecularLevel = uSpecularMaterial.y;
#endif
#if defined(BRIGHTMAP)
	material.Bright = texture(brighttexture, vTexCoord.st);
#endif
	return material;
}

Hello,

Apologies for the necro here, but for anyone else wanting this feature, you can do this with a simple hack. I am unsure what the minimum required gzdoom version is.

[code]HardwareShader TEXNAME
{
Shader "shaders/SHADERLUMP"
Speed 1.0
Texture brighttexture "brightmaps/BMNAME.png"
Define BRIGHTMAP
}[/code]
Should work with old-style Process and new-style ProcessTexel/ProcessMaterial types.

---------------------------------------------
UPDATE:
---------------------------------------------
If you want to have a shader applied, along with a material and brightmapping, you will need to specify the Material type in GLDEFS that matches what gzdoom selects (e.g. "specular" if using normal/specular without brightmaps; "specularbrightmap", if using bright map; "pbr" if using those). Then define the brightmap & material as you normally would.

In your shader, you will need to define ProcessMaterial, and provide an override that supports all of these, since the default does not for some reason.

EXAMPLE GLDEF ENTRY:
[code]HardwareShader texture TEXNAME
{
Shader "shaders/SHADERNAME"
Material specularbrightmap
}
brightmap texture TEXNAME { map "brightmaps/TEXNAME.png" }
material texture TEXNAME
{
normal materials/normalmaps/TEXNAME.png
specular materials/specular/TEXNAME.png

glossiness 100
specularlevel 40
}[/code]

Then the shader:
[code]vec4 ProcessTexel()
{
//this is your shader code here
return texture(tex, vTexCoord.st);
}

//this is basically a copy of shaders/glsl/func_spec.fp
//note: pbr will need to use whats in shaders/glsl/func_pbr.fp instead, which i didnt add here
Material ProcessMaterial()
{
Material material;
material.Base = ProcessTexel();
#ifdef NORMALMAP
material.Normal = ApplyNormalMap(vTexCoord.st);
#endif
#ifdef SPECULAR
material.Specular = texture(speculartexture, vTexCoord.st).rgb;
material.Glossiness = uSpecularMaterial.x;
material.SpecularLevel = uSpecularMaterial.y;
#endif
#if defined(BRIGHTMAP)
material.Bright = texture(brighttexture, vTexCoord.st);
#endif
return material;
}[/code]

This should be doable now with hardware shaders being capable of having multiple layers. Of course it cannot be automatic, the custom shader needs to do the brightmapping itself.

That is truly unfortunate.

It is a known limitation, partly due to the way code dealing with subshaders was written. Maybe it can be dealt with after the materials branch is finished, but until then I don't think this issue will be fixed.

I created a wall texture (not a sprite) that uses both a bright map and a shader.

So long as the shader is applied to the texture, the bright map is ignored. As soon as the shader is removed, the bright map begins working again. This occurs from at least gzdoom 2.3.2 all the way to 3.2.5.

If this is not a bug, please advise on what steps must be taken to apply both.