Buffer overflow in sw renderer draw code

[Edward-san]

I guess you meant here.

[dpJudas]

Yep, that one matches my call stack a lot better. I'm able to consistently trigger it by moving around in the first room of E1M1 if I add the following asserts to vlinec1:

Code: Select all

	DWORD fracstep = dc_iscale;
	DWORD frac = dc_texturefrac;
	DWORD height = rw_pic->GetHeight();
	assert((frac >> vlinebits) < height);
	frac += dc_count * fracstep;
	assert((frac >> vlinebits) < height);

I tried adding some clamping to the wallscan function ala this:

Code: Select all

dc_texturefrac = xs_ToFixed(fracbits, MAX(dc_texturemid + iscale * (y1ve[0] - CenterY + 0.5), 0.0));

This causes my second assert to fail instead of the first one (with a frac value of exactly height). It would seem that the height of the run is exactly one pixel too much in some condition I've not yet tracked down. I think the crash only shows itself if the texture height is a non power-of-two due to the vlinebits shifting.

[Edward-san]

Is it there a way to track down the involved texture?

[dpJudas]

According to my debugger the texture crashing for me right now is called DOOR3. It is 64x72 and it attempts to do a wallscan 'dc_count' run of 171 pixels.

If I understand the comments in the code right, wallscan is never supposed to do a run more than the height of the texture for non-power-of-two textures. I think that its the wallscan_np2 function that somehow fails to break up the run. That's the function I'm looking at right now.

[dpJudas]

Scratch that part about the 171 pixels. I forgot that its doing a stretch of the source texture. Either way, wallscan_np2 is probably the offending function.

[dpJudas]

OK, further looking at this the problem seem be related to loss of precision when doing floating point math.

The 'bot' argument to wallscan_np2 is -0.0 and the partition value calculated becomes 0.0. The while (partition > bot) therefore runs one iteration too many. Probably will have to go through all the wallscan functions with a fine-comb to make sure all float to fixed conversions are correctly clamped and rounded.

[Edward-san]

... these posts should go to the right bug report. Any moderator available?

[wildweasel]

... these posts should go to the right bug report. Any moderator available?

A minor bout of thread-surgery later, I think I got it. I hope I didn't miss anything in the process.

[Edward-san]

It see nothing out of ordinary, thanks.

[dpJudas]

I believe this one is fixed by the new wallscan function.

[Edward-san]

I can confirm everything looks okay now.