Crash Catcher improvements

Post a reply

Smilies
:D :) :( :o :shock: :? 8-) :lol: :x :P :oops: :cry: :evil: :twisted: :roll: :wink: :geek: :ugeek: :!: :?: :idea: :arrow: :| :mrgreen: :3: :wub: >:( :blergh:
View more smilies
BBCode is OFF
Smilies are ON
Topic review
   

Expand view Topic review: Crash Catcher improvements

Re: Crash Catcher improvements

by Edward-san » Sun Jan 02, 2011 6:15 am

nothing changed about the execution, the log file shows like this:
Spoiler:
[edit]
Chris wrote:Note that this thread is about further improvements to the crash catcher and related. Any dissemination of actual bugs should go into the Bugs forum, not a closed feature suggestion thread. I would like to know what others think about the changes, though (the crash log is largely the same, it just goes about making it differently).
I was referring in general, I mean I'd like to know if he wants something more from these crash reports.

Re: Crash Catcher improvements

by Chris » Sun Jan 02, 2011 5:11 am

Well, I realized I'm doing something very wrong. In a signal handler, you're only allowed to use certain system functions due to the asynchronous nature of the event. It's been using functions that aren't guaranteed safe (fopen, (f)printf, etc). So I had to rethink how to work it, and I think I got it. It sets up a pipe and makes the forked process set the read end as its stdin, and the signal handler can write any relevant info which the debugging process can then read and use to do the work.

It's not completely cleared, though. DoomSpecificInfo, which must be called by the signal handler since it handles the game state, calls snprintf which isn't marked as safe. Fixing that may need more invasive changes, if I'm to keep the crash catcher modular. Considering it's not any worse off than before, though, I think this is still at least a positive step.

How well does this patch work?
Attachments
cc-patch2.zip
(4.46 KiB) Downloaded 33 times

Re: Crash Catcher improvements

by Chris » Sat Jan 01, 2011 8:42 pm

Note that this thread is about further improvements to the crash catcher and related. Any dissemination of actual bugs should go into the Bugs forum, not a closed feature suggestion thread. I would like to know what others think about the changes, though (the crash log is largely the same, it just goes about making it differently).

Re: Crash Catcher improvements

by Edward-san » Sat Jan 01, 2011 8:31 pm

Okay, now I'd like to hear what Graf thinks about the output of the last crash log to know if it helps him somehow.

Re: Crash Catcher improvements

by Chris » Sat Jan 01, 2011 7:03 am

As I said, a security measure. It likely limits the amount of memory that can be locked, so the system can't be DoS'd by having a swarm of processes all lock some (making all other processes run directly from swap and bring the system to a crawl).

Re: Crash Catcher improvements

by Edward-san » Sat Jan 01, 2011 6:42 am

Chris wrote:
[edit2]Now that we're talking about linux improvements, when I use fluidsynth, before playing music i have those messages in terminal:

Code: Select all

fluidsynth: warning: Failed to pin the sample data to RAM; swapping is possible.
fluidsynth: warning: Failed to pin the sample data to RAM; swapping is possible.
and it freezes a bit between the two messages... why?
Don't know about the short freeze, but that message sounds like it's trying to mlock some memory and getting an error (possibly due to some more Ubuntu security measures). It just means that it allocates memory for use in time-sensitve code that can potentially get pushed to swap, instead of staying put in physical RAM. Though since ZDoom uses FluidSynth in a threaded, non-real-time manner, any performance hit by it getting swapped should be minimal... that is, unless it keeps getting swapped, but then you'd have bigger issues to worry about (means you're running out of RAM).
The funny thing is that I have 4 GB of RAM...

Re: Crash Catcher improvements

by Chris » Sat Jan 01, 2011 6:29 am

Edward-san wrote:
Chris wrote:How well does this patch work?
I've installed the gxmessage, but it seemed strange before, because it worked with root access (it had a bad graphic, also)...
It was funny that button 'Damn it' instead of 'Okay', but whatever.
I can change the button name back, though it won't work for kdialog/kde messages (kdialog doesn't seem to have an option to specify different button texts). IIRC, I originally set it to that because that's something the Windows crash dialog had (before it was able to make its own full crash reports like it does these days), but it doesn't seem to swear anymore.
Okay, now it works, without the root user access i have this:
Looks good/correct.
It can seem strange, but can you insert a /n at the end of the 'generating %s ...' text in the printf, So that the 'Killed' and other messages can be displayed below?
I kinda like how the "Killed" appears on the same line as the "Generating ..." text. Possible error messages that are printed by ZDoom itself should have a newline before the actual text so it still gets on its own line. I can change it if others really care, though.
[edit2]Now that we're talking about linux improvements, when I use fluidsynth, before playing music i have those messages in terminal:

Code: Select all

fluidsynth: warning: Failed to pin the sample data to RAM; swapping is possible.
fluidsynth: warning: Failed to pin the sample data to RAM; swapping is possible.
and it freezes a bit between the two messages... why?
Don't know about the short freeze, but that message sounds like it's trying to mlock some memory and getting an error (possibly due to some more Ubuntu security measures). It just means that it allocates memory for use in time-sensitve code that can potentially get pushed to swap, instead of staying put in physical RAM. Though since ZDoom uses FluidSynth in a threaded, non-real-time manner, any performance hit by it getting swapped should be minimal... that is, unless it keeps getting swapped, but then you'd have bigger issues to worry about (means you're running out of RAM).

Re: Crash Catcher improvements

by Edward-san » Sat Jan 01, 2011 5:23 am

Chris wrote:How well does this patch work?
I've installed the gxmessage, but it seemed strange before, because it worked with root access (it had a bad graphic, also)...
It was funny that button 'Damn it' instead of 'Okay', but whatever.

Okay, now it works, without the root user access i have this:

Code: Select all

*** Fatal Error ***
Address not mapped to object (signal 11)
Address: 0x40

Generating zdoom-crash.log and killing process 3033, please wait... Killed
And also the gxmessage shows up with all the informations:
Spoiler:
It can seem strange, but can you insert a /n at the end of the 'generating %s ...' text in the printf, So that the 'Killed' and other messages can be displayed below?

[edit]I want to test it with other crashing situations, can someone point me to the problematic threads?
[edit2]Now that we're talking about linux improvements, when I use fluidsynth, before playing music i have those messages in terminal:

Code: Select all

fluidsynth: warning: Failed to pin the sample data to RAM; swapping is possible.
fluidsynth: warning: Failed to pin the sample data to RAM; swapping is possible.
and it freezes a bit between the two messages... why?

Re: Crash Catcher improvements

by Chris » Fri Dec 31, 2010 10:03 pm

Edward-san wrote:Applied patch:
when i run it as normal user, with or without -nosound, it doesn't work, and won't show that gxmessage:
Do you have gxmessage installed? I figured it'd be a normal part of Gnome/GTK. I really wish FDo specified a program to display a message box, like it has for xdg-open to open a file with the right handler.
what can I do about that ptrace problem? All those 'No symbol table info available.' things?
According to this, Ubuntu enables PTrace scoping by default, which prevents a process from "attaching" to another process that's not its child (and since the crashing process is a parent of GDB, it won't allow it). I should be able to allow the spawned process to attach with a Linux-specific call.

As for the "No symbol" thing, IIRC Ubuntu and the like have *-dbg packages which install debug-enabled libs. Not very familiar with how it works, though.
[edit2]if you didn't know this, to exit from freezing issue, do ctrl-alt-f1, then login as normal user, then do this command: "sudo /etc/init.d/gdm stop", then do "sudo /etc/init.d/gdm start", and you're back (you'll lose all the programs you run, so...) (it worked for me, I don't know if it works for you)
You can actually just "killall -9 zdoom" from there to clear it. Still annoying, though. X really needs a key combo to force unlock input.

How well does this patch work?
Attachments
cc-patch2.zip
(3.63 KiB) Downloaded 30 times

Re: Crash Catcher improvements

by Edward-san » Fri Dec 31, 2010 7:31 pm

Chris wrote:I meant using -nosound, but that was a problem here.. you didn't seem to have the freezing issue. How well does the above patch work for you?
Applied patch:
when i run it as normal user, with or without -nosound, it doesn't work, and won't show that gxmessage:

terminal:

Code: Select all

Generating zdoom-crash.log and killing process 0, please wait... Could not attach to process.  If your uid matches the uid of the target
process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try
again as the root user.  For more details, see /etc/sysctl.d/10-ptrace.conf
ptrace: Operation not permitted.
gdb-respfile-aJf7eh:9: Error in sourced command file:
The program has no registers now.
sh: gxmessage: not found
Killed
file zdoom-crash.log
Spoiler:
When I'm root user, it works fine, with or without -nosound, and the gxmessage has the button 'Okay' instead of 'Damn it' (lol):
terminal:

Code: Select all

Generating zdoom-crash.log and killing process 0, please wait... 
warning: not using untrusted file "/home/edward-san/.gdbinit"
Killed
zdoom-crash.log:
Spoiler:
what can I do about that ptrace problem? All those 'No symbol table info available.' things?

[edit]the freezing issue appears only with gdb under these conditions (i don't know any other):
1)mouse enabled
or
2)fullscreen enabled

[edit2]if you didn't know this, to exit from freezing issue, do ctrl-alt-f1, then login as normal user, then do this command: "sudo /etc/init.d/gdm stop", then do "sudo /etc/init.d/gdm start", and you're back (you'll lose all the programs you run, so...) (it worked for me, I don't know if it works for you)

Re: Crash Catcher improvements

by Chris » Fri Dec 31, 2010 5:30 pm

I meant using -nosound, but that was a problem here.. you didn't seem to have the freezing issue. How well does the above patch work for you?

Re: Crash Catcher improvements

by Edward-san » Fri Dec 31, 2010 6:53 am

Chris wrote:There's definitely something odd about that crash. For me, the spawned process is crashing before it gets a chance to kill ZDoom. Seems to be related to audio.. probably something with PulseAudio. But if I disable audio, it "crashes" properly and gives me a full backtrace
What do you mean 'disable audio'? Using -nosound parameter or setting to 'mute' in the ubuntu panel, it gives me the same situation as before:
Spoiler:

Re: Crash Catcher improvements

by Chris » Fri Dec 31, 2010 3:54 am

How well does this work? This makes it do the necessary work before the fork, and makes the child process execlp() into another invocation of ZDoom, though using special switches to dump the debug info of the "crashed" process and exit ASAP. I was able to get rid of the SDL_Quit call in DoomSpecificInfo (which I never liked but was necessary to properly release input), and changed to use checks similar to xdg-open to determine which dialog/messagebox app to call.

If it works, I'll do a final cleanup and make another code submission with a proper patch (this one was again made with git-diff).
Attachments
cc-patch2.zip
(3.31 KiB) Downloaded 33 times

Re: Crash Catcher improvements

by Chris » Thu Dec 30, 2010 8:41 pm

Edward-san wrote:There's something wrong:
look at this example: Doom Tower Defense (here is the topic)

to make zdoom (debug mode compiled with Ubuntu 10.10) crash, do this command:

Code: Select all

./zdoom -iwad doom2.wad -file dtd_2.0.pk3 -skill 1 +map DTD41
There's definitely something odd about that crash. For me, the spawned process is crashing before it gets a chance to kill ZDoom. Seems to be related to audio.. probably something with PulseAudio. But if I disable audio, it "crashes" properly and gives me a full backtrace (which, btw, is:
Spoiler:
so it seems like it's useless using ./zdoom to do debugging
We're kinda stuck with using ./zdoom to dump the crash info, though. Might be able to get away with invoking another program instead of working directly off of fork(), though that would need changes to ZDoom's main() to watch for a special switch to handle everything (the crash handler would, in essence, invoke another run of ZDoom, but with a special switch that doesn't run the game and instead prints the crash info to stdout).. but I'm not sure how I feel about that. It would be safer since the app can be in a pretty bad state by the time a crash happens, but it would still require doing some work in the forked process (to get some info on the current game state) and passing that to the new ZDoom process for it to print out.

Re: Crash Catcher improvements

by Demolisher » Thu Dec 30, 2010 6:48 pm

WELL FUCK, I tried to fix this for hours, this is not a bug on the mod's side.

Top