| commit | df315455b4ee7ffb89806a0c9d58a0ecddb5938f | |
| author | Aleksey Kliger (λgeek) <alklig@microsoft.com> | |
| Wed, 23 Jun 2021 18:45:12 +0000 (23 14:45 -0400) | ||
| committer | GitHub <noreply@github.com> | |
| Wed, 23 Jun 2021 18:45:12 +0000 (23 14:45 -0400) | ||
| tree | def6f75939b6ca38648b5139c7768b9ac0b355b7 | treesnapshot (tar.gz zip) |
| parent | c47db4f43fa8b849063d27c97a3a9f8839131454 | commitdiff |
Start a dedicated thread for MERP crash reporting (#21096)
Related to https://github.com/mono/mono/issues/21009
There are two scenarios:
1. When someone force quits Mono (or just runs `kill -TERM <pid>`), the process can receive the signal on any thread,
2. or - if a thread in the process crashes, but the thread is not attached to the runtime, Mono's signal handlers still run.
The crash reporter assumes that the crashing thread is either attached to the runtime, or at least `mono_thread_info_current` or the JIT TLS data are set for the thread. If the thread is truly foreign and it never interacted with Mono, and it crashes, both of those assumptions are false, but Mono's crash reporter signal handlers still run.
The solution from this PR is: if crash reporting is enabled, start a dedicated thread at process startup that is a "crash report leader" - when a crash happens, the crashing thread (the crash originator) wakes the leader, and the leader collects the crash report. The crash originator does not do any work that requires being attached to the runtime or to the JIT such as iterating over thread IDs or stack walking.
---
* Add a standalone test of crash on a foreign thread
* extra async printfs
* Sketch out crash leader implementation
At process startup, start a separate thread that is attached to the runtime and can collect crash reports. Crashing threads will wake it and wait for it to collect the crash reports
* Start moving responsibilities to the crash leader
* Watch for nil jit_tls data if crash is on a foreign thread
* Start adding a state machine for the crash leader
We need to coordinate the originator and the leader in a few places.
The leader needs to pause to after collecting the thread ids before suspending the non-originator threads, and again while the originator is dumping its own stack.
The originator needs to wait for the leader to collect the thread IDs and to tell it its assigned slot. Then it tells the leader to suspend the others, dumps its own memory, then tell the leader to dump the whole crash report and wait for it to reply when it's done.
* Move remaining summarizer work to the summarizer leader thread
* change some debug printfs
* Successfully summarize a foreign thread crash
* Add test to merp-crash-test.cs
* delete standalone test
* if the crash leader is the originator, collect the report synchronously
either because the crash leader crashed, or because the process got a SIGTERM and it arrived on the crash leader thread
* fixup comments
* turn off crash leader printf debugging
* Don't create leader thread if crash reporting is disabled
* Fix test on win32
* Update mono/tests/libtest.c
* fixup logging and comments around managed stack collection
* Disable the merp dlopen crasher
* Remove crash leader state machine
It's straightline code with two early exits. State machine is overkill
* remove remnants of leader state machine
Related to https://github.com/mono/mono/issues/21009
There are two scenarios:
1. When someone force quits Mono (or just runs `kill -TERM <pid>`), the process can receive the signal on any thread,
2. or - if a thread in the process crashes, but the thread is not attached to the runtime, Mono's signal handlers still run.
The crash reporter assumes that the crashing thread is either attached to the runtime, or at least `mono_thread_info_current` or the JIT TLS data are set for the thread. If the thread is truly foreign and it never interacted with Mono, and it crashes, both of those assumptions are false, but Mono's crash reporter signal handlers still run.
The solution from this PR is: if crash reporting is enabled, start a dedicated thread at process startup that is a "crash report leader" - when a crash happens, the crashing thread (the crash originator) wakes the leader, and the leader collects the crash report. The crash originator does not do any work that requires being attached to the runtime or to the JIT such as iterating over thread IDs or stack walking.
---
* Add a standalone test of crash on a foreign thread
* extra async printfs
* Sketch out crash leader implementation
At process startup, start a separate thread that is attached to the runtime and can collect crash reports. Crashing threads will wake it and wait for it to collect the crash reports
* Start moving responsibilities to the crash leader
* Watch for nil jit_tls data if crash is on a foreign thread
* Start adding a state machine for the crash leader
We need to coordinate the originator and the leader in a few places.
The leader needs to pause to after collecting the thread ids before suspending the non-originator threads, and again while the originator is dumping its own stack.
The originator needs to wait for the leader to collect the thread IDs and to tell it its assigned slot. Then it tells the leader to suspend the others, dumps its own memory, then tell the leader to dump the whole crash report and wait for it to reply when it's done.
* Move remaining summarizer work to the summarizer leader thread
* change some debug printfs
* Successfully summarize a foreign thread crash
* Add test to merp-crash-test.cs
* delete standalone test
* if the crash leader is the originator, collect the report synchronously
either because the crash leader crashed, or because the process got a SIGTERM and it arrived on the crash leader thread
* fixup comments
* turn off crash leader printf debugging
* Don't create leader thread if crash reporting is disabled
* Fix test on win32
* Update mono/tests/libtest.c
* fixup logging and comments around managed stack collection
* Disable the merp dlopen crasher
* Remove crash leader state machine
It's straightline code with two early exits. State machine is overkill
* remove remnants of leader state machine