Plasma Crash Course - Sentry
A while ago a colleague of mine asked about our crash infrastructure in Plasma and whether I could give some overview on it. This seems very useful to others as well, I thought. Here I am, telling you all about it!
Our crash infrastructure is comprised of a number of different components.
- KCrash: a KDE Framework performing crash interception and prepartion for handover to…
- coredumpd: a systemd component performing process core collection and handover to…
- DrKonqi: a GUI for crashes sending data to…
- Sentry: a web service and UI for tracing and presenting crashes for developers
We already looked at KCrash, coredumpd, and DrKonqi. This week it’s time to look at the final piece of the puzzle — Sentry.
Sentry
Sentry is the service we have on the developer side of a crash to manage incoming reports.
As we’ve learned in the DrKonqi post, a Sentry-specific trace created by the preamble is eventually sent off to our Sentry server. In Sentry, yet more processing happens!
Symbolication
If the user hasn’t explicitly opted into it, there usually aren’t debug symbols available on the user system. This would ordinarily render backtraces useless since they are just a bunch of memory addresses without names or references to our source code.
One of the reasons we adopted Sentry is because it can
fill in the gaps through a process called symbolication. For symbolication it essentially needs to know all the loaded libraries and can then fetch the relevant debug symbols via the various debuginfod instances out there. Once it has the debug symbols it can resolve that the address 0x2343244 is really the function int main(int argc, char **argv)
Fingerprinting
When a trace has been symbolicated it’s ready for fingerprinting. This tries to merge together the same crash appearing in different submission into one single sentry event. Generally speaking if two submissions have the same trace frames, they are considered the same crash and merged into one event (in practice the rules are a bit more complicated).
Result
Eventually the crash event appears for us to fix.
In a future blog post, I’ll also give a more detailed guide on how to use Sentry to its full potential.
