Goal of abrt retrace server compared to faf server

[djuarezg]

Maybe I could not fully understand but what is the goal of a retrace server compared to a faf server? A retrace server is intended for full crash dumps and faf is only for mini reports?

[ernestask]

Per the description: retrace-server is only for analyzing core dumps. Micro-reports, sent to FAF, include stack traces, which can be obtained either locally (possibly requiring you to download packages, providing debug symbols) or remotely (e.g. by sending the dump to retrace-server).

[ernestask]

We can make the distinction explicit in some documentation somewhere.

[djuarezg]

So if I want to collect our infra crash reports (potentially thousands of nodes) for posterior analysis a FAF server is the best solution. Retrace server would be for avoid local analysis.

Correct me if I am wrong. Thank you for the clarifications, it really helped!

[ernestask]

So, if you are sending reports from your software directly to FAF (say, by using something like https://github.com/abrt/sABRTooth-python/) as opposed to having a full-blown ABRT installation, retrace-server simply adds no value.

If you do have a full-blown ABRT installation on one of your nodes, then the need for retrace-server depends on a couple of things:

1. You are collecting crash reports from some compiled binary
2. The binaries are built without debugging symbols and they are installable as a package from some repository

This report demonstrates the lack of debugging symbols for the crashed binary: https://retrace.fedoraproject.org/faf/reports/2881701/. You may or may not be able to use the offsets for debugging.

[djuarezg]

So with just a FAF server I cannot do the retrace? I have seen retrace related commands but not sure if they are intended for the same as the retrace server. My goal is to enable by default on thousands of nodes the autoreporting to help our team debug and report issues.

Let me ask it some other way, could I upload all reports to my FAF server and let it do the retracing, supposing that it has access to the rpms and the corresponding debug rpms?

[ernestask]

Oh, you’re right, there is code in FAF to perform retracing as well. I honestly have no idea why there exist two parallel implementations of basically the same thing.

I also cannot tell you how well the retracing in FAF works, but, in theory, it does so in the way you describe. We do have periodic runs of that action in Fedora, though, so it does something for sure.

[ernestask]

I’ll try to help you out with your troubles in getting the FAF container image to work on Monday and we can go from there.

This could be a good starting point to discuss the merits of making retracing more prominent in FAF and putting this to rest.

[djuarezg]

I could start running and sending reports to it. First running on podman as per https://hub.docker.com/r/abrt/faf-image/. I am aware of the possibility of deploying on Openshift and will do on our infra for our prod deployment.

I am now trying to fully understand how retrace works on FAF and how to make it work (i.e. making required packages available on the server for retrace to work), as well as checking how the report-Bugzilla linking works (whether this can be done automagically if the problem already exists upstream). Docs are useful but since we (CERN) are dealing with CentOS it requires some extra configuration compared to plain Fedora.