Error estimation in model.expect_data()

[BlaiseDelaney]

Dear developers,

I wondered if there were a way to return the uncertainties through pyhf when calling model.expected_data(). Supposing I have a complex model with additive and multiplicative modifiers (including normsys and staterror) and the full set of parameters obtained from the likelihood minimisation, how do I propagate the per-parameter uncertainty to the overall model?

Ideally, I would like to understand is whether there exists a method model.expected_data_ufloat() which can accept both the central value and the uncertainty of each parameter and returns two tensors. The first is what is presently available through model.expected_data() and the second would be the tensor of corresponding uncertainties, obtained through proper error propagation in pyhf.

I have attempted to do this by hand, but I am quite unsure on how to deal with the less trivial modifiers, such as normsys.

Any help would be much appreciated.

Thanks!

Blaise

[alexander-held]

Hi, there are two ways of doing this: bootstrapping and error propagation. pyhf itself does not include a method for this at the moment.

For bootstrapping, see #1187 (comment) and #1359 (comment) for an example. You can sample from a multivariate Gaussian and evaluate the model prediction for all samples. In principle you could sample from the full likelihood directly to avoid the Gaussian approximation as well.

Error propagation can for example be done via iminuit.util.propagate as described in this tutorial. Another implementation is provided via the cabinetry library in cabinetry.model_utils.calculate_stdev. See this example notebook showing the use of cabinetry, and in particular the cabinetry.visualize.data_MC call that creates pre- and post-fit model visualization and yield tables with related uncertainties. For the error propagation it is important to also have the parameter correlations available, as correlations typically significantly change the post-fit uncertainty.

[BlaiseDelaney]

Thanks @alexander-held! I'll take some time to review your suggestions and understand them. This was very helpful.