why does the expected limit change when only the observed data changes?

[nfcastro]

I have a probably very naif question.

Take, for instance, the 2 bin example in https://raw.githubusercontent.com/scikit-hep/pyhf/master/docs/examples/json/2-bin_1-channel.json

By running it I get:

{
    "CLs_exp": [
        0.07807427911686156,
        0.17472571775474618,
        0.35998495263681285,
        0.6343568235898907,
        0.8809947004472013
    ],
    "CLs_obs": 0.3599845631401915
}

Now I change only data to { "name": "singlechannel", "data": [55.0, 65.0] }

I would expect the observed limit to change, but not the expected one. But, instead, I get

{
    "CLs_exp": [
        0.08376589347241643,
        0.1837607892960035,
        0.3716612674383095,
        0.6447539814150473,
        0.8859289249492854
    ],
    "CLs_obs": 0.5357344237884727
}

i.e. the expected median changed from 0.35998495263681285 to 0.3716612674383095.

What am I misunderstanding?

Thanks in advance for any feedback

[kratsg]

I think this is always a common confusion (I admit I trip up myself from time to time). Equation 5 from this document: https://twiki.cern.ch/twiki/pub/AtlasProtected/StatisticsTools/Frequentist_Limit_Recommendation.pdf -- shows that

which depends on the observed. The reason is that you're still performing a background-only fit to the data (to get the Asimov dataset) and this depends on the data you provide. If you change the data you fit to, you change how the background is pulled in the fit.

[nfcastro]

Thanks @kratsg, you are right

[matthewfeickert]

As a compliment to @kratsg's answer, for people who aren't on ATLAS or who want a public reference, this is also seen in Equation 76 from "Asymptotic formulae for likelihood-based tests of new physics".

[matthewfeickert]

For posterity, I've learned while discussing this more in Discussion #1619 that there is a public version of the February, 2011 ATLAS Frequentist Limit Recommendation document that @kratsg linked above (in case any non-ATLAS people want to explicitly read that).