prior and liklihood model transforms

[cscherrer]

Say you have a Soss model like

m = @model X,numGroups begin
    numFeatures = size(X,2)
    β ~ Normal() |> iid(numFeatures, numGroups)
    p = softmax.(eachrow(X*β))
    y ~ For(eachindex(p)) do j
            Categorical(p[j])
        end
end

What would you expect prior(m, :y) and likelihood(m, :y) to look like?

We haven't implemented likelihood yet, but for prior we currently get

julia> Soss.prior(m, :y)
@model (X, numGroups) begin
        numFeatures = size(X, 2)
        β ~ Normal() |> iid(numFeatures, numGroups)
        p = softmax.(eachrow(X * β))
    end

Intuitively, I think I'd want this to only include dependencies of β (so no p). It also feels wrong for the prior to include X. But the only way I see to get rid of it is to return a function "on the side" that's something like

deps(nt) = (numFeatures = size(nt.X, 2), )

The I'm trying to be so careful here is that we need to be able to do this sort of thing programmatically, e.g. only "knowing" that a given model turns an X into a y, so we can easily swap it out for another one.

Any thoughts?

@millerjoey @DilumAluthge

[DilumAluthge]

How about storing the numFeatures as part of the model itself, instead of reading it from X?

So, if the full model is:

m = @model X,numFeatures,numGroups begin
    @assert numFeatures == size(X,2)
    β ~ Normal() |> iid(numFeatures, numGroups)
    p = softmax.(eachrow(X*β))
    y ~ For(eachindex(p)) do j
            Categorical(p[j])
        end
end

Then I would expect the prior to be:

@model numFeatures,numGroups begin
    β ~ Normal() |> iid(numFeatures, numGroups)
end

And the likelihood will be:

@model X,numFeatures,numGroups begin
    @assert numFeatures == size(X,2)
    p = softmax.(eachrow(X*β))
    y ~ For(eachindex(p)) do j
            Categorical(p[j])
        end
end

It definitely would seem weird if the prior included X or p.

[cscherrer]

Models written like this are certainly easier to deal with. We can always put more responsibility on the users if we don't see a nice way for Soss to manage this sort of thing. So it's a sensible fallback.

Ideally, I'd love to have it so users don't have to worry about this, and Soss will "do the right thing".

[DilumAluthge]

Hmmm. If we don't have the user specify the number of features, then unfortunately the prior will depend on X, right?

[DilumAluthge]

Or, at least the prior will depend on size(X, 2).

[millerjoey]

Intuitively, I think I'd want this to only include dependencies of β (so no p). It also feels wrong for the prior to include X.

Are we happy with the behavior of before()? Maybe we need a better language to distinguish assignments from stochastic statements to relate it better to the more loaded terms like prior and likelihood.

prior(m::Model, xs...) = before(m, xs..., inclusive = true, strict = true, other_distinguishing_keywords...)

Also does

julia> Soss.before(m, :β, inclusive = true, strict = true)
@model (X, numGroups) begin
        numFeatures = size(X, 2)
        β ~ Normal() |> iid(numFeatures, numGroups)
    end

miss the point? Defaulting prior so it includes the named argument (like how we've discussed you prefer). This gets around the issue of Soss inferring how far "before" to go.

Can you elaborate on how we might do this:

The I'm trying to be so careful here is that we need to be able to do this sort of thing programmatically, e.g. only "knowing" that a given model turns an X into a y, so we can easily swap it out for another one.

It seems interesting but I'm not sure I'm visualizing it right.