Learnable beta for Softplus inverse link

[wulfdewolf]

This might be a pretty niche functionality request, but perhaps some other people want it as well.
There's already functionality for having a Softplus inverse link function.
I wonder if it would be hard to make the beta parameter learnt, just like the GLM weights.
It would change the parameter structure though...

[BalzaniEdoardo]

Hi!

This is in general dangerous territory because having a non-linearity that is parameterized that way would make the learning objective non-convex in general, with interactions between the coefficients and the parameterization that could lead to identifiability issues.

So, in general, it is non-trivial to the point that such a parametric model should not be considered a GLM anymore. If I were to implement it I would probably have a dedicated model class.

For your case, I was trying to see if there were identifiability issues due to the multiplicative interaction of the soft plus parameter with the weights. I ended up with an objective only in the weights but I am not sure it is still convex. I'll post here my calculations, to have some records of it in case I'll end up implementing this model.

$$ \text{softplus}_\beta(x) = \frac{1}{\beta}log \big(1+\exp(\beta x) \big), $$

For $\beta > 0 $. With a Poisson log-likelihood it would take the form

$$ \begin{aligned} \mathcal L (\beta, \mathbf{w}) &= \sum_t y_t \log\left(\text{softplus}_1(\beta \mathbf{w} \cdot X_t)\right) - \text{softplus}_1(\beta \mathbf{w} \cdot X_t)/\beta - y_t \log \beta + const \\ &= f(\beta \mathbf{w}) + \frac{g(\beta \mathbf{w})}{\beta} - N \log\beta + const \end{aligned} $$

With

$$ \begin{aligned} g(\mathbf{w}) &= -\sum_t \text{softplus}_1(\beta \mathbf{w} \cdot X_t) < 0 \\ N &= \sum_t y_t. \end{aligned} $$

Now,

$$\max_{\beta, \mathbf{w}} f(\beta \mathbf{w}) + \frac{g(\beta \mathbf{w})}{\beta} - N \log\beta = \max_{\beta, \mathbf{w}} f(\mathbf{w}) + \frac{g(\mathbf{w})}{\beta} - N \log\beta,$$

because you can absorb $\beta$ into the weights.

If you calculate the partial derivative of $f(\mathbf{w}) + \frac{g(\mathbf{w})}{\beta} - N \log\beta$ in $\beta$, the maximum is obtained in $\hat{\beta}= - \frac{g(\mathbf{w})}{N}$.

So, if you plug that in the previous equation, you get a likelihood purely in $\mathbf{w}$ which can be optimized but I am not sure it is still convex.

Any pair of $(w, \beta)$ that maximizes this new objective can be mapped in one that maximizes the original problem. So if one has a family of solution, so will the other.

Let me know if this makes sense to you!

Cheers,
Edoardo

[BalzaniEdoardo]

Within the framework, you can still define a grid of inverse link functions, one for each beta parameter you want to test, and use cross validation to select the non linearity!

[wulfdewolf]

Hi Edoardo, yes all makes sense!

A grid of inverse link functions is indeed a very usable option to do this, which I'll most likely try.

Having a learnt beta parameter has helped in larger deep learning models, where many neurons are fitted simultaneously.
For example, the winner of the Sensorium 2023 competition managed to get a sizeable increase with this small change alone:
https://openreview.net/forum?id=gViJjwRUlM#discussion

[BalzaniEdoardo]

Interesting! I wanted to check out the paper but I don't have permission to access that link.

Error 403 "You don't have permission to read this forum"

Would you mind pointing to the relative paper?

Thanks again for the insights

[wulfdewolf]

Ah apologies, that one is not public yet. Here is the preprint: https://arxiv.org/abs/2407.09100

[wulfdewolf]

and here is the codebase for the winner, the learnable softplus is explained there:
https://github.com/lRomul/sensorium

[BalzaniEdoardo]

Thanks! Il giorno mar 24 set 2024 alle ore 12:01 Wolf De Wulf < ***@***.***> ha scritto:

…

and here is the codebase for the winner, the learnable softplus is explained there: https://github.com/lRomul/sensorium — Reply to this email directly, view it on GitHub <#217 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG2MOS7Y54KMYQCDDLZ5LPTZYGEGXAVCNFSM6AAAAABNQ3HE6GVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANZUGEZTCMA> . You are receiving this because you commented.Message ID: ***@***.*** com>