Nonlinearity in energy readouts

[svandenhaute]

Is the current default nonlinearity in the final energy readout layer critical? It feels somewhat arbitrary to make the energy readout in the first layer strictly linear, and the readout in the last layer almost linear. Is there perhaps some empirical evidence that suggests that switching to a strictly linear energy readout in all interaction layers makes the model perform systematically worse? Or is there some other logic behind it?

[gabor1]

The linearity in the layers before the last lead to a body ordered potential, and we think this is important for transferability. You can retain the linear readout in the last layer also, and in some early tests this came with a 10% degradation of performance. But comprehensive tests on potentials with a very diverse training set were not done, so I think retaining linear readouts all the way is a good thing to test, especially if you are not just testing on training/validation force accuracy but out of domain stability and inter-molecular interactions which are hard to capture and are not directly correlated with the loss.

[svandenhaute]

Interesting, thanks!

[ilyes319]

In the Design Space paper (https://arxiv.org/pdf/2205.06643.pdf), we have experiments on that in Section VIII. B, and a section related to the body-ordered expansion and the effect of nonlinearities on it. The fact that you get a well-structured body-ordered potential plus a nonlinear correction is very appealing to me. I think one could further justify it theoretically in terms of completeness.

[svandenhaute]

Ah, thanks! I somehow never fully dug into the details in the past.