Shifts in energy prediction for different temperature testset.

[shubhamdongriyal]

Dear MACE developers,

I am trying to train a MACE potential for a system of molecular crystal.
The training dataset only contain structure at 295 K with a fixed lattice vectors.
I tried to evaluate/predict the above trained potential on a test set containing lower temperature structures (80K, 150K, 220K, 295K) with different lattice vectors.

The Force predictions are in perfect agreements. However, there is a constant shift on energy prediction (please find the plots below)
The structures which have perfect agreement in energy prediction plot are from 295 K (same as in training set).

I tried following things, but got the same results.

1. Using stress to train the potential.
2. Adding isolated atoms in training set.
3. Changing various hyperparameters like loss fn, num_channels and so on.

Submission script for training the potential.

srun -n 1 -c 16 mace_run_train \
    --name="250K_potential" \
    --train_file="train.xyz" \
    --valid_fraction=0.05 \
    --E0s="average" \
    --model="MACE" \
    --num_interactions=2 \
    --num_channels=64 \
    --max_L=0 \
    --correlation=3 \
    --r_max=6.0 \
    --compute_stress="True" \
    --forces_weight=1000 \
    --energy_weight=10 \
    --stress_weight=10 \
    --energy_key="REF_energy" \
    --forces_key="REF_forces" \
    --stress_key="REF_stress" \
    --batch_size=4 \
    --valid_batch_size=6 \
    --max_num_epochs=200 \
    --start_swa=150 \
    --scheduler_patience=15 \
    --patience=30 \
    --eval_interval=4 \
    --ema \
    --swa \
    --loss="stress" \
    --error_table='PerAtomMAE' \
    --default_dtype="float32"\
    --device=cuda \
    --seed=123 \
    --restart_latest \
    --save_cpu \

Could someone help me understand, what is the possible reason behind the constant shift in energies for different temperature in testset ?
I will be happy to take any suggestions.

Thank you.

[gabor1]

ML potentials typically are poor at extrapolating in density. So I think what happened is that because your training set is all one density, some interaction energy wasn't partitioned correctly into the various energy terms, because there was no information on how things change when the density changes. So I would expect that if you train on configs that have just a couple of different lattice vectors, then your performance at new lattice vectors will improve.

[gabor1]

Another thing is that I would always train and test using float64, at least until things are converged and all working. then you can experiment with evaluating using float32 to see how much (if any) accuracy you lose in single point (probably little), geometry optimisation and md.

[gabor1]

your forces_weight is very high. we usually start with an E:F:S ratio of 1:10:1 (or even 1:1:1 if you are brave)

[gabor1]

And I would definitely use computed E0s, not "average", because you have a single composition in your data set, so in fact the linear regression to determine E0s automatically is undefined

[gabor1]

(adding the isolated atoms to the training set accomplishes setting the E0s correctly)

[shubhamdongriyal]

ML potentials typically are poor at extrapolating in density. So I think what happened is that because your training set is all one density, some interaction energy wasn't partitioned correctly into the various energy terms, because there was no information on how things change when the density changes. So I would expect that if you train on configs that have just a couple of different lattice vectors, then your performance at new lattice vectors will improve.

Thank you very much Prof @gabor1 for all your suggestions.

To check, I added 20 structures of different densities on top of the existing training set and indeed it fixes the energy curve. We know this can indeed fix it.

However, the original data was generated using VASP-ML active learning protocol and the prediction by VASP-ML on the same above testset (which has diverse density) predicts the energy quite well, even when trained on a single density. Training on the stresses there, though, was essential to get it right, otherwise we got predictions very similar to MACE. So we wonder whether it could also be fixed in MACE.

[gabor1]

It is possible that you could get more out of stress data. have you tried increasing the stress weight? Also, make sure you use the very latest code from the main branch we fixed a bug in the stress weight a month or so ago which meant that before, the stress loss was divided by the number of atoms when it should not have been, effectively making the stress count for very little (unless you used a huge weight). If you are not using the latest code, update and retrain using the above parameters. if you are using the latest code already, try stress weight of 100.

[shubhamdongriyal]

Thank @gabor1. I have the latest mace version installed. I will try with stress weight of 100.

[gabor1]

and also reduce your forces weight to 100 so that it does not overwhelm the loss.