Learned attenuation via direct modelling

[Whichislove]

Hi there,

I am planning to implement direct modelling to predict input dependent observational noise (Aleatoric Uncertainty) together with network's regular outputs. My naive approach was to add the last layer for each regression degree of freedom and then use pretrained weights to train the covariance layer only. However, I am not satisfied with the obtained results. I would like to ask whethere there are people who have implemented direct modelling and can share their experience regarding the training procedure. I am interested in learning how deep from the last layer should I retrain the model in order to obtain good results.