How to optimize more than one loss functions?

[bssrdf]

Hi, I am trying to implement a VAE by following this example. I build a compute graph mimicking theirs (see below). You can see there are two loss functions, klloss and sigmoidcrossentropyloss (please ignore l2loss). How should I optimize using both loss functions? I can see two options:

• add them together to create a final loss and optimize over it (this does not make much sense as klloss should not depend on decoders)

  Turned out this is the right way to optimize (i.e., use the total loss).

• optimize two functions independently. Will the following be correct steps?

  ggml_opt(ctx0, opt_params, klloss);
  
  ggml_opt(ctx0, opt_params, sigmoidcrossentropyloss);
  

Any thoughts? Thanks.