Detach new_covar_cache to enable JIT tracing of models after fantasization

[SaiAakash]

This PR deals with #2604 . It's a one-liner where the new_covar_cache is detached so that models which are conditioned on new data/observations using get_fantasy_model can now be JIT traced/exported to TorchScript.

[gpleiss]

If we always detach the root then we can't backpropagate through to inputs (necessary for BO applications).

We should only detach the root when detach_test_caches context manager is not on.

[SaiAakash]

@gpleiss In all other places in GPyTorch, tensors are detached only when the detach_test_caches setting is on. I am wondering why in this case we have to detach when this setting is not on ?

Should it be something like,

if detach_test_caches.on():
        new_covar_cache = new_lt.root_inv_decomposition().root.detach()
else:
        new_covar_cache = new_lt.root_inv_decomposition().root

[jacobrgardner]

@gpleiss In all other places in GPyTorch, tensors are detached only when the detach_test_caches setting is on. I am wondering why in this case we have to detach when this setting is not on ?

Should it be something like,

if detach_test_caches.on():
        new_covar_cache = new_lt.root_inv_decomposition().root.detach()
else:
        new_covar_cache = new_lt.root_inv_decomposition().root

@SaiAakash Yes, exactly -- we should only detach when the setting is on.

@gpleiss does trace_mode imply detach_test_caches? If no, it probably should I guess?

[SaiAakash]

@jacobrgardner I have made the requested change. The cache is now detached only when the detach_test_caches setting is on.