Add running jobs in Planetary Computer

[jacobbieker]

We would like to run some of our jobs with public data in Planetary Computer, to help with the bandwidth issues with have on Leonardo.

Detailed Description

I've used the kbatch service to submit jobs to Planetary Computer quite successfully, command line args can be passed, as well as code locations. This should allow us to do whatever we want in the VM.

Each VM seems to have: 64GB RAM, 150ish GB storage
We can pass in credentials, including to GCP, to write directly there if need be.

Sometimes, processing seems to fail for an unknown reason, for example it processes HILAM surface NetCDFs fine, but not the pressure ones. But overall, that should just be some debugging and then it will be good to go.

Context

It offloads processing to the cloud, freeing up bandwidth on Leonardo for batch creation or getting of data we can't make public.

Possible Implementation

I have a lot of examples in my planetary-datasets repo

[jacobbieker]

Docs for kbatch: https://kbatch.readthedocs.io/en/latest/examples/index.html
Planetary Computer: https://planetarycomputer.microsoft.com/

[peterdudfield]

Could you point to one specific file where this was done for the Satellite date? Perhaps that is to come?

[jacobbieker]

Sure, here is a config file:
https://github.com/jacobbieker/planetary-datasets/blob/main/pc/eumetsat-0deg.yaml

And to run it, after updating the HF token, EUMETSAT API key and secret:

kbatch job submit -f pc/eumetsat-0deg.yaml

[peterdudfield]

Nice thanks @jacobbieker