Intake data source (public and free)

[espg]

Starting this discussion here so it has a bit more persistence than on slack.

To summarize some discussions about cloud accessible imagery... both @siegfried and @whyjay Zheng have pointed out that the intake catalog that I built for landsat 8 access will switch to 'requester pays' in July, which would make the demo and workflow that we are putting together for EarthCube quite short lived. Luckily it looks like the Landsat (and other) data on Google cloud will stay free, at least for the foreseeable future:

The Cloud Console and gsutil require you to authenticate with Google to access public data. You can authenticate with any Google account; the account does not have to be associated with the project that contains the public data, nor does it need to be signed up for the Cloud Storage service.
By contrast, accessing public data with a Cloud Storage API link does not require authentication. This method is suited for general-purpose links to publicly shared data. For example, API links can be used in a web page or downloaded with a command-line tool such as cURL.

The Google public datasets are pretty useful for our purposes; they include:

• Landsat 8 (Collection 1)
• Also Landsat 4, 5 and 7
• Sentinel 2
• NEXRAD

@tsnow03 is more familiar with the specifics of the Landsat collections, and filled me in on some details of the major collection versions (0, 1, 2). The current catalog that we have references collection 0, which isn't great. AWS also has collection 1, like Google, but the (free) access is going away in July. Collection 2 is the gold standard apparently, and AWS will have that with requester pays soon, while Google should remain free with access collection 1.

Intake yaml files can encode multiple collections and data sources. So, I plan to do the following:

• Keep collection 0 AWS, since it's already coded
• Add google Landsat 8, collection 1 for the EarthCube demo
• Add google Landsat 4, 5, and 7 temples (later)
• Add collection 2 AWS, requester pays (later)
• Add Sentinel 2 and NEXRAD

Let me know if I'm mistaken about Google Cloud free access and/or if there are other generally free cloud datasets that we should add in the future...