guadalupe LSS catalogs

[ashleyjross]

The LSS catalogs here desi/survey/catalogs/edav1/da02/ (organized like sv3) should get copied to the public directory, mirroring sv3.

While I thought we planned for this during the EDR, perhaps not, because the only way I see to make it work with the sv3 data model is to make the version 'guadalupe'. This would put everything in

/global/cfs/cdirs/desi/public/dr1/vac/dr1/lss/guadalupe/

I don't think it works, but an alternative would be

/global/cfs/cdirs/desi/public/dr1/vac/m2/lss/v1.0/

(In the early BAO paper, the sample is referred to as DESI-M2, for 2 months of main survey.)

A couple of minor notes would be needed in the EDR data model, e.g., the alt MTL directory existing only for SV3. I think we should also add a simple text readme within each directory making it clear that these are not the full DR1 LSS catalogs.

[weaverba137]

When you say "I don't think it works", could you please specifically say why not? There are other VACs that do follow this pattern.

A couple of minor notes would be needed in the EDR data model, e.g., the alt MTL directory existing only for SV3.

OK. Is there a PR for this already?

I think we should also add a simple text readme within each directory

Please do this before any data are moved into place. The final data directory will be unwriteable.

[ashleyjross]

Oh, if there are others that follow that pattern and we can use /global/cfs/cdirs/desi/public/dr1/vac/m2/lss/v1.0/, I think it is better. (I find them on my quick look.)

[weaverba137]

/global/cfs/cdirs/desi/public/dr1/vac/m2/lss/v1.0/

No, it would be /global/cfs/cdirs/desi/public/dr1/vac/dr1/lss/guadalupe/v1.0/.

[weaverba137]

When you said "I don't think it works" I assumed you were referring to

/global/cfs/cdirs/desi/public/dr1/vac/dr1/lss/guadalupe/.

[ashleyjross]

Ah, yeah, the problem is that is mismatched with the LSS EDR data model: DESI_ROOT/vac/RELEASE/lss/VERSION/ (https://desidatamodel.readthedocs.io/en/latest/DESI_ROOT/vac/RELEASE/lss/VERSION/index.html) and we want to re-use that (since the catalogs were produced at the same time)

[ashleyjross]

So, I'm assuming /global/cfs/cdirs/desi/public/dr1/vac/dr1/lss/guadalupe (with guadalupe as the version and so no v1.0) is the best we can do. If so, I'm ok with it...

[weaverba137]

I think we're writing past each other just now, so I'm going to wait for a bit.

[ashleyjross]

The issue is, I think, we cannot use

/global/cfs/cdirs/desi/public/dr1/vac/dr1/lss/guadalupe/v1.0/

with the vac data model, because it is DESI_ROOT/vac/RELEASE/lss/VERSION/

(so VERSION would have to be guadalupe/v1.0 , which I don't think is allowed. If it is, then, great, we can actually use it.)

If that is correct, the best option I see is to use

/global/cfs/cdirs/desi/public/dr1/vac/dr1/lss/guadalupe/

and then add a note on the data model page that v1.0 is the only version for EDR, and guadalupe is the only version for DR1 for this data model. Then, we would add a sentence linking to the full DR1 LSS catalog data model.

But, someone might have a better idea!

[weaverba137]

You can absolutely describe VERSION as guadalupe/v1.0.

[ashleyjross]

Ok, that is obviously the best solution then!

[aureliocarnero]

Checking the guadalupe/v1.0 data. There is one directory in clustering named recon. Since this is not documented in the current EDR datamodel, I'm going to remove it from the public repo