Random DLA along the line-of-sight

[londumas]

@fjaviersanchez, I write it there so we don't forget. When we have the time, we could work on improving the random DLA, spatially along the line-of-sight.
I think the proper way would be to:

• select some (ra,dec,z) at random that represent the host quasar and then
  for each of these set multiple random DLAs, as you do for mocked data DLA.
  The consequence is that we will have multiple random DLAs for each random line of sight.
• It would be nice also to have a ID for each random line of sight and another for each random DLA.

This is the current DLA auto-correlation. We can see that there is spurious correlation along the line-of-sight.

[londumas]

@jfarr03 or @fjaviersanchez, could you produce the random catalog of DLA for v5.0.0?

[jfarr03]

@fjaviersanchez I think you've done this previously but I'm happy to take it on in future - did you just use generate_rnd.py in the past? And with what settings?

[fjaviersanchez]

I used a revamped version specific for DLAs. I am sending it to you on slack but feel free to add it to the repo (I would say under the desi directory).

I also generated the random catalog here: /global/projecta/projectdirs/desi/mocks/lya_forest/london/v5.0.0/master_DLA_randoms.fits.gz.

I didn't include any ID but the ra and dec should be from a given line of sight with a DLA.

[londumas]

@fjaviersanchez and @jfarr03, thanks for the randoms. The issue is that we really need the equivalent of MOCKID, i.e., the ID of the line-of-sight. Because without it we can not remove pairs of DLA from the same line-of-sight. If it is possible I would need MOCKID and something like the Z_QSO of the quasar along the line-of-sight where you got you DLA from. if the latter is too complicated, let's start with only MOCKID.

[jfarr03]

That shouldn't be too difficult - I'll get on it now!

[andreufont]

Are you sure you know how to generate DLA randoms? I thought this was non-trivial.

[jfarr03]

Is it not possible to just pick random points on each skewer according to a certain number density n(z)?

[jfarr03]

Looking at the generate_rnd_dla.py code I'm not sure that's what is being done actually: it seems to read the actual DLA redshifts and then add noise to them, to ensure that the n(z) is the same as the actual DLAs. Is it not more straightforward and more reliable to just use the n(z) from pyigm?

[londumas]

Here is what the current randoms give for the auto-correlation of DLA.
It is nearly what we want except along the line-of-sight as expected by the lack of MOCKID in the random file. Very cool!

[jfarr03]

Excellent news! I'm generating a new set of randoms now, using the same method as the old ones but including MOCKID, Z_QSO_RSD and Z_QSO_NO_RSD. The current file is still in the same location but renamed with suffix "_noMOCKID".

[londumas]

@jfarr03, Thanks. Could you create the file directly in *.fits and not in *.fits.gz. I can't read it.
I think it is linked to the same bug I have seen in esheldon/fitsio#199.

running the following code:

import fitsio
h = fitsio.FITS('/global/projecta/projectdirs/desi/mocks/lya_forest/london/v5.0.0/master_DLA_randoms.fits.gz')
h[1].read_header()
h[1]['Z'][:]

I get the error:

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-4-e1d5652c4920> in <module>()
----> 1 h[1]['Z'][:]

/global/common/software/desi/cori/desiconda/20180709-1.2.6-spec/conda/lib/python3.6/site-packages/fitsio/fitslib.py in __getitem__(self, arg)
   3332         if isrows:
   3333             # rows was entered: read all current column subset
-> 3334             return self.read(rows=res)
   3335 
   3336         # columns was entered.  Return a subset objects

/global/common/software/desi/cori/desiconda/20180709-1.2.6-spec/conda/lib/python3.6/site-packages/fitsio/fitslib.py in read(self, **keys)
   3306 
   3307         if self.is_scalar:
-> 3308             data = self.fitshdu.read_column(self.columns, **keys)
   3309         else:
   3310             c=keys.get('columns',None)

/global/common/software/desi/cori/desiconda/20180709-1.2.6-spec/conda/lib/python3.6/site-packages/fitsio/fitslib.py in read_column(self, col, **keys)
   1895         """
   1896 
-> 1897         res = self.read_columns([col], **keys)
   1898         colname = res.dtype.names[0]
   1899         data = res[colname]

/global/common/software/desi/cori/desiconda/20180709-1.2.6-spec/conda/lib/python3.6/site-packages/fitsio/fitslib.py in read_columns(self, columns, **keys)
   2019             colnumsp = colnums[:].copy()
   2020             colnumsp[:] += 1
-> 2021             self._FITS.read_columns_as_rec(self._ext+1, colnumsp, array, rows)
   2022 
   2023             for i in xrange(colnums.size):

OSError: FITSIO status = 107: tried to move past end of file

[fjaviersanchez]

@jfarr03 I think that using the n(z) from pyigm is a good idea but I thought of using this since it is faster and doesn't require pyigm to be installed. @londumas that's very cool!! Thanks a lot.

[londumas]

Fixed with the latest DLA_random_catalog. Thanks @jfarr03 and @fjaviersanchez .