Neaby source db

[tterasaki]

This makes a database which contains flags to indicate whether the obs_id hit radio point sources, including non-planet sources such as RCW38 and galactic center.

An example of config file looks like below:

context_file: '/so/metadata/satp3/contexts/use_this.yaml'
query_text: 'type == "obs"'
min_ctime: null #1700000000
max_ctime: null
#update_delay: 1
query_tags: null
output_dir: '/so/home/tterasaki/code_dev/make_nearby_sources_db/nearby_sources'
distance: 2.

source_list:
 - 'mercury'
 - 'venus'
 - 'mars'
 - 'jupiter'
 - 'saturn'
 - 'uranus'
 - 'neptune'
 - ['taua', 83.6272579, 22.02159891]
 - ['rcw38', 134.7805107, -47.50911231]
 - ['iras16183-4958', 245.5154, -50.09168292]
 - ['iras19078+0901', 287.5575891, 9.107188994]
 - ['rcw122', 260.0339538, -38.95673421]
 - ['cenA', 201.3625336, -43.00797508]
 - ['3c279', 194.0409868, -5.79174024]
 - ['3c273', 187.2775626, 2.053532671]
 - ['G025.36-00.14', 279.5264042, -6.793169326]
 - ['QSO_J2253+1608', 343.4952422, 16.14301323]
 - ['galactic_center', -93.5833, -29.0078]

[tterasaki]

@msilvafe That looks nice. But I want to keep this branch alive.

1. The preprocess takes relatively long as it contains tod processing stuff. That can be overhead when we get want to re-calculate the source tags.
2. The branch looks only calculate if center_on = 'planet', which can overlook point sources in cmb scans
3. I was planning to change the source tags from overall focal plane to each wafer. That is, now I am quering with galactic_center=1, but the new one is galactic_center.ws0=1, which enables us to remove un-hitting wafer. That should be useful.

[tterasaki]

Like this commit. Let me know your thoughts.

[msilvafe]

@msilvafe That looks nice. But I want to keep this branch alive.

1. The preprocess takes relatively long as it contains tod processing stuff. That can be overhead when we get want to re-calculate the source tags.
2. The branch looks only calculate if center_on = 'planet', which can overlook point sources in cmb scans
3. I was planning to change the source tags from overall focal plane to each wafer. That is, now I am quering with galactic_center=1, but the new one is galactic_center.ws0=1, which enables us to remove un-hitting wafer. That should be useful.

I think it's fine to keep this alive. Although I think we should distinguish between preprocess_obs and preprocess_tod in preprocess_obs no detector signal information is used so it is in fact a lot faster. Currently this is just being used by @davidvng to generate the SSO plots and write out the SSO path in xi-eta to a ManifestdB. The reason we brought in this source list and made it a hard coded list/parameters in planets instead of a configuration file or passed list was that when testing the SSO plotting David found that sometimes in an obs tagged moon there would be Jupiter or Uranus etc and so his code code make multiple SSO plots showing the wafer coverage for the different objects in the field. It seems like this is a natural extension to include the rest of these sources. For now I think its ok to have the two scripts running separate but I do think in the long run it makes more sense to combine these as both of them are just looking at the RA-DEC footprint, and finding sources in the field, and then David's code will also make the coverage plot and write out the source path over the focal plane to the ManifestdB and it seems like a simple extension to add columns to the database that just tag the sources and can be used for obsdb querying.

[tterasaki]

Thank you @msilvafe . I did not know preprocess_obs and preprocess_tod are different. That make sense. Let's combine them together.
If @davidvng is ok, I would like to ask you to implement my request into preprocess_obs.

1. Run things both for planet scan and cmb scans. Coverage map is not necessary for cmb scans.
2. Make tags of whether each wafer_slot hit sources even for cmb scans. Like 'jupiter_ws0'. And how about using 'jupiter' tag for a tag to indicate at least one wafer_slot hits the jupiter?

If those two are implemented in the preprocess_obs, I do not have any objection. Please let me know if you have comments.

[davidvng]

@tterasaki @msilvafe
Tested my updates with the following config on obsid = 'obs_1721381147_satp1_1111111':

context_file: '/so/metadata/satp1/contexts/use_this.yaml'

plot_dir: '/so/home/dnguyen/repos/scripts/preprocess/test_plots'

archive:
  index: '/so/home/dnguyen/repos/scripts/preprocess/process_obs_sourcetags_archive.sqlite'
  policy:
    type: 'simple'
    filename: '/so/home/dnguyen/repos/scripts/preprocess/preprocess_obs_sourcetags_archive.h5'
    
telescope: 'SAT'

source_list:
 - 'mercury'
 - 'venus'
 - 'mars'
 - 'jupiter'
 - 'saturn'
 - 'uranus'
 - 'neptune'
 - ['taua', 83.6272579, 22.02159891]
 - ['rcw38', 134.7805107, -47.50911231]
 - ['iras16183-4958', 245.5154, -50.09168292]
 - ['iras19078+0901', 287.5575891, 9.107188994]
 - ['rcw122', 260.0339538, -38.95673421]
 - ['cenA', 201.3625336, -43.00797508]
 - ['3c279', 194.0409868, -5.79174024]
 - ['3c273', 187.2775626, 2.053532671]
 - ['G025.36-00.14', 279.5264042, -6.793169326]
 - ['QSO_J2253+1608', 343.4952422, 16.14301323]
 - ['galactic_center', -93.5833, -29.0078]

process_pipe:
    - name : "dark_dets"
      calc: True
      save: True
      select: True

    - name: "sso_footprint"
      calc:
        # source_list: ['jupiter'] # remove to find nearby sources
        distance: 20
        nstep: 100
        telescope: 'SAT'
        wafer_hit_threshold: 10
      save: True
      plot:
        wafer_offsets: {'ws0': [-2.5, -0.5],
                        'ws1': [-2.5, -13],
                        'ws2': [-13, -7],
                        'ws3': [-13, 5],
                        'ws4': [-2.5, 11.5],
                        'ws5': [8.5, 5],
                        'ws6': [8.5, -7]}
        focal_plane: '/so/home/msilvafe/shared_files/sat_hw_positions.npz'

Running db.inspect({'obs:obs_id': obsid}) gives me:

[{'_id': 1,
  'filename': 'preprocess_obs_sourcetags_archive.h5',
  'obs:obs_id': 'obs_1721381147_satp1_1111111',
  'dataset': 'obs_1721381147_satp1_1111111',
  'mercury': 0,
  'mercury_ws0': 0,
  'mercury_ws1': 0,
  'mercury_ws2': 0,
  'mercury_ws3': 0,
  'mercury_ws4': 0,
  'mercury_ws5': 0,
  'mercury_ws6': 0,
  'venus': 0,
  'venus_ws0': 0,
  'venus_ws1': 0,
  'venus_ws2': 0,
  'venus_ws3': 0,
  'venus_ws4': 0,
  'venus_ws5': 0,
  'venus_ws6': 0,
  'mars': 0,
  'mars_ws0': 0,
  'mars_ws1': 0,
  'mars_ws2': 0,
  'mars_ws3': 0,
  'mars_ws4': 0,
  'mars_ws5': 0,
  'mars_ws6': 0,
  'jupiter': 0,
  'jupiter_ws0': 0,
  'jupiter_ws1': 0,
  'jupiter_ws2': 0,
  'jupiter_ws3': 0,
  'jupiter_ws4': 0,
  'jupiter_ws5': 0,
  'jupiter_ws6': 0,
  'saturn': 1,
  'saturn_ws0': 1,
  'saturn_ws1': 0,
  'saturn_ws2': 0,
  'saturn_ws3': 0,
  'saturn_ws4': 1,
  'saturn_ws5': 1,
  'saturn_ws6': 0,
  'uranus': 0,
  'uranus_ws0': 0,
  'uranus_ws1': 0,
  'uranus_ws2': 0,
  'uranus_ws3': 0,
  'uranus_ws4': 0,
  'uranus_ws5': 0,
  'uranus_ws6': 0,
  'neptune': 1,
  'neptune_ws0': 0,
  'neptune_ws1': 0,
  'neptune_ws2': 0,
  'neptune_ws3': 0,
  'neptune_ws4': 1,
  'neptune_ws5': 1,
  'neptune_ws6': 0,
  'taua': 0,
  'taua_ws0': 0,
  'taua_ws1': 0,
  'taua_ws2': 0,
  'taua_ws3': 0,
  'taua_ws4': 0,
  'taua_ws5': 0,
  'taua_ws6': 0,
  'rcw38': 0,
  'rcw38_ws0': 0,
  'rcw38_ws1': 0,
  'rcw38_ws2': 0,
  'rcw38_ws3': 0,
  'rcw38_ws4': 0,
  'rcw38_ws5': 0,
  'rcw38_ws6': 0,
  'iras16183-4958': 0,
  'iras16183-4958_ws0': 0,
  'iras16183-4958_ws1': 0,
  'iras16183-4958_ws2': 0,
  'iras16183-4958_ws3': 0,
  'iras16183-4958_ws4': 0,
  'iras16183-4958_ws5': 0,
  'iras16183-4958_ws6': 0,
  'iras19078+0901': 0,
  'iras19078+0901_ws0': 0,
  'iras19078+0901_ws1': 0,
  'iras19078+0901_ws2': 0,
  'iras19078+0901_ws3': 0,
  'iras19078+0901_ws4': 0,
  'iras19078+0901_ws5': 0,
  'iras19078+0901_ws6': 0,
  'rcw122': 0,
  'rcw122_ws0': 0,
  'rcw122_ws1': 0,
  'rcw122_ws2': 0,
  'rcw122_ws3': 0,
  'rcw122_ws4': 0,
  'rcw122_ws5': 0,
  'rcw122_ws6': 0,
  'cenA': 0,
  'cenA_ws0': 0,
  'cenA_ws1': 0,
  'cenA_ws2': 0,
  'cenA_ws3': 0,
  'cenA_ws4': 0,
  'cenA_ws5': 0,
  'cenA_ws6': 0,
  '3c279': 0,
  '3c279_ws0': 0,
  '3c279_ws1': 0,
  '3c279_ws2': 0,
  '3c279_ws3': 0,
  '3c279_ws4': 0,
  '3c279_ws5': 0,
  '3c279_ws6': 0,
  '3c273': 0,
  '3c273_ws0': 0,
  '3c273_ws1': 0,
  '3c273_ws2': 0,
  '3c273_ws3': 0,
  '3c273_ws4': 0,
  '3c273_ws5': 0,
  '3c273_ws6': 0,
  'G025.36-00.14': 0,
  'G025.36-00.14_ws0': 0,
  'G025.36-00.14_ws1': 0,
  'G025.36-00.14_ws2': 0,
  'G025.36-00.14_ws3': 0,
  'G025.36-00.14_ws4': 0,
  'G025.36-00.14_ws5': 0,
  'G025.36-00.14_ws6': 0,
  'QSO_J2253+1608': 1,
  'QSO_J2253+1608_ws0': 0,
  'QSO_J2253+1608_ws1': 0,
  'QSO_J2253+1608_ws2': 0,
  'QSO_J2253+1608_ws3': 0,
  'QSO_J2253+1608_ws4': 0,
  'QSO_J2253+1608_ws5': 1,
  'QSO_J2253+1608_ws6': 0,
  'galactic_center': 0,
  'galactic_center_ws0': 0,
  'galactic_center_ws1': 0,
  'galactic_center_ws2': 0,
  'galactic_center_ws3': 0,
  'galactic_center_ws4': 0,
  'galactic_center_ws5': 0,
  'galactic_center_ws6': 0}]

which aligns with the plots generated

[davidvng]

The latest changes creates datasets with the following format:

[{'_id': 1,
  'filename': 'preprocess_obs_sourcetags_archive_v2.h5',
  'obs:obs_id': 'obs_1721381147_satp1_1111111',
  'dataset': 'obs_1721381147_satp1_1111111',
  'coverage': 'saturn:ws0,saturn:ws4,saturn:ws5,neptune:ws4,neptune:ws5,QSO_J2253+1608:ws5'}]

In combination with #856 , in which I added handling for querying an external ManifestDB in which a field, in this case coverage has a string value that is a list comma-separated strings in SQLite format.
I can successfully retrieve the above entry with the following query:

r0 = obsdb.query(
                    subdbs_info_list = [{'filepath': extdb_path,
                                         'query_list': ['saturn:ws0 in coverage'],
                                         'params_list': ['coverage']}]
                   )