MS loader

[marziarivi]

Hi Simon,

I noticed that Montblanc reads the number of antennas from a MS as the number of rows of the ANTENNA table. However, sometimes some of these rows have flag = 1, i.e. must be neglected.
They can be removed from the MS using CASA. However, I just pointed it out in case you want to manage it in the code.

Cheers,
Marzia

[sjperkins]

Thanks for the headsup. We've been having a series of robust discussions about handling flagged or missing data. The new source/sink provider interface in the new version should make this fairly configurable for the user.

I guess the correct behaviour is to flag any baselines the antenna is involved with?

In general I am opposed to removing data from Measurement Sets these days as it makes fitting and reasoning portions of the problem onto GPUs/compute nodes difficult and introduces multiple code paths -- my attitude is that it's better to massage the MS by filling in (and flag missing data points).

/cc @JSKenyon for interests sake

[marziarivi]

In my case those antennas are not used in the other data. But Montblanc assumed a number of antennas different from the real one, because it doesn't consider the flag column. I assume that the MS must be consistent with the actual number of antennas and the baselines, even if some baselines could be flagged as well...

[sjperkins]

In my case those antennas are not used in the other data. But Montblanc assumed a number of antennas different from the real one, because it doesn't consider the flag column. I assume that the MS must be consistent with the actual number of antennas and the baselines, even if some baselines could be flagged as well...

This is more relaxed in the new version in the sense that we don't require the following to hold strictly nbl == na*(na-1)/2 (and nbl == na*(na+1)/2 for autocorrelations) any more. So I don't this will be a problem going forward.

[sjperkins]

Updated my previous comment.

[marziarivi]

Yes I know because I am using v5. However there is a problem when the number of MS rows is different from ntime x nbl x nchan, which makes sense. So I am not sure this must be managed within the MS...

[sjperkins]

Yes I know because I am using v5. However there is a problem when the number of MS rows is different from ntime x nbl x nchan, which makes sense. So I am not sure this must be managed within the MS...

We're still having a think about this. The two solutions we're thinking of are:

1. Support it somehow in the SourceProviders (loaders)
2. Provide a script to fill and flag the missing data points.

[JSKenyon]

In my opinion, the flagged antennas should not have all baselines associated with them removed from the MS. Instead, those rows should be flagged, preserving the ntime, nbl, nchan dimension which you mentioned. Unfortunately, dealing with information which has been removed in its entirety poses certain problems for performant code. As Simon mentioned, it is possible to handle this in the SourceProviders by ensuring the any omitted information is reintroduced and flagged.

@marziarivi I am not sure what you mean when you say "real" number of antennas. Are you referring to the number of unflagged antennas? To me, the true number of antennas is always the total number of antennas in the array, regardless of whether they are functioning properly or not.

[rdeane]

I would strongly support @sjperkins option number 2. This crops up all the time in other applications when we make the switch from using simulated data to real data.