Unable to apply time latency to kmall/mb261 survey data with mbpreprocess #1264
Comments
|
Extracted 10 pings out of the original file with a small python script. Both script and smaller file are here: https://drive.google.com/drive/folders/1WDxXOpdVjjVVlVXDvwNJHDCGAj2G9Aee?usp=sharing |
|
Digging further it looks like this might be a combination of two issues with
Fixing 1 seems easy enough, but I don't know enough about the internals here to deal with 2. Leaving 2 unpatched produces XMS datagrams with the original timestamps, which then propagates weirdness into other areas. In my example it makes mbinfo and mbedit think the very last ping has the original time stamp, while all the others are correctly adjusted. For now, since I don't care about the sidescan packets at the moment, I've just removed that capability. I don't know where to start with part 3. So for now I need to strip out every datagram that isn't MRZ first, then use the below patch. Doing that I can shift survey data as intended. diff --git a/src/mbio/mbsys_kmbes.c b/src/mbio/mbsys_kmbes.c
index 8bc5471ee..d27de1adb 100644
--- a/src/mbio/mbsys_kmbes.c
+++ b/src/mbio/mbsys_kmbes.c
@@ -428,6 +428,7 @@ int mbsys_kmbes_preprocess(int verbose, void *mbio_ptr, void *store_ptr,
int time_i[7];
double time_d;
+ double time_delta;
double navlon;
double navlat;
double sensordepth;
@@ -458,6 +459,7 @@ int mbsys_kmbes_preprocess(int verbose, void *mbio_ptr, void *store_ptr,
mb_get_date(verbose, pars->time_d, time_i);
for (int i = 0; i < 7; i++)
store->time_i[i] = time_i[i];
+ time_delta = store->time_d - pars->time_d;
store->time_d = pars->time_d;
fprintf(stderr,
"Timestamp changed in function %s: "
@@ -488,6 +490,11 @@ int mbsys_kmbes_preprocess(int verbose, void *mbio_ptr, void *store_ptr,
// get time
time_d = ((double)mrz->header.time_sec) + MBSYS_KMBES_NANO * mrz->header.time_nanosec;
+ if (pars->timestamp_changed) {
+ time_d -= time_delta;
+ mrz->header.time_sec = (unsigned int)(floor(time_d));
+ mrz->header.time_nanosec = (unsigned int)((time_d - (double)mrz->header.time_sec) / MBSYS_KMBES_NANO);
+ }
// construct xmt basics
xmt->header = mrz->header;
@@ -769,6 +776,7 @@ int mbsys_kmbes_preprocess(int verbose, void *mbio_ptr, void *store_ptr,
}
}
+#if 0
// generate pseudosidescan
struct mbsys_kmbes_mrz *mrz = (struct mbsys_kmbes_mrz *)&store->mrz[0];
if (xms->pingCnt != mrz->cmnPart.pingCnt) {
@@ -777,6 +785,7 @@ int mbsys_kmbes_preprocess(int verbose, void *mbio_ptr, void *store_ptr,
status = mbsys_kmbes_makess(verbose, mbio_ptr, store_ptr, false,
pixel_size, false, swath_width, 0, error);
}
+#endif
}
if (verbose >= 2) {Edit: I've updated the files on the gdrive share to include other datagrams besides MRZ for further testing |
|
Zac,
Sorry for the slow response on my part - I’m offshore.
In MB-System time latency adjustments are made to the ancillary data streams, never to the primary sensor timestamps. Generally this is done by applying the latency to the nav, attitude, or sensordepth time series in memory before that time series is used to interpolate values onto the desired times. So there is not really even any facility to output data with changed timestamps to the ancillary records.
What is the problem? Do you have an attitude latency showing up as roll artifacts?
Cheers,
Dave
… On Feb 3, 2022, at 9:56 PM, Zac Berkowitz ***@***.***> wrote:
Digging further it looks like this might be a combination of two issues with mbsys_kmbes_preprocess in mbsys_kmbes.c.
• The timestamps in the MRZ and XMT headers are not being adjusted at all
• No timing change information is provided to mbsys_kmbes_makess, so XMS headers can't be adjusted either
• None of of the other many datagrams ("#SVT", "SPO", etc) are shifted in this function. They don't seem to be shifted either, so not sure where that happens.
Fixing 1 seems easy enough, but I don't know enough about the internals here to deal with 2. Leaving 2 unpatched produces XMS datagrams with the original timestamps, which then propagates weirdness into other areas. In my example it makes mbinfo and mbedit think the very last ping has the original time stamp, while all the others are correctly adjusted.
For now, since I don't care about the sidescan packets at the moment, I've just removed that capability.
I don't know where to start with part 3. So for now I need to strip out every datagram that isn't MRZ first, then use the below patch. Doing that I can shift survey data as intended.
diff --git a/src/mbio/mbsys_kmbes.c b/src/mbio/mbsys_kmbes.c
index 8bc5471ee..d27de1adb 100644
--- a/src/mbio/mbsys_kmbes.c
+++ b/src/mbio/mbsys_kmbes.c
@@ -428,6 +428,7 @@
int mbsys_kmbes_preprocess(int verbose, void *mbio_ptr, void *store_ptr,
int time_i[7];
double time_d;
+ double time_delta;
double navlon;
double navlat;
double sensordepth;
@@ -458,6 +459,7 @@
int mbsys_kmbes_preprocess(int verbose, void *mbio_ptr, void *store_ptr,
mb_get_date(verbose, pars->time_d, time_i);
for (int i = 0; i < 7; i++)
store->time_i[i] = time_i[i];
+ time_delta = store->time_d - pars->time_d;
store->time_d = pars->time_d;
fprintf(stderr,
"Timestamp changed in function %s: "
@@ -488,6 +490,11 @@
int mbsys_kmbes_preprocess(int verbose, void *mbio_ptr, void *store_ptr,
// get time
time_d = ((double)mrz->header.time_sec) + MBSYS_KMBES_NANO * mrz->header.time_nanosec;
+ if (pars->timestamp_changed) {
+ time_d -= time_delta;
+ mrz->header.time_sec = (unsigned int)(floor(time_d));
+ mrz->header.time_nanosec = (unsigned int)((time_d - (double)mrz->header.time_sec) / MBSYS_KMBES_NANO);
+ }
// construct xmt basics
xmt->header = mrz->header;
@@ -769,6 +776,7 @@
int mbsys_kmbes_preprocess(int verbose, void *mbio_ptr, void *store_ptr,
}
}
+#if 0
// generate pseudosidescan
struct mbsys_kmbes_mrz *mrz = (struct mbsys_kmbes_mrz *)&store->mrz[0];
if (xms->pingCnt != mrz->cmnPart.pingCnt) {
@@ -777,6 +785,7 @@
int mbsys_kmbes_preprocess(int verbose, void *mbio_ptr, void *store_ptr,
status = mbsys_kmbes_makess(verbose, mbio_ptr, store_ptr, false,
pixel_size, false, swath_width, 0, error);
}
+#endif
}
if (verbose >= 2) {
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you are subscribed to this thread.
----------------------------------------------------
David W. Caress
Principal Engineer
Seafloor Mapping Lab
Monterey Bay Aquarium Research Institute
7700 Sandholdt Road
Moss Landing, CA 95039
***@***.***
http://www.mbari.org/~caress/
Phone: 831-775-1775
|
|
Hah, funny. At sea myself too. We had an issue where the EM2040 clock never sync'd to the main stack. So all the ancillary data has the correct timestamp, but the actual ping data is about 1 hour ahead (eg. a timestamp of 17:00:00 in the file was really collected at 16:00:00). I can use |
|
Zac,
You’re absolutely right- the --time-adjust-apply-survey option should do what you need. I had forgotten about architecting that in. As you found, the problem is in the I/o module. I’ve mostly coded a fix but don’t have it working yet.
Cheers,
Dave
…----------------------------
David Caress
Software Engineer
Monterey Bay Aquarium Research Institute
----------------------------
Sent from my iPhone
On Feb 4, 2022, at 4:57 AM, Zac Berkowitz ***@***.***> wrote:
Hah, funny. At sea myself too.
We had an issue where the EM2040 clock never sync'd to the main stack. So all the ancillary data has the correct timestamp, but the actual ping data is about 1 hour ahead (eg. a timestamp of 17:00:00 in the file was really collected at 16:00:00).
I can use mbprerocess to shift all of the ancillary nav data to match the ping data but that's not the right solution here because the ping timestamp is wrong, not the nav data. And if MB-system never adjusts the primary sensor timestamp, what does --time-adjust-apply-survey do then?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you commented.
|
|
Thanks, Dave. We've been able to shift this dataset as needed using one-off hacks so there isn't a burning need to have this right-this-moment, but of course it'd be great to have it available in MB-System. I can help test your fix prior to merging if you'd like. Zac |
Describe the bug
Unable to apply time latency to kmall/mb261 survey data with mbpreprocess
To Reproduce
Now try to apply a time offset
Same start and end time as without the time shift. Further processing shows that it's not just an issue with mbinfo, the data timestamps have indeed not moved. If I shift all the other data by flipping the sign of the desired shift and using
--time-latency-apply-all-ancilliaryI can get a properly navigated survey but all the timestamps are obviously wrong.Provide information sufficient to reproduce the behavior:
Example kmall file: https://drive.google.com/drive/folders/1WDxXOpdVjjVVlVXDvwNJHDCGAj2G9Aee?usp=sharing
Expected behavior
Should see shifted times in "Start of Data" and "End of Data" from mbinfo
Computing context (please complete the following information):
The text was updated successfully, but these errors were encountered: