Switch back to peakfinding in the matched filter #807
Conversation
skhrg
force-pushed
the
jump_peaks
branch
2 times, most recently
from
5fe973c
to
92a2de8
Compare
|
Could you make a wiki log post that summarizes the core of this PR? Like a description in words (and/or math) of what the different algorithms (current + this PR) are doing, which false positives you were finding with the old algorithm and what those samples look like with this PR for a few representative cases, and some summary comparisons between the two (time to run, memory to run, number of glitches found, etc.)? |
|
Yeah that sounds like a good idea is this the correct place to put it? https://simonsobs.atlassian.net/wiki/spaces/PRO/pages/236421160/TOD+Processing+Analysis+Logbook I kinda think a generic page under https://simonsobs.atlassian.net/wiki/spaces/PRO/pages/236421121/1.06.07.05+TOD+Processing like "[TOD Processing] Algorhitm Documentation" with a page for each operation (jump flag, glitch flag, etc.) where we can stick explanations of math and stuff might be nice but maybe I'm getting ahead of myself. |
Either is fine with me. |
skhrg
force-pushed
the
jump_peaks
branch
2 times, most recently
from
863ae90
to
6c9b30a
Compare
|
Ok see this for details but I think I am happy with this PR and it is a safe move that helps performance and is easier to understand. |
skhrg
force-pushed
the
jump_peaks
branch
3 times, most recently
from
7fdd3b4
to
b7a7c4b
Compare
|
Ok this now also includes speed ups in C++ for twopi jumps. |
skhrg
force-pushed
the
jump_peaks
branch
2 times, most recently
from
ab61e03
to
51953aa
Compare
skhrg
force-pushed
the
jump_peaks
branch
2 times, most recently
from
ce75039
to
4a640f7
Compare
* so3g block_moment * matched filter in so3g * 2pi jumps in so3g * Faster diff buffed * Switch to mean sub * Do more things inplace * Remove unnneeded std check
|
Ok tests pass now that we have the new |
Realized that in some round of modifications I essentially canceled out the matched filter since I was looking for peaks in
np.diff(np.cumsum(x), 2)which is basically justnp.diff(x)(there is some smoothing as well).This is because
np.cumsum(x)makes jumps show up as slope changes unless we cross 0. In the old recursive version this was solved by mean subtracting and finding the largest jump (which crossed 0 when mean subbed) and then recursively repeating between jumps which was slow.Here we solve the problem by detrending the matched filtered data which makes the slope changes look like relative extrema and then usingEDIT: Actually the detrend to make things peaks doesn't work as well as I like, what does work well is to subtract off a mean template.argrelmax(much faster thanfind_peaks) to look for them.Tested this base algorithm on some LAT data and it seems more robust to false positives (especially glitches).