Modules for multitesting #193
Conversation
Added modules for multiple hypothesis testing and stratified multiple hypothesis testing
|
Hi Clayton, thanks so much for this! It looks like a potentially very useful contribution. Could you add some more comments to your code and a few unit tests so we can better evaluate it? |
Added comments for clarity and tests for multitest modules.
|
Hey Amanda, done! I noticed what I thought was strange behavior in the stratified module. On line 130 the number of groups is used to decide whether to use mean or std as the statistic. However I believe this should be the number of conditions. I made this change for the multitest_stratified module, but if I am mistaken I can change it back to groups. I also had the thought it would be more parsimonious to fold the multitest modules into the core and stratified modules, as the desired behavior of the multitest modules with a single test is identical to the original modules. But open to whatever! |
|
@akglazer Did you get a chance to look over the tests? |
|
Hi Clayton! So sorry for the delay on this and thanks so much for all your work! What method are you implementing in this code? It looks like Westfall-Young maxT? |
|
No worries! Yes this is the Westfall-Young method. If I have time in the next few weeks I will speed things up by implementing the Numba library which has worked beautifully for me in the past. |
|
Thanks so much! We really appreciate the time you've put into this and for reminding us of the need for multiple testing corrections. We've been working on implementing Westfall-Young, along with other corrections, and think ultimately it will be better integrated in npc.py so we can leverage the Experiment class and minimize code duplication. |
Removed max statistic multiple hypothesis testing correction and made corresponding adjustments to the multitest_core and multitest_stratified tests. Also touched up some of the tests to make them run smoother.
|
I failed to notice npc and the experiment class! Makes sense. Removed the Westfall-Young method in this PR and updated the tests accordingly. |
|
Hey @akglazer, any chance of merging this soon? |
|
Hi Clayton, we are planning to add multiple testing functionality soon, we are just working out some changes to the code structure still. |
Added modules for multiple hypothesis testing and stratified multiple hypothesis testing.
Would be great to get a second pair of eyes on this and make sure everything checks out!