Add option to trim low abundance reads

[olgabot]

As mentioned here and here both khmer and sourmash support streaming, so to trim low abundance k-mers one could pipe the output of trim-low-abund.py directly into sourmash compute, which would save time as one would not need to wait from trim-low-abund.py to finish (as it is quite slow).

the -M flag is for memory and one could increase it to e.g. 16GiB to make it faster as well

trim-low-abund.py -C 3 -Z 18 -V -M 2e9 ${reads} | \
    sourmash compute ${flags} -

Thoughts, @phoenixAja and @batson ?

[batson]

Makes sense to me! Though in my experience trim-low-abun was about 80% of the runtime, so this would end up saving 20% of total.

…

On Tue, Apr 16, 2019 at 12:15 PM Olga Botvinnik ***@***.***> wrote: As mentioned [here(] https://sourmash.readthedocs.io/en/latest/using-sourmash-a-guide.html#how-should-i-prepare-my-data) and here <sourmash-bio/sourmash#601> both khmer and sourmash support streaming, so to trim low abundance k-mers one could pipe the output of trim-low-abund.py directly into sourmash compute, which would save time as one would not need to wait from trim-low-abund.py to finish (as it is quite slow). the -M flag is for memory and one could increase it to e.g. 16GiB to make it faster as well trim-low-abund.py -C 3 -Z 18 -V -M 2e9 ${reads} | \ sourmash compute ${flags} - Thoughts, @phoenixAja <https://github.com/phoenixAja> and @batson <https://github.com/batson> ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABM-atGWhx6rmISVRCGS56IxkpgK7vGvks5vhiFtgaJpZM4czhSj> .

[olgabot]

Great! I'll take a stab at writing a PR. After I fix these tests first though ... --- Olga Botvinnik, PhD olgabotvinnik.com <http://www.olgabotvinnik.com> On Tue, Apr 16, 2019 at 6:21 PM Joshua Batson <[email protected]> wrote:

…

Makes sense to me! Though in my experience trim-low-abun was about 80% of the runtime, so this would end up saving 20% of total. On Tue, Apr 16, 2019 at 12:15 PM Olga Botvinnik ***@***.***> wrote: > As mentioned [here(] > https://sourmash.readthedocs.io/en/latest/using-sourmash-a-guide.html#how-should-i-prepare-my-data ) > and here <sourmash-bio/sourmash#601> both khmer and > sourmash support streaming, so to trim low abundance k-mers one could pipe > the output of trim-low-abund.py directly into sourmash compute, which > would save time as one would not need to wait from trim-low-abund.py to > finish (as it is quite slow). > > the -M flag is for memory and one could increase it to e.g. 16GiB to make > it faster as well > > trim-low-abund.py -C 3 -Z 18 -V -M 2e9 ${reads} | \ > sourmash compute ${flags} - > > Thoughts, @phoenixAja <https://github.com/phoenixAja> and @batson > <https://github.com/batson> ? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#8>, or mute the > thread > < https://github.com/notifications/unsubscribe-auth/ABM-atGWhx6rmISVRCGS56IxkpgK7vGvks5vhiFtgaJpZM4czhSj > > . > — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#8 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAxNcEAj3DGCBJpe7jAzRiUR0Eia6qwVks5vhk0BgaJpZM4czhSj> .