forked from dib-lab/khmer
-
Notifications
You must be signed in to change notification settings - Fork 0
/
ChangeLog
3157 lines (2277 loc) · 126 KB
/
ChangeLog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
2015-09-28 Lisa Cohen <[email protected]>
* tests/test_read_handling.py: created new file with interleave-reads,
split-paired-reads, and extract-paired-reads functions
* tests/test_scripts.py: removed functions mentioned above
2015-09-19 Titus Brown <[email protected]>
* scripts/filter-abund.py: fixed bug with -o and --gzip used together.
* tests/test_scripts.py: added test for fixed bug.
* sandbox/count-kmers.py: added executable bit.
2015-09-17 Camille Scott <[email protected]>
* scripts/interleave-reads.py: Added --no-reformat flag
to disable format checking and renaming of headers
* tests/{test_scripts.py, test-data/paired.malformat*}: Tests
and test data for --no-reformat flag.
2015-09-11 Russell Y. Neches <[email protected]>
* lib/hashbits.hh: added get_raw_tables to Hashbits
* khmer/_khmer.cc: added get_raw_tables to Hashbits
* tests/test_nodegraph.py: added test for Nodegraph.get_raw_tables
2015-09-11 Camille Scott <[email protected]>
* lib/hashbits.cc: Make hasbits::update_from() correctly update
_occupied_bins.
* lib/test_nodegraph.py: Test Nodegraph.n_occupied() after update.
2015-09-07 Michael R. Crusoe <[email protected]>
* doc/roadmap.rst: Updated for 2.0 release.
* README.rst: drop unstable badges; fix links
* doc/index.rst: point at getting help guide
2015-09-05 Michael R. Crusoe <[email protected]>
* tests/test_scripts.py: make the script version test more specific; double
check that script in question is from the 'khmer' project based upon the
second line of the file.
2015-09-04 Michael R. Crusoe <[email protected]>
* tests/khmer_tst_utils.py: look in "EGG-INFO" for scripts before checking
the PATH
2015-09-04 Michael R. Crusoe <[email protected]>
* doc/release-notes/release-2.0.md: release notes for 2.0
* doc/user/install.txt: correct second invocation of nosetests to match the
first.
* setup.py: update the authors list for PyPI.
* doc/user/known-issues.rst: remove three fixed issues; add a new one
* doc/whats-new-2.0.rst: update to match the release notes
* doc/release-notes/*.rst: refresh using `make convert-release-notes'
2015-09-02 Michael R. Crusoe <[email protected]>
* *: complete audit of license headers
* LICENSE: listings of some of the third-party licenses
* bink.ipynb,lib/graphtest.cc,lib/primes.hh: deleted unused files
* sandbox/assemstats.py: updated screed install instructions
2015-09-02 Michael R. Crusoe <[email protected]>
* *: finish undoing the `load-graph.py` rename
2015-09-02 Micheal R. Crusoe <[email protected]>
* doc/requirements.txt: Update URL to sphinxcontrib-autoprogram tarball
2015-08-31 Michael R. Crusoe <[email protected]>
* khmer/thread_utils.py: Use the robust test for paired reads from
khmer.utils; drop `is_pair()`
* tests/test_threaded_sequence_processor.py: Update to use Screed Record
objects.
2015-08-28 Michael R. Crusoe <[email protected]>
* doc/whats-new-2.0.rst: Many updates from `diff`ing the `--help` output of
version 1.4.1 vs now.
* Makefile: build the project if needed when making a PDF
* doc/index.rst: fixed inter-doc links to be relative
* doc/user/blog-posts.rst: replaced link to 88m-reads.fa.gz with a working
URL
* doc/user/{choosing-table-sizes,guide}.rst: disambiguated :option:
references so that they link properly.
* scripts/README.txt: removed unhelpful file
* scripts/normalize-by-median.py: replace `--force-single` with
`--force_single`. Added help text for `--cutoff`.
2015-08-27 Titus Brown <[email protected]>
* doc/dev/getting-started.rst: fixed a few misspellings.
2015-08-26 Michael R. Crusoe <[email protected]>
* tests/test_scripts.py: removed the unused `_DEBUG_make_graph` function in
favor of other debugging methods like `ipdb`. Added
test_do_partition_no_big_traverse and test_extract_paired_reads_unpaired.
2015-08-24 Michael R. Crusoe <[email protected]>
* khmer/khmer_args.py: Replaced sanitize_epilog() with santize_help() that
reflows the text of ArgParse descriptions and epilog while preserving the
formatting. Enhanced removal of Sphinx directives by replacing double
backticks with the double quote character.
* scripts/*.py: Renamed sanitize_epilog to sanitize_help; leading newlines
from triple-quoted epilogs removed; formatting made consistent;
sanitize_help and ComboFormatter added where it was missing; a couple
script specific epilog reformatting (for use of `:doc:` and a
hyperlink).
* scripts/{count-median,filter-abund-single}.py: Fixed printing of output
file name to do so instead of printing information about the file handle.
* scripts/count-median.py: Added missing command so that example given
actually works.
* scripts/filter-abund-single.py: Removed redundant printing of output file
names.
* scripts/normalize-by-median.py: Removed unused option "-d" from an example
command (left over from the "--dump-frequency" era).
* scripts/{partition-graph.py,do-partition.py}: Fixed erasure of the queue
module name in the worker functions, which is necessary for basic
functionality.
* scripts/{do-partition,abundance-dist,abundance-dist-single,
extract-long-sequences}.py: Added an example command to the epilog.
* tests/khmer_tst_utils.py: Added 'name' attribute to make the fake
sys.stdout more like a read stdout object.
* oxli/__init__.py: removed redundant and unused help text
* scripts/{abundance-dist,annotate-partitions,count-median,
extract-long-sequences,extract-paired-reads,extract-partitions,
fastq-to-fasta,filter-abund,filter-stopgaps,interleave-reads,
load-into-graph,merge-partitions,normalize-by-median,partition-graph,
readstats,sample-reads-randomly,split-paired-reads}.py: made "--version"
and the citation header consistent across the scripts.
* tests/test_scripts.py: added tests for the "--version" and citation
header behavior.
* tests/test_normalize_by_median.py: updated test for 'quiet' mode as
citation header still prints to STDERR.
* setup.py,tests/test_scripts.py: turned off the "oxli" script for v2.0.
2015-08-17 Michael R. Crusoe <[email protected]>
* Makefile: remove BASH shell designation that appears to be incompatible
with OS X Mavericks & virtualenv; refactored out BASH-isms for those whose
default shell isn't BASH (Debian, and others).
* lib/test-read-aligner.cc,read_aligner_test.sh: removed unused test files
found during search for other shell scripts with BASHisms.
2015-08-18 Michael R. Crusoe <[email protected]>
* *: reverted load-into-counting.py -> load-into-countgraph.py and
load-graph.py -> load-into nodegraph.py rename.
* CITATION: unescaped URL
* doc/user/guide.rst: added `:program:` and `:option:` directives as
needed; replaced references to `strip-and-split-for-assembly` with
`extract-paired-reads`. Updated instructions to use `--unpaired-reads` with
`normalize-by-median.py` instead of second round of diginorm.
2015-08-18 Titus Brown <[email protected]>
* doc/index.rst: update introductory text.
* doc/user/biblio.rst: update bibliography link descriptions and text.
* doc/_static/labibi.css: reference new location of base CSS
2015-08-14 Luiz Irber <[email protected]>
* scripts/unique-kmers.py: Rename option --stream-out to --stream-records.
* tests/test_streaming_io.py: Fix unique-kmers tests, use new option.
* khmer/_khmer.cc, lib/hllcounter.cc: Rename some variables for consistency.
2015-08-12 Jacob Fenton <[email protected]>
* doc/dev/{codebase-guide,coding-guidelines-and-review,development,
for-khmer-developers,getting-started,release,scripts-and-sandbox,
binary-file-formats}.rst,doc/{index,introduction,whats-new-2.0,
contributors}.rst,doc/user/{blog-posts,choosing-table-sizes,galaxy,
getting-help,guide,install,known-issues,partitioning-big-data,
scripts}.rst,CITATION: Cleaned up documentation
* scripts/*.py, khmer/khmer_args.py: added epilog sanitation
* scripts/{load-into-counting,load-graph,load-into-countgraph,
load-into-nodegraph}.py, tests/{test_scripts,test_normalize_by_median,
test_streaming_io,test_countgraph}: renamed load-into-counting ->
load-into-countgraph, load-graph -> load-into-nodegraph, fixed tests to not
bork
2015-08-12 Luiz Irber <[email protected]>
* khmer/_khmer.cc: Fix a GCC string initialization warning.
2015-08-12 Michael R. Crusoe <[email protected]>
* CITATION, doc/{index,introduction,user/scripts}.rst, khmer/khmer_args.py:
formatting fixes and new citation for the software as a whole
* Makefile: PDF building hints, tweaked dependencies, update coverity URL,
new target to generate author list for the paper citation
* sort-authors-list.py: helper script for the above
* doc/conf.py: don't generate a module index
* doc/contributors.rst: formatting, remove references to old lab
* doc/dev/coding-guidelines-and-review.rst: add C++ version standard
* doc/dev/getting-started.rst: ccache, git-merge-changelog, and advanced
commit squashing
* doc/user/biblio.rst: added links to khmer citation collections
* doc/user/examples.rst: linked to online version of example scripts
* doc/user/guide.rst: commented out empty section, added BSD note
* lib/counting.{cc},lib/*.hh: c++11 fixes: remove unneeded trailing
semicolons
* scripts/*.py: line-wrap & scrub output
* setup.py: turn optimizations back on and -pedantic
* .mailmap: additional tweaks to author list
2015-08-11 Kevin Murray <[email protected]>
* lib/Makefile: Fix SONAME and ABI versioning to sync with Debian standard
practice.
2015-08-10 Camille Scott <[email protected]>
* lib/traversal.{cc,hh}: Add new files with unified traversal machinery.
Introduce Traverser class to handle finding neighbors and appropriate
bitmasking.
* khmer/_khmer.cc,lib/{counting,hashbits,hashtable,labelhash,subset}.{cc,hh}:
Updated relevant instances of HashIntoType and KMerIterator to use new Kmer
and KmerIterator, respectively.
* lib/Makefile: Add -std=c++11 flag.
* Makefile: Update -std=c++11 flag in libtest target.
* lib/hashtable.{cc,hh}: Update calc_connected_graph_size to use Traverser.
Change kmer_degree to use functions from traversal.cc. Remove redundant
count_kmers_with_radius in favor of calc_connected_graph_size. Update
traverse_from_kmer to use Traverser. Hashtable subclasses KmerFactory.
* lib/{hashtable.hh,kmer_hash.{cc,hh}}: Move KmerIterator from hashtable.hh
to kmer_hash.{cc,hh}. Add Kmer class to store forward, reverse, and
uniqified integer representations of k-mers, and to handle string
conversion. Update KmerIterator to emit objects of type Kmer and to subclass
KmerFactory; add doxygen markup.
* lib/khmer.hh: Forward declare Kmer and typedef new Kmer data structures.
* lib/subset.{cc,hh}: Move constructor definition to .cc file. Remove
queue_neighbors in favor of new traversal machinery. Update find_all_tags,
sweep_for_tags, and find_all_tags_truncate_on_abundance to use Traverser.
* setup.py: Add traversal.{cc,hh} to deps.
2015-08-10 Luiz Irber <[email protected]>
* scripts/unique-kmers.py: use consume_fasta again.
* khmer/_khmer.cc: expose output_records option on HLLCounter consume_fasta.
* lib/hllcounter.{cc,hh}: implement output_records option in consume_fasta.
* lib/read_parsers.{cc,hh}: add Read method write_to, useful for outputting
the read to an output stream.
* doc/whats-new-2.0.rst: Add unique-kmers description.
2015-08-09 Jacob Fenton <[email protected]>
* khmer/khmer_args.py: pep8
* scripts/{interleave-reads,load-graph}.py: Removed unreachable code
* tests/test-data/{paired-broken.fq.badleft,paired-broken.fq.badright,
paired-broken.fq.paired.bad}: added test data files
* tests/{test_normalize_by_median,test_scripts}.py: added tests
2015-08-07 Titus Brown <[email protected]>
* khmer/_khmer.cc,lib/hashbits.{cc,hh}: removed overlap functionality;
eliminated n_entries() as redundant with hashsizes(); removed arguments to
n_occupied(); removed get_kadian_count.
* lib/{hashbits.cc,counting.cc,khmer.hh},tests/test_hashbits.py: updated
save/load of countgraph/nodegraph structures to save _n_occupied.
* lib/{hashtable.hh,counting.hh,hashbits.hh}: promoted n_occupied() to
Hashtable class; fixed CountingHash unique_kmers calculation.
* lib/counting.{cc,hh}: removed get_kadian_count() and moved
n_unique_kmers(); updated countgraph writing to save n_occupied.
* khmer/__init__.py: modified extract_nodegraph_info and
extract_countgraph_info to read in & return n_occupied;
* sandbox/bloom-count-intersection.py,scripts/count-overlap.py,
tests/test-data/overlap.out: removed overlap scripts and test files.
* doc/user/scripts.rst: removed count-overlap.py documentation.
* tests/test_scripts.py: removed count-overlap.py tests.
* sandbox/README.rst: updated with removal of bloom-count-intersection.py.
* tests/test-data/normC20k20.ct: updated file contents to reflect new
format containing _n_occupied.
* tests/test_countgraph.py: removed get_kadian_count tests; added save/load
tests.
* tests/test_counting_single.py: remove n_entries() tests; replace
n_entries() calls with hashsizes() call.
* tests/test_functions.py: updated tests for new extract_*_info functions.
* tests/test_nodegraph.py: update htable etc. to nodegraph; added a
save/load test for n_occupied() on nodegraph.
* tests/{test_normalize_by_median,test_scripts}.py: fixed unique kmers
tests.
2015-08-07 Michael R. Crusoe <[email protected]>
* scripts/*.py,tests/*.py,sandbox/*.py,khmer/*.py,oxli/*.py:
many function and variable renames:
counting_hash, countinghash, hashtable->countgraph;
CountingHash->Countgraph
hashbits->nodegraph; Hashbits->Nodegraph;
check_space_for_hashtable->check_space_for_graph;
hash_args->graph_args
* khmer/_khmer.cc: remove unused 'new_hashtable' method; match renames
* TODO: removed several items
* doc/dev/scripts-and-sandbox.rst: fixed hashbang
2015-08-04 Jacob Fenton <[email protected]>
* khmer/khmer_args.py, oxli/functions.py: migrated estimation functions out
oxli and into khmer_args
* oxli/build_graph.py, tests/test_oxli_functions.py,
sandbox/{estimate_optimal_hash,optimal_args_hashbits}.py,
scripts/{normalize-by-median,unique-kmers}.py: changed to not break on
location change
* tests/{test_normalize_by_median,test_scripts}.py: added tests for
automatic arg setting
* tests/test_script_arguments: changed to play nice with unique_kmers as an
argument
2015-08-04 Titus Brown <[email protected]> and Camille Scott
* khmer/utils.py: added UnpairedReadsError exception.
* scripts/{extract-paired-reads,split-paired-reads}.py: changed --output-dir
argument short form to use '-d'.
* scripts/{split-paired-reads.py} added -0 <filename> to allow orphans; made
'-p'/'--force-paired' default & removed from script.
* scripts/{normalize-by-median,filter-abund,trim-low-abund}.py: changed
long form of '-o' to be '--output'.
* tests/{test_scripts,test_streaming_io}.py: updated and added tests for
new behavior.
2015-08-03 Jacob Fenton <[email protected]>
* doc/dev/coding-guidelines-and-review.rst: added codespell as a possible
spelling tool
2015-08-03 Jacob Fenton <[email protected]>
* Makefile: added oxli to pep257 make target, made clean target wipe out all
.pyc files in scripts/* and tests/* and oxli/*
2015-08-03 Jacob Fenton <[email protected]>
* tests/test_counting_single.py: removed redundant test
2015-08-01 Jacob Fenton <[email protected]> and Titus Brown
* scripts/normalize-by-median.py,khmer/khmer_logger.py: added logging
framework, prototyped in normalize-by-median; added -q/--quiet to
* tests/test_normalize_by_median.py: associated tests.
* khmer/khmer_args.py: Made info function use logging functions.
* tests/khmer_tst_utils.py: removed info reporting in runscript from 'out'
returned.
2015-08-01 Jacob Fenton <[email protected]>
* khmer/kfile.py: added infrastructure for doing compressed output
* khmer/thread_utils.py: switched threaded_sequence_processor to make use of
write_record
* scripts/{extract-long-sequences,extract-paired-reads,
extract-partitions,fastq-to-fasta,filter-abund-single,filter-abund,
interleave-reads,normalize-by-median,sample-reads-randomly,
split-paired-reads,trim-low-abund}.py: added output compression
* tests/{test_functions,test_scripts,test_normalize_by_median}.py: added
tests
* scripts/{load-graph,partition-graph,find-knots.py,
make-initial-stoptags}.py,oxli/build_graph.py: made load-graph no longer
add .pt to graph outfiles, changed partition-graph to not expect .pt's
* doc/whats-new-2.0.rst: doc'd changes to load-graph and partition-graph
* doc/dev/scripts-and-sandbox.rst: updated scripts/ requirements.
2015-08-01 Sherine Awad <[email protected]>
* sandbox/multi-rename.py: updated output of long FASTA sequences to
wrap text at 80 characters.
* tests/test_sandbox_scripts.py: Added a test for multi-rename.py.
2015-07-31 Kevin Murray <[email protected]>
* lib/Makefile,Makefile,lib/*.pc.in,lib/test-compile.cc: Misc debian-based
compatibility changes
* lib/get_version.py: Add crunchbang, chmod +x
2015-07-29 Michael R. Crusoe <[email protected]>
* khmer/_khmer.cc: add more CPyChecker inspired fixes
* lib/*.{cc,hh}: clean up includes and forward declarations
2015-07-29 Luiz Irber <[email protected]>
* Makefile: Adapt Makefile rules for py3 changes.
* jenkins-build.sh: Read PYTHON_EXECUTABLE and TEST_ATTR from environment.
2015-07-29 Amanda Charbonneau <[email protected]>
* scripts/fastq-to-fasta.py: Changed '-n' default description to match
behaviour
2015-07-29 Luiz Irber <[email protected]>
* tests/test_{scripts,streaming_io}.py: Fix the build + add a test
2015-07-28 Titus Brown <[email protected]>
* tests/test_streaming_io.py: new shell cmd tests for streaming/piping.
* tests/khmer_tst_utils.py: refactor/replace runtestredirect(...) with
scriptpath(...) and run_shell_cmd(...).
* scripts/test_scripts.py: remove test_interleave_reads_broken_fq_4 for
only one input file for interleave-reads.py; replace runscriptredirect call
with run_shell_cmd.
* scripts/interleave-reads.py: force exactly two input files.
* scripts/split-paired-reads.py: fix print statement; clarify output.
* scripts/{normalize-by-median.py,sample-reads-randomly.py,
trim-low-abund.py}: if stdin is supplied for input, check that -o
specifies output file.
* scripts/filter-abund.py: if stdin is supplied for input, check that -o
specifies output file; switched -o to use argparse.FileType.
* scripts/extract-long-sequences.py: switched -o to use argparse.FileType.
* scripts/{abundance-dist,count-median}.py: added '-' handling for output.
* khmer/kfile.py: change 'check_input_files' to no longer warn that
'-' doesn't exist'.
* tests/test-data/paired.fq.2: removed extraneous newline from end.
* tests/{test_normalize_by_median,test_script_arguments,test_scripts}.py:
added tests for new code.
* scripts/oxli: added script for running tests in development directory.
* khmer/{__init__,khmer_args}.py,tests/{test_normalize_by_median,
test_script_arguments}.py: refactored out use of AssertionError by not
throwing plain Exceptions when a ValueError or RuntimeError would do.
* oxli/__init__.py: give default help instead of an error when `oxli` is
called with no arguments.
* tests/test_{normalize_by_median,sandbox_scripts,scripts,streaming_io}.py:
always check status code if calling `runscripts` with `fail_ok=True`.
2015-07-28 Luiz Irber <[email protected]>
* sandbox/unique-kmers.py: moved to scripts.
* scripts/unique-kmers.py: fix import bug and initialize to_print earlier.
* tests/test_scripts.py: add tests for unique-kmers.py.
* doc/user/scripts.rst: added unique-kmers.py to script page
2015-07-28 Jacob Fenton <[email protected]>
* scripts/abundance-dist.py: disallowed forcing on the input file check for
the counting table file
2015-07-28 Michael R. Crusoe <[email protected]>
* .mailmap, Makefile: generate a list of authors
2015-07-28 Kevin Murray <[email protected]>
Titus Brown <[email protected]>
* khmer/utils.py: added fix for SRA-style FASTQ output.
* tests/test_scripts.py: tested against a broken version of SRA format.
* tests/test-data/paired-broken4.fq.{1,2}: added test files.
2015-07-28 Michael R. Crusoe <[email protected]>
Titus Brown <[email protected]>
* lib/read_aligner.{cc,hh},tests/{test_read_aligner.py,
test-data/readaligner-{default,k12}.json},khmer/__init__.py: refactor,
read aligner parameters are now configurable & save/load-able. Can do
whole-genome variant finding.
* khmer/_khmer.cc,tests/test_read_aligner.py: ReadAligner.align_forward
method added
* sandbox/correct-errors.py -> sandbox/correct-reads.py: total rewrite
* sandbox/error-correct-pass2.py: new script
* sandbox/readaligner_pairhmm_train.py: new script
* tests/test_sandbox_scripts.py, doc/release-notes/release-1.4.rst:
spelling fixes, import re-arrangement
* sandbox/{Makefile.read_aligner_training,readaligner_pairhmm_train.py}:
Added script to train the aligner
2015-07-27 Titus Brown <[email protected]>
* khmer/khmer_args.py,CITATION: added entry for PeerJ paper on
semi-streaming to citations.
* scripts/{abundance-dist-single.py,abundance-dist.py,count-median.py,
count-overlap.py,filter-abund-single.py,load-into-counting.py}: changed
default behavior to output data in CSV format and report total k-mers.
* tests/test_scripts.py: updated/removed tests for CSV.
* doc/whats-new-2.0.rst: added information about change in columnar output,
along with other minor corrections.
* scripts/normalize-by-median.py: corrected epilog.
* khmer/thread_utils.py,
sandbox/{calc-best-assembly.py,extract-single-partition.py},
scripts/{count-median.py,extract-long-sequences.py,extract-paired-reads.py,
extract-partitions.py,fastq-to-fasta.py,
interleave-reads.py,normalize-by-median.py,readstats.py,
sample-reads-randomly.py,split-paired-reads.py,trim-low-abund.py},
tests/{test_normalize_by_median.py,test_scripts.py}: remove explicit
'parse_description' from screed open calls.
* khmer/_khmer.cc,lib/Makefile,lib/hashtable.{cc,hh},setup.py: removed
WITH_INTERNAL_METRICS and trace_logger/perf_metrics references.
* lib/perf_metrics.{cc,hh},lib/trace_logger.{cc,hh}: removed unused files.
2015-07-24 Jacob Fenton <[email protected]>
* doc/dev/getting-started.rst: added instructions for second contribution
2015-07-22 Jacob Fenton <[email protected]>
* tests/test_read_parsers.py: added workaround for bug in OSX Python
* Makefile: respect that workaround when running the tests
2015-07-21 Jacob Fenton <[email protected]>
* khmer/{kfile,khmer_args}.py: refactored information passing, made it so
space checks happen in the right directory.
* oxli/build_graph.py,sandbox/collect-reads.py,scripts/{
abundance-dist-single,filter-abund-single,load-into-counting,
normalize-by-median,trim-low-abund}.py,tests/test_script_arguments.py:
changed to use new arg structure for checking hashtable save space.
* oxli/functions.py,scripts/saturate-by-median.py: updated error message
to mention --force option.
* scripts/{count-overlap,load-into-counting,make-initial-stoptags,
partition-graph,sample-reads-randomly}.py: removed unnecessary call to
check_space.
2015-07-20 Titus Brown <[email protected]>
* khmer/__init__.py: cleaned up FP rate reporting.
* scripts/normalize-by-median.py: corrected epilog; refactored reporting
to be a bit cleaner; use CSV for reporting file;
added --report-frequency arg.
* tests/test_normalize_by_median.py: updated/added tests for reporting.
2015-07-17 Jacob Fenton <[email protected]>
* oxli/{functions,build_graph}.py,scripts/{load-graph,normalize-by-median,
abundance-dist}.py,tests/test_{normalize_by_median,subset_graph,hashbits,
oxli_function}.py: pylint cleanup.
2015-07-17 Michael R. Crusoe <[email protected]>
* Makefile, tests/test_read_aligner.py: import khmer when pylinting.
2015-07-17 Michael R. Crusoe <[email protected]>
* lib/read_parser.{cc,hh}: use std::string everywhere to match existing
exceptions.
2015-07-10 Jacob Fenton <[email protected]>
* khmer/kfile.py: changed check_valid_file_exists to recognize fifos as
non-empty.
* tests/test_normalize_by_median.py: added test.
2015-07-10 Jacob Fenton <[email protected]>
* oxli/functions.py: changed estimate functions to use correct letter
abbreviations.
* sandbox/estimate_optimal_hash.py: changed to use renamed estimate
functions.
* sandbox/unique-kmers.py: changed to not output recommended HT args by
default.
* tests/test_oxli_functions.py: changed to use renamed estimate functions.
2015-07-10 Jacob Fenton <[email protected]>
* oxli/functions.py: added '--force' check to sanity check.
2015-07-10 Jacob Fenton <[email protected]>
* oxli/functions.py: moved optimization/sanity check func to oxli.
* scripts/normalize-by-median.py,oxli/build_graph.py: added
optimization/sanity checking via oxli estimation funcs.
* tests/test_normalize_by_median.py: updated tests to cover estimation
functions.
2015-07-08 Luiz Irber <[email protected]>
* lib/{counting,hashbits,hashtable,labelhash,subset}.cc: print hexadecimal
representation of the signature read from the file.
2015-07-06 Luiz Irber <[email protected]>
* sandbox/collect-reads.py: Set a default value for coverage based
on the docstring.
* sandbox/count-kmers-single.py, tests/test_{functions,script_arguments}.py:
Replace xrange and cStringIO (not Python 3 compatible).
* lib/*.{hh,cc}, oxli/functions.py, tests/*.py: make format.
2015-07-05 Jacob Fenton <[email protected]>
* doc/whats-new-2.0.rst: added in normalize-by-median.py broken paired
updates.
2015-07-05 Michael R. Crusoe <[email protected]>
* Makefile: fix cppcheck invocation.
* khmer/_khmer.cc: switch to prefix increment for non-primitive objects,
use a C++ cast, adjust scope.
* lib/hashtable.{hh,cc}: make copy constructor no-op explicit. adjust scope
* lib/{ht-diff,test-HashTables,test-Parser}.cc: remove unused test code.
* lib/labelhash.cc,hllcounter.cc: astyle reformatting.
* lib/read_parsers.hh: more explicit constructors.
2015-07-05 Michael R. Crusoe <[email protected]>
* sandbox/{collect-variants,optimal_args_hashbits,sweep-files}.py:
update API usage.
2015-07-05 Titus Brown <[email protected]>
* sandbox/{count-kmers.py,count-kmers-single.py}: added scripts to output
k-mer counts.
* tests/test_sandbox_scripts.py: added tests for count-kmers.py and
count-kmers-single.py.
* sandbox/README.rst: added count-kmers.py and count-kmers-single.py to
sandbox/README.
2015-07-05 Kevin Murray <[email protected]>
* lib/*.{cc,hh},sandbox/*.py,khmer/_khmer.cc,tests/test_*.py: Simplify
exception hierarchy, and ensure all C++ exceptions are converted to python
errors.
* scripts/normalize-by-median.py: Clarify error message.
* tests/khmer_tst_utils.py: Add longify function, converts int => long on
py2, and passes thru list unmodified on py3.
2015-06-30 Jacob Fenton <[email protected]>
* tests/{test_script_arguments,test_functions}.py: changed tests to use
stderr redirection to prevent leaks
* tests/test_normalize_by_median.py: changed to not duplicate a test
* tests/test_script_arguments.py: changed tests to use stderr redirection
2015-06-30 Titus Brown <[email protected]>
* tests/test_normalize_by_median.py: disabled running
test_normalize_by_median_report_fp during normal test running.
2015-06-30 Titus Brown <[email protected]>
* khmer/khmer_args.py: removed incorrect warning for default max_tablesize
when -M is used.
* tests/test_scripts.py: added test for correct max_tablesize behavior.
2015-06-30 Titus Brown <[email protected]>
* setup.cfg: changed 'stop=TRUE' to 'stop=FALSE', so that tests do not
stop running at first failure.
2015-06-30 Kevin Murray <[email protected]>
* scripts/{extract-paired-reads,split-paired-reads}.py: Fix creation of
default output files even when output files were provided on CLI.
2015-06-29 Sherine Awad <[email protected]>
* khmer/utils.py: Fix bug in naming in interleave-reads.py
* tests/test_scripts.py: Add a test function for the new behavior
* tests/test-data/*.fq: Add 3 test files needed for the testing
2015-06-28 Jacob Fenton <[email protected]>
* tests/test_sandbox_scripts.py: made error more informative and not crashy
* sandbox/{estimate_optimal_hash,optimal_args_hashbits}.py: minor cleanups
2015-06-28 Qingpeng Zhang <[email protected]>
* sandbox/{estimate_optimal_hash,optimal_args_hashbits}.py: added sandbox
methods for estimating memory usage based on desired fp rate, etc.
2015-06-27 Kevin Murray <[email protected]>
* doc/dev/binary-file-formats.rst: Fix issue in ksize documentation for
Countgraph
2015-06-27 Kevin Murray <[email protected]>
* README.rst: Fix link to virtualenv installation instructions.
2015-06-19 Titus Brown <[email protected]>
* khmer/__init__.py: split CountingHash into _CountingHash (CPython) and
CountingHash to mimic Hashbits behavior; pass IOError through
extract_countinghash_info and extract_hashbits_info so that
file-does-not-exist errors are correctly reported; fixed FP rate reporting;
changed to using get_n_primes_near_x to build hashtable sizes; removed
get_n_primes_above_x, new_hashbits, and new_counting_hash functions.
* khmer/_khmer.cc: changed tp_flags for KCountingHash so that it could
be a base class.
* khmer/khmer_args.py: removed environment variable override for hash size
defaults; added -M/--max_memory_usage, and functions create_nodegraph()
and create_countgraph(). Also renamed --min-tablesize to --max-tablesize.
* khmer/kfile.py: fixed check_space_for_hashtable to depend on args obj.
* oxli/build_graph.py, scripts/{annotate-partitions.py,count-overlap.py,
do-partition.py,filter-stoptags.py,
merge-partitions.py}, sandbox/{assembly-diff.py,assembly-diff-2.py,
bloom-count-intersection.py,bloom-count.py,build-sparse-graph.py,
collect-reads.py,saturate-by-median.py, graph-size.py,print-stoptags.py,
print-tagset.py,stoptags-by-position.py, subset-report.py,
sweep-out-reads-with-contigs.py,sweep-reads2.py,sweep-reads3.py}: changed
hashtype over to 'nodegraph' and 'countgraph' in call to report_on_config;
replaced counting hash/hashbits creation with new khmer_args create*
functions, and/or new_counting_hash/new_hashbits with CountingHash/Hashbits.
* doc/scripts.rst: updated hashtable size help text.
* doc/whats-new-2.0.rst: updated with description of -M/--max-memory-usage.
* tests/test*.py: switched from new_counting_hash to CountingHash, and
new_hashbits to Hashbits; adjusts tests for new behavior of hashtable
size calculation.
* tests/test_hashbits_obj.py: merged into test_hashbits.py and removed file.
* tests/test_script_arguments.py: updated for new check_space_for_hashtable
behavior; added tests for create_countgraph and create_nodegraph.
* tests/test_counting_single.py: fixed countgraph size & palindrome testing
beahavior in test_complete_no_collision.
2015-06-19 Titus Brown <[email protected]>
* Makefile: temporarily disable 'huge' tests on Linux.
2015-06-17 Titus Brown <[email protected]>
* scripts/normalize-by-median.py: changed DEFAULT_DESIRED_COVERAGE to 20,
and corrected options help.
* tests/{test_scripts.py,test_normalize_by_median.py}: moved
normalize-by-median.py tests into a their own file.
* tests/test-data/{dn-test-all-paired-all-keep.fa,dn-test-none-paired.fa,
dn-test-some-paired-all-keep.fa}: added test data files for specific
pairing/saturation behavior.
2015-06-16 Kevin Murray <[email protected]>
* doc/dev/binary-file-formats.rst: Add documentation of khmer's binary file
formats.
* doc/dev/index.rst: Add above docs to developer documentation index.
2015-06-14 Michael R. Crusoe <[email protected]>
* khmer/__init__.py,lib/{counting,hashbits,hashtable,subset,labelhash}.cc,
lib/khmer.hh: add signature to beginning of all binary file types
* tests/test-data/{normC20k20.ct,badversion-k32.tagset,
goodversion-k32.tagset}: update to new format by prepending "OXLI" to the
data stream
* tests/test_{counting_hash,functions,scripts,hashbits,hashbits_obj,
labelhash}.py: tests should fail, not error (add try, except + assert
blocks). Adapted other tests to cope with the new file formats
* lib/magic: new, teaches the unix `file` command about khmer file types
* doc/index.rst,doc/whats-new-2.0.rst: document these changes
2015-06-14 Titus Brown <[email protected]>
* scripts/extract-paired-reads.py: added --output_dir, --paired-output,
and --single-output arguments to change output file details; script
now accepts stdin, and will output to stdout upon request.
* scripts/split-paired-reads.py: changed script to output to stdout upon
request; added '-' as stdin input.
* tests/test_scripts.py: added tests for new extract-paired-reads.py
behavior.
2015-06-14 Titus Brown <[email protected]>
* tests/test_counting_hash.py: fixed duplicated test
'get_kmer_counts_too_short' by changing to 'get_kmer_hashes_too_short'.
2015-06-14 Jacob Fenton <[email protected]>
* scripts/abundance-dist.py: added weird bigcount circumstance detection
* tests/test_scripts.py: added test for the above
2015-06-14 Kevin Murray <[email protected]>
* lib/counting.cc: Fix infinite loop in gzipped CountingHash I/O
* tests/test_counting_hash.py: Add test of large CountingHash I/O
* setup.cfg: Skip tests with the 'huge' label by default
2015-06-13 Michael R. Crusoe <[email protected]>
* Makefile, build-jenkins.sh: unify sphinx dependencies
* scripts/readstats.py: fix typo
2015-06-13 Titus Brown <[email protected]>
* doc/dev/getting-started.rst: update instructions for creating a new
branch name to preferred practice (fix/brief_issue_description, instead
of fix/issuenum).
2015-06-13 Michael R. Crusoe <[email protected]>
* doc/dev/release.rst: remove false positive from version check
* tests/test_{counting_hash,scripts}.py: remove scriptpath no-op method
2015-06-12 Luiz Irber <[email protected]>
* setup.py: revert changes to zlib compilation.
* setup.cfg: nose should stop on first error by default.
* Makefile, tests/test_threaded_sequence_processor.py,
scripts/{do-partition,partition-graph}.py, khmer/thread_utils.py: Remove
dependency on future package.
2015-06-12 Michael R. Crusoe <[email protected]>
* setup.py: update screed version to 0.9
2015-06-12 Luiz Irber <[email protected]>
* *.py: refactor for Python 3 compatibility. Clear separation of Unicode
and Byte strings, use __future__ imports for compatibility (print function,
absolute imports, unicode_literals), fix tests to consider changes to random
number generator between Python versions.
* khmer/_khmer.cc: rename file, methods return Unicode strings instead of
Bytestrings.
2015-06-12 Luiz Irber <[email protected]>
* khmer/{khmermodule.cc},tests/test_hashbits.py: Add Unicode support to
hashbits.get method.
* tests/test_hll.py: Avoid using translate for revcomp calculation.
2015-06-12 Sarah Guermond <[email protected]>
* scripts/trim-low-abund.py: changed _screed_record_dict to Record
2015-06-11 Sherine Awad <[email protected]>
* Change split-paired-reads.py to accept input from stdin.
* Add test function to test new behavior of split-paired.
2015-06-10 Camille Scott <[email protected]>
* lib/hashtable.cc: Tweaked median_at_least to reduce number of
conditional checks.
2015-06-10 Titus Brown <[email protected]>
* scripts/find-knots.py: fixed invocation of check_space to take correct
arguments.
* tests/test_scripts.py: added simple test of find-knots.py execution.
2015-06-09 Jacob Fenton <[email protected]>
* scripts/normalize-by-median.py: implemented broken_paired_reader
* tests/test_scripts.py: modified tests to properly use new args
* khmer/utils.py: added force-paired option to broken_paired_reader (@ctb)
2015-06-09 Luiz Irber <[email protected]>
* khmer/_khmermodule.cc, lib/hashtable.{cc,hh}: astyle fixes.
2015-06-09 Titus Brown <[email protected]>
* khmer/_khmermodule.cc: fixed nasty Hashtable.get() bug.
* lib/hashtable.{cc,hh}: add Hashtable::get_kmers(), get_kmer_hashes(),
and get_kmer_counts().
* khmer/_khmermodule.cc: add CPython functions for get_kmers(),
get_kmer_hashes(), and get_kmer_counts(); reorganize hashtable_methods.
* tests/test_counting_hash.py: add tests for get_kmers(), get_kmer_hashes(),
and get_kmer_counts(), as well as for nasty Hashtable.get() bug.
2015-06-08 Camille Scott <[email protected]>
* lib/hashtable.{cc,hh}: Add filter_on_median method to check
if median k-mer count is above a cutoff
* khmer/_khmermodule.cc: Expose filter_on_median to python-land
* scripts/normalize-by-median.py: Switch to new filter_on_median
* tests/test_counting_hash.py: Tests for new method
2015-06-08 Luiz Irber <[email protected]>
* tests/test_hll.py: test return values from consume_{string,fasta}.
2015-06-06 Titus Brown <[email protected]>
* khmer/_khmermodule.cc: added hllcounter_merge.
* tests/test_hll.py: added merge tests.
* lib/hllcounter.cc: changed HLLCounter::consume_string to uppercase input.
* sandbox/unique-kmers.py: added --stream-out option; updated to print out
k-mers per file as well as k-mer size used.
2015-06-04 Titus Brown <[email protected]>
* khmer/_khmermodule.cc: added error handling to load_partitionmap.
* lib/subset.cc: modified partitionmap format to detect truncated files;
changed untestable sanity checks to assertions.
* tests/{test_counting_hash,test_hashbits,test_subset_graph}.py: added
tests to try loading all possible truncations of binary save files.
2015-06-04 Titus Brown <[email protected]>
* khmer/_khmermodule.cc,lib/hashbits.{cc,hh}: add Hashbits::update_from()
and Hashbits.update().
* tests/test_hashbits.py: associated tests.
2015-06-01 Jacob Fenton <[email protected]>
* scripts/normalize-by-median.py: major refactoring to use context
managers and classes; fixed -R
* tests/test_scripts.py: added test for normalize's -R arg
2015-06-01 Tamer Mansour <[email protected]>
* scripts/normalize-by-median.py: changed to count kmers from both PE reads
when either one of them is below the coverage cutoff
* tests/test_scripts.py: Added test for new behaviour
2015-05-26 Titus Brown <[email protected]>
* khmer/_khmermodule.cc: refactor CPython layer so that KHashtable
is at base of CountingHash and Hashbits.
* lib/hashbits.hh: add n_entries() function from Hashtable::n_entries.
* lib/hashtable.hh: add several virtual functions to Hashtable that exist in
CountingHash and Hashbits.
2015-05-26 Titus Brown <[email protected]>
* khmer/{__init__.py,_khmermodule.cc},lib/labelhash.{cc,hh},
lib/{hashtable,khmer}.hh: changed LabelHash to be a "friend" of Hashtable,
rather than a subclass; allowed initialization with either a CountingHash
or a Hashbits; added 'graph' attribute to the Python object to store a
reference to host object.
* lib/labelhash.{cc,hh}: changed TagPtr maps to Tag maps to fix disastrous
bug.
* lib/labelhash.{cc,hh}: added save/load_tags_and_labels functions for
saving and loading labels.
* tests/test_labelhash.py: removed unnecessary tests; added tests for save
and load.
* sandbox/sweep-reads.py: updated with LabelHash changes.
2015-05-26 Kevin Murray <[email protected]>
* lib/Makefile: Remove old libkhmer.so versions during make clean
2015-05-25 Kevin Murray <[email protected]>
* Makefile: Fix issue with 'lib' target not building by using FORCE
2015-05-20 Jacob Fenton <[email protected]>
* oxli/{__init__,khmer_api,common}.py,scripts/build-graph.py,
tests/test_scripts.py: added oxli module, oxlified load_graph script, tests
* scripts/load-graph.py: replaced with oxlified version
* setup.py: added oxli module and entry point
2015-05-20 Kevin Murray <[email protected]>
* .gitignore: Add htmlcov/ and diff-cover.html to gitignore
* Makefile: Use rm -f to remove files to quash error messages on
non-existant files
2015-05-18 Sherine Awad <[email protected]>
* tests/test_scripts.py: Test loading of compressed counting table
with bigcounts,and test abundance with bigcounts
2015-05-18 Michael R. Crusoe <[email protected]>
* all files: references to github.com/ged-lab changed to
github.com/dib-lab. All GitHub URLs normalized to use HTTPS
* README.rst: broken landscape.io badge removed
* doc/user/known-issues.rst: removed two known issues fixed in v1.4 release
2015-05-18 Titus Brown <[email protected]>
* sandbox/{assembly-diff-2.py,sandbox/collect-reads.py},
scripts/{count-median.py,filter-abund-single.py,filter-abund.py}: changed
sequence-reading behavior to replace 'N' with 'A', to be consistent with
rest of code base.
* scripts/{filter-abund.py,filter-abund-single.py}: changed behavior of
scripts to keep sequences with 'N's in them, and count them as 'A's.
* tests/test_scripts.py: added tests for new
filter-abund/filter-abund-single behavior.
* tests/test-data/test-filter-abund-Ns.fq: new test file for new tests.
2015-05-13 Scott Sievert <[email protected]>
* tests/*,scripts/*,lib/*,sandbox/*,khmer/*: changed "doc/LICENSE.txt" to
"LICENSE" in copyright header.
2015-05-13 Michael R. Crusoe <[email protected]>
* doc/dev/getting-started.rst: added missing dev tools to install list
2015-05-12 Kevin Murray <[email protected]>
* scripts/load-into-counting.py,test/test_scripts.py: Add the number of
reads processed to the machine readable output files of --summary-info.
2015-05-11 Titus Brown <[email protected]>
* scripts/sample-reads-randomly.py: fixed boundary error in
sample-reads-randomly.py.
* tests/test_scripts.py: updated tests to correspond with correct
behavior of sample-reads-randomly.py.
2015-04-23 Lex Nederbragt <[email protected]>
* tests/test_scripts.py: added a test for extract-partitions:
whitespace in fasta header.
2015-04-21 Daniel Standage <[email protected]>
* scripts/sample-reads-randomly.py: use broken paired reader to provide
paired-end read support.
* tests/test_scripts.py: change test results to compensate for the change in
implementation.
2015-04-17 Jessica Mizzi <[email protected]>