csvtk:-
+
- add the shortcut
-Xfor the flag--infile-list. #249
+
csvtk pretty:-
-
- fix the bug of empty first row with
-H/--no-header-row, introduced in v0.27.0.
- - new style
3linefor three-line table.
+ - support field ranges for
-m/--align-centerand-r/--align-right. #244
csvtk csv2xlsx:-
-
- binaries compiled with go1.21 would result in a broken xlsx file. #243 +
csvtk spread:-
+
- support values sharing the same keys. #248
-csvtk splitxlsx:-
-
- fix the error of
invalid worksheet index. #1617
+ csvtk join:-
+
- a new flag
-P/--prefix-duplicates: add filenames as colname prefixes only for duplicated colnames. #246
+
+- a new flag
csvtk mutate2:-
+
- fix changing the order of the header row, the code was accidentally missing during code refactoring in v0.27.0. #252 +
+csvtk xlsx2csv:-
+
- fix
open /tmp/excelize-: no such file or directoryerror for big.xlsxfiles. #251
+
+- fix
csvtk comb:-
+
- fix the empty result bug for alphabet sizes greater than 64.
Links
Linux 32-bit -csvtk_linux_386.tar.gz, -
中国镜像
csvtk_linux_386.tar.gz, +
中国镜像Linux 64-bit -csvtk_linux_amd64.tar.gz, -
中国镜像
csvtk_linux_amd64.tar.gz, +
中国镜像Linux 64-bit -csvtk_linux_arm64.tar.gz, -
中国镜像
csvtk_linux_arm64.tar.gz, +
中国镜像macOS 64-bit -csvtk_darwin_amd64.tar.gz, -
中国镜像
csvtk_darwin_amd64.tar.gz, +
中国镜像macOS arm64 -csvtk_darwin_arm64.tar.gz, -
中国镜像
csvtk_darwin_arm64.tar.gz, +
中国镜像Windows 32-bit -csvtk_windows_386.exe.tar.gz, -
中国镜像
csvtk_windows_386.exe.tar.gz, +
中国镜像Windows 64-bit -csvtk_windows_amd64.exe.tar.gz, -
中国镜像
csvtk_windows_amd64.exe.tar.gz, +
中国镜像Shell-completion
Release history
-
+
- csvtk v0.27.2
+
+
csvtk pretty:-
+
- fix the bug of empty first row with
-H/--no-header-row, introduced in v0.27.0.
+ - new style
3linefor three-line table.
+
+- fix the bug of empty first row with
csvtk csv2xlsx:-
+
- binaries compiled with go1.21 would result in a broken xlsx file. #243 +
+csvtk splitxlsx:-
+
- fix the error of
invalid worksheet index. #1617
+
- fix the error of
- csvtk v0.27.1
csvtk filter2/mutate2:-
diff --git a/index.html b/index.html
index 3e7018e..aba9577 100644
--- a/index.html
+++ b/index.html
@@ -966,20 +966,20 @@
Examples
NA Robert Abel 123 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -$ csvtk pretty names.csv -S bold -┏━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┓ -┃ id ┃ first_name ┃ last_name ┃ username ┃ -┣━━━━╋━━━━━━━━━━━━╋━━━━━━━━━━━╋━━━━━━━━━━┫ -┃ 11 ┃ Rob ┃ Pike ┃ rob ┃ -┣━━━━╋━━━━━━━━━━━━╋━━━━━━━━━━━╋━━━━━━━━━━┫ -┃ 2 ┃ Ken ┃ Thompson ┃ ken ┃ -┣━━━━╋━━━━━━━━━━━━╋━━━━━━━━━━━╋━━━━━━━━━━┫ -┃ 4 ┃ Robert ┃ Griesemer ┃ gri ┃ -┣━━━━╋━━━━━━━━━━━━╋━━━━━━━━━━━╋━━━━━━━━━━┫ -┃ 1 ┃ Robert ┃ Thompson ┃ abc ┃ -┣━━━━╋━━━━━━━━━━━━╋━━━━━━━━━━━╋━━━━━━━━━━┫ -┃ NA ┃ Robert ┃ Abel ┃ 123 ┃ -┗━━━━┻━━━━━━━━━━━━┻━━━━━━━━━━━┻━━━━━━━━━━┛ +$ csvtk pretty names.csv -S bold -w 5 -m 1- +┏━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┓ +┃ id ┃ first_name ┃ last_name ┃ username ┃ +┣━━━━━━━╋━━━━━━━━━━━━╋━━━━━━━━━━━╋━━━━━━━━━━┫ +┃ 11 ┃ Rob ┃ Pike ┃ rob ┃ +┣━━━━━━━╋━━━━━━━━━━━━╋━━━━━━━━━━━╋━━━━━━━━━━┫ +┃ 2 ┃ Ken ┃ Thompson ┃ ken ┃ +┣━━━━━━━╋━━━━━━━━━━━━╋━━━━━━━━━━━╋━━━━━━━━━━┫ +┃ 4 ┃ Robert ┃ Griesemer ┃ gri ┃ +┣━━━━━━━╋━━━━━━━━━━━━╋━━━━━━━━━━━╋━━━━━━━━━━┫ +┃ 1 ┃ Robert ┃ Thompson ┃ abc ┃ +┣━━━━━━━╋━━━━━━━━━━━━╋━━━━━━━━━━━╋━━━━━━━━━━┫ +┃ NA ┃ Robert ┃ Abel ┃ 123 ┃ +┗━━━━━━━┻━━━━━━━━━━━━┻━━━━━━━━━━━┻━━━━━━━━━━┛-
diff --git a/search/search_index.json b/search/search_index.json
index d6a0ef4..795a625 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -1 +1 @@
-{"config":{"indexing":"full","lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"csvtk - a cross-platform, efficient and practical CSV/TSV toolkit Documents: http://bioinf.shenwei.me/csvtk ( Usage and Tutorial ). \u4e2d\u6587\u4ecb\u7ecd Source code: https://github.com/shenwei356/csvtk Latest version: Introduction Similar to FASTA/Q format in field of Bioinformatics, CSV/TSV formats are basic and ubiquitous file formats in both Bioinformatics and data science. People usually use spreadsheet software like MS Excel to process table data. However this is all by clicking and typing, which is not automated and is time-consuming to repeat , especially when you want to apply similar operations with different datasets or purposes. You can also accomplish some CSV/TSV manipulations using shell commands, but more code is needed to handle the header line. Shell commands do not support selecting columns with column names either. csvtk is convenient for rapid data investigation and also easy to integrate into analysis pipelines . It could save you lots of time in (not) writing Python/R scripts. Table of Contents Features Subcommands Installation Command-line completion Compared to csvkit Examples Acknowledgements Contact License Starchart Features Cross-platform (Linux/Windows/Mac OS X/OpenBSD/FreeBSD) Light weight and out-of-the-box, no dependencies, no compilation, no configuration Fast , multiple-CPUs supported (some commands) Practical functions provided by N subcommands Support STDIN and gziped input/output file, easy being used in pipe Most of the subcommands support unselecting fields and fuzzy fields , e.g. -f \"-id,-name\" for all fields except \"id\" and \"name\", -F -f \"a.*\" for all fields with prefix \"a.\". Support some common plots (see usage ) Seamlessly support for data with meta line (e.g., sep=, ) of separator declaration used by MS Excel Subcommands 51 subcommands in total. Information headers : prints headers dim : dimensions of CSV file nrow : print number of records ncol : print number of columns summary : summary statistics of selected numeric or text fields (groupby group fields) watch : online monitoring and histogram of selected field corr : calculate Pearson correlation between numeric columns Format conversion pretty : converts CSV to a readable aligned table csv2tab : converts CSV to tabular format tab2csv : converts tabular format to CSV space2tab : converts space delimited format to TSVW transpose : transposes CSV data csv2md : converts CSV to markdown format csv2rst : converts CSV to reStructuredText format csv2json : converts CSV to JSON format csv2xlsx : converts CSV/TSV files to XLSX file xlsx2csv : converts XLSX to CSV format Set operations head : prints first N records concat : concatenates CSV/TSV files by rows sample : sampling by proportion cut : select and arrange fields grep : greps data by selected fields with patterns/regular expressions uniq : unique data without sorting freq : frequencies of selected fields inter : intersection of multiple files filter : filters rows by values of selected fields with arithmetic expression filter2 : filters rows by awk-like arithmetic/string expressions join : join files by selected fields (inner, left and outer join) split splits CSV/TSV into multiple files according to column values splitxlsx : splits XLSX sheet into multiple sheets according to column values comb : compute combinations of items at every row Edit fix : fix CSV/TSV with different numbers of columns in rows add-header : add column names del-header : delete column names rename : renames column names with new names rename2 : renames column names by regular expression replace : replaces data of selected fields by regular expression round : round float to n decimal places mutate : creates new columns from selected fields by regular expression mutate2 : creates a new column from selected fields by awk-like arithmetic/string expressions sep : separate column into multiple columns gather : gather columns into key-value pairs, like tidyr::gather/pivot_longer spread : spread a key-value pair across multiple columns, like tidyr::spread/pivot_wider unfold : unfold multiple values in cells of a field fold : fold multiple values of a field into cells of groups fmtdate : format date of selected fields Ordering sort : sorts by selected fields Ploting plot see usage plot hist histogram plot box boxplot plot line line plot and scatter plot Misc cat stream file and report progress version print version information and check for update genautocomplete generate shell autocompletion script (bash|zsh|fish|powershell) Installation Download Page csvtk is implemented in Go programming language, executable binary files for most popular operating systems are freely available in release page. Method 1: Download binaries (latest stable/dev version) Just download compressed executable file of your operating system, and decompress it with tar -zxvf *.tar.gz command or other tools. And then: For Linux-like systems If you have root privilege simply copy it to /usr/local/bin : sudo cp csvtk /usr/local/bin/ Or copy to anywhere in the environment variable PATH : mkdir -p $HOME/bin/; cp csvtk $HOME/bin/ For windows , just copy csvtk.exe to C:\\WINDOWS\\system32 . Method 2: Install via conda (latest stable version) conda install -c bioconda csvtk Method 3: Install via homebrew brew install csvtk Method 4: For Go developer (latest stable/dev version) go get -u github.com/shenwei356/csvtk/csvtk Method 5: For ArchLinux AUR users (may be not the latest) yaourt -S csvtk Command-line completion Bash: # generate completion shell csvtk genautocomplete --shell bash # configure if never did. # install bash-completion if the \"complete\" command is not found. echo \"for bcfile in ~/.bash_completion.d/* ; do source \\$bcfile; done\" >> ~/.bash_completion echo \"source ~/.bash_completion\" >> ~/.bashrc Zsh: # generate completion shell csvtk genautocomplete --shell zsh --file ~/.zfunc/_csvtk # configure if never did echo 'fpath=( ~/.zfunc \"${fpath[@]}\" )' >> ~/.zshrc echo \"autoload -U compinit; compinit\" >> ~/.zshrc fish: csvtk genautocomplete --shell fish --file ~/.config/fish/completions/csvtk.fish Compared to csvkit csvkit , attention: this table wasn't updated for 2 years. Features csvtk csvkit Note Read Gzip Yes Yes read gzip files Fields ranges Yes Yes e.g. -f 1-4,6 Unselect fileds Yes -- e.g. -1 for excluding first column Fuzzy fields Yes -- e.g. ab* for columns with name prefix \"ab\" Reorder fields Yes Yes it means -f 1,2 is different from -f 2,1 Rename columns Yes -- rename with new name(s) or from existed names Sort by multiple keys Yes Yes bash sort like operations Sort by number Yes -- e.g. -k 1:n Multiple sort Yes -- e.g. -k 2:r -k 1:nr Pretty output Yes Yes convert CSV to readable aligned table Unique data Yes -- unique data of selected fields frequency Yes -- frequencies of selected fields Sampling Yes -- sampling by proportion Mutate fields Yes -- create new columns from selected fields Replace Yes -- replace data of selected fields Similar tools: csvkit - A suite of utilities for converting to and working with CSV, the king of tabular file formats. http://csvkit.rtfd.org/ xsv - A fast CSV toolkit written in Rust. miller - Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV and tabular JSON http://johnkerl.org/miller tsv-utils - Command line utilities for tab-separated value files written in the D programming language. Examples More examples and tutorial . Attention The CSV parser requires all the lines have same number of fields/columns. Even lines with spaces will cause error. Use '-I/--ignore-illegal-row' to skip these lines if neccessary. By default, csvtk thinks your files have header row, if not, switch flag -H on. Column names better be unique. By default, lines starting with # will be ignored, if the header row starts with # , please assign flag -C another rare symbol, e.g. '$' . By default, csvtk handles CSV files, use flag -t for tab-delimited files. If \" exists in tab-delimited files, use flag -l . Do not mix use field (column) numbers and names. Examples Pretty result $ csvtk pretty names.csv id first_name last_name username -- ---------- --------- -------- 11 Rob Pike rob 2 Ken Thompson ken 4 Robert Griesemer gri 1 Robert Thompson abc NA Robert Abel 123 $ csvtk pretty names.csv -S 3line \u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501 id first_name last_name username ---------------------------------------- 11 Rob Pike rob 2 Ken Thompson ken 4 Robert Griesemer gri 1 Robert Thompson abc NA Robert Abel 123 \u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501 $ csvtk pretty names.csv -S bold \u250f\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513 \u2503 id \u2503 first_name \u2503 last_name \u2503 username \u2503 \u2523\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u252b \u2503 11 \u2503 Rob \u2503 Pike \u2503 rob \u2503 \u2523\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u252b \u2503 2 \u2503 Ken \u2503 Thompson \u2503 ken \u2503 \u2523\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u252b \u2503 4 \u2503 Robert \u2503 Griesemer \u2503 gri \u2503 \u2523\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u252b \u2503 1 \u2503 Robert \u2503 Thompson \u2503 abc \u2503 \u2523\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u252b \u2503 NA \u2503 Robert \u2503 Abel \u2503 123 \u2503 \u2517\u2501\u2501\u2501\u2501\u253b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u253b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u253b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u251b Summary of selected numeric fields, supporting \"group-by\" $ cat testdata/digitals2.csv \\ | csvtk summary -i -f f4:sum,f5:sum -g f1,f2 \\ | csvtk pretty f1 f2 f4:sum f5:sum bar xyz 7.00 106.00 bar xyz2 4.00 4.00 foo bar 6.00 3.00 foo bar2 4.50 5.00 Select fields/columns ( cut ) By index: csvtk cut -f 1,2 By names: csvtk cut -f first_name,username Unselect : csvtk cut -f -1,-2 or csvtk cut -f -first_name Fuzzy fields : csvtk cut -F -f \"*_name,username\" Field ranges: csvtk cut -f 2-4 for column 2,3,4 or csvtk cut -f -3--1 for discarding column 1,2,3 All fields: csvtk cut -f 1- or csvtk cut -F -f \"*\" Search by selected fields ( grep ) (matched parts will be highlighted as red) By exactly matching: csvtk grep -f first_name -p Robert -p Rob By regular expression: csvtk grep -f first_name -r -p Rob By pattern list: csvtk grep -f first_name -P name_list.txt Remore rows containing missing data (NA): csvtk grep -F -f \"*\" -r -p \"^$\" -v Rename column names ( rename and rename2 ) Setting new names: csvtk rename -f A,B -n a,b or csvtk rename -f 1-3 -n a,b,c Replacing with original names by regular express: csvtk rename2 -f 1- -p \"(.*)\" -r 'prefix_$1' for adding prefix to all column names. Edit data with regular expression ( replace ) Remove Chinese charactors: csvtk replace -F -f \"*_name\" -p \"\\p{Han}+\" -r \"\" Create new column from selected fields by regular expression ( mutate ) In default, copy a column: csvtk mutate -f id Extract prefix of data as group name (get \"A\" from \"A.1\" as group name): csvtk mutate -f sample -n group -p \"^(.+?)\\.\" --after sample Sort by multiple keys ( sort ) By single column : csvtk sort -k 1 or csvtk sort -k last_name By multiple columns: csvtk sort -k 1,2 or csvtk sort -k 1 -k 2 or csvtk sort -k last_name,age Sort by number: csvtk sort -k 1:n or csvtk sort -k 1:nr for reverse number Complex sort: csvtk sort -k region -k age:n -k id:nr In natural order: csvtk sort -k chr:N Join multiple files by keys ( join ) All files have same key column: csvtk join -f id file1.csv file2.csv Files have different key columns: csvtk join -f \"username;username;name\" names.csv phone.csv adress.csv -k Filter by numbers ( filter ) Single field: csvtk filter -f \"id>0\" Multiple fields : csvtk filter -f \"1-3>0\" Using --any to print record if any of the field satisfy the condition: csvtk filter -f \"1-3>0\" --any fuzzy fields : csvtk filter -F -f \"A*!=0\" Filter rows by awk-like arithmetic/string expressions ( filter2 ) Using field index: csvtk filter2 -f '$3>0' Using column names: csvtk filter2 -f '$id > 0' Both arithmetic and string expressions: csvtk filter2 -f '$id > 3 || $username==\"ken\"' More complicated: csvtk filter2 -H -t -f '$1 > 2 && $2 % 2 == 0' Ploting plot histogram with data of the second column: csvtk -t plot hist testdata/grouped_data.tsv.gz -f 2 | display plot boxplot with data of the \"GC Content\" (third) column, group information is the \"Group\" column. csvtk -t plot box testdata/grouped_data.tsv.gz -g \"Group\" \\ -f \"GC Content\" --width 3 | display plot horiz boxplot with data of the \"Length\" (second) column, group information is the \"Group\" column. csvtk -t plot box testdata/grouped_data.tsv.gz -g \"Group\" -f \"Length\" \\ --height 3 --width 5 --horiz --title \"Horiz box plot\" | display plot line plot with X-Y data csvtk -t plot line testdata/xy.tsv -x X -y Y -g Group | display plot scatter plot with X-Y data csvtk -t plot line testdata/xy.tsv -x X -y Y -g Group --scatter | display Acknowledgements We are grateful to Zhiluo Deng and Li Peng for suggesting features and reporting bugs. Thanks Albert Vilella for features suggestion, which makes csvtk feature-rich\u3002 Contact Create an issue to report bugs, propose new functions or ask for help. Or leave a comment . License MIT License Starchart","title":"Home"},{"location":"#csvtk-a-cross-platform-efficient-and-practical-csvtsv-toolkit","text":"Documents: http://bioinf.shenwei.me/csvtk ( Usage and Tutorial ). \u4e2d\u6587\u4ecb\u7ecd Source code: https://github.com/shenwei356/csvtk Latest version:","title":"csvtk - a cross-platform, efficient and practical CSV/TSV toolkit"},{"location":"#introduction","text":"Similar to FASTA/Q format in field of Bioinformatics, CSV/TSV formats are basic and ubiquitous file formats in both Bioinformatics and data science. People usually use spreadsheet software like MS Excel to process table data. However this is all by clicking and typing, which is not automated and is time-consuming to repeat , especially when you want to apply similar operations with different datasets or purposes. You can also accomplish some CSV/TSV manipulations using shell commands, but more code is needed to handle the header line. Shell commands do not support selecting columns with column names either. csvtk is convenient for rapid data investigation and also easy to integrate into analysis pipelines . It could save you lots of time in (not) writing Python/R scripts.","title":"Introduction"},{"location":"#table-of-contents","text":"Features Subcommands Installation Command-line completion Compared to csvkit Examples Acknowledgements Contact License Starchart","title":"Table of Contents"},{"location":"#features","text":"Cross-platform (Linux/Windows/Mac OS X/OpenBSD/FreeBSD) Light weight and out-of-the-box, no dependencies, no compilation, no configuration Fast , multiple-CPUs supported (some commands) Practical functions provided by N subcommands Support STDIN and gziped input/output file, easy being used in pipe Most of the subcommands support unselecting fields and fuzzy fields , e.g. -f \"-id,-name\" for all fields except \"id\" and \"name\", -F -f \"a.*\" for all fields with prefix \"a.\". Support some common plots (see usage ) Seamlessly support for data with meta line (e.g., sep=, ) of separator declaration used by MS Excel","title":"Features"},{"location":"#subcommands","text":"51 subcommands in total. Information headers : prints headers dim : dimensions of CSV file nrow : print number of records ncol : print number of columns summary : summary statistics of selected numeric or text fields (groupby group fields) watch : online monitoring and histogram of selected field corr : calculate Pearson correlation between numeric columns Format conversion pretty : converts CSV to a readable aligned table csv2tab : converts CSV to tabular format tab2csv : converts tabular format to CSV space2tab : converts space delimited format to TSVW transpose : transposes CSV data csv2md : converts CSV to markdown format csv2rst : converts CSV to reStructuredText format csv2json : converts CSV to JSON format csv2xlsx : converts CSV/TSV files to XLSX file xlsx2csv : converts XLSX to CSV format Set operations head : prints first N records concat : concatenates CSV/TSV files by rows sample : sampling by proportion cut : select and arrange fields grep : greps data by selected fields with patterns/regular expressions uniq : unique data without sorting freq : frequencies of selected fields inter : intersection of multiple files filter : filters rows by values of selected fields with arithmetic expression filter2 : filters rows by awk-like arithmetic/string expressions join : join files by selected fields (inner, left and outer join) split splits CSV/TSV into multiple files according to column values splitxlsx : splits XLSX sheet into multiple sheets according to column values comb : compute combinations of items at every row Edit fix : fix CSV/TSV with different numbers of columns in rows add-header : add column names del-header : delete column names rename : renames column names with new names rename2 : renames column names by regular expression replace : replaces data of selected fields by regular expression round : round float to n decimal places mutate : creates new columns from selected fields by regular expression mutate2 : creates a new column from selected fields by awk-like arithmetic/string expressions sep : separate column into multiple columns gather : gather columns into key-value pairs, like tidyr::gather/pivot_longer spread : spread a key-value pair across multiple columns, like tidyr::spread/pivot_wider unfold : unfold multiple values in cells of a field fold : fold multiple values of a field into cells of groups fmtdate : format date of selected fields Ordering sort : sorts by selected fields Ploting plot see usage plot hist histogram plot box boxplot plot line line plot and scatter plot Misc cat stream file and report progress version print version information and check for update genautocomplete generate shell autocompletion script (bash|zsh|fish|powershell)","title":"Subcommands"},{"location":"#installation","text":"Download Page csvtk is implemented in Go programming language, executable binary files for most popular operating systems are freely available in release page.","title":"Installation"},{"location":"#method-1-download-binaries-latest-stabledev-version","text":"Just download compressed executable file of your operating system, and decompress it with tar -zxvf *.tar.gz command or other tools. And then: For Linux-like systems If you have root privilege simply copy it to /usr/local/bin : sudo cp csvtk /usr/local/bin/ Or copy to anywhere in the environment variable PATH : mkdir -p $HOME/bin/; cp csvtk $HOME/bin/ For windows , just copy csvtk.exe to C:\\WINDOWS\\system32 .","title":"Method 1: Download binaries (latest stable/dev version)"},{"location":"#method-2-install-via-conda-latest-stable-version","text":"conda install -c bioconda csvtk","title":"Method 2: Install via conda (latest stable version)"},{"location":"#method-3-install-via-homebrew","text":"brew install csvtk","title":"Method 3: Install via homebrew"},{"location":"#method-4-for-go-developer-latest-stabledev-version","text":"go get -u github.com/shenwei356/csvtk/csvtk","title":"Method 4: For Go developer (latest stable/dev version)"},{"location":"#method-5-for-archlinux-aur-users-may-be-not-the-latest","text":"yaourt -S csvtk","title":"Method 5: For ArchLinux AUR users (may be not the latest)"},{"location":"#command-line-completion","text":"Bash: # generate completion shell csvtk genautocomplete --shell bash # configure if never did. # install bash-completion if the \"complete\" command is not found. echo \"for bcfile in ~/.bash_completion.d/* ; do source \\$bcfile; done\" >> ~/.bash_completion echo \"source ~/.bash_completion\" >> ~/.bashrc Zsh: # generate completion shell csvtk genautocomplete --shell zsh --file ~/.zfunc/_csvtk # configure if never did echo 'fpath=( ~/.zfunc \"${fpath[@]}\" )' >> ~/.zshrc echo \"autoload -U compinit; compinit\" >> ~/.zshrc fish: csvtk genautocomplete --shell fish --file ~/.config/fish/completions/csvtk.fish","title":"Command-line completion"},{"location":"#compared-to-csvkit","text":"csvkit , attention: this table wasn't updated for 2 years. Features csvtk csvkit Note Read Gzip Yes Yes read gzip files Fields ranges Yes Yes e.g. -f 1-4,6 Unselect fileds Yes -- e.g. -1 for excluding first column Fuzzy fields Yes -- e.g. ab* for columns with name prefix \"ab\" Reorder fields Yes Yes it means -f 1,2 is different from -f 2,1 Rename columns Yes -- rename with new name(s) or from existed names Sort by multiple keys Yes Yes bash sort like operations Sort by number Yes -- e.g. -k 1:n Multiple sort Yes -- e.g. -k 2:r -k 1:nr Pretty output Yes Yes convert CSV to readable aligned table Unique data Yes -- unique data of selected fields frequency Yes -- frequencies of selected fields Sampling Yes -- sampling by proportion Mutate fields Yes -- create new columns from selected fields Replace Yes -- replace data of selected fields Similar tools: csvkit - A suite of utilities for converting to and working with CSV, the king of tabular file formats. http://csvkit.rtfd.org/ xsv - A fast CSV toolkit written in Rust. miller - Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV and tabular JSON http://johnkerl.org/miller tsv-utils - Command line utilities for tab-separated value files written in the D programming language.","title":"Compared to csvkit"},{"location":"#examples","text":"More examples and tutorial . Attention The CSV parser requires all the lines have same number of fields/columns. Even lines with spaces will cause error. Use '-I/--ignore-illegal-row' to skip these lines if neccessary. By default, csvtk thinks your files have header row, if not, switch flag -H on. Column names better be unique. By default, lines starting with # will be ignored, if the header row starts with # , please assign flag -C another rare symbol, e.g. '$' . By default, csvtk handles CSV files, use flag -t for tab-delimited files. If \" exists in tab-delimited files, use flag -l . Do not mix use field (column) numbers and names. Examples Pretty result $ csvtk pretty names.csv id first_name last_name username -- ---------- --------- -------- 11 Rob Pike rob 2 Ken Thompson ken 4 Robert Griesemer gri 1 Robert Thompson abc NA Robert Abel 123 $ csvtk pretty names.csv -S 3line \u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501 id first_name last_name username ---------------------------------------- 11 Rob Pike rob 2 Ken Thompson ken 4 Robert Griesemer gri 1 Robert Thompson abc NA Robert Abel 123 \u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501 $ csvtk pretty names.csv -S bold \u250f\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513 \u2503 id \u2503 first_name \u2503 last_name \u2503 username \u2503 \u2523\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u252b \u2503 11 \u2503 Rob \u2503 Pike \u2503 rob \u2503 \u2523\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u252b \u2503 2 \u2503 Ken \u2503 Thompson \u2503 ken \u2503 \u2523\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u252b \u2503 4 \u2503 Robert \u2503 Griesemer \u2503 gri \u2503 \u2523\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u252b \u2503 1 \u2503 Robert \u2503 Thompson \u2503 abc \u2503 \u2523\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u252b \u2503 NA \u2503 Robert \u2503 Abel \u2503 123 \u2503 \u2517\u2501\u2501\u2501\u2501\u253b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u253b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u253b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u251b Summary of selected numeric fields, supporting \"group-by\" $ cat testdata/digitals2.csv \\ | csvtk summary -i -f f4:sum,f5:sum -g f1,f2 \\ | csvtk pretty f1 f2 f4:sum f5:sum bar xyz 7.00 106.00 bar xyz2 4.00 4.00 foo bar 6.00 3.00 foo bar2 4.50 5.00 Select fields/columns ( cut ) By index: csvtk cut -f 1,2 By names: csvtk cut -f first_name,username Unselect : csvtk cut -f -1,-2 or csvtk cut -f -first_name Fuzzy fields : csvtk cut -F -f \"*_name,username\" Field ranges: csvtk cut -f 2-4 for column 2,3,4 or csvtk cut -f -3--1 for discarding column 1,2,3 All fields: csvtk cut -f 1- or csvtk cut -F -f \"*\" Search by selected fields ( grep ) (matched parts will be highlighted as red) By exactly matching: csvtk grep -f first_name -p Robert -p Rob By regular expression: csvtk grep -f first_name -r -p Rob By pattern list: csvtk grep -f first_name -P name_list.txt Remore rows containing missing data (NA): csvtk grep -F -f \"*\" -r -p \"^$\" -v Rename column names ( rename and rename2 ) Setting new names: csvtk rename -f A,B -n a,b or csvtk rename -f 1-3 -n a,b,c Replacing with original names by regular express: csvtk rename2 -f 1- -p \"(.*)\" -r 'prefix_$1' for adding prefix to all column names. Edit data with regular expression ( replace ) Remove Chinese charactors: csvtk replace -F -f \"*_name\" -p \"\\p{Han}+\" -r \"\" Create new column from selected fields by regular expression ( mutate ) In default, copy a column: csvtk mutate -f id Extract prefix of data as group name (get \"A\" from \"A.1\" as group name): csvtk mutate -f sample -n group -p \"^(.+?)\\.\" --after sample Sort by multiple keys ( sort ) By single column : csvtk sort -k 1 or csvtk sort -k last_name By multiple columns: csvtk sort -k 1,2 or csvtk sort -k 1 -k 2 or csvtk sort -k last_name,age Sort by number: csvtk sort -k 1:n or csvtk sort -k 1:nr for reverse number Complex sort: csvtk sort -k region -k age:n -k id:nr In natural order: csvtk sort -k chr:N Join multiple files by keys ( join ) All files have same key column: csvtk join -f id file1.csv file2.csv Files have different key columns: csvtk join -f \"username;username;name\" names.csv phone.csv adress.csv -k Filter by numbers ( filter ) Single field: csvtk filter -f \"id>0\" Multiple fields : csvtk filter -f \"1-3>0\" Using --any to print record if any of the field satisfy the condition: csvtk filter -f \"1-3>0\" --any fuzzy fields : csvtk filter -F -f \"A*!=0\" Filter rows by awk-like arithmetic/string expressions ( filter2 ) Using field index: csvtk filter2 -f '$3>0' Using column names: csvtk filter2 -f '$id > 0' Both arithmetic and string expressions: csvtk filter2 -f '$id > 3 || $username==\"ken\"' More complicated: csvtk filter2 -H -t -f '$1 > 2 && $2 % 2 == 0' Ploting plot histogram with data of the second column: csvtk -t plot hist testdata/grouped_data.tsv.gz -f 2 | display plot boxplot with data of the \"GC Content\" (third) column, group information is the \"Group\" column. csvtk -t plot box testdata/grouped_data.tsv.gz -g \"Group\" \\ -f \"GC Content\" --width 3 | display plot horiz boxplot with data of the \"Length\" (second) column, group information is the \"Group\" column. csvtk -t plot box testdata/grouped_data.tsv.gz -g \"Group\" -f \"Length\" \\ --height 3 --width 5 --horiz --title \"Horiz box plot\" | display plot line plot with X-Y data csvtk -t plot line testdata/xy.tsv -x X -y Y -g Group | display plot scatter plot with X-Y data csvtk -t plot line testdata/xy.tsv -x X -y Y -g Group --scatter | display","title":"Examples"},{"location":"#acknowledgements","text":"We are grateful to Zhiluo Deng and Li Peng for suggesting features and reporting bugs. Thanks Albert Vilella for features suggestion, which makes csvtk feature-rich\u3002","title":"Acknowledgements"},{"location":"#contact","text":"Create an issue to report bugs, propose new functions or ask for help. Or leave a comment .","title":"Contact"},{"location":"#license","text":"MIT License","title":"License"},{"location":"#starchart","text":"","title":"Starchart"},{"location":"bioinf/","text":"","title":"Bioinf"},{"location":"chinese/","text":"\u5982\u540c\u751f\u7269\u4fe1\u606f\u9886\u57df\u4e2d\u7684FASTA/Q\u683c\u5f0f\u4e00\u6837\uff0cCSV/TSV\u4f5c\u4e3a\u8ba1\u7b97\u673a\u3001\u6570\u636e\u79d1\u5b66\u548c\u751f\u7269\u4fe1\u606f\u7684\u57fa\u672c\u683c\u5f0f\uff0c\u5e94\u7528\u975e\u5e38\u5e7f\u6cdb\u3002\u5e38\u7528\u7684\u5904\u7406\u8f6f\u4ef6\u5305\u62ec\uff1a \u4ee5\u5fae\u8f6fExcel\u4e3a\u4ee3\u8868\u7684\u7535\u5b50\u8868\u683c\u8f6f\u4ef6 Notepad++/SublimeText\u7b49\u6587\u672c\u7f16\u8f91\u5668 sed/awk/cut\u7b49Shell\u547d\u4ee4 \u5404\u79cd\u7f16\u7a0b\u8bed\u8a00\u7684\u6570\u636e\u5904\u7406\u5e93\u3002 \u7136\u800c\uff0c\u7535\u5b50\u8868\u683c\u8f6f\u4ef6\u548c\u6587\u672c\u7f16\u8f91\u5668\u56fa\u7136\u5f3a\u5927\uff0c\u4f46\u4f9d\u8d56\u9f20\u6807\u64cd\u4f5c\uff0c\u4e0d\u9002\u5408\u6279\u91cf\u5904\u7406\uff1b sed/awk/cut\u7b49Shell\u547d\u4ee4\u4e3b\u8981\u7528\u4e8e\u901a\u7528\u7684\u8868\u683c\u6570\u636e\uff0c\u4e0d\u9002\u5408\u542b\u6709\u6807\u9898\u884c\u7684CSV\u683c\u5f0f \uff1b \u4e3a\u4e86\u4e00\u4e2a\u5c0f\u64cd\u4f5c\u5199Python/R\u811a\u672c\u4e5f\u6709\u70b9\u5c0f\u9898\u5927\u4f5c\uff0c\u4e14\u96be\u4ee5\u590d\u7528 \u3002 \u5f00\u53d1csvtk\u524d\u73b0\u6709\u7684\u5de5\u5177\u4e3b\u8981\u662fPython\u5199\u7684csvkit\uff0cRust\u5199\u7684xsv\uff0cC\u8bed\u8a00\u5199\u7684miller\uff0c\u90fd\u5404\u6709\u4f18\u52a3\u3002\u5f53\u65f6\u6211\u521a\u5f00\u53d1\u5b8cseqkit\uff0c\u6295\u6587\u7ae0\u8fc7\u7a0b\u4e2d\u65f6\u95f4\u5145\u8db3\uff0c\u4fbf\u60f3\u8d81\u70ed\u518d\u9020\u4e00\u4e2a\u8f6e\u5b50\u3002 \u6240\u4ee5\u6211\u51b3\u5b9a\u5199\u4e00\u4e2a\u547d\u4ee4\u884c\u5de5\u5177\u6765\u6ee1\u8db3CSV/TSV\u683c\u5f0f\u7684\u5e38\u89c1\u64cd\u4f5c\uff0c\u8fd9\u5c31\u662fcsvtk\u4e86\u3002 \u4ecb\u7ecd \u57fa\u672c\u4fe1\u606f \u5de5\u5177\u7c7b\u578b: \u547d\u4ee4\u884c\u5de5\u5177\uff0c\u5b50\u547d\u4ee4\u7ed3\u6784 \u652f\u6301\u683c\u5f0f: CSV/TSV, plain/gzip-compressed \u7f16\u7a0b\u8bed\u8a00: Go \u652f\u6301\u5e73\u53f0: Linux, OS X\uff0c Windows \u7b49 \u53d1\u5e03\u65b9\u5f0f: \u5355\u4e00\u53ef\u6267\u884c\u4e8c\u8fdb\u5236\u6587\u4ef6\uff0c\u4e0b\u8f7d\u5373\u7528 \u53d1\u5e03\u5e73\u53f0: Github, Bioconda \u9879\u76ee\u4e3b\u9875: http://bioinf.shenwei.me/csvtk/ \u5f00\u6e90\u5730\u5740: https://github.com/shenwei356/csvtk \u7279\u6027 \u8de8\u5e73\u53f0 \u8f7b\u91cf\uff0c\u65e0\u4efb\u4f55\u4f9d\u8d56\uff0c\u65e0\u9700\u7f16\u8bd1\u3001\u914d\u7f6e\uff0c\u4e0b\u8f7d\u5373\u7528 \u5feb\u901f \u652f\u6301stdin\u548cgzip\u538b\u7f29\u7684\u8f93\u5165\u548c\u8f93\u51fa\u6587\u4ef6\uff0c\u4fbf\u4e8e\u6d41\u5904\u7406 27\u4e2a\u5b50\u547d\u4ee4\u63d0\u4f9b\u591a\u79cd\u5b9e\u7528\u7684\u529f\u80fd\uff0c\u4e14\u80fd\u901a\u8fc7\u547d\u4ee4\u884c\u7ba1\u9053\u7ec4\u5408 \u652f\u6301Bash\u81ea\u52a8\u8865\u5168 \u652f\u6301\u7b80\u5355\u7684\u7ed8\u56fe \u529f\u80fd \u5728\u5f00\u53d1csvtk\u4e4b\u524d\u7684\u4e24\u4e09\u5e74\u95f4\uff0c\u6211\u5df2\u7ecf\u5199\u4e86\u51e0\u4e2a\u53ef\u4ee5\u590d\u7528\u7684Python/Perl\u811a\u672c\uff08https://github.com/shenwei356/datakit\uff09 \uff0c\u5305\u62eccsv2tab\u3001csvtk_grep\u3001csv_join\u3001csv_melt\uff0cintersection\uff0cunique\u3002\u6240\u4ee5\u6211\u7684\u8ba1\u5212\u662f\u9996\u5148\u96c6\u6210\u8fd9\u4e9b\u5df2\u6709\u7684\u529f\u80fd\uff0c\u968f\u540e\u6839\u636e\u9700\u6c42\u8fdb\u884c\u6269\u5c55\u3002 \u5230\u76ee\u524d\u4e3a\u6b62\uff0ccsvtk\u5df2\u670927\u4e2a\u5b50\u547d\u4ee4\uff0c\u5206\u4e3a\u4ee5\u4e0b\u51e0\u5927\u7c7b\uff1a \u4fe1\u606f headers \u76f4\u89c2\u6253\u5370\u6807\u9898\u884c\uff08 \u64cd\u4f5c\u5217\u6570\u8f83\u591a\u7684CSV\u524d\u4f7f\u7528\u6700\u4f73 \uff09 stats \u57fa\u672c\u7edf\u8ba1 stats2 \u5bf9\u9009\u5b9a\u7684\u6570\u503c\u5217\u8fdb\u884c\u57fa\u672c\u7edf\u8ba1 \u683c\u5f0f\u8f6c\u5316 pretty \u8f6c\u4e3a\u7f8e\u89c2\u3001\u53ef\u8bfb\u6027\u5f3a\u7684\u683c\u5f0f\uff08 \u6700\u5e38\u7528\u547d\u4ee4\u4e4b\u4e00 \uff09 csv2tab \u8f6cCSV\u4e3a\u5236\u8868\u7b26\u5206\u5272\u683c\u5f0f\uff08TSV\uff09 tab2csv \u8f6cTSV\u4e3aCSV space2tab \u8f6c\u7a7a\u683c\u5206\u5272\u683c\u5f0f\u4e3aTSV transpose \u8f6c\u7f6eCSV/TSV csv2md \u8f6cCSV/TSV\u4e3amakrdown\u683c\u5f0f\uff08 \u5199\u6587\u6863\u5e38\u7528 \uff09 \u96c6\u5408\u64cd\u4f5c head \u6253\u5370\u524dN\u6761\u8bb0\u5f55 sample \u6309\u6bd4\u4f8b\u968f\u673a\u91c7\u6837 cut \u9009\u62e9\u7279\u5b9a\u5217\uff0c\u652f\u6301 \u6309\u5217\u6216\u5217\u540d\u8fdb\u884c\u57fa\u672c\u9009\u62e9\u3001\u8303\u56f4\u9009\u62e9\u3001\u6a21\u7cca\u9009\u62e9\u3001\u8d1f\u5411\u9009\u62e9 \uff08 \u6700\u5e38\u7528\u547d\u4ee4\u4e4b\u4e00\uff0c\u975e\u5e38\u5f3a\u5927 \uff09 uniq \u65e0\u987b\u6392\u5e8f\uff0c\u8fd4\u56de\u6309\u6307\u5b9a\uff08\u591a\uff09\u5217\u4f5c\u4e3akey\u7684\u552f\u4e00\u8bb0\u5f55\uff08\u597d\u7ed5\u3002\u3002\uff09 freq \u6309\u6307\u5b9a\uff08\u591a\uff09\u5217\u8fdb\u884c\u8ba1\u6570\uff08 \u5e38\u7528 \uff09 inter \u591a\u4e2a\u6587\u4ef6\u95f4\u7684\u4ea4\u96c6 grep \u6307\u5b9a\uff08\u591a\uff09\u5217\u4e3aKey\u8fdb\u884c\u641c\u7d22\uff08 \u6700\u5e38\u7528\u547d\u4ee4\u4e4b\u4e00\uff0c\u53ef\u6309\u6307\u5b9a\u5217\u641c\u7d22 \uff09 filter \u6309\u6307\u5b9a\uff08\u591a\uff09\u5217\u7684\u6570\u503c\u8fdb\u884c\u8fc7\u6ee4 filter2 \u7528\u7c7b\u4f3cawk\u7684\u6570\u503c/\u8868\u8fbe\u5f0f\uff0c\u6309\u6307\u5b9a\uff08\u591a\uff09\u5217\u7684\u6570\u503c\u8fdb\u884c\u8fc7\u6ee4 join \u5408\u5e76\u591a\u4e2a\u6587\u4ef6\uff08 \u5e38\u7528 \uff09 \u7f16\u8f91 rename \u76f4\u63a5\u91cd\u547d\u540d\u6307\u5b9a\uff08\u591a\uff09\u5217\u540d\uff08 \u7b80\u5355\u800c\u5b9e\u7528 \uff09 rename2 \u4ee5\u6b63\u5219\u8868\u8fbe\u5f0f\u91cd\u547d\u540d\u6307\u5b9a\uff08\u591a\uff09\u5217\u540d\uff08 \u7b80\u5355\u800c\u5b9e\u7528 \uff09 replace \u4ee5\u6b63\u5219\u8868\u8fbe\u5f0f\u5bf9\u6307\u5b9a\uff08\u591a\uff09\u5217\u8fdb\u884c\u66ff\u6362\u7f16\u8f91\uff08 \u6700\u5e38\u7528\u547d\u4ee4\u4e4b\u4e00\uff0c\u53ef\u6309\u6307\u5b9a\u5217\u7f16\u8f91 \uff09 mutate \u4ee5\u6b63\u5219\u8868\u8fbe\u5f0f\u57fa\u4e8e\u5df2\u6709\u5217\u521b\u5efa\u65b0\u7684\u4e00\u5217\uff08 \u5e38\u7528\u4e8e\u751f\u6210\u591a\u5217\u6d4b\u8bd5\u6570\u636e \uff09 mutate2 \u7528\u7c7b\u4f3cawk\u7684\u6570\u503c/\u8868\u8fbe\u5f0f\uff0c\u4ee5\u6b63\u5219\u8868\u8fbe\u5f0f\u57fa\u4e8e\u5df2\u6709\uff08\u591a\uff09\u5217\u521b\u5efa\u65b0\u7684\u4e00\u5217\uff08 \u5e38\u7528 \uff09 gather \u7c7b\u4f3c\u4e8eR\u91cc\u9762tidyr\u5305\u7684gather\u65b9\u6cd5 \u6392\u5e8f sort \u6309\u6307\u5b9a\uff08\u591a\uff09\u5217\u8fdb\u884c\u6392\u5e8f \u7ed8\u56fe plot \u57fa\u672c\u7ed8\u56fe plot hist histogram plot box boxplot plot line line plot and scatter plot \u5176\u5b83 version \u7248\u672c\u4fe1\u606f\u548c\u68c0\u67e5\u65b0\u7248\u672c genautocomplete \u751f\u6210\u652f\u6301Bash\u81ea\u52a8\u8865\u5168\u7684\u914d\u7f6e\u6587\u4ef6\uff0c\u91cd\u542fTerminal\u751f\u6548\u3002 \u4f7f\u7528 \u8f93\u5165\u6570\u636e\u8981\u6c42\u6bcf\u884c\u7684\u5217\u6570\u4e00\u81f4\uff0c\u7a7a\u884c\u4e5f\u4f1a\u62a5\u9519 csvtk\u9ed8\u8ba4\u8f93\u5165\u6570\u636e\u542b\u6709\u6807\u9898\u884c\uff0c\u5982\u6ca1\u6709\u8bf7\u5f00\u542f\u5168\u5c40\u53c2\u6570 -H csvtk\u9ed8\u8ba4\u8f93\u5165\u6570\u636e\u4e3aCSV\u683c\u5f0f\uff0c\u5982\u4e3aTSV\u8bf7\u5f00\u542f\u5168\u5c40\u53c2\u6570 -t \u8f93\u5165\u6570\u636e\u5217\u540d\u6700\u597d\u552f\u4e00\u65e0\u91cd\u590d \u5982\u679cTSV\u4e2d\u5b58\u5728\u53cc\u5f15\u53f7 \"\" \uff0c\u8bf7\u5f00\u542f\u5168\u5c40\u53c2\u6570 -l csvtk\u9ed8\u8ba4\u4ee5 # \u5f00\u59cb\u7684\u4e3a\u6ce8\u91ca\u884c\uff0c\u82e5\u6807\u9898\u884c\u542b # \uff0c\u8bf7\u7ed9\u5168\u5c40\u53c2\u6570 -C \u6307\u5b9a\u53e6\u4e00\u4e2a\u4e0d\u5e38\u89c1\u7684\u5b57\u7b26\uff08\u5982 $ \uff09 \u4f8b\u5b50 \u4ec5\u63d0\u4f9b\u5c11\u91cf\u4f8b\u5b50\uff0c\u66f4\u591a\u4f8b\u5b50\u8bf7\u770b\u4f7f\u7528\u624b\u518c http://bioinf.shenwei.me/csvtk/usage/ \u3002 \u793a\u4f8b\u6570\u636e $ cat names.csv id,first_name,last_name,username 11,\"Rob\",\"Pike\",rob 2,Ken,Thompson,ken 4,\"Robert\",\"Griesemer\",\"gri\" 1,\"Robert\",\"Thompson\",\"abc\" NA,\"Robert\",\"Abel\",\"123\" \u589e\u5f3a\u53ef\u8bfb\u6027 $ cat names.csv | csvtk pretty id first_name last_name username 11 Rob Pike rob 2 Ken Thompson ken 4 Robert Griesemer gri 1 Robert Thompson abc NA Robert Abel 123 \u8f6c\u4e3amarkdown $ cat names.csv | csvtk csv2md id |first_name|last_name|username :--|:---------|:--------|:------- 11 |Rob |Pike |rob 2 |Ken |Thompson |ken 4 |Robert |Griesemer|gri 1 |Robert |Thompson |abc NA |Robert |Abel |123 \u6548\u679c id first_name last_name username 11 Rob Pike rob 2 Ken Thompson ken 4 Robert Griesemer gri 1 Robert Thompson abc NA Robert Abel 123 \u7528\u5217\u6216\u5217\u540d\u6765\u9009\u62e9\u6307\u5b9a\u5217\uff0c\u53ef\u6539\u53d8\u5217\u7684\u987a\u5e8f $ cat names.csv | csvtk cut -f 3,1 | csvtk pretty $ cat names.csv | csvtk cut -f last_name,id | csvtk pretty last_name id Pike 11 Thompson 2 Griesemer 4 Thompson 1 Abel NA \u7528\u901a\u914d\u7b26\u9009\u62e9\u591a\u5217 $ cat names.csv | csvtk cut -F -f '*name,id' | csvtk pretty first_name last_name username id Rob Pike rob 11 Ken Thompson ken 2 Robert Griesemer gri 4 Robert Thompson abc 1 Robert Abel 123 NA \u5220\u9664\u7b2c2\uff0c3\u5217\uff08 \u4e0b\u5217\u7b2c\u4e8c\u79cd\u65b9\u6cd5\u662f\u9009\u5b9a\u8303\u56f4\uff0c\u4f46-3\u5728\u524d,-2\u5728\u540e \uff09 $ cat names.csv | csvtk cut -f -2,-3 | csvtk pretty $ cat names.csv | csvtk cut -f -3--2 | csvtk pretty $ cat names.csv | csvtk cut -f -first_name,-last_name | csvtk pretty id username 11 rob 2 ken 4 gri 1 abc NA 123 \u6309\u6307\u5b9a\u5217\u641c\u7d22\uff0c \u9ed8\u8ba4\u7cbe\u786e\u5339\u914d $ cat names.csv | csvtk grep -f id -p 1 | csvtk pretty id first_name last_name username 1 Robert Thompson abc \u6a21\u7cca\u641c\u7d22\uff08\u6b63\u5219\u8868\u8fbe\u5f0f\uff09 $ cat names.csv | csvtk grep -f id -p 1 -r | csvtk pretty id first_name last_name username 11 Rob Pike rob 1 Robert Thompson abc \u7528\u6587\u4ef6\u4f5c\u4e3a\u6a21\u5f0f\u6765\u6e90 $ cat names.csv | csvtk grep -f id -P id-files.txt \u5bf9\u6307\u5b9a\u5217\u505a\u7b80\u5355\u66ff\u6362 $ cat names.csv | csvtk replace -f id -p '(\\d+)' -r 'ID: $1' \\ | csvtk pretty id first_name last_name username ID: 11 Rob Pike rob ID: 2 Ken Thompson ken ID: 4 Robert Griesemer gri ID: 1 Robert Thompson abc NA Robert Abel 123 \u7528key-value\u6587\u4ef6\u6765\u66ff\u6362\uff08seqkit\u548cbrename\u90fd\u652f\u6301\u7c7b\u4f3c\u64cd\u4f5c\uff09 $ cat data.tsv name id A ID001 B ID002 C ID004 $ cat alias.tsv 001 Tom 002 Bob 003 Jim $ csvtk replace -t -f 2 -p \"ID(.+)\" -r \"N: {nr}, alias: {kv}\" -k \\ alias.tsv data.tsv name id A N: 1, alias: Tom B N: 2, alias: Bob C N: 3, alias: 004 \u5408\u5e76\u8868\u683c\uff0c\u9700\u8981\u5206\u522b\u6307\u5b9a\u5404\u6587\u4ef6\u4e2d\u7684key\u5217\uff1a\u9ed8\u8ba4\u5747\u4e3a\u7b2c\u4e00\u5217\uff1b\u82e5\u5217\uff08\u540d\uff09\u76f8\u540c\u63d0\u4f9b\u4e00\u4e2a\uff1b\u82e5\u4e0d\u540c\u7528\u5206\u53f7\u5206\u5272 $ cat testdata/phones.csv username,phone gri,11111 rob,12345 ken,22222 shenwei,999999 $ csvtk join -f 'username;username' --keep-unmatched names.csv phones.csv \\ | csvtk pretty id first_name last_name username phone 11 Rob Pike rob 12345 2 Ken Thompson ken 22222 4 Robert Griesemer gri 11111 1 Robert Thompson abc NA Robert Abel 123","title":"\u4e2d\u6587\u4ecb\u7ecd"},{"location":"chinese/#_1","text":"\u57fa\u672c\u4fe1\u606f \u5de5\u5177\u7c7b\u578b: \u547d\u4ee4\u884c\u5de5\u5177\uff0c\u5b50\u547d\u4ee4\u7ed3\u6784 \u652f\u6301\u683c\u5f0f: CSV/TSV, plain/gzip-compressed \u7f16\u7a0b\u8bed\u8a00: Go \u652f\u6301\u5e73\u53f0: Linux, OS X\uff0c Windows \u7b49 \u53d1\u5e03\u65b9\u5f0f: \u5355\u4e00\u53ef\u6267\u884c\u4e8c\u8fdb\u5236\u6587\u4ef6\uff0c\u4e0b\u8f7d\u5373\u7528 \u53d1\u5e03\u5e73\u53f0: Github, Bioconda \u9879\u76ee\u4e3b\u9875: http://bioinf.shenwei.me/csvtk/ \u5f00\u6e90\u5730\u5740: https://github.com/shenwei356/csvtk \u7279\u6027 \u8de8\u5e73\u53f0 \u8f7b\u91cf\uff0c\u65e0\u4efb\u4f55\u4f9d\u8d56\uff0c\u65e0\u9700\u7f16\u8bd1\u3001\u914d\u7f6e\uff0c\u4e0b\u8f7d\u5373\u7528 \u5feb\u901f \u652f\u6301stdin\u548cgzip\u538b\u7f29\u7684\u8f93\u5165\u548c\u8f93\u51fa\u6587\u4ef6\uff0c\u4fbf\u4e8e\u6d41\u5904\u7406 27\u4e2a\u5b50\u547d\u4ee4\u63d0\u4f9b\u591a\u79cd\u5b9e\u7528\u7684\u529f\u80fd\uff0c\u4e14\u80fd\u901a\u8fc7\u547d\u4ee4\u884c\u7ba1\u9053\u7ec4\u5408 \u652f\u6301Bash\u81ea\u52a8\u8865\u5168 \u652f\u6301\u7b80\u5355\u7684\u7ed8\u56fe","title":"\u4ecb\u7ecd"},{"location":"chinese/#_2","text":"\u5728\u5f00\u53d1csvtk\u4e4b\u524d\u7684\u4e24\u4e09\u5e74\u95f4\uff0c\u6211\u5df2\u7ecf\u5199\u4e86\u51e0\u4e2a\u53ef\u4ee5\u590d\u7528\u7684Python/Perl\u811a\u672c\uff08https://github.com/shenwei356/datakit\uff09 \uff0c\u5305\u62eccsv2tab\u3001csvtk_grep\u3001csv_join\u3001csv_melt\uff0cintersection\uff0cunique\u3002\u6240\u4ee5\u6211\u7684\u8ba1\u5212\u662f\u9996\u5148\u96c6\u6210\u8fd9\u4e9b\u5df2\u6709\u7684\u529f\u80fd\uff0c\u968f\u540e\u6839\u636e\u9700\u6c42\u8fdb\u884c\u6269\u5c55\u3002 \u5230\u76ee\u524d\u4e3a\u6b62\uff0ccsvtk\u5df2\u670927\u4e2a\u5b50\u547d\u4ee4\uff0c\u5206\u4e3a\u4ee5\u4e0b\u51e0\u5927\u7c7b\uff1a \u4fe1\u606f headers \u76f4\u89c2\u6253\u5370\u6807\u9898\u884c\uff08 \u64cd\u4f5c\u5217\u6570\u8f83\u591a\u7684CSV\u524d\u4f7f\u7528\u6700\u4f73 \uff09 stats \u57fa\u672c\u7edf\u8ba1 stats2 \u5bf9\u9009\u5b9a\u7684\u6570\u503c\u5217\u8fdb\u884c\u57fa\u672c\u7edf\u8ba1 \u683c\u5f0f\u8f6c\u5316 pretty \u8f6c\u4e3a\u7f8e\u89c2\u3001\u53ef\u8bfb\u6027\u5f3a\u7684\u683c\u5f0f\uff08 \u6700\u5e38\u7528\u547d\u4ee4\u4e4b\u4e00 \uff09 csv2tab \u8f6cCSV\u4e3a\u5236\u8868\u7b26\u5206\u5272\u683c\u5f0f\uff08TSV\uff09 tab2csv \u8f6cTSV\u4e3aCSV space2tab \u8f6c\u7a7a\u683c\u5206\u5272\u683c\u5f0f\u4e3aTSV transpose \u8f6c\u7f6eCSV/TSV csv2md \u8f6cCSV/TSV\u4e3amakrdown\u683c\u5f0f\uff08 \u5199\u6587\u6863\u5e38\u7528 \uff09 \u96c6\u5408\u64cd\u4f5c head \u6253\u5370\u524dN\u6761\u8bb0\u5f55 sample \u6309\u6bd4\u4f8b\u968f\u673a\u91c7\u6837 cut \u9009\u62e9\u7279\u5b9a\u5217\uff0c\u652f\u6301 \u6309\u5217\u6216\u5217\u540d\u8fdb\u884c\u57fa\u672c\u9009\u62e9\u3001\u8303\u56f4\u9009\u62e9\u3001\u6a21\u7cca\u9009\u62e9\u3001\u8d1f\u5411\u9009\u62e9 \uff08 \u6700\u5e38\u7528\u547d\u4ee4\u4e4b\u4e00\uff0c\u975e\u5e38\u5f3a\u5927 \uff09 uniq \u65e0\u987b\u6392\u5e8f\uff0c\u8fd4\u56de\u6309\u6307\u5b9a\uff08\u591a\uff09\u5217\u4f5c\u4e3akey\u7684\u552f\u4e00\u8bb0\u5f55\uff08\u597d\u7ed5\u3002\u3002\uff09 freq \u6309\u6307\u5b9a\uff08\u591a\uff09\u5217\u8fdb\u884c\u8ba1\u6570\uff08 \u5e38\u7528 \uff09 inter \u591a\u4e2a\u6587\u4ef6\u95f4\u7684\u4ea4\u96c6 grep \u6307\u5b9a\uff08\u591a\uff09\u5217\u4e3aKey\u8fdb\u884c\u641c\u7d22\uff08 \u6700\u5e38\u7528\u547d\u4ee4\u4e4b\u4e00\uff0c\u53ef\u6309\u6307\u5b9a\u5217\u641c\u7d22 \uff09 filter \u6309\u6307\u5b9a\uff08\u591a\uff09\u5217\u7684\u6570\u503c\u8fdb\u884c\u8fc7\u6ee4 filter2 \u7528\u7c7b\u4f3cawk\u7684\u6570\u503c/\u8868\u8fbe\u5f0f\uff0c\u6309\u6307\u5b9a\uff08\u591a\uff09\u5217\u7684\u6570\u503c\u8fdb\u884c\u8fc7\u6ee4 join \u5408\u5e76\u591a\u4e2a\u6587\u4ef6\uff08 \u5e38\u7528 \uff09 \u7f16\u8f91 rename \u76f4\u63a5\u91cd\u547d\u540d\u6307\u5b9a\uff08\u591a\uff09\u5217\u540d\uff08 \u7b80\u5355\u800c\u5b9e\u7528 \uff09 rename2 \u4ee5\u6b63\u5219\u8868\u8fbe\u5f0f\u91cd\u547d\u540d\u6307\u5b9a\uff08\u591a\uff09\u5217\u540d\uff08 \u7b80\u5355\u800c\u5b9e\u7528 \uff09 replace \u4ee5\u6b63\u5219\u8868\u8fbe\u5f0f\u5bf9\u6307\u5b9a\uff08\u591a\uff09\u5217\u8fdb\u884c\u66ff\u6362\u7f16\u8f91\uff08 \u6700\u5e38\u7528\u547d\u4ee4\u4e4b\u4e00\uff0c\u53ef\u6309\u6307\u5b9a\u5217\u7f16\u8f91 \uff09 mutate \u4ee5\u6b63\u5219\u8868\u8fbe\u5f0f\u57fa\u4e8e\u5df2\u6709\u5217\u521b\u5efa\u65b0\u7684\u4e00\u5217\uff08 \u5e38\u7528\u4e8e\u751f\u6210\u591a\u5217\u6d4b\u8bd5\u6570\u636e \uff09 mutate2 \u7528\u7c7b\u4f3cawk\u7684\u6570\u503c/\u8868\u8fbe\u5f0f\uff0c\u4ee5\u6b63\u5219\u8868\u8fbe\u5f0f\u57fa\u4e8e\u5df2\u6709\uff08\u591a\uff09\u5217\u521b\u5efa\u65b0\u7684\u4e00\u5217\uff08 \u5e38\u7528 \uff09 gather \u7c7b\u4f3c\u4e8eR\u91cc\u9762tidyr\u5305\u7684gather\u65b9\u6cd5 \u6392\u5e8f sort \u6309\u6307\u5b9a\uff08\u591a\uff09\u5217\u8fdb\u884c\u6392\u5e8f \u7ed8\u56fe plot \u57fa\u672c\u7ed8\u56fe plot hist histogram plot box boxplot plot line line plot and scatter plot \u5176\u5b83 version \u7248\u672c\u4fe1\u606f\u548c\u68c0\u67e5\u65b0\u7248\u672c genautocomplete \u751f\u6210\u652f\u6301Bash\u81ea\u52a8\u8865\u5168\u7684\u914d\u7f6e\u6587\u4ef6\uff0c\u91cd\u542fTerminal\u751f\u6548\u3002","title":"\u529f\u80fd"},{"location":"chinese/#_3","text":"\u8f93\u5165\u6570\u636e\u8981\u6c42\u6bcf\u884c\u7684\u5217\u6570\u4e00\u81f4\uff0c\u7a7a\u884c\u4e5f\u4f1a\u62a5\u9519 csvtk\u9ed8\u8ba4\u8f93\u5165\u6570\u636e\u542b\u6709\u6807\u9898\u884c\uff0c\u5982\u6ca1\u6709\u8bf7\u5f00\u542f\u5168\u5c40\u53c2\u6570 -H csvtk\u9ed8\u8ba4\u8f93\u5165\u6570\u636e\u4e3aCSV\u683c\u5f0f\uff0c\u5982\u4e3aTSV\u8bf7\u5f00\u542f\u5168\u5c40\u53c2\u6570 -t \u8f93\u5165\u6570\u636e\u5217\u540d\u6700\u597d\u552f\u4e00\u65e0\u91cd\u590d \u5982\u679cTSV\u4e2d\u5b58\u5728\u53cc\u5f15\u53f7 \"\" \uff0c\u8bf7\u5f00\u542f\u5168\u5c40\u53c2\u6570 -l csvtk\u9ed8\u8ba4\u4ee5 # \u5f00\u59cb\u7684\u4e3a\u6ce8\u91ca\u884c\uff0c\u82e5\u6807\u9898\u884c\u542b # \uff0c\u8bf7\u7ed9\u5168\u5c40\u53c2\u6570 -C \u6307\u5b9a\u53e6\u4e00\u4e2a\u4e0d\u5e38\u89c1\u7684\u5b57\u7b26\uff08\u5982 $ \uff09","title":"\u4f7f\u7528"},{"location":"chinese/#_4","text":"\u4ec5\u63d0\u4f9b\u5c11\u91cf\u4f8b\u5b50\uff0c\u66f4\u591a\u4f8b\u5b50\u8bf7\u770b\u4f7f\u7528\u624b\u518c http://bioinf.shenwei.me/csvtk/usage/ \u3002 \u793a\u4f8b\u6570\u636e $ cat names.csv id,first_name,last_name,username 11,\"Rob\",\"Pike\",rob 2,Ken,Thompson,ken 4,\"Robert\",\"Griesemer\",\"gri\" 1,\"Robert\",\"Thompson\",\"abc\" NA,\"Robert\",\"Abel\",\"123\" \u589e\u5f3a\u53ef\u8bfb\u6027 $ cat names.csv | csvtk pretty id first_name last_name username 11 Rob Pike rob 2 Ken Thompson ken 4 Robert Griesemer gri 1 Robert Thompson abc NA Robert Abel 123 \u8f6c\u4e3amarkdown $ cat names.csv | csvtk csv2md id |first_name|last_name|username :--|:---------|:--------|:------- 11 |Rob |Pike |rob 2 |Ken |Thompson |ken 4 |Robert |Griesemer|gri 1 |Robert |Thompson |abc NA |Robert |Abel |123 \u6548\u679c id first_name last_name username 11 Rob Pike rob 2 Ken Thompson ken 4 Robert Griesemer gri 1 Robert Thompson abc NA Robert Abel 123 \u7528\u5217\u6216\u5217\u540d\u6765\u9009\u62e9\u6307\u5b9a\u5217\uff0c\u53ef\u6539\u53d8\u5217\u7684\u987a\u5e8f $ cat names.csv | csvtk cut -f 3,1 | csvtk pretty $ cat names.csv | csvtk cut -f last_name,id | csvtk pretty last_name id Pike 11 Thompson 2 Griesemer 4 Thompson 1 Abel NA \u7528\u901a\u914d\u7b26\u9009\u62e9\u591a\u5217 $ cat names.csv | csvtk cut -F -f '*name,id' | csvtk pretty first_name last_name username id Rob Pike rob 11 Ken Thompson ken 2 Robert Griesemer gri 4 Robert Thompson abc 1 Robert Abel 123 NA \u5220\u9664\u7b2c2\uff0c3\u5217\uff08 \u4e0b\u5217\u7b2c\u4e8c\u79cd\u65b9\u6cd5\u662f\u9009\u5b9a\u8303\u56f4\uff0c\u4f46-3\u5728\u524d,-2\u5728\u540e \uff09 $ cat names.csv | csvtk cut -f -2,-3 | csvtk pretty $ cat names.csv | csvtk cut -f -3--2 | csvtk pretty $ cat names.csv | csvtk cut -f -first_name,-last_name | csvtk pretty id username 11 rob 2 ken 4 gri 1 abc NA 123 \u6309\u6307\u5b9a\u5217\u641c\u7d22\uff0c \u9ed8\u8ba4\u7cbe\u786e\u5339\u914d $ cat names.csv | csvtk grep -f id -p 1 | csvtk pretty id first_name last_name username 1 Robert Thompson abc \u6a21\u7cca\u641c\u7d22\uff08\u6b63\u5219\u8868\u8fbe\u5f0f\uff09 $ cat names.csv | csvtk grep -f id -p 1 -r | csvtk pretty id first_name last_name username 11 Rob Pike rob 1 Robert Thompson abc \u7528\u6587\u4ef6\u4f5c\u4e3a\u6a21\u5f0f\u6765\u6e90 $ cat names.csv | csvtk grep -f id -P id-files.txt \u5bf9\u6307\u5b9a\u5217\u505a\u7b80\u5355\u66ff\u6362 $ cat names.csv | csvtk replace -f id -p '(\\d+)' -r 'ID: $1' \\ | csvtk pretty id first_name last_name username ID: 11 Rob Pike rob ID: 2 Ken Thompson ken ID: 4 Robert Griesemer gri ID: 1 Robert Thompson abc NA Robert Abel 123 \u7528key-value\u6587\u4ef6\u6765\u66ff\u6362\uff08seqkit\u548cbrename\u90fd\u652f\u6301\u7c7b\u4f3c\u64cd\u4f5c\uff09 $ cat data.tsv name id A ID001 B ID002 C ID004 $ cat alias.tsv 001 Tom 002 Bob 003 Jim $ csvtk replace -t -f 2 -p \"ID(.+)\" -r \"N: {nr}, alias: {kv}\" -k \\ alias.tsv data.tsv name id A N: 1, alias: Tom B N: 2, alias: Bob C N: 3, alias: 004 \u5408\u5e76\u8868\u683c\uff0c\u9700\u8981\u5206\u522b\u6307\u5b9a\u5404\u6587\u4ef6\u4e2d\u7684key\u5217\uff1a\u9ed8\u8ba4\u5747\u4e3a\u7b2c\u4e00\u5217\uff1b\u82e5\u5217\uff08\u540d\uff09\u76f8\u540c\u63d0\u4f9b\u4e00\u4e2a\uff1b\u82e5\u4e0d\u540c\u7528\u5206\u53f7\u5206\u5272 $ cat testdata/phones.csv username,phone gri,11111 rob,12345 ken,22222 shenwei,999999 $ csvtk join -f 'username;username' --keep-unmatched names.csv phones.csv \\ | csvtk pretty id first_name last_name username phone 11 Rob Pike rob 12345 2 Ken Thompson ken 22222 4 Robert Griesemer gri 11111 1 Robert Thompson abc NA Robert Abel 123","title":"\u4f8b\u5b50"},{"location":"download/","text":"Download csvtk is implemented in Go programming language, executable binary files for most popular operating system are freely available in release page. Current Version csvtk v0.27.2 csvtk pretty : fix the bug of empty first row with -H/--no-header-row , introduced in v0.27.0. new style 3line for three-line table. csvtk csv2xlsx : binaries compiled with go1.21 would result in a broken xlsx file. #243 csvtk splitxlsx : fix the error of invalid worksheet index . #1617 Links OS Arch File, \u4e2d\u56fd\u955c\u50cf Download Count Linux 32-bit csvtk_linux_386.tar.gz , \u4e2d\u56fd\u955c\u50cf Linux 64-bit csvtk_linux_amd64.tar.gz , \u4e2d\u56fd\u955c\u50cf Linux 64-bit csvtk_linux_arm64.tar.gz , \u4e2d\u56fd\u955c\u50cf macOS 64-bit csvtk_darwin_amd64.tar.gz , \u4e2d\u56fd\u955c\u50cf macOS arm64 csvtk_darwin_arm64.tar.gz , \u4e2d\u56fd\u955c\u50cf Windows 32-bit csvtk_windows_386.exe.tar.gz , \u4e2d\u56fd\u955c\u50cf Windows 64-bit csvtk_windows_amd64.exe.tar.gz , \u4e2d\u56fd\u955c\u50cf Notes run csvtk version to check update !!! run csvtk genautocomplete to update Bash completion !!! Installation Download Page csvtk is implemented in Go programming language, executable binary files for most popular operating systems are freely available in release page. Method 1: Download binaries (latest stable/dev version) Just download compressed executable file of your operating system, and decompress it with tar -zxvf *.tar.gz command or other tools. And then: For Linux-like systems If you have root privilege simply copy it to /usr/local/bin : sudo cp csvtk /usr/local/bin/ Or copy to anywhere in the environment variable PATH : mkdir -p $HOME/bin/; cp csvtk $HOME/bin/ For windows , just copy csvtk.exe to C:\\WINDOWS\\system32 . Method 2: Install via conda (latest stable version) conda install -c bioconda csvtk Method 3: Install via homebrew (may be not the latest) brew install csvtk Method 4: For Go developer (latest stable/dev version) go get -u github.com/shenwei356/csvtk/csvtk Method 5: For ArchLinux AUR users (may be not the latest) yaourt -S csvtk Method 6: Compiling from source (latest stable/dev version) # ------------------- install golang ----------------- # download Go from https://go.dev/dl wget https://go.dev/dl/go1.17.12.linux-amd64.tar.gz tar -zxf go1.17.12.linux-amd64.tar.gz -C $HOME/ # or # echo \"export PATH=$PATH:$HOME/go/bin\" >> ~/.bashrc # source ~/.bashrc export PATH=$PATH:$HOME/go/bin # ------------- the latest stable version ------------- go get -v -u github.com/shenwei356/csvtk/csvtk # The executable binary file is located in: # ~/go/bin/csvtk # You can also move it to anywhere in the $PATH mkdir -p $HOME/bin cp ~/go/bin/csvtk $HOME/bin/ # --------------- the development version -------------- git clone https://github.com/shenwei356/csvtk cd csvtk/csvtk/ go build # The executable binary file is located in: # ./csvtk # You can also move it to anywhere in the $PATH mkdir -p $HOME/bin cp ./csvtk $HOME/bin/ Shell-completion Bash: # generate completion shell csvtk genautocomplete --shell bash # configure if never did. # install bash-completion if the \"complete\" command is not found. echo \"for bcfile in ~/.bash_completion.d/* ; do source \\$bcfile; done\" >> ~/.bash_completion echo \"source ~/.bash_completion\" >> ~/.bashrc Zsh: # generate completion shell csvtk genautocomplete --shell zsh --file ~/.zfunc/_csvtk # configure if never did echo 'fpath=( ~/.zfunc \"${fpath[@]}\" )' >> ~/.zshrc echo \"autoload -U compinit; compinit\" >> ~/.zshrc fish: csvtk genautocomplete --shell fish --file ~/.config/fish/completions/csvtk.fish Release history csvtk v0.27.1 csvtk filter2/mutate2 : fix the bug of selecting with field numbers. #242 csvtk v0.27.0 csvtk : code refactoring and simplifying code, with 16% less code. most commands support open column range syntax , e.g., csvtk grep -f 2- . #120 only selected column names are not allowed to be duplicated in the input data : box, corr, filter, filter2, fold, freq, gather, historysort, inter, join, line, mutate, mutate2, rename, replace, sep, split, summary, unfold, uniq, watch. Other commands do not have the restriction. #235 add a new global flag -Z/--show-row-number , supported commands: cut, csv2tab, csv2xlsx, tab2csv, pretty. the colum name of row number changes from \"n\" to \"row\": csv2xlsx, csv2tab, cut, filter, filter2, grep, pretty, sample, tab2csv. new command : csvtk spread : spread a key-value pair across multiple columns, like tidyr::spread/pivot_wider . #91 , #236 , #239 csvtk mutate/mutate2 : new flags --at , --before , --after for specifying the position of the new column . #193 csvtk cut : fix unselect range error. #234 fix -i/--ignore-case . csvtk pretty : allow align-center and align-right for specific columns . #240 csvtk round : fix bug of failing to round scientific notation with value small than one, e.g., 7.1E-1 . csvtk summary : fix duplicated columns. fix result error when multiple stats applied to the same column. csvtk corr/watch : rewrite and fix bug, support choosing fields with column names. csvtk v0.26.0 csvtk : near all commands skip empty files now . #204 the global flag --infile-list accepts stdin \"-\". #210 new command csvtk fix : fix CSV/TSV with different numbers of columns in rows . #226 csvtk pretty : rewrite to support wrapping cells . #206 #209 #228 csvtk cut/fmtdate/freq/grep/rename/rename2/replace/round : allow duplicated column names. csvtk csv2xlsx : optionally stores numbers as float. #217 csvtk xlsx2csv : fix bug where xlsx2csv treats small number (padj < 1e-25) as 0. It's solved by updating the excelize package. #261 csvtk join : a new flag for adding filename as column name prefix. by @tetedange13 #202 csvtk mutate2 : fix wrongly treating strings like E10 as numbers in scientific notation. #219 csvtk sep : fix the logic. #218 csvtk space2tab : fix \"bufio.Scanner: token too long\". #231 csvtk v0.25.0 csvtk : report empty files. csvtk join : fix loading file with no records. csvtk filter2/muate2 : support variable format of ${var} with special charactors including commas, spaces, and parentheses, e.g., ${a,b} , ${a b} , or ${a (b)} . #186 csvtk sort : fix checking non-existed fileds. csvtk plot box/hist/line : new flag --skip-na for skipping missing data. #188 csvtk csv2xlsx : stores number as float. #192 csvtk summary : new functions argmin and argmax . #181 csvtk v0.24.0 Incompatible changes : csvtk mutate2/summary : mutate2 : remove the option -L/--digits . use the same option -w/--decimal-width to limit floats to N decimal points. new command csvtk fmtdate : format date of selected fields. #159 csvtk grep : fix bug for searching with -r -p . . csvtk csv2rst : fix bug for data containing unicode. #137 csvtk filter2 : fix bug for date expression. #146 csvtk mutate2/filter2 : change the way of rexpression evaluation. add custom functions: len() . #153 fix bug when using two or more columns with common prefixes in column names . #173 fix value with single or double quotes. #174 csvtk cut : new flags -m/--allow-missing-col and -b/--blank-missing-col . #156 csvtk pretty : still add header row for empty column. csvtk csv2md : better format. csvtk join : new flag -n/--ignore-null . #163 csvtk v0.23.0 new comand: csvtk csv2rst for converting CSV to reStructuredText format. #137 csvtk pretty : add header separator line. #123 csvtk mutate2/summary : fix message and doc. Thanks @VladimirAlexiev #127 csvtk mutate2 : fix null coalescence: ??. #129 csvtk genautocomplete : supports bash|zsh|fish|powershell. #126 csvtk cat : fix progress bar. #130 csvtk grep : new flag immediate-output . csvtk csv2xlsx : fix bug for table with > 26 columns. 138 csvtk v0.22.0 csvtk : global flag -t does not overide -D anymore . #114 If the executable/symlink name is tsvtk the -t/--tabs option for tab input is set . Thanks @bsipos. #117 new command: csvtk csv2xlsx for converting CSV/TSV file(s) to a single .xlsx file. new command: csvtk unfold for unfolding multiple values in cells of a field. #103 rename csvtk collapse to csvtk fold , for folding multiple values of a field into cells of groups. csvtk cut : support range format 2- to choose 2nd column to the end . #106 csvtk round : fix bug of failing to round scientific notation with value small than one, e.g., 7.1E-1 . csvtk v0.21.0 new command: csvtk nrow/ncol for printing number of rows or columns. new command: round to round float to n decimal places. #112 csvtk headers : file name and column index is optional outputted with new flag -v/--verbose . csvtk dim : new flags --tabluar , --cols , --rows , -n/--no-files . csvtk dim/ncol/nrow : can handle empty files now. #108 csvtk csv2json #104 : new flag -b/--blank : do not convert \"\", \"na\", \"n/a\", \"none\", \"null\", \".\" to null new flag -n/--parse-num : parse numeric values for nth column(s), multiple values are supported and \"a\"/\"all\" for all columns. csvtk xlsx2csv : fix output for ragged table. #110 csvtk join : fix bug for joining >2 files. csvtk uniq : new flag -n/--keep-n for keeping first N records of every key. csvtk cut : support repeatedly selecting columns. #106 csvtk v0.20.0 new command csvtk comb : compute combinations of items at every row. new command csvtk sep : separate column into multiple columns. #96 csvtk : list lines' number of illegal ( -I ) and empty ( -E ) rows. #97 new flag --infile-list for giving file of input files list (one file per line), if given, they are appended to files from cli arguments csvtk join : reenable flag -i/--ignore-case . #99 outer join is supported . #23 new flag -L/--left-join : left join, equals to -k/--keep-unmatched, exclusive with --outer-join new flag -O/--outer-join : outer join, exclusive with --left-join rename flag --fill to --na . csvtk filter2 : fix bug when column names start with digits, e.g., 1000g2015aug . Thank @VorontsovIE ( #44 ) csvtk concat : allow one input file. #98 csvtk mutate : new flag -R/--remove for removing input column. csvtk v0.19.1 csvtk : fix checking file existence. show friendly error message when giving empty field like csvtk cut -f a, b . csvtk summary : fix err of q1 and q3. #90 csvtk version : making checking update optional. csvtk v0.19.0 new commands by @bsipos : watch : online monitoring and histogram of selected field. corr : calculate Pearson correlation between numeric columns. cat : stream file and report progress. csvtk split : fix bug of repeatedly output header line when number of output files exceed value of --buf-groups . #83 csvtk plot hist : new option --percentiles to add percentiles to histogram x label. #88 csvtk v0.18.2 csvtk replace/rename2/splitxlsx : fix flag conflicts with global flag -I since v0.18.0. csvtk replace/rename2 : removing shorthand flag -I for --key-capt-idx . csvtk splitxlsx : changing shorthand flag of --sheet-index from -I to -N . csvtk v0.18.1 csvtk sort : fix mutiple-key-sort containing natural order sorting. #79 csvtk xlsx2csv : reacts to global flags -t , -T , -D and -E . #78 csvtk v0.18.0 csvtk : add new flag --ignore-illegal-row to skip illegal rows. #72 csvtk summary : add more textual/numeric operations. #64 csvtk sort : fix bug for sorting by columns with empty values. #70 csvtk grep : add new flag --delete-matched to delete a pattern right after being matched, this keeps the firstly matched data and speedups when using regular expressions. #77 csvtk v0.17.0 new command: csvtk add-header and csvtk del-header for adding/deleting column names. [#62] csvtk v0.16.0 new command: csvtk csv2json : convert CSV to JSON format. remove comand: csvtk stats2 . new command csvtk summary : summary statistics of selected digital fields (groupby group fields), usage and examples . #59 csvtk replace : add flag --nr-width : minimum width for {nr} in flag -r/--replacement. e.g., formating \"1\" to \"001\" by --nr-width 3 (default 1) csvtk rename2/replace : add flag -A, --kv-file-all-left-columns-as-value , for treating all columns except 1th one as value for kv-file with more than 2 columns. #56 csvtk v0.15.0 csvtk : add global flag -E/--ignore-empty-row to skip empty row. #50 csvtk mutate2 : add flag -s/--digits-as-string for not converting big digits into scientific notation. #46 csvtk sort : add support for sorting in natural order. #49 csvtk v0.14.0 csvtk : supporting multi-line fields by replacing multicorecsv with standard library encoding/csv , while losing support for metaline which was supported since v0.7.0 . It also gain a little speedup. csvtk sample : add flag -n/--line-number to print line number as the first column (\"n\") csvtk filter2 : fix bug when column names start with digits, e.g., 1000g2015aug ( #44 ) csvtk rename2 : add support for similar repalecement symbols {kv} and {nr} in csvtk replace csvtk v0.13.0 new command concat for concatenating CSV/TSV files by rows #38 csvtk : add support for environment variables for frequently used global flags #39 CSVTK_T for flag -t/--tabs CSVTK_H for flag -H/--no-header-row mutate2 : add support for eval expression WITHOUT column index symbol, so we can add some string constants #37 pretty : better support for files with duplicated column names csvtk v0.12.0 new command collapse : collapsing one field with selected fields as keys freq : keeping orignal order of keys by default split : performance improvement add option -G/--out-gzip for forcing output gzipped file csvtk v0.11.0 add command split to split CSV/TSV into multiple files according to column values add command splitxlxs to split XLSX sheet into multiple sheets according to column values csvtk , automatically check BOM (byte-order mark) and discard it csvtk v0.10.0 add subcommand xlsx2csv to convert XLSX to CSV format grep , filter , filter2 : add flag -n/--line-number to print line-number as the first column cut : add flag -i/--ignore-case to ignore case of column name csvtk v0.9.1 csvtk replace : fix bug when replacing with key-value pairs brought in v0.8.0 csvtk v0.9.0 add subcommand csvtk mutate2 : create new column from selected fields by awk-like arithmetic/string expressions add new command genautocomplete to generate shell autocompletion script! csvtk v0.8.0 new command csvtk gather for gathering columns into key-value pairs . csvtk sort : support sorting by user-defined order . fix bug of unselecting field : wrongly reporting error of fields not existing. affected commands: cut , filter , fitler2 , freq , grep , inter , mutate , rename , rename2 , replace , stats2 , uniq . update help message of flag -F/--fuzzy-fields . update help message of global flag -t , which overrides both -d and -D . If you want other delimiter for tabular input, use -t $'\\t' -D \"delimiter\" . csvtk v0.7.1 csvtk plot box and csvtk plot line : fix bugs for special cases of input compile with go1.8.1 csvtk v0.7.0 fig bug of \"stricter field checking\" in v0.6.0 and v0.6.1 when using flag -F/--fuzzy-fields csvtk pretty and csvtk csv2md : add attention that these commands treat the first row as header line and require them to be unique. csvtk stat renamed to csvtk stats , old name is still available as an alias. csvtk stat2 renamed to csvtk stats2 , old name is still available as an alias. issues/13 seamlessly support for data with meta line of separator declaration used by MS Excel . csvtk v0.6.1 csvtk cut : minor bug: panic when no fields given. i.e., csvtk cut . All relevant commands have been fixed. csvtk v0.6.0 csvtk grep : large performance improvement by discarding goroutine (multiple threads), and keeping output in order of input . Better column name checking and stricter field checking, fields out of range are not ignored now . Affected commands include cut , filter , freq , grep , inter , mutate , rename , rename2 , replace , stat2 , and uniq . New command: csvtk filter2 , filtering rows by arithmetic/string expressions like awk . csvtk v0.5.0 csvtk cut : delete flag -n/--names , move it to a new command csvtk headers new command: csvtk headers new command: csvtk head new command: csvtk sample csvtk v0.4.6 csvtk grep : fix result highlight when flag -v is on. csvtk v0.4.5 csvtk join : support the 2nd or later files with entries with same ID. csvtk v0.4.4 add command csvtk freq : frequencies of selected fields add lots of examples in usage page csvtk v0.4.3 improvement of using experience: flag -n is not required anymore when flag -H in csvtk mutate csvtk v0.4.2 fix highlight bug of csvtk grep : if the pattern matches multiple parts, the text will be wrongly edited. changes: disable highlight when pattern file given. change the default output of all ploting commands to STDOUT, now you can pipe the image to \"display\" command of Imagemagic. csvtk v0.4.1 Nothing changed. Just fix the links due to inappropriate deployment of v0.4.0 csvtk v0.4.0 add flag for csvtk replace : -K ( --keep-key ) keep the key as value when no value found for the key. This is open in default in previous versions. csvtk v0.3.9 fix bug: header row incomplete in csvtk sort result csvtk v0.3.8.1 fix bug of flag parsing library pflag , detail . The bug affected the csvtk grep -r -p , when value of -p contain \"[\" and \"]\" at the beginning or end, they are wrongly parsed. csvtk v0.3.8 new feature: csvtk cut supports ordered fields output. e.g., csvtk cut -f 2,1 outputs the 2nd column in front of 1th column. new commands: csvtk plot can plot three types of plots by subcommands: csvtk plot hist : histogram csvtk plot box : boxplot csvtk plot line : line plot and scatter plot csvtk v0.3.7 fix a serious bug of using negative field of column name, e.g. -f \"-id\" csvtk v0.3.6 csvtk replace support replacement symbols {nr} (record number) and {kv} (corresponding value of the key ($1) by key-value file) csvtk v0.3.5.2 add flag --fill for csvtk join , so we can fill the unmatched data fix typo csvtk v0.3.5.1 fix minor bug of reading lines ending with \\r\\n from a dependency package csvtk v0.3.5 fix minor bug of csv2md add subcommand version which could check for update csvtk v0.3.4 fix bug of csvtk replace that head row should not be edited. csvtk v0.3.3 fix bug of csvtk grep -t -P csvtk v0.3.2 fix bug of inter csvtk v0.3.1 add support of search multiple fields for grep csvtk v0.3 add subcommand csv2md csvtk v0.2.9 add more flags to subcommand pretty fix bug of csvtk cut -n add subcommand filter csvtk v0.2.8 add subcommand pretty -- convert CSV to readable aligned table csvtk v0.2.7 fix highlight failing in windows csvtk v0.2.6 fix one error message of grep highlight matched fields in result of grep csvtk v0.2.5 fix bug of stat that failed to considerate files with header row add subcommand stat2 - summary of selected number fields make the output of stat prettier csvtk v0.2.4 fix bug of handling comment lines add some notes before using csvtk csvtk v0.2.3 add flag --colnames to cut flag -f ( --fields ) of join supports single value now csvtk v0.2.2 add flag --keep-unmathed to join csvtk v0.2 finish almost functions csvtk v0.2.1 fix bug of mutate /** * RECOMMENDED CONFIGURATION VARIABLES: EDIT AND UNCOMMENT THE SECTION BELOW TO INSERT DYNAMIC VALUES FROM YOUR PLATFORM OR CMS. * LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT: https://disqus.com/admin/universalcode/#configuration-variables */ /* var disqus_config = function () { this.page.url = PAGE_URL; // Replace PAGE_URL with your page's canonical URL variable this.page.identifier = PAGE_IDENTIFIER; // Replace PAGE_IDENTIFIER with your page's unique identifier variable }; */ (function() { // DON'T EDIT BELOW THIS LINE var d = document, s = d.createElement('script'); s.src = '//csvtk.disqus.com/embed.js'; s.setAttribute('data-timestamp', +new Date()); (d.head || d.body).appendChild(s); })(); Please enable JavaScript to view the comments powered by Disqus.","title":"Download"},{"location":"download/#download","text":"csvtk is implemented in Go programming language, executable binary files for most popular operating system are freely available in release page.","title":"Download"},{"location":"download/#current-version","text":"csvtk v0.27.2 csvtk pretty : fix the bug of empty first row with -H/--no-header-row , introduced in v0.27.0. new style 3line for three-line table. csvtk csv2xlsx : binaries compiled with go1.21 would result in a broken xlsx file. #243 csvtk splitxlsx : fix the error of invalid worksheet index . #1617","title":"Current Version"},{"location":"download/#links","text":"OS Arch File, \u4e2d\u56fd\u955c\u50cf Download Count Linux 32-bit csvtk_linux_386.tar.gz , \u4e2d\u56fd\u955c\u50cf Linux 64-bit csvtk_linux_amd64.tar.gz , \u4e2d\u56fd\u955c\u50cf Linux 64-bit csvtk_linux_arm64.tar.gz , \u4e2d\u56fd\u955c\u50cf macOS 64-bit csvtk_darwin_amd64.tar.gz , \u4e2d\u56fd\u955c\u50cf macOS arm64 csvtk_darwin_arm64.tar.gz , \u4e2d\u56fd\u955c\u50cf Windows 32-bit csvtk_windows_386.exe.tar.gz , \u4e2d\u56fd\u955c\u50cf Windows 64-bit csvtk_windows_amd64.exe.tar.gz , \u4e2d\u56fd\u955c\u50cf Notes run csvtk version to check update !!! run csvtk genautocomplete to update Bash completion !!!","title":"Links"},{"location":"download/#installation","text":"Download Page csvtk is implemented in Go programming language, executable binary files for most popular operating systems are freely available in release page.","title":"Installation"},{"location":"download/#method-1-download-binaries-latest-stabledev-version","text":"Just download compressed executable file of your operating system, and decompress it with tar -zxvf *.tar.gz command or other tools. And then: For Linux-like systems If you have root privilege simply copy it to /usr/local/bin : sudo cp csvtk /usr/local/bin/ Or copy to anywhere in the environment variable PATH : mkdir -p $HOME/bin/; cp csvtk $HOME/bin/ For windows , just copy csvtk.exe to C:\\WINDOWS\\system32 .","title":"Method 1: Download binaries (latest stable/dev version)"},{"location":"download/#method-2-install-via-conda-latest-stable-version","text":"conda install -c bioconda csvtk","title":"Method 2: Install via conda (latest stable version)"},{"location":"download/#method-3-install-via-homebrew-may-be-not-the-latest","text":"brew install csvtk","title":"Method 3: Install via homebrew (may be not the latest)"},{"location":"download/#method-4-for-go-developer-latest-stabledev-version","text":"go get -u github.com/shenwei356/csvtk/csvtk","title":"Method 4: For Go developer (latest stable/dev version)"},{"location":"download/#method-5-for-archlinux-aur-users-may-be-not-the-latest","text":"yaourt -S csvtk","title":"Method 5: For ArchLinux AUR users (may be not the latest)"},{"location":"download/#method-6-compiling-from-source-latest-stabledev-version","text":"# ------------------- install golang ----------------- # download Go from https://go.dev/dl wget https://go.dev/dl/go1.17.12.linux-amd64.tar.gz tar -zxf go1.17.12.linux-amd64.tar.gz -C $HOME/ # or # echo \"export PATH=$PATH:$HOME/go/bin\" >> ~/.bashrc # source ~/.bashrc export PATH=$PATH:$HOME/go/bin # ------------- the latest stable version ------------- go get -v -u github.com/shenwei356/csvtk/csvtk # The executable binary file is located in: # ~/go/bin/csvtk # You can also move it to anywhere in the $PATH mkdir -p $HOME/bin cp ~/go/bin/csvtk $HOME/bin/ # --------------- the development version -------------- git clone https://github.com/shenwei356/csvtk cd csvtk/csvtk/ go build # The executable binary file is located in: # ./csvtk # You can also move it to anywhere in the $PATH mkdir -p $HOME/bin cp ./csvtk $HOME/bin/","title":"Method 6: Compiling from source (latest stable/dev version)"},{"location":"download/#shell-completion","text":"Bash: # generate completion shell csvtk genautocomplete --shell bash # configure if never did. # install bash-completion if the \"complete\" command is not found. echo \"for bcfile in ~/.bash_completion.d/* ; do source \\$bcfile; done\" >> ~/.bash_completion echo \"source ~/.bash_completion\" >> ~/.bashrc Zsh: # generate completion shell csvtk genautocomplete --shell zsh --file ~/.zfunc/_csvtk # configure if never did echo 'fpath=( ~/.zfunc \"${fpath[@]}\" )' >> ~/.zshrc echo \"autoload -U compinit; compinit\" >> ~/.zshrc fish: csvtk genautocomplete --shell fish --file ~/.config/fish/completions/csvtk.fish","title":"Shell-completion"},{"location":"download/#release-history","text":"csvtk v0.27.1 csvtk filter2/mutate2 : fix the bug of selecting with field numbers. #242 csvtk v0.27.0 csvtk : code refactoring and simplifying code, with 16% less code. most commands support open column range syntax , e.g., csvtk grep -f 2- . #120 only selected column names are not allowed to be duplicated in the input data : box, corr, filter, filter2, fold, freq, gather, historysort, inter, join, line, mutate, mutate2, rename, replace, sep, split, summary, unfold, uniq, watch. Other commands do not have the restriction. #235 add a new global flag -Z/--show-row-number , supported commands: cut, csv2tab, csv2xlsx, tab2csv, pretty. the colum name of row number changes from \"n\" to \"row\": csv2xlsx, csv2tab, cut, filter, filter2, grep, pretty, sample, tab2csv. new command : csvtk spread : spread a key-value pair across multiple columns, like tidyr::spread/pivot_wider . #91 , #236 , #239 csvtk mutate/mutate2 : new flags --at , --before , --after for specifying the position of the new column . #193 csvtk cut : fix unselect range error. #234 fix -i/--ignore-case . csvtk pretty : allow align-center and align-right for specific columns . #240 csvtk round : fix bug of failing to round scientific notation with value small than one, e.g., 7.1E-1 . csvtk summary : fix duplicated columns. fix result error when multiple stats applied to the same column. csvtk corr/watch : rewrite and fix bug, support choosing fields with column names. csvtk v0.26.0 csvtk : near all commands skip empty files now . #204 the global flag --infile-list accepts stdin \"-\". #210 new command csvtk fix : fix CSV/TSV with different numbers of columns in rows . #226 csvtk pretty : rewrite to support wrapping cells . #206 #209 #228 csvtk cut/fmtdate/freq/grep/rename/rename2/replace/round : allow duplicated column names. csvtk csv2xlsx : optionally stores numbers as float. #217 csvtk xlsx2csv : fix bug where xlsx2csv treats small number (padj < 1e-25) as 0. It's solved by updating the excelize package. #261 csvtk join : a new flag for adding filename as column name prefix. by @tetedange13 #202 csvtk mutate2 : fix wrongly treating strings like E10 as numbers in scientific notation. #219 csvtk sep : fix the logic. #218 csvtk space2tab : fix \"bufio.Scanner: token too long\". #231 csvtk v0.25.0 csvtk : report empty files. csvtk join : fix loading file with no records. csvtk filter2/muate2 : support variable format of ${var} with special charactors including commas, spaces, and parentheses, e.g., ${a,b} , ${a b} , or ${a (b)} . #186 csvtk sort : fix checking non-existed fileds. csvtk plot box/hist/line : new flag --skip-na for skipping missing data. #188 csvtk csv2xlsx : stores number as float. #192 csvtk summary : new functions argmin and argmax . #181 csvtk v0.24.0 Incompatible changes : csvtk mutate2/summary : mutate2 : remove the option -L/--digits . use the same option -w/--decimal-width to limit floats to N decimal points. new command csvtk fmtdate : format date of selected fields. #159 csvtk grep : fix bug for searching with -r -p . . csvtk csv2rst : fix bug for data containing unicode. #137 csvtk filter2 : fix bug for date expression. #146 csvtk mutate2/filter2 : change the way of rexpression evaluation. add custom functions: len() . #153 fix bug when using two or more columns with common prefixes in column names . #173 fix value with single or double quotes. #174 csvtk cut : new flags -m/--allow-missing-col and -b/--blank-missing-col . #156 csvtk pretty : still add header row for empty column. csvtk csv2md : better format. csvtk join : new flag -n/--ignore-null . #163 csvtk v0.23.0 new comand: csvtk csv2rst for converting CSV to reStructuredText format. #137 csvtk pretty : add header separator line. #123 csvtk mutate2/summary : fix message and doc. Thanks @VladimirAlexiev #127 csvtk mutate2 : fix null coalescence: ??. #129 csvtk genautocomplete : supports bash|zsh|fish|powershell. #126 csvtk cat : fix progress bar. #130 csvtk grep : new flag immediate-output . csvtk csv2xlsx : fix bug for table with > 26 columns. 138 csvtk v0.22.0 csvtk : global flag -t does not overide -D anymore . #114 If the executable/symlink name is tsvtk the -t/--tabs option for tab input is set . Thanks @bsipos. #117 new command: csvtk csv2xlsx for converting CSV/TSV file(s) to a single .xlsx file. new command: csvtk unfold for unfolding multiple values in cells of a field. #103 rename csvtk collapse to csvtk fold , for folding multiple values of a field into cells of groups. csvtk cut : support range format 2- to choose 2nd column to the end . #106 csvtk round : fix bug of failing to round scientific notation with value small than one, e.g., 7.1E-1 . csvtk v0.21.0 new command: csvtk nrow/ncol for printing number of rows or columns. new command: round to round float to n decimal places. #112 csvtk headers : file name and column index is optional outputted with new flag -v/--verbose . csvtk dim : new flags --tabluar , --cols , --rows , -n/--no-files . csvtk dim/ncol/nrow : can handle empty files now. #108 csvtk csv2json #104 : new flag -b/--blank : do not convert \"\", \"na\", \"n/a\", \"none\", \"null\", \".\" to null new flag -n/--parse-num : parse numeric values for nth column(s), multiple values are supported and \"a\"/\"all\" for all columns. csvtk xlsx2csv : fix output for ragged table. #110 csvtk join : fix bug for joining >2 files. csvtk uniq : new flag -n/--keep-n for keeping first N records of every key. csvtk cut : support repeatedly selecting columns. #106 csvtk v0.20.0 new command csvtk comb : compute combinations of items at every row. new command csvtk sep : separate column into multiple columns. #96 csvtk : list lines' number of illegal ( -I ) and empty ( -E ) rows. #97 new flag --infile-list for giving file of input files list (one file per line), if given, they are appended to files from cli arguments csvtk join : reenable flag -i/--ignore-case . #99 outer join is supported . #23 new flag -L/--left-join : left join, equals to -k/--keep-unmatched, exclusive with --outer-join new flag -O/--outer-join : outer join, exclusive with --left-join rename flag --fill to --na . csvtk filter2 : fix bug when column names start with digits, e.g., 1000g2015aug . Thank @VorontsovIE ( #44 ) csvtk concat : allow one input file. #98 csvtk mutate : new flag -R/--remove for removing input column. csvtk v0.19.1 csvtk : fix checking file existence. show friendly error message when giving empty field like csvtk cut -f a, b . csvtk summary : fix err of q1 and q3. #90 csvtk version : making checking update optional. csvtk v0.19.0 new commands by @bsipos : watch : online monitoring and histogram of selected field. corr : calculate Pearson correlation between numeric columns. cat : stream file and report progress. csvtk split : fix bug of repeatedly output header line when number of output files exceed value of --buf-groups . #83 csvtk plot hist : new option --percentiles to add percentiles to histogram x label. #88 csvtk v0.18.2 csvtk replace/rename2/splitxlsx : fix flag conflicts with global flag -I since v0.18.0. csvtk replace/rename2 : removing shorthand flag -I for --key-capt-idx . csvtk splitxlsx : changing shorthand flag of --sheet-index from -I to -N . csvtk v0.18.1 csvtk sort : fix mutiple-key-sort containing natural order sorting. #79 csvtk xlsx2csv : reacts to global flags -t , -T , -D and -E . #78 csvtk v0.18.0 csvtk : add new flag --ignore-illegal-row to skip illegal rows. #72 csvtk summary : add more textual/numeric operations. #64 csvtk sort : fix bug for sorting by columns with empty values. #70 csvtk grep : add new flag --delete-matched to delete a pattern right after being matched, this keeps the firstly matched data and speedups when using regular expressions. #77 csvtk v0.17.0 new command: csvtk add-header and csvtk del-header for adding/deleting column names. [#62] csvtk v0.16.0 new command: csvtk csv2json : convert CSV to JSON format. remove comand: csvtk stats2 . new command csvtk summary : summary statistics of selected digital fields (groupby group fields), usage and examples . #59 csvtk replace : add flag --nr-width : minimum width for {nr} in flag -r/--replacement. e.g., formating \"1\" to \"001\" by --nr-width 3 (default 1) csvtk rename2/replace : add flag -A, --kv-file-all-left-columns-as-value , for treating all columns except 1th one as value for kv-file with more than 2 columns. #56 csvtk v0.15.0 csvtk : add global flag -E/--ignore-empty-row to skip empty row. #50 csvtk mutate2 : add flag -s/--digits-as-string for not converting big digits into scientific notation. #46 csvtk sort : add support for sorting in natural order. #49 csvtk v0.14.0 csvtk : supporting multi-line fields by replacing multicorecsv with standard library encoding/csv , while losing support for metaline which was supported since v0.7.0 . It also gain a little speedup. csvtk sample : add flag -n/--line-number to print line number as the first column (\"n\") csvtk filter2 : fix bug when column names start with digits, e.g., 1000g2015aug ( #44 ) csvtk rename2 : add support for similar repalecement symbols {kv} and {nr} in csvtk replace csvtk v0.13.0 new command concat for concatenating CSV/TSV files by rows #38 csvtk : add support for environment variables for frequently used global flags #39 CSVTK_T for flag -t/--tabs CSVTK_H for flag -H/--no-header-row mutate2 : add support for eval expression WITHOUT column index symbol, so we can add some string constants #37 pretty : better support for files with duplicated column names csvtk v0.12.0 new command collapse : collapsing one field with selected fields as keys freq : keeping orignal order of keys by default split : performance improvement add option -G/--out-gzip for forcing output gzipped file csvtk v0.11.0 add command split to split CSV/TSV into multiple files according to column values add command splitxlxs to split XLSX sheet into multiple sheets according to column values csvtk , automatically check BOM (byte-order mark) and discard it csvtk v0.10.0 add subcommand xlsx2csv to convert XLSX to CSV format grep , filter , filter2 : add flag -n/--line-number to print line-number as the first column cut : add flag -i/--ignore-case to ignore case of column name csvtk v0.9.1 csvtk replace : fix bug when replacing with key-value pairs brought in v0.8.0 csvtk v0.9.0 add subcommand csvtk mutate2 : create new column from selected fields by awk-like arithmetic/string expressions add new command genautocomplete to generate shell autocompletion script! csvtk v0.8.0 new command csvtk gather for gathering columns into key-value pairs . csvtk sort : support sorting by user-defined order . fix bug of unselecting field : wrongly reporting error of fields not existing. affected commands: cut , filter , fitler2 , freq , grep , inter , mutate , rename , rename2 , replace , stats2 , uniq . update help message of flag -F/--fuzzy-fields . update help message of global flag -t , which overrides both -d and -D . If you want other delimiter for tabular input, use -t $'\\t' -D \"delimiter\" . csvtk v0.7.1 csvtk plot box and csvtk plot line : fix bugs for special cases of input compile with go1.8.1 csvtk v0.7.0 fig bug of \"stricter field checking\" in v0.6.0 and v0.6.1 when using flag -F/--fuzzy-fields csvtk pretty and csvtk csv2md : add attention that these commands treat the first row as header line and require them to be unique. csvtk stat renamed to csvtk stats , old name is still available as an alias. csvtk stat2 renamed to csvtk stats2 , old name is still available as an alias. issues/13 seamlessly support for data with meta line of separator declaration used by MS Excel . csvtk v0.6.1 csvtk cut : minor bug: panic when no fields given. i.e., csvtk cut . All relevant commands have been fixed. csvtk v0.6.0 csvtk grep : large performance improvement by discarding goroutine (multiple threads), and keeping output in order of input . Better column name checking and stricter field checking, fields out of range are not ignored now . Affected commands include cut , filter , freq , grep , inter , mutate , rename , rename2 , replace , stat2 , and uniq . New command: csvtk filter2 , filtering rows by arithmetic/string expressions like awk . csvtk v0.5.0 csvtk cut : delete flag -n/--names , move it to a new command csvtk headers new command: csvtk headers new command: csvtk head new command: csvtk sample csvtk v0.4.6 csvtk grep : fix result highlight when flag -v is on. csvtk v0.4.5 csvtk join : support the 2nd or later files with entries with same ID. csvtk v0.4.4 add command csvtk freq : frequencies of selected fields add lots of examples in usage page csvtk v0.4.3 improvement of using experience: flag -n is not required anymore when flag -H in csvtk mutate csvtk v0.4.2 fix highlight bug of csvtk grep : if the pattern matches multiple parts, the text will be wrongly edited. changes: disable highlight when pattern file given. change the default output of all ploting commands to STDOUT, now you can pipe the image to \"display\" command of Imagemagic. csvtk v0.4.1 Nothing changed. Just fix the links due to inappropriate deployment of v0.4.0 csvtk v0.4.0 add flag for csvtk replace : -K ( --keep-key ) keep the key as value when no value found for the key. This is open in default in previous versions. csvtk v0.3.9 fix bug: header row incomplete in csvtk sort result csvtk v0.3.8.1 fix bug of flag parsing library pflag , detail . The bug affected the csvtk grep -r -p , when value of -p contain \"[\" and \"]\" at the beginning or end, they are wrongly parsed. csvtk v0.3.8 new feature: csvtk cut supports ordered fields output. e.g., csvtk cut -f 2,1 outputs the 2nd column in front of 1th column. new commands: csvtk plot can plot three types of plots by subcommands: csvtk plot hist : histogram csvtk plot box : boxplot csvtk plot line : line plot and scatter plot csvtk v0.3.7 fix a serious bug of using negative field of column name, e.g. -f \"-id\" csvtk v0.3.6 csvtk replace support replacement symbols {nr} (record number) and {kv} (corresponding value of the key ($1) by key-value file) csvtk v0.3.5.2 add flag --fill for csvtk join , so we can fill the unmatched data fix typo csvtk v0.3.5.1 fix minor bug of reading lines ending with \\r\\n from a dependency package csvtk v0.3.5 fix minor bug of csv2md add subcommand version which could check for update csvtk v0.3.4 fix bug of csvtk replace that head row should not be edited. csvtk v0.3.3 fix bug of csvtk grep -t -P csvtk v0.3.2 fix bug of inter csvtk v0.3.1 add support of search multiple fields for grep csvtk v0.3 add subcommand csv2md csvtk v0.2.9 add more flags to subcommand pretty fix bug of csvtk cut -n add subcommand filter csvtk v0.2.8 add subcommand pretty -- convert CSV to readable aligned table csvtk v0.2.7 fix highlight failing in windows csvtk v0.2.6 fix one error message of grep highlight matched fields in result of grep csvtk v0.2.5 fix bug of stat that failed to considerate files with header row add subcommand stat2 - summary of selected number fields make the output of stat prettier csvtk v0.2.4 fix bug of handling comment lines add some notes before using csvtk csvtk v0.2.3 add flag --colnames to cut flag -f ( --fields ) of join supports single value now csvtk v0.2.2 add flag --keep-unmathed to join csvtk v0.2 finish almost functions csvtk v0.2.1 fix bug of mutate /** * RECOMMENDED CONFIGURATION VARIABLES: EDIT AND UNCOMMENT THE SECTION BELOW TO INSERT DYNAMIC VALUES FROM YOUR PLATFORM OR CMS. * LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT: https://disqus.com/admin/universalcode/#configuration-variables */ /* var disqus_config = function () { this.page.url = PAGE_URL; // Replace PAGE_URL with your page's canonical URL variable this.page.identifier = PAGE_IDENTIFIER; // Replace PAGE_IDENTIFIER with your page's unique identifier variable }; */ (function() { // DON'T EDIT BELOW THIS LINE var d = document, s = d.createElement('script'); s.src = '//csvtk.disqus.com/embed.js'; s.setAttribute('data-timestamp', +new Date()); (d.head || d.body).appendChild(s); })(); Please enable JavaScript to view the comments powered by Disqus.","title":"Release history"},{"location":"tutorial/","text":"Tutorial Analyzing OTU table Data Here is a mock OTU table from 16S rRNA sequencing result. Columns are sample IDs in format of \"GROUP.ID\" $ cat otu_table.csv Taxonomy,A.1,A.2,A.3,B.1,B.2,B.3,C.1,C.2 Proteobacteria,.13,.29,.13,.16,.13,.22,.30,.23 Firmicutes,.42,.06,.49,.41,.55,.41,.32,.38 Bacteroidetes,.19,.62,.12,.33,.16,.29,.34,.35 Deferribacteres,.17,.00,.24,.01,.01,.01,.01,.01 What a mess! Let's make it prettier! $ csvtk pretty otu_table.csv Taxonomy A.1 A.2 A.3 B.1 B.2 B.3 C.1 C.2 Proteobacteria .13 .29 .13 .16 .13 .22 .30 .23 Firmicutes .42 .06 .49 .41 .55 .41 .32 .38 Bacteroidetes .19 .62 .12 .33 .16 .29 .34 .35 Deferribacteres .17 .00 .24 .01 .01 .01 .01 .01 Steps Counting $ csvtk stat otu_table.csv file num_cols num_rows otu_table.csv 9 4 Column names $ csvtk headers otu_table.csv # otu_table.csv 1 Taxonomy 2 A.1 3 A.2 4 A.3 5 B.1 6 B.2 7 B.3 8 C.1 9 C.2 Convert to tab-delimited table $ csvtk csv2tab otu_table.csv Taxonomy A.1 A.2 A.3 B.1 B.2 B.3 C.1 C.2 Proteobacteria .13 .29 .13 .16 .13 .22 .30 .23 Firmicutes .42 .06 .49 .41 .55 .41 .32 .38 Bacteroidetes .19 .62 .12 .33 .16 .29 .34 .35 Deferribacteres .17 .00 .24 .01 .01 .01 .01 .01 Extract data of group A and B and save to file -o otu_table.gAB.csv $ csvtk cut -F -f \"Taxonomy,A.*,B.*\" otu_table.csv -o otu_table.gAB.csv $ csvtk pretty otu_table.gAB.csv Taxonomy A.1 A.2 A.3 B.1 B.2 B.3 Proteobacteria .13 .29 .13 .16 .13 .22 Firmicutes .42 .06 .49 .41 .55 .41 Bacteroidetes .19 .62 .12 .33 .16 .29 Deferribacteres .17 .00 .24 .01 .01 .01 Search some rows by fields. Matched parts will be highlighted as red $ csvtk grep -f Taxonomy -r -p \"tes\" otu_table.gAB.csv -T Result: Transpose $ csvtk transpose otu_table.gAB.csv -o otu_table.gAB.t.csv $ csvtk pretty otu_table.gAB.t.csv Taxonomy Proteobacteria Firmicutes Bacteroidetes Deferribacteres A.1 .13 .42 .19 .17 A.2 .29 .06 .62 .00 A.3 .13 .49 .12 .24 B.1 .16 .41 .33 .01 B.2 .13 .55 .16 .01 B.3 .22 .41 .29 .01 Rename name of the first column $ csvtk rename -f 1 -n \"sample\" otu_table.gAB.t.csv -o otu_table.gAB.t.r.csv $ csvtk pretty otu_table.gAB.t.r.csv sample Proteobacteria Firmicutes Bacteroidetes Deferribacteres A.1 .13 .42 .19 .17 A.2 .29 .06 .62 .00 A.3 .13 .49 .12 .24 B.1 .16 .41 .33 .01 B.2 .13 .55 .16 .01 B.3 .22 .41 .29 .01 Add group column $ csvtk mutate -p \"(.+?)\\.\" -n group otu_table.gAB.t.r.csv -o otu_table2.csv $ csvtk pretty otu_table2.csv sample Proteobacteria Firmicutes Bacteroidetes Deferribacteres group A.1 .13 .42 .19 .17 A A.2 .29 .06 .62 .00 A A.3 .13 .49 .12 .24 A B.1 .16 .41 .33 .01 B B.2 .13 .55 .16 .01 B B.3 .22 .41 .29 .01 B Rename groups: $ csvtk replace -f group -p \"A\" -r \"Ctrl\" otu_table2.csv \\ | csvtk replace -f group -p \"B\" -r \"Treatment\" \\ > otu_table3.csv $ csvtk pretty -s \" \" otu_table3.csv sample Proteobacteria Firmicutes Bacteroidetes Deferribacteres group A.1 .13 .42 .19 .17 Ctrl A.2 .29 .06 .62 .00 Ctrl A.3 .13 .49 .12 .24 Ctrl B.1 .16 .41 .33 .01 Treatment B.2 .13 .55 .16 .01 Treatment B.3 .22 .41 .29 .01 Treatment Sort by abundance of Proteobacteria in descending order. $ csvtk sort -k Proteobacteria:nr otu_table3.csv \\ | csvtk pretty -s \" \" sample Proteobacteria Firmicutes Bacteroidetes Deferribacteres group A.2 .29 .06 .62 .00 Ctrl B.3 .22 .41 .29 .01 Treatment B.1 .16 .41 .33 .01 Treatment B.2 .13 .55 .16 .01 Treatment A.3 .13 .49 .12 .24 Ctrl A.1 .13 .42 .19 .17 Ctrl Sort by abundance of Proteobacteria in descending order and Firmicutes in ascending order $ csvtk sort -k Proteobacteria:nr -k Firmicutes:n otu_table3.csv \\ | csvtk pretty -s \" \" sample Proteobacteria Firmicutes Bacteroidetes Deferribacteres group A.2 .29 .06 .62 .00 Ctrl B.3 .22 .41 .29 .01 Treatment B.1 .16 .41 .33 .01 Treatment A.1 .13 .42 .19 .17 Ctrl A.3 .13 .49 .12 .24 Ctrl B.2 .13 .55 .16 .01 Treatment Filter samples with abundance greater than 0 in all taxons (columns except for sample and group, you can also use -f \"2-5>0\" ). $ cat otu_table3.csv \\ | csvtk filter -f \"2-5>0\" \\ | csvtk pretty -s \" \" sample Proteobacteria Firmicutes Bacteroidetes Deferribacteres group A.1 .13 .42 .19 .17 Ctrl A.3 .13 .49 .12 .24 Ctrl B.1 .16 .41 .33 .01 Treatment B.2 .13 .55 .16 .01 Treatment B.3 .22 .41 .29 .01 Treatment Most of the time, we may want to remove samples with abundance of 0 in all taxons. $ cat otu_table3.csv \\ | csvtk filter -f \"2-5>0\" --any \\ | csvtk pretty -s \" \" sample Proteobacteria Firmicutes Bacteroidetes Deferribacteres group A.1 .13 .42 .19 .17 Ctrl A.2 .29 .06 .62 .00 Ctrl A.3 .13 .49 .12 .24 Ctrl B.1 .16 .41 .33 .01 Treatment B.2 .13 .55 .16 .01 Treatment B.3 .22 .41 .29 .01 Treatment /** * RECOMMENDED CONFIGURATION VARIABLES: EDIT AND UNCOMMENT THE SECTION BELOW TO INSERT DYNAMIC VALUES FROM YOUR PLATFORM OR CMS. * LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT: https://disqus.com/admin/universalcode/#configuration-variables */ /* var disqus_config = function () { this.page.url = PAGE_URL; // Replace PAGE_URL with your page's canonical URL variable this.page.identifier = PAGE_IDENTIFIER; // Replace PAGE_IDENTIFIER with your page's unique identifier variable }; */ (function() { // DON'T EDIT BELOW THIS LINE var d = document, s = d.createElement('script'); s.src = '//csvtk.disqus.com/embed.js'; s.setAttribute('data-timestamp', +new Date()); (d.head || d.body).appendChild(s); })(); Please enable JavaScript to view the comments powered by Disqus.","title":"Tutorial"},{"location":"tutorial/#tutorial","text":"","title":"Tutorial"},{"location":"tutorial/#analyzing-otu-table","text":"","title":"Analyzing OTU table"},{"location":"tutorial/#data","text":"Here is a mock OTU table from 16S rRNA sequencing result. Columns are sample IDs in format of \"GROUP.ID\" $ cat otu_table.csv Taxonomy,A.1,A.2,A.3,B.1,B.2,B.3,C.1,C.2 Proteobacteria,.13,.29,.13,.16,.13,.22,.30,.23 Firmicutes,.42,.06,.49,.41,.55,.41,.32,.38 Bacteroidetes,.19,.62,.12,.33,.16,.29,.34,.35 Deferribacteres,.17,.00,.24,.01,.01,.01,.01,.01 What a mess! Let's make it prettier! $ csvtk pretty otu_table.csv Taxonomy A.1 A.2 A.3 B.1 B.2 B.3 C.1 C.2 Proteobacteria .13 .29 .13 .16 .13 .22 .30 .23 Firmicutes .42 .06 .49 .41 .55 .41 .32 .38 Bacteroidetes .19 .62 .12 .33 .16 .29 .34 .35 Deferribacteres .17 .00 .24 .01 .01 .01 .01 .01","title":"Data"},{"location":"tutorial/#steps","text":"Counting $ csvtk stat otu_table.csv file num_cols num_rows otu_table.csv 9 4 Column names $ csvtk headers otu_table.csv # otu_table.csv 1 Taxonomy 2 A.1 3 A.2 4 A.3 5 B.1 6 B.2 7 B.3 8 C.1 9 C.2 Convert to tab-delimited table $ csvtk csv2tab otu_table.csv Taxonomy A.1 A.2 A.3 B.1 B.2 B.3 C.1 C.2 Proteobacteria .13 .29 .13 .16 .13 .22 .30 .23 Firmicutes .42 .06 .49 .41 .55 .41 .32 .38 Bacteroidetes .19 .62 .12 .33 .16 .29 .34 .35 Deferribacteres .17 .00 .24 .01 .01 .01 .01 .01 Extract data of group A and B and save to file -o otu_table.gAB.csv $ csvtk cut -F -f \"Taxonomy,A.*,B.*\" otu_table.csv -o otu_table.gAB.csv $ csvtk pretty otu_table.gAB.csv Taxonomy A.1 A.2 A.3 B.1 B.2 B.3 Proteobacteria .13 .29 .13 .16 .13 .22 Firmicutes .42 .06 .49 .41 .55 .41 Bacteroidetes .19 .62 .12 .33 .16 .29 Deferribacteres .17 .00 .24 .01 .01 .01 Search some rows by fields. Matched parts will be highlighted as red $ csvtk grep -f Taxonomy -r -p \"tes\" otu_table.gAB.csv -T Result: Transpose $ csvtk transpose otu_table.gAB.csv -o otu_table.gAB.t.csv $ csvtk pretty otu_table.gAB.t.csv Taxonomy Proteobacteria Firmicutes Bacteroidetes Deferribacteres A.1 .13 .42 .19 .17 A.2 .29 .06 .62 .00 A.3 .13 .49 .12 .24 B.1 .16 .41 .33 .01 B.2 .13 .55 .16 .01 B.3 .22 .41 .29 .01 Rename name of the first column $ csvtk rename -f 1 -n \"sample\" otu_table.gAB.t.csv -o otu_table.gAB.t.r.csv $ csvtk pretty otu_table.gAB.t.r.csv sample Proteobacteria Firmicutes Bacteroidetes Deferribacteres A.1 .13 .42 .19 .17 A.2 .29 .06 .62 .00 A.3 .13 .49 .12 .24 B.1 .16 .41 .33 .01 B.2 .13 .55 .16 .01 B.3 .22 .41 .29 .01 Add group column $ csvtk mutate -p \"(.+?)\\.\" -n group otu_table.gAB.t.r.csv -o otu_table2.csv $ csvtk pretty otu_table2.csv sample Proteobacteria Firmicutes Bacteroidetes Deferribacteres group A.1 .13 .42 .19 .17 A A.2 .29 .06 .62 .00 A A.3 .13 .49 .12 .24 A B.1 .16 .41 .33 .01 B B.2 .13 .55 .16 .01 B B.3 .22 .41 .29 .01 B Rename groups: $ csvtk replace -f group -p \"A\" -r \"Ctrl\" otu_table2.csv \\ | csvtk replace -f group -p \"B\" -r \"Treatment\" \\ > otu_table3.csv $ csvtk pretty -s \" \" otu_table3.csv sample Proteobacteria Firmicutes Bacteroidetes Deferribacteres group A.1 .13 .42 .19 .17 Ctrl A.2 .29 .06 .62 .00 Ctrl A.3 .13 .49 .12 .24 Ctrl B.1 .16 .41 .33 .01 Treatment B.2 .13 .55 .16 .01 Treatment B.3 .22 .41 .29 .01 Treatment Sort by abundance of Proteobacteria in descending order. $ csvtk sort -k Proteobacteria:nr otu_table3.csv \\ | csvtk pretty -s \" \" sample Proteobacteria Firmicutes Bacteroidetes Deferribacteres group A.2 .29 .06 .62 .00 Ctrl B.3 .22 .41 .29 .01 Treatment B.1 .16 .41 .33 .01 Treatment B.2 .13 .55 .16 .01 Treatment A.3 .13 .49 .12 .24 Ctrl A.1 .13 .42 .19 .17 Ctrl Sort by abundance of Proteobacteria in descending order and Firmicutes in ascending order $ csvtk sort -k Proteobacteria:nr -k Firmicutes:n otu_table3.csv \\ | csvtk pretty -s \" \" sample Proteobacteria Firmicutes Bacteroidetes Deferribacteres group A.2 .29 .06 .62 .00 Ctrl B.3 .22 .41 .29 .01 Treatment B.1 .16 .41 .33 .01 Treatment A.1 .13 .42 .19 .17 Ctrl A.3 .13 .49 .12 .24 Ctrl B.2 .13 .55 .16 .01 Treatment Filter samples with abundance greater than 0 in all taxons (columns except for sample and group, you can also use -f \"2-5>0\" ). $ cat otu_table3.csv \\ | csvtk filter -f \"2-5>0\" \\ | csvtk pretty -s \" \" sample Proteobacteria Firmicutes Bacteroidetes Deferribacteres group A.1 .13 .42 .19 .17 Ctrl A.3 .13 .49 .12 .24 Ctrl B.1 .16 .41 .33 .01 Treatment B.2 .13 .55 .16 .01 Treatment B.3 .22 .41 .29 .01 Treatment Most of the time, we may want to remove samples with abundance of 0 in all taxons. $ cat otu_table3.csv \\ | csvtk filter -f \"2-5>0\" --any \\ | csvtk pretty -s \" \" sample Proteobacteria Firmicutes Bacteroidetes Deferribacteres group A.1 .13 .42 .19 .17 Ctrl A.2 .29 .06 .62 .00 Ctrl A.3 .13 .49 .12 .24 Ctrl B.1 .16 .41 .33 .01 Treatment B.2 .13 .55 .16 .01 Treatment B.3 .22 .41 .29 .01 Treatment /** * RECOMMENDED CONFIGURATION VARIABLES: EDIT AND UNCOMMENT THE SECTION BELOW TO INSERT DYNAMIC VALUES FROM YOUR PLATFORM OR CMS. * LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT: https://disqus.com/admin/universalcode/#configuration-variables */ /* var disqus_config = function () { this.page.url = PAGE_URL; // Replace PAGE_URL with your page's canonical URL variable this.page.identifier = PAGE_IDENTIFIER; // Replace PAGE_IDENTIFIER with your page's unique identifier variable }; */ (function() { // DON'T EDIT BELOW THIS LINE var d = document, s = d.createElement('script'); s.src = '//csvtk.disqus.com/embed.js'; s.setAttribute('data-timestamp', +new Date()); (d.head || d.body).appendChild(s); })(); Please enable JavaScript to view the comments powered by Disqus.","title":"Steps"},{"location":"usage/","text":"Usage and Examples Before use Attention The CSV parser requires all the lines have same number of fields/columns. Even lines with spaces will cause error. Use '-I/--ignore-illegal-row' to skip these lines if neccessary. You can also use 'csvtk fix' to fix files with different numbers of columns in rows. By default, csvtk thinks your files have header row, if not, switch flag \"-H\" on. Column names better be unique. By default, lines starting with \"#\" will be ignored, if the header row starts with \"#\", please assign flag \"-C\" another rare symbol, e.g. '$'. By default, csvtk handles CSV files, use flag \"-t\" for tab-delimited files. If double quotes exist in fields, use flag \"-l\". Do not mix use field (column) numbers and names. Table of Contents csvtk Information headers dim/nrow/ncol summary corr watch Format conversion pretty transpose space2tab csv2md csv2json csv2xlsx xlsx2csv Set operations head concat sample cut grep uniq freq inter filter filter2 join split splitxlsx comb Edit fix add-header del-header rename rename2 replace round mutate mutate2 sep gather spread unfold fold Ordering sort Ploting plot plot hist plot box plot line Misc cat genautocomplete csvtk Usage csvtk -- a cross-platform, efficient and practical CSV/TSV toolkit Version: 0.27.2 Author: Wei Shen
: . e.g., \"-k name:u -L name:level.txt Examples data $ cat testdata/names.csv id,first_name,last_name,username 11,\"Rob\",\"Pike\",rob 2,Ken,Thompson,ken 4,\"Robert\",\"Griesemer\",\"gri\" 1,\"Robert\",\"Thompson\",\"abc\" NA,\"Robert\",\"Abel\",\"123\" By single column : csvtk sort -k 1 or csvtk sort -k last_name in alphabetical order $ cat testdata/names.csv \\ | csvtk sort -k first_name id,first_name,last_name,username 2,Ken,Thompson,ken 11,Rob,Pike,rob NA,Robert,Abel,123 1,Robert,Thompson,abc 4,Robert,Griesemer,gri in reversed alphabetical order ( key:r ) $ cat testdata/names.csv \\ | csvtk sort -k first_name:r id,first_name,last_name,username NA,Robert,Abel,123 1,Robert,Thompson,abc 4,Robert,Griesemer,gri 11,Rob,Pike,rob 2,Ken,Thompson,ken in numerical order ( key:n ) $ cat testdata/names.csv \\ | csvtk sort -k id:n id,first_name,last_name,username NA,Robert,Abel,123 1,Robert,Thompson,abc 2,Ken,Thompson,ken 4,Robert,Griesemer,gri 11,Rob,Pike,rob in natural order ( key:N ) $ cat testdata/names.csv | csvtk sort -k id:N id,first_name,last_name,username 1,Robert,Thompson,abc 2,Ken,Thompson,ken 4,Robert,Griesemer,gri 11,Rob,Pike,rob NA,Robert,Abel,123 in natural order ( key:N ), a bioinformatics example $ echo \"X,Y,1,10,2,M,11,1_c,Un_g,1_g\" | csvtk transpose X Y 1 10 2 M 11 1_c Un_g 1_g $ echo \"X,Y,1,10,2,M,11,1_c,Un_g,1_g\" \\ | csvtk transpose \\ | csvtk sort -H -k 1:N 1 1_c 1_g 2 10 11 M Un_g X Y By multiple columns: csvtk sort -k 1,2 or csvtk sort -k 1 -k 2 or csvtk sort -k last_name,age # by first_name and then last_name $ cat testdata/names.csv | csvtk sort -k first_name -k last_name id,first_name,last_name,username 2,Ken,Thompson,ken 11,Rob,Pike,rob NA,Robert,Abel,123 4,Robert,Griesemer,gri 1,Robert,Thompson,abc # by first_name and then ID $ cat testdata/names.csv | csvtk sort -k first_name -k id:n id,first_name,last_name,username 2,Ken,Thompson,ken 11,Rob,Pike,rob NA,Robert,Abel,123 1,Robert,Thompson,abc 4,Robert,Griesemer,gri By user-defined order # user-defined order/level $ cat testdata/size_level.txt tiny mini small medium big # original data $ cat testdata/size.csv id,size 1,Huge 2,Tiny 3,Big 4,Small 5,Medium $ csvtk sort -k 2:u -i -L 2:testdata/size_level.txt testdata/size.csv id,size 2,Tiny 4,Small 5,Medium 3,Big 1,Huge plot Usage plot common figures Notes: 1. Output file can be set by flag -o/--out-file. 2. File format is determined by the out file suffix. Supported formats: eps, jpg|jpeg, pdf, png, svg, and tif|tiff 3. If flag -o/--out-file not set (default), image is written to stdout, you can display the image by pipping to \"display\" command of Imagemagic or just redirect to file. Usage: csvtk plot [command] Available Commands: box plot boxplot hist plot histogram line line plot and scatter plot Flags: --axis-width float axis width (default 1.5) -f, --data-field string column index or column name of data (default \"1\") --format string image format for stdout when flag -o/--out-file not given. available values: eps, jpg|jpeg, pdf, png, svg, and tif|tiff. (default \"png\") -g, --group-field string column index or column name of group --height float Figure height (default 4.5) -h, --help help for plot --label-size int label font size (default 14) --na-values strings NA values, case ignored (default [,NA,N/A]) --skip-na skip NA values in --na-values --tick-width float axis tick width (default 1.5) --title string Figure title --title-size int title font size (default 16) --width float Figure width (default 6) --x-max string maximum value of X axis --x-min string minimum value of X axis --xlab string x label text --y-max string maximum value of Y axis --y-min string minimum value of Y axis --ylab string y label text Note that most of the flags of plot are global flags of the subcommands hist , box and line Notes of image output Output file can be set by flag -o/--out-file. File format is determined by the out file suffix. Supported formats: eps, jpg|jpeg, pdf, png, svg, and tif|tiff If flag -o/--out-file not set (default), image is written to stdout, you can display the image by pipping to display command of Imagemagic or just redirect to file. plot hist Usage plot histogram Notes: 1. Output file can be set by flag -o/--out-file. 2. File format is determined by the out file suffix. Supported formats: eps, jpg|jpeg, pdf, png, svg, and tif|tiff 3. If flag -o/--out-file not set (default), image is written to stdout, you can display the image by pipping to \"display\" command of Imagemagic or just redirect to file. Usage: csvtk plot hist [flags] Flags: --bins int number of bins (default 50) --color-index int color index, 1-7 (default 1) Examples example data $ zcat testdata/grouped_data.tsv.gz | head -n 5 | csvtk -t pretty Group Length GC Content Group A 97 57.73 Group A 95 49.47 Group A 97 49.48 Group A 100 51.00 plot histogram with data of the second column: $ csvtk -t plot hist testdata/grouped_data.tsv.gz -f 2 \\ --title Histogram -o histogram.png You can also write image to stdout and pipe to \"display\" command of Imagemagic: $ csvtk -t plot hist testdata/grouped_data.tsv.gz -f 2 | display plot box Usage plot boxplot Notes: 1. Output file can be set by flag -o/--out-file. 2. File format is determined by the out file suffix. Supported formats: eps, jpg|jpeg, pdf, png, svg, and tif|tiff 3. If flag -o/--out-file not set (default), image is written to stdout, you can display the image by pipping to \"display\" command of Imagemagic or just redirect to file. Usage: csvtk plot box [flags] Flags: --box-width float box width --horiz horize box plot Examples plot boxplot with data of the \"GC Content\" (third) column, group information is the \"Group\" column. csvtk -t plot box testdata/grouped_data.tsv.gz -g \"Group\" -f \"GC Content\" \\ --width 3 --title \"Box plot\" \\ > boxplot.png plot horiz boxplot with data of the \"Length\" (second) column, group information is the \"Group\" column. $ csvtk -t plot box testdata/grouped_data.tsv.gz -g \"Group\" -f \"Length\" \\ --height 3 --width 5 --horiz --title \"Horiz box plot\" \\ > boxplot2.png` plot line Usage line plot and scatter plot Notes: 1. Output file can be set by flag -o/--out-file. 2. File format is determined by the out file suffix. Supported formats: eps, jpg|jpeg, pdf, png, svg, and tif|tiff 3. If flag -o/--out-file not set (default), image is written to stdout, you can display the image by pipping to \"display\" command of Imagemagic or just redirect to file. Usage: csvtk plot line [flags] Flags: -x, --data-field-x string column index or column name of X for command line -y, --data-field-y string column index or column name of Y for command line --legend-left locate legend along the left edge of the plot --legend-top locate legend along the top edge of the plot --line-width float line width (default 1.5) --point-size float point size (default 3) --scatter only plot points Examples example data $ head -n 5 testdata/xy.tsv Group X Y A 0 1 A 1 1.3 A 1.5 1.5 A 2.0 2 plot line plot with X-Y data $ csvtk -t plot line testdata/xy.tsv -x X -y Y -g Group \\ --title \"Line plot\" \\ > lineplot.png plot scatter $ csvtk -t plot line testdata/xy.tsv -x X -y Y -g Group \\ --title \"Scatter\" --scatter \\ > lineplot.png cat Usage stream file to stdout and report progress on stderr Usage: csvtk cat [flags] Flags: -b, --buffsize int buffer size (default 8192) -h, --help help for cat -L, --lines count lines instead of bytes -p, --print-freq int print frequency (-1 for print after parsing) (default 1) -s, --total int expected total bytes/lines (default -1) Examples Stream file, report progress in bytes csvtk cat file.tsv Stream file from stdin, report progress in lines tac input.tsv | csvtk cat -L -s `wc -l < input.tsv` - genautocomplete Usage generate shell autocompletion script Supported shell: bash|zsh|fish|powershell Bash: # generate completion shell csvtk genautocomplete --shell bash # configure if never did. # install bash-completion if the \"complete\" command is not found. echo \"for bcfile in ~/.bash_completion.d/* ; do source \\$bcfile; done\" >> ~/.bash_completion echo \"source ~/.bash_completion\" >> ~/.bashrc Zsh: # generate completion shell csvtk genautocomplete --shell zsh --file ~/.zfunc/_csvtk # configure if never did echo 'fpath=( ~/.zfunc \"${fpath[@]}\" )' >> ~/.zshrc echo \"autoload -U compinit; compinit\" >> ~/.zshrc fish: csvtk genautocomplete --shell fish --file ~/.config/fish/completions/csvtk.fish Usage: csvtk genautocomplete [flags] Flags: --file string autocompletion file (default \"/home/shenwei/.bash_completion.d/csvtk.sh\") -h, --help help for genautocomplete --shell string autocompletion type (bash|zsh|fish|powershell) (default \"bash\") /** * RECOMMENDED CONFIGURATION VARIABLES: EDIT AND UNCOMMENT THE SECTION BELOW TO INSERT DYNAMIC VALUES FROM YOUR PLATFORM OR CMS. * LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT: https://disqus.com/admin/universalcode/#configuration-variables */ /* var disqus_config = function () { this.page.url = PAGE_URL; // Replace PAGE_URL with your page's canonical URL variable this.page.identifier = PAGE_IDENTIFIER; // Replace PAGE_IDENTIFIER with your page's unique identifier variable }; */ (function() { // DON'T EDIT BELOW THIS LINE var d = document, s = d.createElement('script'); s.src = '//csvtk.disqus.com/embed.js'; s.setAttribute('data-timestamp', +new Date()); (d.head || d.body).appendChild(s); })(); Please enable JavaScript to view the comments powered by Disqus.","title":"Usage"},{"location":"usage/#usage-and-examples","text":"","title":"Usage and Examples"},{"location":"usage/#before-use","text":"Attention The CSV parser requires all the lines have same number of fields/columns. Even lines with spaces will cause error. Use '-I/--ignore-illegal-row' to skip these lines if neccessary. You can also use 'csvtk fix' to fix files with different numbers of columns in rows. By default, csvtk thinks your files have header row, if not, switch flag \"-H\" on. Column names better be unique. By default, lines starting with \"#\" will be ignored, if the header row starts with \"#\", please assign flag \"-C\" another rare symbol, e.g. '$'. By default, csvtk handles CSV files, use flag \"-t\" for tab-delimited files. If double quotes exist in fields, use flag \"-l\". Do not mix use field (column) numbers and names.","title":"Before use"},{"location":"usage/#table-of-contents","text":"csvtk Information headers dim/nrow/ncol summary corr watch Format conversion pretty transpose space2tab csv2md csv2json csv2xlsx xlsx2csv Set operations head concat sample cut grep uniq freq inter filter filter2 join split splitxlsx comb Edit fix add-header del-header rename rename2 replace round mutate mutate2 sep gather spread unfold fold Ordering sort Ploting plot plot hist plot box plot line Misc cat genautocomplete","title":"Table of Contents"},{"location":"usage/#csvtk","text":"Usage csvtk -- a cross-platform, efficient and practical CSV/TSV toolkit Version: 0.27.2 Author: Wei Shen : . e.g., \"-k name:u -L name:level.txt Examples data $ cat testdata/names.csv id,first_name,last_name,username 11,\"Rob\",\"Pike\",rob 2,Ken,Thompson,ken 4,\"Robert\",\"Griesemer\",\"gri\" 1,\"Robert\",\"Thompson\",\"abc\" NA,\"Robert\",\"Abel\",\"123\" By single column : csvtk sort -k 1 or csvtk sort -k last_name in alphabetical order $ cat testdata/names.csv \\ | csvtk sort -k first_name id,first_name,last_name,username 2,Ken,Thompson,ken 11,Rob,Pike,rob NA,Robert,Abel,123 1,Robert,Thompson,abc 4,Robert,Griesemer,gri in reversed alphabetical order ( key:r ) $ cat testdata/names.csv \\ | csvtk sort -k first_name:r id,first_name,last_name,username NA,Robert,Abel,123 1,Robert,Thompson,abc 4,Robert,Griesemer,gri 11,Rob,Pike,rob 2,Ken,Thompson,ken in numerical order ( key:n ) $ cat testdata/names.csv \\ | csvtk sort -k id:n id,first_name,last_name,username NA,Robert,Abel,123 1,Robert,Thompson,abc 2,Ken,Thompson,ken 4,Robert,Griesemer,gri 11,Rob,Pike,rob in natural order ( key:N ) $ cat testdata/names.csv | csvtk sort -k id:N id,first_name,last_name,username 1,Robert,Thompson,abc 2,Ken,Thompson,ken 4,Robert,Griesemer,gri 11,Rob,Pike,rob NA,Robert,Abel,123 in natural order ( key:N ), a bioinformatics example $ echo \"X,Y,1,10,2,M,11,1_c,Un_g,1_g\" | csvtk transpose X Y 1 10 2 M 11 1_c Un_g 1_g $ echo \"X,Y,1,10,2,M,11,1_c,Un_g,1_g\" \\ | csvtk transpose \\ | csvtk sort -H -k 1:N 1 1_c 1_g 2 10 11 M Un_g X Y By multiple columns: csvtk sort -k 1,2 or csvtk sort -k 1 -k 2 or csvtk sort -k last_name,age # by first_name and then last_name $ cat testdata/names.csv | csvtk sort -k first_name -k last_name id,first_name,last_name,username 2,Ken,Thompson,ken 11,Rob,Pike,rob NA,Robert,Abel,123 4,Robert,Griesemer,gri 1,Robert,Thompson,abc # by first_name and then ID $ cat testdata/names.csv | csvtk sort -k first_name -k id:n id,first_name,last_name,username 2,Ken,Thompson,ken 11,Rob,Pike,rob NA,Robert,Abel,123 1,Robert,Thompson,abc 4,Robert,Griesemer,gri By user-defined order # user-defined order/level $ cat testdata/size_level.txt tiny mini small medium big # original data $ cat testdata/size.csv id,size 1,Huge 2,Tiny 3,Big 4,Small 5,Medium $ csvtk sort -k 2:u -i -L 2:testdata/size_level.txt testdata/size.csv id,size 2,Tiny 4,Small 5,Medium 3,Big 1,Huge","title":"sort"},{"location":"usage/#plot","text":"Usage plot common figures Notes: 1. Output file can be set by flag -o/--out-file. 2. File format is determined by the out file suffix. Supported formats: eps, jpg|jpeg, pdf, png, svg, and tif|tiff 3. If flag -o/--out-file not set (default), image is written to stdout, you can display the image by pipping to \"display\" command of Imagemagic or just redirect to file. Usage: csvtk plot [command] Available Commands: box plot boxplot hist plot histogram line line plot and scatter plot Flags: --axis-width float axis width (default 1.5) -f, --data-field string column index or column name of data (default \"1\") --format string image format for stdout when flag -o/--out-file not given. available values: eps, jpg|jpeg, pdf, png, svg, and tif|tiff. (default \"png\") -g, --group-field string column index or column name of group --height float Figure height (default 4.5) -h, --help help for plot --label-size int label font size (default 14) --na-values strings NA values, case ignored (default [,NA,N/A]) --skip-na skip NA values in --na-values --tick-width float axis tick width (default 1.5) --title string Figure title --title-size int title font size (default 16) --width float Figure width (default 6) --x-max string maximum value of X axis --x-min string minimum value of X axis --xlab string x label text --y-max string maximum value of Y axis --y-min string minimum value of Y axis --ylab string y label text Note that most of the flags of plot are global flags of the subcommands hist , box and line Notes of image output Output file can be set by flag -o/--out-file. File format is determined by the out file suffix. Supported formats: eps, jpg|jpeg, pdf, png, svg, and tif|tiff If flag -o/--out-file not set (default), image is written to stdout, you can display the image by pipping to display command of Imagemagic or just redirect to file.","title":"plot"},{"location":"usage/#plot-hist","text":"Usage plot histogram Notes: 1. Output file can be set by flag -o/--out-file. 2. File format is determined by the out file suffix. Supported formats: eps, jpg|jpeg, pdf, png, svg, and tif|tiff 3. If flag -o/--out-file not set (default), image is written to stdout, you can display the image by pipping to \"display\" command of Imagemagic or just redirect to file. Usage: csvtk plot hist [flags] Flags: --bins int number of bins (default 50) --color-index int color index, 1-7 (default 1) Examples example data $ zcat testdata/grouped_data.tsv.gz | head -n 5 | csvtk -t pretty Group Length GC Content Group A 97 57.73 Group A 95 49.47 Group A 97 49.48 Group A 100 51.00 plot histogram with data of the second column: $ csvtk -t plot hist testdata/grouped_data.tsv.gz -f 2 \\ --title Histogram -o histogram.png You can also write image to stdout and pipe to \"display\" command of Imagemagic: $ csvtk -t plot hist testdata/grouped_data.tsv.gz -f 2 | display","title":"plot hist"},{"location":"usage/#plot-box","text":"Usage plot boxplot Notes: 1. Output file can be set by flag -o/--out-file. 2. File format is determined by the out file suffix. Supported formats: eps, jpg|jpeg, pdf, png, svg, and tif|tiff 3. If flag -o/--out-file not set (default), image is written to stdout, you can display the image by pipping to \"display\" command of Imagemagic or just redirect to file. Usage: csvtk plot box [flags] Flags: --box-width float box width --horiz horize box plot Examples plot boxplot with data of the \"GC Content\" (third) column, group information is the \"Group\" column. csvtk -t plot box testdata/grouped_data.tsv.gz -g \"Group\" -f \"GC Content\" \\ --width 3 --title \"Box plot\" \\ > boxplot.png plot horiz boxplot with data of the \"Length\" (second) column, group information is the \"Group\" column. $ csvtk -t plot box testdata/grouped_data.tsv.gz -g \"Group\" -f \"Length\" \\ --height 3 --width 5 --horiz --title \"Horiz box plot\" \\ > boxplot2.png`","title":"plot box"},{"location":"usage/#plot-line","text":"Usage line plot and scatter plot Notes: 1. Output file can be set by flag -o/--out-file. 2. File format is determined by the out file suffix. Supported formats: eps, jpg|jpeg, pdf, png, svg, and tif|tiff 3. If flag -o/--out-file not set (default), image is written to stdout, you can display the image by pipping to \"display\" command of Imagemagic or just redirect to file. Usage: csvtk plot line [flags] Flags: -x, --data-field-x string column index or column name of X for command line -y, --data-field-y string column index or column name of Y for command line --legend-left locate legend along the left edge of the plot --legend-top locate legend along the top edge of the plot --line-width float line width (default 1.5) --point-size float point size (default 3) --scatter only plot points Examples example data $ head -n 5 testdata/xy.tsv Group X Y A 0 1 A 1 1.3 A 1.5 1.5 A 2.0 2 plot line plot with X-Y data $ csvtk -t plot line testdata/xy.tsv -x X -y Y -g Group \\ --title \"Line plot\" \\ > lineplot.png plot scatter $ csvtk -t plot line testdata/xy.tsv -x X -y Y -g Group \\ --title \"Scatter\" --scatter \\ > lineplot.png","title":"plot line"},{"location":"usage/#cat","text":"Usage stream file to stdout and report progress on stderr Usage: csvtk cat [flags] Flags: -b, --buffsize int buffer size (default 8192) -h, --help help for cat -L, --lines count lines instead of bytes -p, --print-freq int print frequency (-1 for print after parsing) (default 1) -s, --total int expected total bytes/lines (default -1) Examples Stream file, report progress in bytes csvtk cat file.tsv Stream file from stdin, report progress in lines tac input.tsv | csvtk cat -L -s `wc -l < input.tsv` -","title":"cat"},{"location":"usage/#genautocomplete","text":"Usage generate shell autocompletion script Supported shell: bash|zsh|fish|powershell Bash: # generate completion shell csvtk genautocomplete --shell bash # configure if never did. # install bash-completion if the \"complete\" command is not found. echo \"for bcfile in ~/.bash_completion.d/* ; do source \\$bcfile; done\" >> ~/.bash_completion echo \"source ~/.bash_completion\" >> ~/.bashrc Zsh: # generate completion shell csvtk genautocomplete --shell zsh --file ~/.zfunc/_csvtk # configure if never did echo 'fpath=( ~/.zfunc \"${fpath[@]}\" )' >> ~/.zshrc echo \"autoload -U compinit; compinit\" >> ~/.zshrc fish: csvtk genautocomplete --shell fish --file ~/.config/fish/completions/csvtk.fish Usage: csvtk genautocomplete [flags] Flags: --file string autocompletion file (default \"/home/shenwei/.bash_completion.d/csvtk.sh\") -h, --help help for genautocomplete --shell string autocompletion type (bash|zsh|fish|powershell) (default \"bash\") /** * RECOMMENDED CONFIGURATION VARIABLES: EDIT AND UNCOMMENT THE SECTION BELOW TO INSERT DYNAMIC VALUES FROM YOUR PLATFORM OR CMS. * LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT: https://disqus.com/admin/universalcode/#configuration-variables */ /* var disqus_config = function () { this.page.url = PAGE_URL; // Replace PAGE_URL with your page's canonical URL variable this.page.identifier = PAGE_IDENTIFIER; // Replace PAGE_IDENTIFIER with your page's unique identifier variable }; */ (function() { // DON'T EDIT BELOW THIS LINE var d = document, s = d.createElement('script'); s.src = '//csvtk.disqus.com/embed.js'; s.setAttribute('data-timestamp', +new Date()); (d.head || d.body).appendChild(s); })(); Please enable JavaScript to view the comments powered by Disqus.","title":"genautocomplete"}]} \ No newline at end of file +{"config":{"indexing":"full","lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"csvtk - a cross-platform, efficient and practical CSV/TSV toolkit Documents: http://bioinf.shenwei.me/csvtk ( Usage and Tutorial ). \u4e2d\u6587\u4ecb\u7ecd Source code: https://github.com/shenwei356/csvtk Latest version: Introduction Similar to FASTA/Q format in field of Bioinformatics, CSV/TSV formats are basic and ubiquitous file formats in both Bioinformatics and data science. People usually use spreadsheet software like MS Excel to process table data. However this is all by clicking and typing, which is not automated and is time-consuming to repeat , especially when you want to apply similar operations with different datasets or purposes. You can also accomplish some CSV/TSV manipulations using shell commands, but more code is needed to handle the header line. Shell commands do not support selecting columns with column names either. csvtk is convenient for rapid data investigation and also easy to integrate into analysis pipelines . It could save you lots of time in (not) writing Python/R scripts. Table of Contents Features Subcommands Installation Command-line completion Compared to csvkit Examples Acknowledgements Contact License Starchart Features Cross-platform (Linux/Windows/Mac OS X/OpenBSD/FreeBSD) Light weight and out-of-the-box, no dependencies, no compilation, no configuration Fast , multiple-CPUs supported (some commands) Practical functions provided by N subcommands Support STDIN and gziped input/output file, easy being used in pipe Most of the subcommands support unselecting fields and fuzzy fields , e.g. -f \"-id,-name\" for all fields except \"id\" and \"name\", -F -f \"a.*\" for all fields with prefix \"a.\". Support some common plots (see usage ) Seamlessly support for data with meta line (e.g., sep=, ) of separator declaration used by MS Excel Subcommands 51 subcommands in total. Information headers : prints headers dim : dimensions of CSV file nrow : print number of records ncol : print number of columns summary : summary statistics of selected numeric or text fields (groupby group fields) watch : online monitoring and histogram of selected field corr : calculate Pearson correlation between numeric columns Format conversion pretty : converts CSV to a readable aligned table csv2tab : converts CSV to tabular format tab2csv : converts tabular format to CSV space2tab : converts space delimited format to TSVW transpose : transposes CSV data csv2md : converts CSV to markdown format csv2rst : converts CSV to reStructuredText format csv2json : converts CSV to JSON format csv2xlsx : converts CSV/TSV files to XLSX file xlsx2csv : converts XLSX to CSV format Set operations head : prints first N records concat : concatenates CSV/TSV files by rows sample : sampling by proportion cut : select and arrange fields grep : greps data by selected fields with patterns/regular expressions uniq : unique data without sorting freq : frequencies of selected fields inter : intersection of multiple files filter : filters rows by values of selected fields with arithmetic expression filter2 : filters rows by awk-like arithmetic/string expressions join : join files by selected fields (inner, left and outer join) split splits CSV/TSV into multiple files according to column values splitxlsx : splits XLSX sheet into multiple sheets according to column values comb : compute combinations of items at every row Edit fix : fix CSV/TSV with different numbers of columns in rows add-header : add column names del-header : delete column names rename : renames column names with new names rename2 : renames column names by regular expression replace : replaces data of selected fields by regular expression round : round float to n decimal places mutate : creates new columns from selected fields by regular expression mutate2 : creates a new column from selected fields by awk-like arithmetic/string expressions sep : separate column into multiple columns gather : gather columns into key-value pairs, like tidyr::gather/pivot_longer spread : spread a key-value pair across multiple columns, like tidyr::spread/pivot_wider unfold : unfold multiple values in cells of a field fold : fold multiple values of a field into cells of groups fmtdate : format date of selected fields Ordering sort : sorts by selected fields Ploting plot see usage plot hist histogram plot box boxplot plot line line plot and scatter plot Misc cat stream file and report progress version print version information and check for update genautocomplete generate shell autocompletion script (bash|zsh|fish|powershell) Installation Download Page csvtk is implemented in Go programming language, executable binary files for most popular operating systems are freely available in release page. Method 1: Download binaries (latest stable/dev version) Just download compressed executable file of your operating system, and decompress it with tar -zxvf *.tar.gz command or other tools. And then: For Linux-like systems If you have root privilege simply copy it to /usr/local/bin : sudo cp csvtk /usr/local/bin/ Or copy to anywhere in the environment variable PATH : mkdir -p $HOME/bin/; cp csvtk $HOME/bin/ For windows , just copy csvtk.exe to C:\\WINDOWS\\system32 . Method 2: Install via conda (latest stable version) conda install -c bioconda csvtk Method 3: Install via homebrew brew install csvtk Method 4: For Go developer (latest stable/dev version) go get -u github.com/shenwei356/csvtk/csvtk Method 5: For ArchLinux AUR users (may be not the latest) yaourt -S csvtk Command-line completion Bash: # generate completion shell csvtk genautocomplete --shell bash # configure if never did. # install bash-completion if the \"complete\" command is not found. echo \"for bcfile in ~/.bash_completion.d/* ; do source \\$bcfile; done\" >> ~/.bash_completion echo \"source ~/.bash_completion\" >> ~/.bashrc Zsh: # generate completion shell csvtk genautocomplete --shell zsh --file ~/.zfunc/_csvtk # configure if never did echo 'fpath=( ~/.zfunc \"${fpath[@]}\" )' >> ~/.zshrc echo \"autoload -U compinit; compinit\" >> ~/.zshrc fish: csvtk genautocomplete --shell fish --file ~/.config/fish/completions/csvtk.fish Compared to csvkit csvkit , attention: this table wasn't updated for 2 years. Features csvtk csvkit Note Read Gzip Yes Yes read gzip files Fields ranges Yes Yes e.g. -f 1-4,6 Unselect fileds Yes -- e.g. -1 for excluding first column Fuzzy fields Yes -- e.g. ab* for columns with name prefix \"ab\" Reorder fields Yes Yes it means -f 1,2 is different from -f 2,1 Rename columns Yes -- rename with new name(s) or from existed names Sort by multiple keys Yes Yes bash sort like operations Sort by number Yes -- e.g. -k 1:n Multiple sort Yes -- e.g. -k 2:r -k 1:nr Pretty output Yes Yes convert CSV to readable aligned table Unique data Yes -- unique data of selected fields frequency Yes -- frequencies of selected fields Sampling Yes -- sampling by proportion Mutate fields Yes -- create new columns from selected fields Replace Yes -- replace data of selected fields Similar tools: csvkit - A suite of utilities for converting to and working with CSV, the king of tabular file formats. http://csvkit.rtfd.org/ xsv - A fast CSV toolkit written in Rust. miller - Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV and tabular JSON http://johnkerl.org/miller tsv-utils - Command line utilities for tab-separated value files written in the D programming language. Examples More examples and tutorial . Attention The CSV parser requires all the lines have same number of fields/columns. Even lines with spaces will cause error. Use '-I/--ignore-illegal-row' to skip these lines if neccessary. By default, csvtk thinks your files have header row, if not, switch flag -H on. Column names better be unique. By default, lines starting with # will be ignored, if the header row starts with # , please assign flag -C another rare symbol, e.g. '$' . By default, csvtk handles CSV files, use flag -t for tab-delimited files. If \" exists in tab-delimited files, use flag -l . Do not mix use field (column) numbers and names. Examples Pretty result $ csvtk pretty names.csv id first_name last_name username -- ---------- --------- -------- 11 Rob Pike rob 2 Ken Thompson ken 4 Robert Griesemer gri 1 Robert Thompson abc NA Robert Abel 123 $ csvtk pretty names.csv -S 3line \u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501 id first_name last_name username ---------------------------------------- 11 Rob Pike rob 2 Ken Thompson ken 4 Robert Griesemer gri 1 Robert Thompson abc NA Robert Abel 123 \u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501 $ csvtk pretty names.csv -S bold -w 5 -m 1- \u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513 \u2503 id \u2503 first_name \u2503 last_name \u2503 username \u2503 \u2523\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u252b \u2503 11 \u2503 Rob \u2503 Pike \u2503 rob \u2503 \u2523\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u252b \u2503 2 \u2503 Ken \u2503 Thompson \u2503 ken \u2503 \u2523\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u252b \u2503 4 \u2503 Robert \u2503 Griesemer \u2503 gri \u2503 \u2523\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u252b \u2503 1 \u2503 Robert \u2503 Thompson \u2503 abc \u2503 \u2523\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u252b \u2503 NA \u2503 Robert \u2503 Abel \u2503 123 \u2503 \u2517\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u253b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u253b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u253b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u251b Summary of selected numeric fields, supporting \"group-by\" $ cat testdata/digitals2.csv \\ | csvtk summary -i -f f4:sum,f5:sum -g f1,f2 \\ | csvtk pretty f1 f2 f4:sum f5:sum bar xyz 7.00 106.00 bar xyz2 4.00 4.00 foo bar 6.00 3.00 foo bar2 4.50 5.00 Select fields/columns ( cut ) By index: csvtk cut -f 1,2 By names: csvtk cut -f first_name,username Unselect : csvtk cut -f -1,-2 or csvtk cut -f -first_name Fuzzy fields : csvtk cut -F -f \"*_name,username\" Field ranges: csvtk cut -f 2-4 for column 2,3,4 or csvtk cut -f -3--1 for discarding column 1,2,3 All fields: csvtk cut -f 1- or csvtk cut -F -f \"*\" Search by selected fields ( grep ) (matched parts will be highlighted as red) By exactly matching: csvtk grep -f first_name -p Robert -p Rob By regular expression: csvtk grep -f first_name -r -p Rob By pattern list: csvtk grep -f first_name -P name_list.txt Remore rows containing missing data (NA): csvtk grep -F -f \"*\" -r -p \"^$\" -v Rename column names ( rename and rename2 ) Setting new names: csvtk rename -f A,B -n a,b or csvtk rename -f 1-3 -n a,b,c Replacing with original names by regular express: csvtk rename2 -f 1- -p \"(.*)\" -r 'prefix_$1' for adding prefix to all column names. Edit data with regular expression ( replace ) Remove Chinese charactors: csvtk replace -F -f \"*_name\" -p \"\\p{Han}+\" -r \"\" Create new column from selected fields by regular expression ( mutate ) In default, copy a column: csvtk mutate -f id Extract prefix of data as group name (get \"A\" from \"A.1\" as group name): csvtk mutate -f sample -n group -p \"^(.+?)\\.\" --after sample Sort by multiple keys ( sort ) By single column : csvtk sort -k 1 or csvtk sort -k last_name By multiple columns: csvtk sort -k 1,2 or csvtk sort -k 1 -k 2 or csvtk sort -k last_name,age Sort by number: csvtk sort -k 1:n or csvtk sort -k 1:nr for reverse number Complex sort: csvtk sort -k region -k age:n -k id:nr In natural order: csvtk sort -k chr:N Join multiple files by keys ( join ) All files have same key column: csvtk join -f id file1.csv file2.csv Files have different key columns: csvtk join -f \"username;username;name\" names.csv phone.csv adress.csv -k Filter by numbers ( filter ) Single field: csvtk filter -f \"id>0\" Multiple fields : csvtk filter -f \"1-3>0\" Using --any to print record if any of the field satisfy the condition: csvtk filter -f \"1-3>0\" --any fuzzy fields : csvtk filter -F -f \"A*!=0\" Filter rows by awk-like arithmetic/string expressions ( filter2 ) Using field index: csvtk filter2 -f '$3>0' Using column names: csvtk filter2 -f '$id > 0' Both arithmetic and string expressions: csvtk filter2 -f '$id > 3 || $username==\"ken\"' More complicated: csvtk filter2 -H -t -f '$1 > 2 && $2 % 2 == 0' Ploting plot histogram with data of the second column: csvtk -t plot hist testdata/grouped_data.tsv.gz -f 2 | display plot boxplot with data of the \"GC Content\" (third) column, group information is the \"Group\" column. csvtk -t plot box testdata/grouped_data.tsv.gz -g \"Group\" \\ -f \"GC Content\" --width 3 | display plot horiz boxplot with data of the \"Length\" (second) column, group information is the \"Group\" column. csvtk -t plot box testdata/grouped_data.tsv.gz -g \"Group\" -f \"Length\" \\ --height 3 --width 5 --horiz --title \"Horiz box plot\" | display plot line plot with X-Y data csvtk -t plot line testdata/xy.tsv -x X -y Y -g Group | display plot scatter plot with X-Y data csvtk -t plot line testdata/xy.tsv -x X -y Y -g Group --scatter | display Acknowledgements We are grateful to Zhiluo Deng and Li Peng for suggesting features and reporting bugs. Thanks Albert Vilella for features suggestion, which makes csvtk feature-rich\u3002 Contact Create an issue to report bugs, propose new functions or ask for help. Or leave a comment . License MIT License Starchart","title":"Home"},{"location":"#csvtk-a-cross-platform-efficient-and-practical-csvtsv-toolkit","text":"Documents: http://bioinf.shenwei.me/csvtk ( Usage and Tutorial ). \u4e2d\u6587\u4ecb\u7ecd Source code: https://github.com/shenwei356/csvtk Latest version:","title":"csvtk - a cross-platform, efficient and practical CSV/TSV toolkit"},{"location":"#introduction","text":"Similar to FASTA/Q format in field of Bioinformatics, CSV/TSV formats are basic and ubiquitous file formats in both Bioinformatics and data science. People usually use spreadsheet software like MS Excel to process table data. However this is all by clicking and typing, which is not automated and is time-consuming to repeat , especially when you want to apply similar operations with different datasets or purposes. You can also accomplish some CSV/TSV manipulations using shell commands, but more code is needed to handle the header line. Shell commands do not support selecting columns with column names either. csvtk is convenient for rapid data investigation and also easy to integrate into analysis pipelines . It could save you lots of time in (not) writing Python/R scripts.","title":"Introduction"},{"location":"#table-of-contents","text":"Features Subcommands Installation Command-line completion Compared to csvkit Examples Acknowledgements Contact License Starchart","title":"Table of Contents"},{"location":"#features","text":"Cross-platform (Linux/Windows/Mac OS X/OpenBSD/FreeBSD) Light weight and out-of-the-box, no dependencies, no compilation, no configuration Fast , multiple-CPUs supported (some commands) Practical functions provided by N subcommands Support STDIN and gziped input/output file, easy being used in pipe Most of the subcommands support unselecting fields and fuzzy fields , e.g. -f \"-id,-name\" for all fields except \"id\" and \"name\", -F -f \"a.*\" for all fields with prefix \"a.\". Support some common plots (see usage ) Seamlessly support for data with meta line (e.g., sep=, ) of separator declaration used by MS Excel","title":"Features"},{"location":"#subcommands","text":"51 subcommands in total. Information headers : prints headers dim : dimensions of CSV file nrow : print number of records ncol : print number of columns summary : summary statistics of selected numeric or text fields (groupby group fields) watch : online monitoring and histogram of selected field corr : calculate Pearson correlation between numeric columns Format conversion pretty : converts CSV to a readable aligned table csv2tab : converts CSV to tabular format tab2csv : converts tabular format to CSV space2tab : converts space delimited format to TSVW transpose : transposes CSV data csv2md : converts CSV to markdown format csv2rst : converts CSV to reStructuredText format csv2json : converts CSV to JSON format csv2xlsx : converts CSV/TSV files to XLSX file xlsx2csv : converts XLSX to CSV format Set operations head : prints first N records concat : concatenates CSV/TSV files by rows sample : sampling by proportion cut : select and arrange fields grep : greps data by selected fields with patterns/regular expressions uniq : unique data without sorting freq : frequencies of selected fields inter : intersection of multiple files filter : filters rows by values of selected fields with arithmetic expression filter2 : filters rows by awk-like arithmetic/string expressions join : join files by selected fields (inner, left and outer join) split splits CSV/TSV into multiple files according to column values splitxlsx : splits XLSX sheet into multiple sheets according to column values comb : compute combinations of items at every row Edit fix : fix CSV/TSV with different numbers of columns in rows add-header : add column names del-header : delete column names rename : renames column names with new names rename2 : renames column names by regular expression replace : replaces data of selected fields by regular expression round : round float to n decimal places mutate : creates new columns from selected fields by regular expression mutate2 : creates a new column from selected fields by awk-like arithmetic/string expressions sep : separate column into multiple columns gather : gather columns into key-value pairs, like tidyr::gather/pivot_longer spread : spread a key-value pair across multiple columns, like tidyr::spread/pivot_wider unfold : unfold multiple values in cells of a field fold : fold multiple values of a field into cells of groups fmtdate : format date of selected fields Ordering sort : sorts by selected fields Ploting plot see usage plot hist histogram plot box boxplot plot line line plot and scatter plot Misc cat stream file and report progress version print version information and check for update genautocomplete generate shell autocompletion script (bash|zsh|fish|powershell)","title":"Subcommands"},{"location":"#installation","text":"Download Page csvtk is implemented in Go programming language, executable binary files for most popular operating systems are freely available in release page.","title":"Installation"},{"location":"#method-1-download-binaries-latest-stabledev-version","text":"Just download compressed executable file of your operating system, and decompress it with tar -zxvf *.tar.gz command or other tools. And then: For Linux-like systems If you have root privilege simply copy it to /usr/local/bin : sudo cp csvtk /usr/local/bin/ Or copy to anywhere in the environment variable PATH : mkdir -p $HOME/bin/; cp csvtk $HOME/bin/ For windows , just copy csvtk.exe to C:\\WINDOWS\\system32 .","title":"Method 1: Download binaries (latest stable/dev version)"},{"location":"#method-2-install-via-conda-latest-stable-version","text":"conda install -c bioconda csvtk","title":"Method 2: Install via conda (latest stable version)"},{"location":"#method-3-install-via-homebrew","text":"brew install csvtk","title":"Method 3: Install via homebrew"},{"location":"#method-4-for-go-developer-latest-stabledev-version","text":"go get -u github.com/shenwei356/csvtk/csvtk","title":"Method 4: For Go developer (latest stable/dev version)"},{"location":"#method-5-for-archlinux-aur-users-may-be-not-the-latest","text":"yaourt -S csvtk","title":"Method 5: For ArchLinux AUR users (may be not the latest)"},{"location":"#command-line-completion","text":"Bash: # generate completion shell csvtk genautocomplete --shell bash # configure if never did. # install bash-completion if the \"complete\" command is not found. echo \"for bcfile in ~/.bash_completion.d/* ; do source \\$bcfile; done\" >> ~/.bash_completion echo \"source ~/.bash_completion\" >> ~/.bashrc Zsh: # generate completion shell csvtk genautocomplete --shell zsh --file ~/.zfunc/_csvtk # configure if never did echo 'fpath=( ~/.zfunc \"${fpath[@]}\" )' >> ~/.zshrc echo \"autoload -U compinit; compinit\" >> ~/.zshrc fish: csvtk genautocomplete --shell fish --file ~/.config/fish/completions/csvtk.fish","title":"Command-line completion"},{"location":"#compared-to-csvkit","text":"csvkit , attention: this table wasn't updated for 2 years. Features csvtk csvkit Note Read Gzip Yes Yes read gzip files Fields ranges Yes Yes e.g. -f 1-4,6 Unselect fileds Yes -- e.g. -1 for excluding first column Fuzzy fields Yes -- e.g. ab* for columns with name prefix \"ab\" Reorder fields Yes Yes it means -f 1,2 is different from -f 2,1 Rename columns Yes -- rename with new name(s) or from existed names Sort by multiple keys Yes Yes bash sort like operations Sort by number Yes -- e.g. -k 1:n Multiple sort Yes -- e.g. -k 2:r -k 1:nr Pretty output Yes Yes convert CSV to readable aligned table Unique data Yes -- unique data of selected fields frequency Yes -- frequencies of selected fields Sampling Yes -- sampling by proportion Mutate fields Yes -- create new columns from selected fields Replace Yes -- replace data of selected fields Similar tools: csvkit - A suite of utilities for converting to and working with CSV, the king of tabular file formats. http://csvkit.rtfd.org/ xsv - A fast CSV toolkit written in Rust. miller - Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV and tabular JSON http://johnkerl.org/miller tsv-utils - Command line utilities for tab-separated value files written in the D programming language.","title":"Compared to csvkit"},{"location":"#examples","text":"More examples and tutorial . Attention The CSV parser requires all the lines have same number of fields/columns. Even lines with spaces will cause error. Use '-I/--ignore-illegal-row' to skip these lines if neccessary. By default, csvtk thinks your files have header row, if not, switch flag -H on. Column names better be unique. By default, lines starting with # will be ignored, if the header row starts with # , please assign flag -C another rare symbol, e.g. '$' . By default, csvtk handles CSV files, use flag -t for tab-delimited files. If \" exists in tab-delimited files, use flag -l . Do not mix use field (column) numbers and names. Examples Pretty result $ csvtk pretty names.csv id first_name last_name username -- ---------- --------- -------- 11 Rob Pike rob 2 Ken Thompson ken 4 Robert Griesemer gri 1 Robert Thompson abc NA Robert Abel 123 $ csvtk pretty names.csv -S 3line \u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501 id first_name last_name username ---------------------------------------- 11 Rob Pike rob 2 Ken Thompson ken 4 Robert Griesemer gri 1 Robert Thompson abc NA Robert Abel 123 \u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501 $ csvtk pretty names.csv -S bold -w 5 -m 1- \u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513 \u2503 id \u2503 first_name \u2503 last_name \u2503 username \u2503 \u2523\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u252b \u2503 11 \u2503 Rob \u2503 Pike \u2503 rob \u2503 \u2523\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u252b \u2503 2 \u2503 Ken \u2503 Thompson \u2503 ken \u2503 \u2523\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u252b \u2503 4 \u2503 Robert \u2503 Griesemer \u2503 gri \u2503 \u2523\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u252b \u2503 1 \u2503 Robert \u2503 Thompson \u2503 abc \u2503 \u2523\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u254b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u252b \u2503 NA \u2503 Robert \u2503 Abel \u2503 123 \u2503 \u2517\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u253b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u253b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u253b\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u251b Summary of selected numeric fields, supporting \"group-by\" $ cat testdata/digitals2.csv \\ | csvtk summary -i -f f4:sum,f5:sum -g f1,f2 \\ | csvtk pretty f1 f2 f4:sum f5:sum bar xyz 7.00 106.00 bar xyz2 4.00 4.00 foo bar 6.00 3.00 foo bar2 4.50 5.00 Select fields/columns ( cut ) By index: csvtk cut -f 1,2 By names: csvtk cut -f first_name,username Unselect : csvtk cut -f -1,-2 or csvtk cut -f -first_name Fuzzy fields : csvtk cut -F -f \"*_name,username\" Field ranges: csvtk cut -f 2-4 for column 2,3,4 or csvtk cut -f -3--1 for discarding column 1,2,3 All fields: csvtk cut -f 1- or csvtk cut -F -f \"*\" Search by selected fields ( grep ) (matched parts will be highlighted as red) By exactly matching: csvtk grep -f first_name -p Robert -p Rob By regular expression: csvtk grep -f first_name -r -p Rob By pattern list: csvtk grep -f first_name -P name_list.txt Remore rows containing missing data (NA): csvtk grep -F -f \"*\" -r -p \"^$\" -v Rename column names ( rename and rename2 ) Setting new names: csvtk rename -f A,B -n a,b or csvtk rename -f 1-3 -n a,b,c Replacing with original names by regular express: csvtk rename2 -f 1- -p \"(.*)\" -r 'prefix_$1' for adding prefix to all column names. Edit data with regular expression ( replace ) Remove Chinese charactors: csvtk replace -F -f \"*_name\" -p \"\\p{Han}+\" -r \"\" Create new column from selected fields by regular expression ( mutate ) In default, copy a column: csvtk mutate -f id Extract prefix of data as group name (get \"A\" from \"A.1\" as group name): csvtk mutate -f sample -n group -p \"^(.+?)\\.\" --after sample Sort by multiple keys ( sort ) By single column : csvtk sort -k 1 or csvtk sort -k last_name By multiple columns: csvtk sort -k 1,2 or csvtk sort -k 1 -k 2 or csvtk sort -k last_name,age Sort by number: csvtk sort -k 1:n or csvtk sort -k 1:nr for reverse number Complex sort: csvtk sort -k region -k age:n -k id:nr In natural order: csvtk sort -k chr:N Join multiple files by keys ( join ) All files have same key column: csvtk join -f id file1.csv file2.csv Files have different key columns: csvtk join -f \"username;username;name\" names.csv phone.csv adress.csv -k Filter by numbers ( filter ) Single field: csvtk filter -f \"id>0\" Multiple fields : csvtk filter -f \"1-3>0\" Using --any to print record if any of the field satisfy the condition: csvtk filter -f \"1-3>0\" --any fuzzy fields : csvtk filter -F -f \"A*!=0\" Filter rows by awk-like arithmetic/string expressions ( filter2 ) Using field index: csvtk filter2 -f '$3>0' Using column names: csvtk filter2 -f '$id > 0' Both arithmetic and string expressions: csvtk filter2 -f '$id > 3 || $username==\"ken\"' More complicated: csvtk filter2 -H -t -f '$1 > 2 && $2 % 2 == 0' Ploting plot histogram with data of the second column: csvtk -t plot hist testdata/grouped_data.tsv.gz -f 2 | display plot boxplot with data of the \"GC Content\" (third) column, group information is the \"Group\" column. csvtk -t plot box testdata/grouped_data.tsv.gz -g \"Group\" \\ -f \"GC Content\" --width 3 | display plot horiz boxplot with data of the \"Length\" (second) column, group information is the \"Group\" column. csvtk -t plot box testdata/grouped_data.tsv.gz -g \"Group\" -f \"Length\" \\ --height 3 --width 5 --horiz --title \"Horiz box plot\" | display plot line plot with X-Y data csvtk -t plot line testdata/xy.tsv -x X -y Y -g Group | display plot scatter plot with X-Y data csvtk -t plot line testdata/xy.tsv -x X -y Y -g Group --scatter | display","title":"Examples"},{"location":"#acknowledgements","text":"We are grateful to Zhiluo Deng and Li Peng for suggesting features and reporting bugs. Thanks Albert Vilella for features suggestion, which makes csvtk feature-rich\u3002","title":"Acknowledgements"},{"location":"#contact","text":"Create an issue to report bugs, propose new functions or ask for help. Or leave a comment .","title":"Contact"},{"location":"#license","text":"MIT License","title":"License"},{"location":"#starchart","text":"","title":"Starchart"},{"location":"bioinf/","text":"","title":"Bioinf"},{"location":"chinese/","text":"\u5982\u540c\u751f\u7269\u4fe1\u606f\u9886\u57df\u4e2d\u7684FASTA/Q\u683c\u5f0f\u4e00\u6837\uff0cCSV/TSV\u4f5c\u4e3a\u8ba1\u7b97\u673a\u3001\u6570\u636e\u79d1\u5b66\u548c\u751f\u7269\u4fe1\u606f\u7684\u57fa\u672c\u683c\u5f0f\uff0c\u5e94\u7528\u975e\u5e38\u5e7f\u6cdb\u3002\u5e38\u7528\u7684\u5904\u7406\u8f6f\u4ef6\u5305\u62ec\uff1a \u4ee5\u5fae\u8f6fExcel\u4e3a\u4ee3\u8868\u7684\u7535\u5b50\u8868\u683c\u8f6f\u4ef6 Notepad++/SublimeText\u7b49\u6587\u672c\u7f16\u8f91\u5668 sed/awk/cut\u7b49Shell\u547d\u4ee4 \u5404\u79cd\u7f16\u7a0b\u8bed\u8a00\u7684\u6570\u636e\u5904\u7406\u5e93\u3002 \u7136\u800c\uff0c\u7535\u5b50\u8868\u683c\u8f6f\u4ef6\u548c\u6587\u672c\u7f16\u8f91\u5668\u56fa\u7136\u5f3a\u5927\uff0c\u4f46\u4f9d\u8d56\u9f20\u6807\u64cd\u4f5c\uff0c\u4e0d\u9002\u5408\u6279\u91cf\u5904\u7406\uff1b sed/awk/cut\u7b49Shell\u547d\u4ee4\u4e3b\u8981\u7528\u4e8e\u901a\u7528\u7684\u8868\u683c\u6570\u636e\uff0c\u4e0d\u9002\u5408\u542b\u6709\u6807\u9898\u884c\u7684CSV\u683c\u5f0f \uff1b \u4e3a\u4e86\u4e00\u4e2a\u5c0f\u64cd\u4f5c\u5199Python/R\u811a\u672c\u4e5f\u6709\u70b9\u5c0f\u9898\u5927\u4f5c\uff0c\u4e14\u96be\u4ee5\u590d\u7528 \u3002 \u5f00\u53d1csvtk\u524d\u73b0\u6709\u7684\u5de5\u5177\u4e3b\u8981\u662fPython\u5199\u7684csvkit\uff0cRust\u5199\u7684xsv\uff0cC\u8bed\u8a00\u5199\u7684miller\uff0c\u90fd\u5404\u6709\u4f18\u52a3\u3002\u5f53\u65f6\u6211\u521a\u5f00\u53d1\u5b8cseqkit\uff0c\u6295\u6587\u7ae0\u8fc7\u7a0b\u4e2d\u65f6\u95f4\u5145\u8db3\uff0c\u4fbf\u60f3\u8d81\u70ed\u518d\u9020\u4e00\u4e2a\u8f6e\u5b50\u3002 \u6240\u4ee5\u6211\u51b3\u5b9a\u5199\u4e00\u4e2a\u547d\u4ee4\u884c\u5de5\u5177\u6765\u6ee1\u8db3CSV/TSV\u683c\u5f0f\u7684\u5e38\u89c1\u64cd\u4f5c\uff0c\u8fd9\u5c31\u662fcsvtk\u4e86\u3002 \u4ecb\u7ecd \u57fa\u672c\u4fe1\u606f \u5de5\u5177\u7c7b\u578b: \u547d\u4ee4\u884c\u5de5\u5177\uff0c\u5b50\u547d\u4ee4\u7ed3\u6784 \u652f\u6301\u683c\u5f0f: CSV/TSV, plain/gzip-compressed \u7f16\u7a0b\u8bed\u8a00: Go \u652f\u6301\u5e73\u53f0: Linux, OS X\uff0c Windows \u7b49 \u53d1\u5e03\u65b9\u5f0f: \u5355\u4e00\u53ef\u6267\u884c\u4e8c\u8fdb\u5236\u6587\u4ef6\uff0c\u4e0b\u8f7d\u5373\u7528 \u53d1\u5e03\u5e73\u53f0: Github, Bioconda \u9879\u76ee\u4e3b\u9875: http://bioinf.shenwei.me/csvtk/ \u5f00\u6e90\u5730\u5740: https://github.com/shenwei356/csvtk \u7279\u6027 \u8de8\u5e73\u53f0 \u8f7b\u91cf\uff0c\u65e0\u4efb\u4f55\u4f9d\u8d56\uff0c\u65e0\u9700\u7f16\u8bd1\u3001\u914d\u7f6e\uff0c\u4e0b\u8f7d\u5373\u7528 \u5feb\u901f \u652f\u6301stdin\u548cgzip\u538b\u7f29\u7684\u8f93\u5165\u548c\u8f93\u51fa\u6587\u4ef6\uff0c\u4fbf\u4e8e\u6d41\u5904\u7406 27\u4e2a\u5b50\u547d\u4ee4\u63d0\u4f9b\u591a\u79cd\u5b9e\u7528\u7684\u529f\u80fd\uff0c\u4e14\u80fd\u901a\u8fc7\u547d\u4ee4\u884c\u7ba1\u9053\u7ec4\u5408 \u652f\u6301Bash\u81ea\u52a8\u8865\u5168 \u652f\u6301\u7b80\u5355\u7684\u7ed8\u56fe \u529f\u80fd \u5728\u5f00\u53d1csvtk\u4e4b\u524d\u7684\u4e24\u4e09\u5e74\u95f4\uff0c\u6211\u5df2\u7ecf\u5199\u4e86\u51e0\u4e2a\u53ef\u4ee5\u590d\u7528\u7684Python/Perl\u811a\u672c\uff08https://github.com/shenwei356/datakit\uff09 \uff0c\u5305\u62eccsv2tab\u3001csvtk_grep\u3001csv_join\u3001csv_melt\uff0cintersection\uff0cunique\u3002\u6240\u4ee5\u6211\u7684\u8ba1\u5212\u662f\u9996\u5148\u96c6\u6210\u8fd9\u4e9b\u5df2\u6709\u7684\u529f\u80fd\uff0c\u968f\u540e\u6839\u636e\u9700\u6c42\u8fdb\u884c\u6269\u5c55\u3002 \u5230\u76ee\u524d\u4e3a\u6b62\uff0ccsvtk\u5df2\u670927\u4e2a\u5b50\u547d\u4ee4\uff0c\u5206\u4e3a\u4ee5\u4e0b\u51e0\u5927\u7c7b\uff1a \u4fe1\u606f headers \u76f4\u89c2\u6253\u5370\u6807\u9898\u884c\uff08 \u64cd\u4f5c\u5217\u6570\u8f83\u591a\u7684CSV\u524d\u4f7f\u7528\u6700\u4f73 \uff09 stats \u57fa\u672c\u7edf\u8ba1 stats2 \u5bf9\u9009\u5b9a\u7684\u6570\u503c\u5217\u8fdb\u884c\u57fa\u672c\u7edf\u8ba1 \u683c\u5f0f\u8f6c\u5316 pretty \u8f6c\u4e3a\u7f8e\u89c2\u3001\u53ef\u8bfb\u6027\u5f3a\u7684\u683c\u5f0f\uff08 \u6700\u5e38\u7528\u547d\u4ee4\u4e4b\u4e00 \uff09 csv2tab \u8f6cCSV\u4e3a\u5236\u8868\u7b26\u5206\u5272\u683c\u5f0f\uff08TSV\uff09 tab2csv \u8f6cTSV\u4e3aCSV space2tab \u8f6c\u7a7a\u683c\u5206\u5272\u683c\u5f0f\u4e3aTSV transpose \u8f6c\u7f6eCSV/TSV csv2md \u8f6cCSV/TSV\u4e3amakrdown\u683c\u5f0f\uff08 \u5199\u6587\u6863\u5e38\u7528 \uff09 \u96c6\u5408\u64cd\u4f5c head \u6253\u5370\u524dN\u6761\u8bb0\u5f55 sample \u6309\u6bd4\u4f8b\u968f\u673a\u91c7\u6837 cut \u9009\u62e9\u7279\u5b9a\u5217\uff0c\u652f\u6301 \u6309\u5217\u6216\u5217\u540d\u8fdb\u884c\u57fa\u672c\u9009\u62e9\u3001\u8303\u56f4\u9009\u62e9\u3001\u6a21\u7cca\u9009\u62e9\u3001\u8d1f\u5411\u9009\u62e9 \uff08 \u6700\u5e38\u7528\u547d\u4ee4\u4e4b\u4e00\uff0c\u975e\u5e38\u5f3a\u5927 \uff09 uniq \u65e0\u987b\u6392\u5e8f\uff0c\u8fd4\u56de\u6309\u6307\u5b9a\uff08\u591a\uff09\u5217\u4f5c\u4e3akey\u7684\u552f\u4e00\u8bb0\u5f55\uff08\u597d\u7ed5\u3002\u3002\uff09 freq \u6309\u6307\u5b9a\uff08\u591a\uff09\u5217\u8fdb\u884c\u8ba1\u6570\uff08 \u5e38\u7528 \uff09 inter \u591a\u4e2a\u6587\u4ef6\u95f4\u7684\u4ea4\u96c6 grep \u6307\u5b9a\uff08\u591a\uff09\u5217\u4e3aKey\u8fdb\u884c\u641c\u7d22\uff08 \u6700\u5e38\u7528\u547d\u4ee4\u4e4b\u4e00\uff0c\u53ef\u6309\u6307\u5b9a\u5217\u641c\u7d22 \uff09 filter \u6309\u6307\u5b9a\uff08\u591a\uff09\u5217\u7684\u6570\u503c\u8fdb\u884c\u8fc7\u6ee4 filter2 \u7528\u7c7b\u4f3cawk\u7684\u6570\u503c/\u8868\u8fbe\u5f0f\uff0c\u6309\u6307\u5b9a\uff08\u591a\uff09\u5217\u7684\u6570\u503c\u8fdb\u884c\u8fc7\u6ee4 join \u5408\u5e76\u591a\u4e2a\u6587\u4ef6\uff08 \u5e38\u7528 \uff09 \u7f16\u8f91 rename \u76f4\u63a5\u91cd\u547d\u540d\u6307\u5b9a\uff08\u591a\uff09\u5217\u540d\uff08 \u7b80\u5355\u800c\u5b9e\u7528 \uff09 rename2 \u4ee5\u6b63\u5219\u8868\u8fbe\u5f0f\u91cd\u547d\u540d\u6307\u5b9a\uff08\u591a\uff09\u5217\u540d\uff08 \u7b80\u5355\u800c\u5b9e\u7528 \uff09 replace \u4ee5\u6b63\u5219\u8868\u8fbe\u5f0f\u5bf9\u6307\u5b9a\uff08\u591a\uff09\u5217\u8fdb\u884c\u66ff\u6362\u7f16\u8f91\uff08 \u6700\u5e38\u7528\u547d\u4ee4\u4e4b\u4e00\uff0c\u53ef\u6309\u6307\u5b9a\u5217\u7f16\u8f91 \uff09 mutate \u4ee5\u6b63\u5219\u8868\u8fbe\u5f0f\u57fa\u4e8e\u5df2\u6709\u5217\u521b\u5efa\u65b0\u7684\u4e00\u5217\uff08 \u5e38\u7528\u4e8e\u751f\u6210\u591a\u5217\u6d4b\u8bd5\u6570\u636e \uff09 mutate2 \u7528\u7c7b\u4f3cawk\u7684\u6570\u503c/\u8868\u8fbe\u5f0f\uff0c\u4ee5\u6b63\u5219\u8868\u8fbe\u5f0f\u57fa\u4e8e\u5df2\u6709\uff08\u591a\uff09\u5217\u521b\u5efa\u65b0\u7684\u4e00\u5217\uff08 \u5e38\u7528 \uff09 gather \u7c7b\u4f3c\u4e8eR\u91cc\u9762tidyr\u5305\u7684gather\u65b9\u6cd5 \u6392\u5e8f sort \u6309\u6307\u5b9a\uff08\u591a\uff09\u5217\u8fdb\u884c\u6392\u5e8f \u7ed8\u56fe plot \u57fa\u672c\u7ed8\u56fe plot hist histogram plot box boxplot plot line line plot and scatter plot \u5176\u5b83 version \u7248\u672c\u4fe1\u606f\u548c\u68c0\u67e5\u65b0\u7248\u672c genautocomplete \u751f\u6210\u652f\u6301Bash\u81ea\u52a8\u8865\u5168\u7684\u914d\u7f6e\u6587\u4ef6\uff0c\u91cd\u542fTerminal\u751f\u6548\u3002 \u4f7f\u7528 \u8f93\u5165\u6570\u636e\u8981\u6c42\u6bcf\u884c\u7684\u5217\u6570\u4e00\u81f4\uff0c\u7a7a\u884c\u4e5f\u4f1a\u62a5\u9519 csvtk\u9ed8\u8ba4\u8f93\u5165\u6570\u636e\u542b\u6709\u6807\u9898\u884c\uff0c\u5982\u6ca1\u6709\u8bf7\u5f00\u542f\u5168\u5c40\u53c2\u6570 -H csvtk\u9ed8\u8ba4\u8f93\u5165\u6570\u636e\u4e3aCSV\u683c\u5f0f\uff0c\u5982\u4e3aTSV\u8bf7\u5f00\u542f\u5168\u5c40\u53c2\u6570 -t \u8f93\u5165\u6570\u636e\u5217\u540d\u6700\u597d\u552f\u4e00\u65e0\u91cd\u590d \u5982\u679cTSV\u4e2d\u5b58\u5728\u53cc\u5f15\u53f7 \"\" \uff0c\u8bf7\u5f00\u542f\u5168\u5c40\u53c2\u6570 -l csvtk\u9ed8\u8ba4\u4ee5 # \u5f00\u59cb\u7684\u4e3a\u6ce8\u91ca\u884c\uff0c\u82e5\u6807\u9898\u884c\u542b # \uff0c\u8bf7\u7ed9\u5168\u5c40\u53c2\u6570 -C \u6307\u5b9a\u53e6\u4e00\u4e2a\u4e0d\u5e38\u89c1\u7684\u5b57\u7b26\uff08\u5982 $ \uff09 \u4f8b\u5b50 \u4ec5\u63d0\u4f9b\u5c11\u91cf\u4f8b\u5b50\uff0c\u66f4\u591a\u4f8b\u5b50\u8bf7\u770b\u4f7f\u7528\u624b\u518c http://bioinf.shenwei.me/csvtk/usage/ \u3002 \u793a\u4f8b\u6570\u636e $ cat names.csv id,first_name,last_name,username 11,\"Rob\",\"Pike\",rob 2,Ken,Thompson,ken 4,\"Robert\",\"Griesemer\",\"gri\" 1,\"Robert\",\"Thompson\",\"abc\" NA,\"Robert\",\"Abel\",\"123\" \u589e\u5f3a\u53ef\u8bfb\u6027 $ cat names.csv | csvtk pretty id first_name last_name username 11 Rob Pike rob 2 Ken Thompson ken 4 Robert Griesemer gri 1 Robert Thompson abc NA Robert Abel 123 \u8f6c\u4e3amarkdown $ cat names.csv | csvtk csv2md id |first_name|last_name|username :--|:---------|:--------|:------- 11 |Rob |Pike |rob 2 |Ken |Thompson |ken 4 |Robert |Griesemer|gri 1 |Robert |Thompson |abc NA |Robert |Abel |123 \u6548\u679c id first_name last_name username 11 Rob Pike rob 2 Ken Thompson ken 4 Robert Griesemer gri 1 Robert Thompson abc NA Robert Abel 123 \u7528\u5217\u6216\u5217\u540d\u6765\u9009\u62e9\u6307\u5b9a\u5217\uff0c\u53ef\u6539\u53d8\u5217\u7684\u987a\u5e8f $ cat names.csv | csvtk cut -f 3,1 | csvtk pretty $ cat names.csv | csvtk cut -f last_name,id | csvtk pretty last_name id Pike 11 Thompson 2 Griesemer 4 Thompson 1 Abel NA \u7528\u901a\u914d\u7b26\u9009\u62e9\u591a\u5217 $ cat names.csv | csvtk cut -F -f '*name,id' | csvtk pretty first_name last_name username id Rob Pike rob 11 Ken Thompson ken 2 Robert Griesemer gri 4 Robert Thompson abc 1 Robert Abel 123 NA \u5220\u9664\u7b2c2\uff0c3\u5217\uff08 \u4e0b\u5217\u7b2c\u4e8c\u79cd\u65b9\u6cd5\u662f\u9009\u5b9a\u8303\u56f4\uff0c\u4f46-3\u5728\u524d,-2\u5728\u540e \uff09 $ cat names.csv | csvtk cut -f -2,-3 | csvtk pretty $ cat names.csv | csvtk cut -f -3--2 | csvtk pretty $ cat names.csv | csvtk cut -f -first_name,-last_name | csvtk pretty id username 11 rob 2 ken 4 gri 1 abc NA 123 \u6309\u6307\u5b9a\u5217\u641c\u7d22\uff0c \u9ed8\u8ba4\u7cbe\u786e\u5339\u914d $ cat names.csv | csvtk grep -f id -p 1 | csvtk pretty id first_name last_name username 1 Robert Thompson abc \u6a21\u7cca\u641c\u7d22\uff08\u6b63\u5219\u8868\u8fbe\u5f0f\uff09 $ cat names.csv | csvtk grep -f id -p 1 -r | csvtk pretty id first_name last_name username 11 Rob Pike rob 1 Robert Thompson abc \u7528\u6587\u4ef6\u4f5c\u4e3a\u6a21\u5f0f\u6765\u6e90 $ cat names.csv | csvtk grep -f id -P id-files.txt \u5bf9\u6307\u5b9a\u5217\u505a\u7b80\u5355\u66ff\u6362 $ cat names.csv | csvtk replace -f id -p '(\\d+)' -r 'ID: $1' \\ | csvtk pretty id first_name last_name username ID: 11 Rob Pike rob ID: 2 Ken Thompson ken ID: 4 Robert Griesemer gri ID: 1 Robert Thompson abc NA Robert Abel 123 \u7528key-value\u6587\u4ef6\u6765\u66ff\u6362\uff08seqkit\u548cbrename\u90fd\u652f\u6301\u7c7b\u4f3c\u64cd\u4f5c\uff09 $ cat data.tsv name id A ID001 B ID002 C ID004 $ cat alias.tsv 001 Tom 002 Bob 003 Jim $ csvtk replace -t -f 2 -p \"ID(.+)\" -r \"N: {nr}, alias: {kv}\" -k \\ alias.tsv data.tsv name id A N: 1, alias: Tom B N: 2, alias: Bob C N: 3, alias: 004 \u5408\u5e76\u8868\u683c\uff0c\u9700\u8981\u5206\u522b\u6307\u5b9a\u5404\u6587\u4ef6\u4e2d\u7684key\u5217\uff1a\u9ed8\u8ba4\u5747\u4e3a\u7b2c\u4e00\u5217\uff1b\u82e5\u5217\uff08\u540d\uff09\u76f8\u540c\u63d0\u4f9b\u4e00\u4e2a\uff1b\u82e5\u4e0d\u540c\u7528\u5206\u53f7\u5206\u5272 $ cat testdata/phones.csv username,phone gri,11111 rob,12345 ken,22222 shenwei,999999 $ csvtk join -f 'username;username' --keep-unmatched names.csv phones.csv \\ | csvtk pretty id first_name last_name username phone 11 Rob Pike rob 12345 2 Ken Thompson ken 22222 4 Robert Griesemer gri 11111 1 Robert Thompson abc NA Robert Abel 123","title":"\u4e2d\u6587\u4ecb\u7ecd"},{"location":"chinese/#_1","text":"\u57fa\u672c\u4fe1\u606f \u5de5\u5177\u7c7b\u578b: \u547d\u4ee4\u884c\u5de5\u5177\uff0c\u5b50\u547d\u4ee4\u7ed3\u6784 \u652f\u6301\u683c\u5f0f: CSV/TSV, plain/gzip-compressed \u7f16\u7a0b\u8bed\u8a00: Go \u652f\u6301\u5e73\u53f0: Linux, OS X\uff0c Windows \u7b49 \u53d1\u5e03\u65b9\u5f0f: \u5355\u4e00\u53ef\u6267\u884c\u4e8c\u8fdb\u5236\u6587\u4ef6\uff0c\u4e0b\u8f7d\u5373\u7528 \u53d1\u5e03\u5e73\u53f0: Github, Bioconda \u9879\u76ee\u4e3b\u9875: http://bioinf.shenwei.me/csvtk/ \u5f00\u6e90\u5730\u5740: https://github.com/shenwei356/csvtk \u7279\u6027 \u8de8\u5e73\u53f0 \u8f7b\u91cf\uff0c\u65e0\u4efb\u4f55\u4f9d\u8d56\uff0c\u65e0\u9700\u7f16\u8bd1\u3001\u914d\u7f6e\uff0c\u4e0b\u8f7d\u5373\u7528 \u5feb\u901f \u652f\u6301stdin\u548cgzip\u538b\u7f29\u7684\u8f93\u5165\u548c\u8f93\u51fa\u6587\u4ef6\uff0c\u4fbf\u4e8e\u6d41\u5904\u7406 27\u4e2a\u5b50\u547d\u4ee4\u63d0\u4f9b\u591a\u79cd\u5b9e\u7528\u7684\u529f\u80fd\uff0c\u4e14\u80fd\u901a\u8fc7\u547d\u4ee4\u884c\u7ba1\u9053\u7ec4\u5408 \u652f\u6301Bash\u81ea\u52a8\u8865\u5168 \u652f\u6301\u7b80\u5355\u7684\u7ed8\u56fe","title":"\u4ecb\u7ecd"},{"location":"chinese/#_2","text":"\u5728\u5f00\u53d1csvtk\u4e4b\u524d\u7684\u4e24\u4e09\u5e74\u95f4\uff0c\u6211\u5df2\u7ecf\u5199\u4e86\u51e0\u4e2a\u53ef\u4ee5\u590d\u7528\u7684Python/Perl\u811a\u672c\uff08https://github.com/shenwei356/datakit\uff09 \uff0c\u5305\u62eccsv2tab\u3001csvtk_grep\u3001csv_join\u3001csv_melt\uff0cintersection\uff0cunique\u3002\u6240\u4ee5\u6211\u7684\u8ba1\u5212\u662f\u9996\u5148\u96c6\u6210\u8fd9\u4e9b\u5df2\u6709\u7684\u529f\u80fd\uff0c\u968f\u540e\u6839\u636e\u9700\u6c42\u8fdb\u884c\u6269\u5c55\u3002 \u5230\u76ee\u524d\u4e3a\u6b62\uff0ccsvtk\u5df2\u670927\u4e2a\u5b50\u547d\u4ee4\uff0c\u5206\u4e3a\u4ee5\u4e0b\u51e0\u5927\u7c7b\uff1a \u4fe1\u606f headers \u76f4\u89c2\u6253\u5370\u6807\u9898\u884c\uff08 \u64cd\u4f5c\u5217\u6570\u8f83\u591a\u7684CSV\u524d\u4f7f\u7528\u6700\u4f73 \uff09 stats \u57fa\u672c\u7edf\u8ba1 stats2 \u5bf9\u9009\u5b9a\u7684\u6570\u503c\u5217\u8fdb\u884c\u57fa\u672c\u7edf\u8ba1 \u683c\u5f0f\u8f6c\u5316 pretty \u8f6c\u4e3a\u7f8e\u89c2\u3001\u53ef\u8bfb\u6027\u5f3a\u7684\u683c\u5f0f\uff08 \u6700\u5e38\u7528\u547d\u4ee4\u4e4b\u4e00 \uff09 csv2tab \u8f6cCSV\u4e3a\u5236\u8868\u7b26\u5206\u5272\u683c\u5f0f\uff08TSV\uff09 tab2csv \u8f6cTSV\u4e3aCSV space2tab \u8f6c\u7a7a\u683c\u5206\u5272\u683c\u5f0f\u4e3aTSV transpose \u8f6c\u7f6eCSV/TSV csv2md \u8f6cCSV/TSV\u4e3amakrdown\u683c\u5f0f\uff08 \u5199\u6587\u6863\u5e38\u7528 \uff09 \u96c6\u5408\u64cd\u4f5c head \u6253\u5370\u524dN\u6761\u8bb0\u5f55 sample \u6309\u6bd4\u4f8b\u968f\u673a\u91c7\u6837 cut \u9009\u62e9\u7279\u5b9a\u5217\uff0c\u652f\u6301 \u6309\u5217\u6216\u5217\u540d\u8fdb\u884c\u57fa\u672c\u9009\u62e9\u3001\u8303\u56f4\u9009\u62e9\u3001\u6a21\u7cca\u9009\u62e9\u3001\u8d1f\u5411\u9009\u62e9 \uff08 \u6700\u5e38\u7528\u547d\u4ee4\u4e4b\u4e00\uff0c\u975e\u5e38\u5f3a\u5927 \uff09 uniq \u65e0\u987b\u6392\u5e8f\uff0c\u8fd4\u56de\u6309\u6307\u5b9a\uff08\u591a\uff09\u5217\u4f5c\u4e3akey\u7684\u552f\u4e00\u8bb0\u5f55\uff08\u597d\u7ed5\u3002\u3002\uff09 freq \u6309\u6307\u5b9a\uff08\u591a\uff09\u5217\u8fdb\u884c\u8ba1\u6570\uff08 \u5e38\u7528 \uff09 inter \u591a\u4e2a\u6587\u4ef6\u95f4\u7684\u4ea4\u96c6 grep \u6307\u5b9a\uff08\u591a\uff09\u5217\u4e3aKey\u8fdb\u884c\u641c\u7d22\uff08 \u6700\u5e38\u7528\u547d\u4ee4\u4e4b\u4e00\uff0c\u53ef\u6309\u6307\u5b9a\u5217\u641c\u7d22 \uff09 filter \u6309\u6307\u5b9a\uff08\u591a\uff09\u5217\u7684\u6570\u503c\u8fdb\u884c\u8fc7\u6ee4 filter2 \u7528\u7c7b\u4f3cawk\u7684\u6570\u503c/\u8868\u8fbe\u5f0f\uff0c\u6309\u6307\u5b9a\uff08\u591a\uff09\u5217\u7684\u6570\u503c\u8fdb\u884c\u8fc7\u6ee4 join \u5408\u5e76\u591a\u4e2a\u6587\u4ef6\uff08 \u5e38\u7528 \uff09 \u7f16\u8f91 rename \u76f4\u63a5\u91cd\u547d\u540d\u6307\u5b9a\uff08\u591a\uff09\u5217\u540d\uff08 \u7b80\u5355\u800c\u5b9e\u7528 \uff09 rename2 \u4ee5\u6b63\u5219\u8868\u8fbe\u5f0f\u91cd\u547d\u540d\u6307\u5b9a\uff08\u591a\uff09\u5217\u540d\uff08 \u7b80\u5355\u800c\u5b9e\u7528 \uff09 replace \u4ee5\u6b63\u5219\u8868\u8fbe\u5f0f\u5bf9\u6307\u5b9a\uff08\u591a\uff09\u5217\u8fdb\u884c\u66ff\u6362\u7f16\u8f91\uff08 \u6700\u5e38\u7528\u547d\u4ee4\u4e4b\u4e00\uff0c\u53ef\u6309\u6307\u5b9a\u5217\u7f16\u8f91 \uff09 mutate \u4ee5\u6b63\u5219\u8868\u8fbe\u5f0f\u57fa\u4e8e\u5df2\u6709\u5217\u521b\u5efa\u65b0\u7684\u4e00\u5217\uff08 \u5e38\u7528\u4e8e\u751f\u6210\u591a\u5217\u6d4b\u8bd5\u6570\u636e \uff09 mutate2 \u7528\u7c7b\u4f3cawk\u7684\u6570\u503c/\u8868\u8fbe\u5f0f\uff0c\u4ee5\u6b63\u5219\u8868\u8fbe\u5f0f\u57fa\u4e8e\u5df2\u6709\uff08\u591a\uff09\u5217\u521b\u5efa\u65b0\u7684\u4e00\u5217\uff08 \u5e38\u7528 \uff09 gather \u7c7b\u4f3c\u4e8eR\u91cc\u9762tidyr\u5305\u7684gather\u65b9\u6cd5 \u6392\u5e8f sort \u6309\u6307\u5b9a\uff08\u591a\uff09\u5217\u8fdb\u884c\u6392\u5e8f \u7ed8\u56fe plot \u57fa\u672c\u7ed8\u56fe plot hist histogram plot box boxplot plot line line plot and scatter plot \u5176\u5b83 version \u7248\u672c\u4fe1\u606f\u548c\u68c0\u67e5\u65b0\u7248\u672c genautocomplete \u751f\u6210\u652f\u6301Bash\u81ea\u52a8\u8865\u5168\u7684\u914d\u7f6e\u6587\u4ef6\uff0c\u91cd\u542fTerminal\u751f\u6548\u3002","title":"\u529f\u80fd"},{"location":"chinese/#_3","text":"\u8f93\u5165\u6570\u636e\u8981\u6c42\u6bcf\u884c\u7684\u5217\u6570\u4e00\u81f4\uff0c\u7a7a\u884c\u4e5f\u4f1a\u62a5\u9519 csvtk\u9ed8\u8ba4\u8f93\u5165\u6570\u636e\u542b\u6709\u6807\u9898\u884c\uff0c\u5982\u6ca1\u6709\u8bf7\u5f00\u542f\u5168\u5c40\u53c2\u6570 -H csvtk\u9ed8\u8ba4\u8f93\u5165\u6570\u636e\u4e3aCSV\u683c\u5f0f\uff0c\u5982\u4e3aTSV\u8bf7\u5f00\u542f\u5168\u5c40\u53c2\u6570 -t \u8f93\u5165\u6570\u636e\u5217\u540d\u6700\u597d\u552f\u4e00\u65e0\u91cd\u590d \u5982\u679cTSV\u4e2d\u5b58\u5728\u53cc\u5f15\u53f7 \"\" \uff0c\u8bf7\u5f00\u542f\u5168\u5c40\u53c2\u6570 -l csvtk\u9ed8\u8ba4\u4ee5 # \u5f00\u59cb\u7684\u4e3a\u6ce8\u91ca\u884c\uff0c\u82e5\u6807\u9898\u884c\u542b # \uff0c\u8bf7\u7ed9\u5168\u5c40\u53c2\u6570 -C \u6307\u5b9a\u53e6\u4e00\u4e2a\u4e0d\u5e38\u89c1\u7684\u5b57\u7b26\uff08\u5982 $ \uff09","title":"\u4f7f\u7528"},{"location":"chinese/#_4","text":"\u4ec5\u63d0\u4f9b\u5c11\u91cf\u4f8b\u5b50\uff0c\u66f4\u591a\u4f8b\u5b50\u8bf7\u770b\u4f7f\u7528\u624b\u518c http://bioinf.shenwei.me/csvtk/usage/ \u3002 \u793a\u4f8b\u6570\u636e $ cat names.csv id,first_name,last_name,username 11,\"Rob\",\"Pike\",rob 2,Ken,Thompson,ken 4,\"Robert\",\"Griesemer\",\"gri\" 1,\"Robert\",\"Thompson\",\"abc\" NA,\"Robert\",\"Abel\",\"123\" \u589e\u5f3a\u53ef\u8bfb\u6027 $ cat names.csv | csvtk pretty id first_name last_name username 11 Rob Pike rob 2 Ken Thompson ken 4 Robert Griesemer gri 1 Robert Thompson abc NA Robert Abel 123 \u8f6c\u4e3amarkdown $ cat names.csv | csvtk csv2md id |first_name|last_name|username :--|:---------|:--------|:------- 11 |Rob |Pike |rob 2 |Ken |Thompson |ken 4 |Robert |Griesemer|gri 1 |Robert |Thompson |abc NA |Robert |Abel |123 \u6548\u679c id first_name last_name username 11 Rob Pike rob 2 Ken Thompson ken 4 Robert Griesemer gri 1 Robert Thompson abc NA Robert Abel 123 \u7528\u5217\u6216\u5217\u540d\u6765\u9009\u62e9\u6307\u5b9a\u5217\uff0c\u53ef\u6539\u53d8\u5217\u7684\u987a\u5e8f $ cat names.csv | csvtk cut -f 3,1 | csvtk pretty $ cat names.csv | csvtk cut -f last_name,id | csvtk pretty last_name id Pike 11 Thompson 2 Griesemer 4 Thompson 1 Abel NA \u7528\u901a\u914d\u7b26\u9009\u62e9\u591a\u5217 $ cat names.csv | csvtk cut -F -f '*name,id' | csvtk pretty first_name last_name username id Rob Pike rob 11 Ken Thompson ken 2 Robert Griesemer gri 4 Robert Thompson abc 1 Robert Abel 123 NA \u5220\u9664\u7b2c2\uff0c3\u5217\uff08 \u4e0b\u5217\u7b2c\u4e8c\u79cd\u65b9\u6cd5\u662f\u9009\u5b9a\u8303\u56f4\uff0c\u4f46-3\u5728\u524d,-2\u5728\u540e \uff09 $ cat names.csv | csvtk cut -f -2,-3 | csvtk pretty $ cat names.csv | csvtk cut -f -3--2 | csvtk pretty $ cat names.csv | csvtk cut -f -first_name,-last_name | csvtk pretty id username 11 rob 2 ken 4 gri 1 abc NA 123 \u6309\u6307\u5b9a\u5217\u641c\u7d22\uff0c \u9ed8\u8ba4\u7cbe\u786e\u5339\u914d $ cat names.csv | csvtk grep -f id -p 1 | csvtk pretty id first_name last_name username 1 Robert Thompson abc \u6a21\u7cca\u641c\u7d22\uff08\u6b63\u5219\u8868\u8fbe\u5f0f\uff09 $ cat names.csv | csvtk grep -f id -p 1 -r | csvtk pretty id first_name last_name username 11 Rob Pike rob 1 Robert Thompson abc \u7528\u6587\u4ef6\u4f5c\u4e3a\u6a21\u5f0f\u6765\u6e90 $ cat names.csv | csvtk grep -f id -P id-files.txt \u5bf9\u6307\u5b9a\u5217\u505a\u7b80\u5355\u66ff\u6362 $ cat names.csv | csvtk replace -f id -p '(\\d+)' -r 'ID: $1' \\ | csvtk pretty id first_name last_name username ID: 11 Rob Pike rob ID: 2 Ken Thompson ken ID: 4 Robert Griesemer gri ID: 1 Robert Thompson abc NA Robert Abel 123 \u7528key-value\u6587\u4ef6\u6765\u66ff\u6362\uff08seqkit\u548cbrename\u90fd\u652f\u6301\u7c7b\u4f3c\u64cd\u4f5c\uff09 $ cat data.tsv name id A ID001 B ID002 C ID004 $ cat alias.tsv 001 Tom 002 Bob 003 Jim $ csvtk replace -t -f 2 -p \"ID(.+)\" -r \"N: {nr}, alias: {kv}\" -k \\ alias.tsv data.tsv name id A N: 1, alias: Tom B N: 2, alias: Bob C N: 3, alias: 004 \u5408\u5e76\u8868\u683c\uff0c\u9700\u8981\u5206\u522b\u6307\u5b9a\u5404\u6587\u4ef6\u4e2d\u7684key\u5217\uff1a\u9ed8\u8ba4\u5747\u4e3a\u7b2c\u4e00\u5217\uff1b\u82e5\u5217\uff08\u540d\uff09\u76f8\u540c\u63d0\u4f9b\u4e00\u4e2a\uff1b\u82e5\u4e0d\u540c\u7528\u5206\u53f7\u5206\u5272 $ cat testdata/phones.csv username,phone gri,11111 rob,12345 ken,22222 shenwei,999999 $ csvtk join -f 'username;username' --keep-unmatched names.csv phones.csv \\ | csvtk pretty id first_name last_name username phone 11 Rob Pike rob 12345 2 Ken Thompson ken 22222 4 Robert Griesemer gri 11111 1 Robert Thompson abc NA Robert Abel 123","title":"\u4f8b\u5b50"},{"location":"download/","text":"Download csvtk is implemented in Go programming language, executable binary files for most popular operating system are freely available in release page. Current Version csvtk v0.28.0 csvtk : add the shortcut -X for the flag --infile-list . #249 csvtk pretty : support field ranges for -m/--align-center and -r/--align-right . #244 csvtk spread : support values sharing the same keys. #248 csvtk join : a new flag -P/--prefix-duplicates : add filenames as colname prefixes only for duplicated colnames. #246 csvtk mutate2 : fix changing the order of the header row, the code was accidentally missing during code refactoring in v0.27.0. #252 csvtk xlsx2csv : fix open /tmp/excelize-: no such file or directory error for big .xlsx files. #251 csvtk comb : fix the empty result bug for alphabet sizes greater than 64. Links OS Arch File, \u4e2d\u56fd\u955c\u50cf Download Count Linux 32-bit csvtk_linux_386.tar.gz , \u4e2d\u56fd\u955c\u50cf Linux 64-bit csvtk_linux_amd64.tar.gz , \u4e2d\u56fd\u955c\u50cf Linux 64-bit csvtk_linux_arm64.tar.gz , \u4e2d\u56fd\u955c\u50cf macOS 64-bit csvtk_darwin_amd64.tar.gz , \u4e2d\u56fd\u955c\u50cf macOS arm64 csvtk_darwin_arm64.tar.gz , \u4e2d\u56fd\u955c\u50cf Windows 32-bit csvtk_windows_386.exe.tar.gz , \u4e2d\u56fd\u955c\u50cf Windows 64-bit csvtk_windows_amd64.exe.tar.gz , \u4e2d\u56fd\u955c\u50cf Notes run csvtk version to check update !!! run csvtk genautocomplete to update Bash completion !!! Installation Download Page csvtk is implemented in Go programming language, executable binary files for most popular operating systems are freely available in release page. Method 1: Download binaries (latest stable/dev version) Just download compressed executable file of your operating system, and decompress it with tar -zxvf *.tar.gz command or other tools. And then: For Linux-like systems If you have root privilege simply copy it to /usr/local/bin : sudo cp csvtk /usr/local/bin/ Or copy to anywhere in the environment variable PATH : mkdir -p $HOME/bin/; cp csvtk $HOME/bin/ For windows , just copy csvtk.exe to C:\\WINDOWS\\system32 . Method 2: Install via conda (latest stable version) conda install -c bioconda csvtk Method 3: Install via homebrew (may be not the latest) brew install csvtk Method 4: For Go developer (latest stable/dev version) go get -u github.com/shenwei356/csvtk/csvtk Method 5: For ArchLinux AUR users (may be not the latest) yaourt -S csvtk Method 6: Compiling from source (latest stable/dev version) # ------------------- install golang ----------------- # download Go from https://go.dev/dl wget https://go.dev/dl/go1.17.12.linux-amd64.tar.gz tar -zxf go1.17.12.linux-amd64.tar.gz -C $HOME/ # or # echo \"export PATH=$PATH:$HOME/go/bin\" >> ~/.bashrc # source ~/.bashrc export PATH=$PATH:$HOME/go/bin # ------------- the latest stable version ------------- go get -v -u github.com/shenwei356/csvtk/csvtk # The executable binary file is located in: # ~/go/bin/csvtk # You can also move it to anywhere in the $PATH mkdir -p $HOME/bin cp ~/go/bin/csvtk $HOME/bin/ # --------------- the development version -------------- git clone https://github.com/shenwei356/csvtk cd csvtk/csvtk/ go build # The executable binary file is located in: # ./csvtk # You can also move it to anywhere in the $PATH mkdir -p $HOME/bin cp ./csvtk $HOME/bin/ Shell-completion Bash: # generate completion shell csvtk genautocomplete --shell bash # configure if never did. # install bash-completion if the \"complete\" command is not found. echo \"for bcfile in ~/.bash_completion.d/* ; do source \\$bcfile; done\" >> ~/.bash_completion echo \"source ~/.bash_completion\" >> ~/.bashrc Zsh: # generate completion shell csvtk genautocomplete --shell zsh --file ~/.zfunc/_csvtk # configure if never did echo 'fpath=( ~/.zfunc \"${fpath[@]}\" )' >> ~/.zshrc echo \"autoload -U compinit; compinit\" >> ~/.zshrc fish: csvtk genautocomplete --shell fish --file ~/.config/fish/completions/csvtk.fish Release history csvtk v0.27.2 csvtk pretty : fix the bug of empty first row with -H/--no-header-row , introduced in v0.27.0. new style 3line for three-line table. csvtk csv2xlsx : binaries compiled with go1.21 would result in a broken xlsx file. #243 csvtk splitxlsx : fix the error of invalid worksheet index . #1617 csvtk v0.27.1 csvtk filter2/mutate2 : fix the bug of selecting with field numbers. #242 csvtk v0.27.0 csvtk : code refactoring and simplifying code, with 16% less code. most commands support open column range syntax , e.g., csvtk grep -f 2- . #120 only selected column names are not allowed to be duplicated in the input data : box, corr, filter, filter2, fold, freq, gather, historysort, inter, join, line, mutate, mutate2, rename, replace, sep, split, summary, unfold, uniq, watch. Other commands do not have the restriction. #235 add a new global flag -Z/--show-row-number , supported commands: cut, csv2tab, csv2xlsx, tab2csv, pretty. the colum name of row number changes from \"n\" to \"row\": csv2xlsx, csv2tab, cut, filter, filter2, grep, pretty, sample, tab2csv. new command : csvtk spread : spread a key-value pair across multiple columns, like tidyr::spread/pivot_wider . #91 , #236 , #239 csvtk mutate/mutate2 : new flags --at , --before , --after for specifying the position of the new column . #193 csvtk cut : fix unselect range error. #234 fix -i/--ignore-case . csvtk pretty : allow align-center and align-right for specific columns . #240 csvtk round : fix bug of failing to round scientific notation with value small than one, e.g., 7.1E-1 . csvtk summary : fix duplicated columns. fix result error when multiple stats applied to the same column. csvtk corr/watch : rewrite and fix bug, support choosing fields with column names. csvtk v0.26.0 csvtk : near all commands skip empty files now . #204 the global flag --infile-list accepts stdin \"-\". #210 new command csvtk fix : fix CSV/TSV with different numbers of columns in rows . #226 csvtk pretty : rewrite to support wrapping cells . #206 #209 #228 csvtk cut/fmtdate/freq/grep/rename/rename2/replace/round : allow duplicated column names. csvtk csv2xlsx : optionally stores numbers as float. #217 csvtk xlsx2csv : fix bug where xlsx2csv treats small number (padj < 1e-25) as 0. It's solved by updating the excelize package. #261 csvtk join : a new flag for adding filename as column name prefix. by @tetedange13 #202 csvtk mutate2 : fix wrongly treating strings like E10 as numbers in scientific notation. #219 csvtk sep : fix the logic. #218 csvtk space2tab : fix \"bufio.Scanner: token too long\". #231 csvtk v0.25.0 csvtk : report empty files. csvtk join : fix loading file with no records. csvtk filter2/muate2 : support variable format of ${var} with special charactors including commas, spaces, and parentheses, e.g., ${a,b} , ${a b} , or ${a (b)} . #186 csvtk sort : fix checking non-existed fileds. csvtk plot box/hist/line : new flag --skip-na for skipping missing data. #188 csvtk csv2xlsx : stores number as float. #192 csvtk summary : new functions argmin and argmax . #181 csvtk v0.24.0 Incompatible changes : csvtk mutate2/summary : mutate2 : remove the option -L/--digits . use the same option -w/--decimal-width to limit floats to N decimal points. new command csvtk fmtdate : format date of selected fields. #159 csvtk grep : fix bug for searching with -r -p . . csvtk csv2rst : fix bug for data containing unicode. #137 csvtk filter2 : fix bug for date expression. #146 csvtk mutate2/filter2 : change the way of rexpression evaluation. add custom functions: len() . #153 fix bug when using two or more columns with common prefixes in column names . #173 fix value with single or double quotes. #174 csvtk cut : new flags -m/--allow-missing-col and -b/--blank-missing-col . #156 csvtk pretty : still add header row for empty column. csvtk csv2md : better format. csvtk join : new flag -n/--ignore-null . #163 csvtk v0.23.0 new comand: csvtk csv2rst for converting CSV to reStructuredText format. #137 csvtk pretty : add header separator line. #123 csvtk mutate2/summary : fix message and doc. Thanks @VladimirAlexiev #127 csvtk mutate2 : fix null coalescence: ??. #129 csvtk genautocomplete : supports bash|zsh|fish|powershell. #126 csvtk cat : fix progress bar. #130 csvtk grep : new flag immediate-output . csvtk csv2xlsx : fix bug for table with > 26 columns. 138 csvtk v0.22.0 csvtk : global flag -t does not overide -D anymore . #114 If the executable/symlink name is tsvtk the -t/--tabs option for tab input is set . Thanks @bsipos. #117 new command: csvtk csv2xlsx for converting CSV/TSV file(s) to a single .xlsx file. new command: csvtk unfold for unfolding multiple values in cells of a field. #103 rename csvtk collapse to csvtk fold , for folding multiple values of a field into cells of groups. csvtk cut : support range format 2- to choose 2nd column to the end . #106 csvtk round : fix bug of failing to round scientific notation with value small than one, e.g., 7.1E-1 . csvtk v0.21.0 new command: csvtk nrow/ncol for printing number of rows or columns. new command: round to round float to n decimal places. #112 csvtk headers : file name and column index is optional outputted with new flag -v/--verbose . csvtk dim : new flags --tabluar , --cols , --rows , -n/--no-files . csvtk dim/ncol/nrow : can handle empty files now. #108 csvtk csv2json #104 : new flag -b/--blank : do not convert \"\", \"na\", \"n/a\", \"none\", \"null\", \".\" to null new flag -n/--parse-num : parse numeric values for nth column(s), multiple values are supported and \"a\"/\"all\" for all columns. csvtk xlsx2csv : fix output for ragged table. #110 csvtk join : fix bug for joining >2 files. csvtk uniq : new flag -n/--keep-n for keeping first N records of every key. csvtk cut : support repeatedly selecting columns. #106 csvtk v0.20.0 new command csvtk comb : compute combinations of items at every row. new command csvtk sep : separate column into multiple columns. #96 csvtk : list lines' number of illegal ( -I ) and empty ( -E ) rows. #97 new flag --infile-list for giving file of input files list (one file per line), if given, they are appended to files from cli arguments csvtk join : reenable flag -i/--ignore-case . #99 outer join is supported . #23 new flag -L/--left-join : left join, equals to -k/--keep-unmatched, exclusive with --outer-join new flag -O/--outer-join : outer join, exclusive with --left-join rename flag --fill to --na . csvtk filter2 : fix bug when column names start with digits, e.g., 1000g2015aug . Thank @VorontsovIE ( #44 ) csvtk concat : allow one input file. #98 csvtk mutate : new flag -R/--remove for removing input column. csvtk v0.19.1 csvtk : fix checking file existence. show friendly error message when giving empty field like csvtk cut -f a, b . csvtk summary : fix err of q1 and q3. #90 csvtk version : making checking update optional. csvtk v0.19.0 new commands by @bsipos : watch : online monitoring and histogram of selected field. corr : calculate Pearson correlation between numeric columns. cat : stream file and report progress. csvtk split : fix bug of repeatedly output header line when number of output files exceed value of --buf-groups . #83 csvtk plot hist : new option --percentiles to add percentiles to histogram x label. #88 csvtk v0.18.2 csvtk replace/rename2/splitxlsx : fix flag conflicts with global flag -I since v0.18.0. csvtk replace/rename2 : removing shorthand flag -I for --key-capt-idx . csvtk splitxlsx : changing shorthand flag of --sheet-index from -I to -N . csvtk v0.18.1 csvtk sort : fix mutiple-key-sort containing natural order sorting. #79 csvtk xlsx2csv : reacts to global flags -t , -T , -D and -E . #78 csvtk v0.18.0 csvtk : add new flag --ignore-illegal-row to skip illegal rows. #72 csvtk summary : add more textual/numeric operations. #64 csvtk sort : fix bug for sorting by columns with empty values. #70 csvtk grep : add new flag --delete-matched to delete a pattern right after being matched, this keeps the firstly matched data and speedups when using regular expressions. #77 csvtk v0.17.0 new command: csvtk add-header and csvtk del-header for adding/deleting column names. [#62] csvtk v0.16.0 new command: csvtk csv2json : convert CSV to JSON format. remove comand: csvtk stats2 . new command csvtk summary : summary statistics of selected digital fields (groupby group fields), usage and examples . #59 csvtk replace : add flag --nr-width : minimum width for {nr} in flag -r/--replacement. e.g., formating \"1\" to \"001\" by --nr-width 3 (default 1) csvtk rename2/replace : add flag -A, --kv-file-all-left-columns-as-value , for treating all columns except 1th one as value for kv-file with more than 2 columns. #56 csvtk v0.15.0 csvtk : add global flag -E/--ignore-empty-row to skip empty row. #50 csvtk mutate2 : add flag -s/--digits-as-string for not converting big digits into scientific notation. #46 csvtk sort : add support for sorting in natural order. #49 csvtk v0.14.0 csvtk : supporting multi-line fields by replacing multicorecsv with standard library encoding/csv , while losing support for metaline which was supported since v0.7.0 . It also gain a little speedup. csvtk sample : add flag -n/--line-number to print line number as the first column (\"n\") csvtk filter2 : fix bug when column names start with digits, e.g., 1000g2015aug ( #44 ) csvtk rename2 : add support for similar repalecement symbols {kv} and {nr} in csvtk replace csvtk v0.13.0 new command concat for concatenating CSV/TSV files by rows #38 csvtk : add support for environment variables for frequently used global flags #39 CSVTK_T for flag -t/--tabs CSVTK_H for flag -H/--no-header-row mutate2 : add support for eval expression WITHOUT column index symbol, so we can add some string constants #37 pretty : better support for files with duplicated column names csvtk v0.12.0 new command collapse : collapsing one field with selected fields as keys freq : keeping orignal order of keys by default split : performance improvement add option -G/--out-gzip for forcing output gzipped file csvtk v0.11.0 add command split to split CSV/TSV into multiple files according to column values add command splitxlxs to split XLSX sheet into multiple sheets according to column values csvtk , automatically check BOM (byte-order mark) and discard it csvtk v0.10.0 add subcommand xlsx2csv to convert XLSX to CSV format grep , filter , filter2 : add flag -n/--line-number to print line-number as the first column cut : add flag -i/--ignore-case to ignore case of column name csvtk v0.9.1 csvtk replace : fix bug when replacing with key-value pairs brought in v0.8.0 csvtk v0.9.0 add subcommand csvtk mutate2 : create new column from selected fields by awk-like arithmetic/string expressions add new command genautocomplete to generate shell autocompletion script! csvtk v0.8.0 new command csvtk gather for gathering columns into key-value pairs . csvtk sort : support sorting by user-defined order . fix bug of unselecting field : wrongly reporting error of fields not existing. affected commands: cut , filter , fitler2 , freq , grep , inter , mutate , rename , rename2 , replace , stats2 , uniq . update help message of flag -F/--fuzzy-fields . update help message of global flag -t , which overrides both -d and -D . If you want other delimiter for tabular input, use -t $'\\t' -D \"delimiter\" . csvtk v0.7.1 csvtk plot box and csvtk plot line : fix bugs for special cases of input compile with go1.8.1 csvtk v0.7.0 fig bug of \"stricter field checking\" in v0.6.0 and v0.6.1 when using flag -F/--fuzzy-fields csvtk pretty and csvtk csv2md : add attention that these commands treat the first row as header line and require them to be unique. csvtk stat renamed to csvtk stats , old name is still available as an alias. csvtk stat2 renamed to csvtk stats2 , old name is still available as an alias. issues/13 seamlessly support for data with meta line of separator declaration used by MS Excel . csvtk v0.6.1 csvtk cut : minor bug: panic when no fields given. i.e., csvtk cut . All relevant commands have been fixed. csvtk v0.6.0 csvtk grep : large performance improvement by discarding goroutine (multiple threads), and keeping output in order of input . Better column name checking and stricter field checking, fields out of range are not ignored now . Affected commands include cut , filter , freq , grep , inter , mutate , rename , rename2 , replace , stat2 , and uniq . New command: csvtk filter2 , filtering rows by arithmetic/string expressions like awk . csvtk v0.5.0 csvtk cut : delete flag -n/--names , move it to a new command csvtk headers new command: csvtk headers new command: csvtk head new command: csvtk sample csvtk v0.4.6 csvtk grep : fix result highlight when flag -v is on. csvtk v0.4.5 csvtk join : support the 2nd or later files with entries with same ID. csvtk v0.4.4 add command csvtk freq : frequencies of selected fields add lots of examples in usage page csvtk v0.4.3 improvement of using experience: flag -n is not required anymore when flag -H in csvtk mutate csvtk v0.4.2 fix highlight bug of csvtk grep : if the pattern matches multiple parts, the text will be wrongly edited. changes: disable highlight when pattern file given. change the default output of all ploting commands to STDOUT, now you can pipe the image to \"display\" command of Imagemagic. csvtk v0.4.1 Nothing changed. Just fix the links due to inappropriate deployment of v0.4.0 csvtk v0.4.0 add flag for csvtk replace : -K ( --keep-key ) keep the key as value when no value found for the key. This is open in default in previous versions. csvtk v0.3.9 fix bug: header row incomplete in csvtk sort result csvtk v0.3.8.1 fix bug of flag parsing library pflag , detail . The bug affected the csvtk grep -r -p , when value of -p contain \"[\" and \"]\" at the beginning or end, they are wrongly parsed. csvtk v0.3.8 new feature: csvtk cut supports ordered fields output. e.g., csvtk cut -f 2,1 outputs the 2nd column in front of 1th column. new commands: csvtk plot can plot three types of plots by subcommands: csvtk plot hist : histogram csvtk plot box : boxplot csvtk plot line : line plot and scatter plot csvtk v0.3.7 fix a serious bug of using negative field of column name, e.g. -f \"-id\" csvtk v0.3.6 csvtk replace support replacement symbols {nr} (record number) and {kv} (corresponding value of the key ($1) by key-value file) csvtk v0.3.5.2 add flag --fill for csvtk join , so we can fill the unmatched data fix typo csvtk v0.3.5.1 fix minor bug of reading lines ending with \\r\\n from a dependency package csvtk v0.3.5 fix minor bug of csv2md add subcommand version which could check for update csvtk v0.3.4 fix bug of csvtk replace that head row should not be edited. csvtk v0.3.3 fix bug of csvtk grep -t -P csvtk v0.3.2 fix bug of inter csvtk v0.3.1 add support of search multiple fields for grep csvtk v0.3 add subcommand csv2md csvtk v0.2.9 add more flags to subcommand pretty fix bug of csvtk cut -n add subcommand filter csvtk v0.2.8 add subcommand pretty -- convert CSV to readable aligned table csvtk v0.2.7 fix highlight failing in windows csvtk v0.2.6 fix one error message of grep highlight matched fields in result of grep csvtk v0.2.5 fix bug of stat that failed to considerate files with header row add subcommand stat2 - summary of selected number fields make the output of stat prettier csvtk v0.2.4 fix bug of handling comment lines add some notes before using csvtk csvtk v0.2.3 add flag --colnames to cut flag -f ( --fields ) of join supports single value now csvtk v0.2.2 add flag --keep-unmathed to join csvtk v0.2 finish almost functions csvtk v0.2.1 fix bug of mutate /** * RECOMMENDED CONFIGURATION VARIABLES: EDIT AND UNCOMMENT THE SECTION BELOW TO INSERT DYNAMIC VALUES FROM YOUR PLATFORM OR CMS. * LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT: https://disqus.com/admin/universalcode/#configuration-variables */ /* var disqus_config = function () { this.page.url = PAGE_URL; // Replace PAGE_URL with your page's canonical URL variable this.page.identifier = PAGE_IDENTIFIER; // Replace PAGE_IDENTIFIER with your page's unique identifier variable }; */ (function() { // DON'T EDIT BELOW THIS LINE var d = document, s = d.createElement('script'); s.src = '//csvtk.disqus.com/embed.js'; s.setAttribute('data-timestamp', +new Date()); (d.head || d.body).appendChild(s); })(); Please enable JavaScript to view the comments powered by Disqus.","title":"Download"},{"location":"download/#download","text":"csvtk is implemented in Go programming language, executable binary files for most popular operating system are freely available in release page.","title":"Download"},{"location":"download/#current-version","text":"csvtk v0.28.0 csvtk : add the shortcut -X for the flag --infile-list . #249 csvtk pretty : support field ranges for -m/--align-center and -r/--align-right . #244 csvtk spread : support values sharing the same keys. #248 csvtk join : a new flag -P/--prefix-duplicates : add filenames as colname prefixes only for duplicated colnames. #246 csvtk mutate2 : fix changing the order of the header row, the code was accidentally missing during code refactoring in v0.27.0. #252 csvtk xlsx2csv : fix open /tmp/excelize-: no such file or directory error for big .xlsx files. #251 csvtk comb : fix the empty result bug for alphabet sizes greater than 64.","title":"Current Version"},{"location":"download/#links","text":"OS Arch File, \u4e2d\u56fd\u955c\u50cf Download Count Linux 32-bit csvtk_linux_386.tar.gz , \u4e2d\u56fd\u955c\u50cf Linux 64-bit csvtk_linux_amd64.tar.gz , \u4e2d\u56fd\u955c\u50cf Linux 64-bit csvtk_linux_arm64.tar.gz , \u4e2d\u56fd\u955c\u50cf macOS 64-bit csvtk_darwin_amd64.tar.gz , \u4e2d\u56fd\u955c\u50cf macOS arm64 csvtk_darwin_arm64.tar.gz , \u4e2d\u56fd\u955c\u50cf Windows 32-bit csvtk_windows_386.exe.tar.gz , \u4e2d\u56fd\u955c\u50cf Windows 64-bit csvtk_windows_amd64.exe.tar.gz , \u4e2d\u56fd\u955c\u50cf Notes run csvtk version to check update !!! run csvtk genautocomplete to update Bash completion !!!","title":"Links"},{"location":"download/#installation","text":"Download Page csvtk is implemented in Go programming language, executable binary files for most popular operating systems are freely available in release page.","title":"Installation"},{"location":"download/#method-1-download-binaries-latest-stabledev-version","text":"Just download compressed executable file of your operating system, and decompress it with tar -zxvf *.tar.gz command or other tools. And then: For Linux-like systems If you have root privilege simply copy it to /usr/local/bin : sudo cp csvtk /usr/local/bin/ Or copy to anywhere in the environment variable PATH : mkdir -p $HOME/bin/; cp csvtk $HOME/bin/ For windows , just copy csvtk.exe to C:\\WINDOWS\\system32 .","title":"Method 1: Download binaries (latest stable/dev version)"},{"location":"download/#method-2-install-via-conda-latest-stable-version","text":"conda install -c bioconda csvtk","title":"Method 2: Install via conda (latest stable version)"},{"location":"download/#method-3-install-via-homebrew-may-be-not-the-latest","text":"brew install csvtk","title":"Method 3: Install via homebrew (may be not the latest)"},{"location":"download/#method-4-for-go-developer-latest-stabledev-version","text":"go get -u github.com/shenwei356/csvtk/csvtk","title":"Method 4: For Go developer (latest stable/dev version)"},{"location":"download/#method-5-for-archlinux-aur-users-may-be-not-the-latest","text":"yaourt -S csvtk","title":"Method 5: For ArchLinux AUR users (may be not the latest)"},{"location":"download/#method-6-compiling-from-source-latest-stabledev-version","text":"# ------------------- install golang ----------------- # download Go from https://go.dev/dl wget https://go.dev/dl/go1.17.12.linux-amd64.tar.gz tar -zxf go1.17.12.linux-amd64.tar.gz -C $HOME/ # or # echo \"export PATH=$PATH:$HOME/go/bin\" >> ~/.bashrc # source ~/.bashrc export PATH=$PATH:$HOME/go/bin # ------------- the latest stable version ------------- go get -v -u github.com/shenwei356/csvtk/csvtk # The executable binary file is located in: # ~/go/bin/csvtk # You can also move it to anywhere in the $PATH mkdir -p $HOME/bin cp ~/go/bin/csvtk $HOME/bin/ # --------------- the development version -------------- git clone https://github.com/shenwei356/csvtk cd csvtk/csvtk/ go build # The executable binary file is located in: # ./csvtk # You can also move it to anywhere in the $PATH mkdir -p $HOME/bin cp ./csvtk $HOME/bin/","title":"Method 6: Compiling from source (latest stable/dev version)"},{"location":"download/#shell-completion","text":"Bash: # generate completion shell csvtk genautocomplete --shell bash # configure if never did. # install bash-completion if the \"complete\" command is not found. echo \"for bcfile in ~/.bash_completion.d/* ; do source \\$bcfile; done\" >> ~/.bash_completion echo \"source ~/.bash_completion\" >> ~/.bashrc Zsh: # generate completion shell csvtk genautocomplete --shell zsh --file ~/.zfunc/_csvtk # configure if never did echo 'fpath=( ~/.zfunc \"${fpath[@]}\" )' >> ~/.zshrc echo \"autoload -U compinit; compinit\" >> ~/.zshrc fish: csvtk genautocomplete --shell fish --file ~/.config/fish/completions/csvtk.fish","title":"Shell-completion"},{"location":"download/#release-history","text":"csvtk v0.27.2 csvtk pretty : fix the bug of empty first row with -H/--no-header-row , introduced in v0.27.0. new style 3line for three-line table. csvtk csv2xlsx : binaries compiled with go1.21 would result in a broken xlsx file. #243 csvtk splitxlsx : fix the error of invalid worksheet index . #1617 csvtk v0.27.1 csvtk filter2/mutate2 : fix the bug of selecting with field numbers. #242 csvtk v0.27.0 csvtk : code refactoring and simplifying code, with 16% less code. most commands support open column range syntax , e.g., csvtk grep -f 2- . #120 only selected column names are not allowed to be duplicated in the input data : box, corr, filter, filter2, fold, freq, gather, historysort, inter, join, line, mutate, mutate2, rename, replace, sep, split, summary, unfold, uniq, watch. Other commands do not have the restriction. #235 add a new global flag -Z/--show-row-number , supported commands: cut, csv2tab, csv2xlsx, tab2csv, pretty. the colum name of row number changes from \"n\" to \"row\": csv2xlsx, csv2tab, cut, filter, filter2, grep, pretty, sample, tab2csv. new command : csvtk spread : spread a key-value pair across multiple columns, like tidyr::spread/pivot_wider . #91 , #236 , #239 csvtk mutate/mutate2 : new flags --at , --before , --after for specifying the position of the new column . #193 csvtk cut : fix unselect range error. #234 fix -i/--ignore-case . csvtk pretty : allow align-center and align-right for specific columns . #240 csvtk round : fix bug of failing to round scientific notation with value small than one, e.g., 7.1E-1 . csvtk summary : fix duplicated columns. fix result error when multiple stats applied to the same column. csvtk corr/watch : rewrite and fix bug, support choosing fields with column names. csvtk v0.26.0 csvtk : near all commands skip empty files now . #204 the global flag --infile-list accepts stdin \"-\". #210 new command csvtk fix : fix CSV/TSV with different numbers of columns in rows . #226 csvtk pretty : rewrite to support wrapping cells . #206 #209 #228 csvtk cut/fmtdate/freq/grep/rename/rename2/replace/round : allow duplicated column names. csvtk csv2xlsx : optionally stores numbers as float. #217 csvtk xlsx2csv : fix bug where xlsx2csv treats small number (padj < 1e-25) as 0. It's solved by updating the excelize package. #261 csvtk join : a new flag for adding filename as column name prefix. by @tetedange13 #202 csvtk mutate2 : fix wrongly treating strings like E10 as numbers in scientific notation. #219 csvtk sep : fix the logic. #218 csvtk space2tab : fix \"bufio.Scanner: token too long\". #231 csvtk v0.25.0 csvtk : report empty files. csvtk join : fix loading file with no records. csvtk filter2/muate2 : support variable format of ${var} with special charactors including commas, spaces, and parentheses, e.g., ${a,b} , ${a b} , or ${a (b)} . #186 csvtk sort : fix checking non-existed fileds. csvtk plot box/hist/line : new flag --skip-na for skipping missing data. #188 csvtk csv2xlsx : stores number as float. #192 csvtk summary : new functions argmin and argmax . #181 csvtk v0.24.0 Incompatible changes : csvtk mutate2/summary : mutate2 : remove the option -L/--digits . use the same option -w/--decimal-width to limit floats to N decimal points. new command csvtk fmtdate : format date of selected fields. #159 csvtk grep : fix bug for searching with -r -p . . csvtk csv2rst : fix bug for data containing unicode. #137 csvtk filter2 : fix bug for date expression. #146 csvtk mutate2/filter2 : change the way of rexpression evaluation. add custom functions: len() . #153 fix bug when using two or more columns with common prefixes in column names . #173 fix value with single or double quotes. #174 csvtk cut : new flags -m/--allow-missing-col and -b/--blank-missing-col . #156 csvtk pretty : still add header row for empty column. csvtk csv2md : better format. csvtk join : new flag -n/--ignore-null . #163 csvtk v0.23.0 new comand: csvtk csv2rst for converting CSV to reStructuredText format. #137 csvtk pretty : add header separator line. #123 csvtk mutate2/summary : fix message and doc. Thanks @VladimirAlexiev #127 csvtk mutate2 : fix null coalescence: ??. #129 csvtk genautocomplete : supports bash|zsh|fish|powershell. #126 csvtk cat : fix progress bar. #130 csvtk grep : new flag immediate-output . csvtk csv2xlsx : fix bug for table with > 26 columns. 138 csvtk v0.22.0 csvtk : global flag -t does not overide -D anymore . #114 If the executable/symlink name is tsvtk the -t/--tabs option for tab input is set . Thanks @bsipos. #117 new command: csvtk csv2xlsx for converting CSV/TSV file(s) to a single .xlsx file. new command: csvtk unfold for unfolding multiple values in cells of a field. #103 rename csvtk collapse to csvtk fold , for folding multiple values of a field into cells of groups. csvtk cut : support range format 2- to choose 2nd column to the end . #106 csvtk round : fix bug of failing to round scientific notation with value small than one, e.g., 7.1E-1 . csvtk v0.21.0 new command: csvtk nrow/ncol for printing number of rows or columns. new command: round to round float to n decimal places. #112 csvtk headers : file name and column index is optional outputted with new flag -v/--verbose . csvtk dim : new flags --tabluar , --cols , --rows , -n/--no-files . csvtk dim/ncol/nrow : can handle empty files now. #108 csvtk csv2json #104 : new flag -b/--blank : do not convert \"\", \"na\", \"n/a\", \"none\", \"null\", \".\" to null new flag -n/--parse-num : parse numeric values for nth column(s), multiple values are supported and \"a\"/\"all\" for all columns. csvtk xlsx2csv : fix output for ragged table. #110 csvtk join : fix bug for joining >2 files. csvtk uniq : new flag -n/--keep-n for keeping first N records of every key. csvtk cut : support repeatedly selecting columns. #106 csvtk v0.20.0 new command csvtk comb : compute combinations of items at every row. new command csvtk sep : separate column into multiple columns. #96 csvtk : list lines' number of illegal ( -I ) and empty ( -E ) rows. #97 new flag --infile-list for giving file of input files list (one file per line), if given, they are appended to files from cli arguments csvtk join : reenable flag -i/--ignore-case . #99 outer join is supported . #23 new flag -L/--left-join : left join, equals to -k/--keep-unmatched, exclusive with --outer-join new flag -O/--outer-join : outer join, exclusive with --left-join rename flag --fill to --na . csvtk filter2 : fix bug when column names start with digits, e.g., 1000g2015aug . Thank @VorontsovIE ( #44 ) csvtk concat : allow one input file. #98 csvtk mutate : new flag -R/--remove for removing input column. csvtk v0.19.1 csvtk : fix checking file existence. show friendly error message when giving empty field like csvtk cut -f a, b . csvtk summary : fix err of q1 and q3. #90 csvtk version : making checking update optional. csvtk v0.19.0 new commands by @bsipos : watch : online monitoring and histogram of selected field. corr : calculate Pearson correlation between numeric columns. cat : stream file and report progress. csvtk split : fix bug of repeatedly output header line when number of output files exceed value of --buf-groups . #83 csvtk plot hist : new option --percentiles to add percentiles to histogram x label. #88 csvtk v0.18.2 csvtk replace/rename2/splitxlsx : fix flag conflicts with global flag -I since v0.18.0. csvtk replace/rename2 : removing shorthand flag -I for --key-capt-idx . csvtk splitxlsx : changing shorthand flag of --sheet-index from -I to -N . csvtk v0.18.1 csvtk sort : fix mutiple-key-sort containing natural order sorting. #79 csvtk xlsx2csv : reacts to global flags -t , -T , -D and -E . #78 csvtk v0.18.0 csvtk : add new flag --ignore-illegal-row to skip illegal rows. #72 csvtk summary : add more textual/numeric operations. #64 csvtk sort : fix bug for sorting by columns with empty values. #70 csvtk grep : add new flag --delete-matched to delete a pattern right after being matched, this keeps the firstly matched data and speedups when using regular expressions. #77 csvtk v0.17.0 new command: csvtk add-header and csvtk del-header for adding/deleting column names. [#62] csvtk v0.16.0 new command: csvtk csv2json : convert CSV to JSON format. remove comand: csvtk stats2 . new command csvtk summary : summary statistics of selected digital fields (groupby group fields), usage and examples . #59 csvtk replace : add flag --nr-width : minimum width for {nr} in flag -r/--replacement. e.g., formating \"1\" to \"001\" by --nr-width 3 (default 1) csvtk rename2/replace : add flag -A, --kv-file-all-left-columns-as-value , for treating all columns except 1th one as value for kv-file with more than 2 columns. #56 csvtk v0.15.0 csvtk : add global flag -E/--ignore-empty-row to skip empty row. #50 csvtk mutate2 : add flag -s/--digits-as-string for not converting big digits into scientific notation. #46 csvtk sort : add support for sorting in natural order. #49 csvtk v0.14.0 csvtk : supporting multi-line fields by replacing multicorecsv with standard library encoding/csv , while losing support for metaline which was supported since v0.7.0 . It also gain a little speedup. csvtk sample : add flag -n/--line-number to print line number as the first column (\"n\") csvtk filter2 : fix bug when column names start with digits, e.g., 1000g2015aug ( #44 ) csvtk rename2 : add support for similar repalecement symbols {kv} and {nr} in csvtk replace csvtk v0.13.0 new command concat for concatenating CSV/TSV files by rows #38 csvtk : add support for environment variables for frequently used global flags #39 CSVTK_T for flag -t/--tabs CSVTK_H for flag -H/--no-header-row mutate2 : add support for eval expression WITHOUT column index symbol, so we can add some string constants #37 pretty : better support for files with duplicated column names csvtk v0.12.0 new command collapse : collapsing one field with selected fields as keys freq : keeping orignal order of keys by default split : performance improvement add option -G/--out-gzip for forcing output gzipped file csvtk v0.11.0 add command split to split CSV/TSV into multiple files according to column values add command splitxlxs to split XLSX sheet into multiple sheets according to column values csvtk , automatically check BOM (byte-order mark) and discard it csvtk v0.10.0 add subcommand xlsx2csv to convert XLSX to CSV format grep , filter , filter2 : add flag -n/--line-number to print line-number as the first column cut : add flag -i/--ignore-case to ignore case of column name csvtk v0.9.1 csvtk replace : fix bug when replacing with key-value pairs brought in v0.8.0 csvtk v0.9.0 add subcommand csvtk mutate2 : create new column from selected fields by awk-like arithmetic/string expressions add new command genautocomplete to generate shell autocompletion script! csvtk v0.8.0 new command csvtk gather for gathering columns into key-value pairs . csvtk sort : support sorting by user-defined order . fix bug of unselecting field : wrongly reporting error of fields not existing. affected commands: cut , filter , fitler2 , freq , grep , inter , mutate , rename , rename2 , replace , stats2 , uniq . update help message of flag -F/--fuzzy-fields . update help message of global flag -t , which overrides both -d and -D . If you want other delimiter for tabular input, use -t $'\\t' -D \"delimiter\" . csvtk v0.7.1 csvtk plot box and csvtk plot line : fix bugs for special cases of input compile with go1.8.1 csvtk v0.7.0 fig bug of \"stricter field checking\" in v0.6.0 and v0.6.1 when using flag -F/--fuzzy-fields csvtk pretty and csvtk csv2md : add attention that these commands treat the first row as header line and require them to be unique. csvtk stat renamed to csvtk stats , old name is still available as an alias. csvtk stat2 renamed to csvtk stats2 , old name is still available as an alias. issues/13 seamlessly support for data with meta line of separator declaration used by MS Excel . csvtk v0.6.1 csvtk cut : minor bug: panic when no fields given. i.e., csvtk cut . All relevant commands have been fixed. csvtk v0.6.0 csvtk grep : large performance improvement by discarding goroutine (multiple threads), and keeping output in order of input . Better column name checking and stricter field checking, fields out of range are not ignored now . Affected commands include cut , filter , freq , grep , inter , mutate , rename , rename2 , replace , stat2 , and uniq . New command: csvtk filter2 , filtering rows by arithmetic/string expressions like awk . csvtk v0.5.0 csvtk cut : delete flag -n/--names , move it to a new command csvtk headers new command: csvtk headers new command: csvtk head new command: csvtk sample csvtk v0.4.6 csvtk grep : fix result highlight when flag -v is on. csvtk v0.4.5 csvtk join : support the 2nd or later files with entries with same ID. csvtk v0.4.4 add command csvtk freq : frequencies of selected fields add lots of examples in usage page csvtk v0.4.3 improvement of using experience: flag -n is not required anymore when flag -H in csvtk mutate csvtk v0.4.2 fix highlight bug of csvtk grep : if the pattern matches multiple parts, the text will be wrongly edited. changes: disable highlight when pattern file given. change the default output of all ploting commands to STDOUT, now you can pipe the image to \"display\" command of Imagemagic. csvtk v0.4.1 Nothing changed. Just fix the links due to inappropriate deployment of v0.4.0 csvtk v0.4.0 add flag for csvtk replace : -K ( --keep-key ) keep the key as value when no value found for the key. This is open in default in previous versions. csvtk v0.3.9 fix bug: header row incomplete in csvtk sort result csvtk v0.3.8.1 fix bug of flag parsing library pflag , detail . The bug affected the csvtk grep -r -p , when value of -p contain \"[\" and \"]\" at the beginning or end, they are wrongly parsed. csvtk v0.3.8 new feature: csvtk cut supports ordered fields output. e.g., csvtk cut -f 2,1 outputs the 2nd column in front of 1th column. new commands: csvtk plot can plot three types of plots by subcommands: csvtk plot hist : histogram csvtk plot box : boxplot csvtk plot line : line plot and scatter plot csvtk v0.3.7 fix a serious bug of using negative field of column name, e.g. -f \"-id\" csvtk v0.3.6 csvtk replace support replacement symbols {nr} (record number) and {kv} (corresponding value of the key ($1) by key-value file) csvtk v0.3.5.2 add flag --fill for csvtk join , so we can fill the unmatched data fix typo csvtk v0.3.5.1 fix minor bug of reading lines ending with \\r\\n from a dependency package csvtk v0.3.5 fix minor bug of csv2md add subcommand version which could check for update csvtk v0.3.4 fix bug of csvtk replace that head row should not be edited. csvtk v0.3.3 fix bug of csvtk grep -t -P csvtk v0.3.2 fix bug of inter csvtk v0.3.1 add support of search multiple fields for grep csvtk v0.3 add subcommand csv2md csvtk v0.2.9 add more flags to subcommand pretty fix bug of csvtk cut -n add subcommand filter csvtk v0.2.8 add subcommand pretty -- convert CSV to readable aligned table csvtk v0.2.7 fix highlight failing in windows csvtk v0.2.6 fix one error message of grep highlight matched fields in result of grep csvtk v0.2.5 fix bug of stat that failed to considerate files with header row add subcommand stat2 - summary of selected number fields make the output of stat prettier csvtk v0.2.4 fix bug of handling comment lines add some notes before using csvtk csvtk v0.2.3 add flag --colnames to cut flag -f ( --fields ) of join supports single value now csvtk v0.2.2 add flag --keep-unmathed to join csvtk v0.2 finish almost functions csvtk v0.2.1 fix bug of mutate /** * RECOMMENDED CONFIGURATION VARIABLES: EDIT AND UNCOMMENT THE SECTION BELOW TO INSERT DYNAMIC VALUES FROM YOUR PLATFORM OR CMS. * LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT: https://disqus.com/admin/universalcode/#configuration-variables */ /* var disqus_config = function () { this.page.url = PAGE_URL; // Replace PAGE_URL with your page's canonical URL variable this.page.identifier = PAGE_IDENTIFIER; // Replace PAGE_IDENTIFIER with your page's unique identifier variable }; */ (function() { // DON'T EDIT BELOW THIS LINE var d = document, s = d.createElement('script'); s.src = '//csvtk.disqus.com/embed.js'; s.setAttribute('data-timestamp', +new Date()); (d.head || d.body).appendChild(s); })(); Please enable JavaScript to view the comments powered by Disqus.","title":"Release history"},{"location":"tutorial/","text":"Tutorial Analyzing OTU table Data Here is a mock OTU table from 16S rRNA sequencing result. Columns are sample IDs in format of \"GROUP.ID\" $ cat otu_table.csv Taxonomy,A.1,A.2,A.3,B.1,B.2,B.3,C.1,C.2 Proteobacteria,.13,.29,.13,.16,.13,.22,.30,.23 Firmicutes,.42,.06,.49,.41,.55,.41,.32,.38 Bacteroidetes,.19,.62,.12,.33,.16,.29,.34,.35 Deferribacteres,.17,.00,.24,.01,.01,.01,.01,.01 What a mess! Let's make it prettier! $ csvtk pretty otu_table.csv Taxonomy A.1 A.2 A.3 B.1 B.2 B.3 C.1 C.2 Proteobacteria .13 .29 .13 .16 .13 .22 .30 .23 Firmicutes .42 .06 .49 .41 .55 .41 .32 .38 Bacteroidetes .19 .62 .12 .33 .16 .29 .34 .35 Deferribacteres .17 .00 .24 .01 .01 .01 .01 .01 Steps Counting $ csvtk stat otu_table.csv file num_cols num_rows otu_table.csv 9 4 Column names $ csvtk headers otu_table.csv # otu_table.csv 1 Taxonomy 2 A.1 3 A.2 4 A.3 5 B.1 6 B.2 7 B.3 8 C.1 9 C.2 Convert to tab-delimited table $ csvtk csv2tab otu_table.csv Taxonomy A.1 A.2 A.3 B.1 B.2 B.3 C.1 C.2 Proteobacteria .13 .29 .13 .16 .13 .22 .30 .23 Firmicutes .42 .06 .49 .41 .55 .41 .32 .38 Bacteroidetes .19 .62 .12 .33 .16 .29 .34 .35 Deferribacteres .17 .00 .24 .01 .01 .01 .01 .01 Extract data of group A and B and save to file -o otu_table.gAB.csv $ csvtk cut -F -f \"Taxonomy,A.*,B.*\" otu_table.csv -o otu_table.gAB.csv $ csvtk pretty otu_table.gAB.csv Taxonomy A.1 A.2 A.3 B.1 B.2 B.3 Proteobacteria .13 .29 .13 .16 .13 .22 Firmicutes .42 .06 .49 .41 .55 .41 Bacteroidetes .19 .62 .12 .33 .16 .29 Deferribacteres .17 .00 .24 .01 .01 .01 Search some rows by fields. Matched parts will be highlighted as red $ csvtk grep -f Taxonomy -r -p \"tes\" otu_table.gAB.csv -T Result: Transpose $ csvtk transpose otu_table.gAB.csv -o otu_table.gAB.t.csv $ csvtk pretty otu_table.gAB.t.csv Taxonomy Proteobacteria Firmicutes Bacteroidetes Deferribacteres A.1 .13 .42 .19 .17 A.2 .29 .06 .62 .00 A.3 .13 .49 .12 .24 B.1 .16 .41 .33 .01 B.2 .13 .55 .16 .01 B.3 .22 .41 .29 .01 Rename name of the first column $ csvtk rename -f 1 -n \"sample\" otu_table.gAB.t.csv -o otu_table.gAB.t.r.csv $ csvtk pretty otu_table.gAB.t.r.csv sample Proteobacteria Firmicutes Bacteroidetes Deferribacteres A.1 .13 .42 .19 .17 A.2 .29 .06 .62 .00 A.3 .13 .49 .12 .24 B.1 .16 .41 .33 .01 B.2 .13 .55 .16 .01 B.3 .22 .41 .29 .01 Add group column $ csvtk mutate -p \"(.+?)\\.\" -n group otu_table.gAB.t.r.csv -o otu_table2.csv $ csvtk pretty otu_table2.csv sample Proteobacteria Firmicutes Bacteroidetes Deferribacteres group A.1 .13 .42 .19 .17 A A.2 .29 .06 .62 .00 A A.3 .13 .49 .12 .24 A B.1 .16 .41 .33 .01 B B.2 .13 .55 .16 .01 B B.3 .22 .41 .29 .01 B Rename groups: $ csvtk replace -f group -p \"A\" -r \"Ctrl\" otu_table2.csv \\ | csvtk replace -f group -p \"B\" -r \"Treatment\" \\ > otu_table3.csv $ csvtk pretty -s \" \" otu_table3.csv sample Proteobacteria Firmicutes Bacteroidetes Deferribacteres group A.1 .13 .42 .19 .17 Ctrl A.2 .29 .06 .62 .00 Ctrl A.3 .13 .49 .12 .24 Ctrl B.1 .16 .41 .33 .01 Treatment B.2 .13 .55 .16 .01 Treatment B.3 .22 .41 .29 .01 Treatment Sort by abundance of Proteobacteria in descending order. $ csvtk sort -k Proteobacteria:nr otu_table3.csv \\ | csvtk pretty -s \" \" sample Proteobacteria Firmicutes Bacteroidetes Deferribacteres group A.2 .29 .06 .62 .00 Ctrl B.3 .22 .41 .29 .01 Treatment B.1 .16 .41 .33 .01 Treatment B.2 .13 .55 .16 .01 Treatment A.3 .13 .49 .12 .24 Ctrl A.1 .13 .42 .19 .17 Ctrl Sort by abundance of Proteobacteria in descending order and Firmicutes in ascending order $ csvtk sort -k Proteobacteria:nr -k Firmicutes:n otu_table3.csv \\ | csvtk pretty -s \" \" sample Proteobacteria Firmicutes Bacteroidetes Deferribacteres group A.2 .29 .06 .62 .00 Ctrl B.3 .22 .41 .29 .01 Treatment B.1 .16 .41 .33 .01 Treatment A.1 .13 .42 .19 .17 Ctrl A.3 .13 .49 .12 .24 Ctrl B.2 .13 .55 .16 .01 Treatment Filter samples with abundance greater than 0 in all taxons (columns except for sample and group, you can also use -f \"2-5>0\" ). $ cat otu_table3.csv \\ | csvtk filter -f \"2-5>0\" \\ | csvtk pretty -s \" \" sample Proteobacteria Firmicutes Bacteroidetes Deferribacteres group A.1 .13 .42 .19 .17 Ctrl A.3 .13 .49 .12 .24 Ctrl B.1 .16 .41 .33 .01 Treatment B.2 .13 .55 .16 .01 Treatment B.3 .22 .41 .29 .01 Treatment Most of the time, we may want to remove samples with abundance of 0 in all taxons. $ cat otu_table3.csv \\ | csvtk filter -f \"2-5>0\" --any \\ | csvtk pretty -s \" \" sample Proteobacteria Firmicutes Bacteroidetes Deferribacteres group A.1 .13 .42 .19 .17 Ctrl A.2 .29 .06 .62 .00 Ctrl A.3 .13 .49 .12 .24 Ctrl B.1 .16 .41 .33 .01 Treatment B.2 .13 .55 .16 .01 Treatment B.3 .22 .41 .29 .01 Treatment /** * RECOMMENDED CONFIGURATION VARIABLES: EDIT AND UNCOMMENT THE SECTION BELOW TO INSERT DYNAMIC VALUES FROM YOUR PLATFORM OR CMS. * LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT: https://disqus.com/admin/universalcode/#configuration-variables */ /* var disqus_config = function () { this.page.url = PAGE_URL; // Replace PAGE_URL with your page's canonical URL variable this.page.identifier = PAGE_IDENTIFIER; // Replace PAGE_IDENTIFIER with your page's unique identifier variable }; */ (function() { // DON'T EDIT BELOW THIS LINE var d = document, s = d.createElement('script'); s.src = '//csvtk.disqus.com/embed.js'; s.setAttribute('data-timestamp', +new Date()); (d.head || d.body).appendChild(s); })(); Please enable JavaScript to view the comments powered by Disqus.","title":"Tutorial"},{"location":"tutorial/#tutorial","text":"","title":"Tutorial"},{"location":"tutorial/#analyzing-otu-table","text":"","title":"Analyzing OTU table"},{"location":"tutorial/#data","text":"Here is a mock OTU table from 16S rRNA sequencing result. Columns are sample IDs in format of \"GROUP.ID\" $ cat otu_table.csv Taxonomy,A.1,A.2,A.3,B.1,B.2,B.3,C.1,C.2 Proteobacteria,.13,.29,.13,.16,.13,.22,.30,.23 Firmicutes,.42,.06,.49,.41,.55,.41,.32,.38 Bacteroidetes,.19,.62,.12,.33,.16,.29,.34,.35 Deferribacteres,.17,.00,.24,.01,.01,.01,.01,.01 What a mess! Let's make it prettier! $ csvtk pretty otu_table.csv Taxonomy A.1 A.2 A.3 B.1 B.2 B.3 C.1 C.2 Proteobacteria .13 .29 .13 .16 .13 .22 .30 .23 Firmicutes .42 .06 .49 .41 .55 .41 .32 .38 Bacteroidetes .19 .62 .12 .33 .16 .29 .34 .35 Deferribacteres .17 .00 .24 .01 .01 .01 .01 .01","title":"Data"},{"location":"tutorial/#steps","text":"Counting $ csvtk stat otu_table.csv file num_cols num_rows otu_table.csv 9 4 Column names $ csvtk headers otu_table.csv # otu_table.csv 1 Taxonomy 2 A.1 3 A.2 4 A.3 5 B.1 6 B.2 7 B.3 8 C.1 9 C.2 Convert to tab-delimited table $ csvtk csv2tab otu_table.csv Taxonomy A.1 A.2 A.3 B.1 B.2 B.3 C.1 C.2 Proteobacteria .13 .29 .13 .16 .13 .22 .30 .23 Firmicutes .42 .06 .49 .41 .55 .41 .32 .38 Bacteroidetes .19 .62 .12 .33 .16 .29 .34 .35 Deferribacteres .17 .00 .24 .01 .01 .01 .01 .01 Extract data of group A and B and save to file -o otu_table.gAB.csv $ csvtk cut -F -f \"Taxonomy,A.*,B.*\" otu_table.csv -o otu_table.gAB.csv $ csvtk pretty otu_table.gAB.csv Taxonomy A.1 A.2 A.3 B.1 B.2 B.3 Proteobacteria .13 .29 .13 .16 .13 .22 Firmicutes .42 .06 .49 .41 .55 .41 Bacteroidetes .19 .62 .12 .33 .16 .29 Deferribacteres .17 .00 .24 .01 .01 .01 Search some rows by fields. Matched parts will be highlighted as red $ csvtk grep -f Taxonomy -r -p \"tes\" otu_table.gAB.csv -T Result: Transpose $ csvtk transpose otu_table.gAB.csv -o otu_table.gAB.t.csv $ csvtk pretty otu_table.gAB.t.csv Taxonomy Proteobacteria Firmicutes Bacteroidetes Deferribacteres A.1 .13 .42 .19 .17 A.2 .29 .06 .62 .00 A.3 .13 .49 .12 .24 B.1 .16 .41 .33 .01 B.2 .13 .55 .16 .01 B.3 .22 .41 .29 .01 Rename name of the first column $ csvtk rename -f 1 -n \"sample\" otu_table.gAB.t.csv -o otu_table.gAB.t.r.csv $ csvtk pretty otu_table.gAB.t.r.csv sample Proteobacteria Firmicutes Bacteroidetes Deferribacteres A.1 .13 .42 .19 .17 A.2 .29 .06 .62 .00 A.3 .13 .49 .12 .24 B.1 .16 .41 .33 .01 B.2 .13 .55 .16 .01 B.3 .22 .41 .29 .01 Add group column $ csvtk mutate -p \"(.+?)\\.\" -n group otu_table.gAB.t.r.csv -o otu_table2.csv $ csvtk pretty otu_table2.csv sample Proteobacteria Firmicutes Bacteroidetes Deferribacteres group A.1 .13 .42 .19 .17 A A.2 .29 .06 .62 .00 A A.3 .13 .49 .12 .24 A B.1 .16 .41 .33 .01 B B.2 .13 .55 .16 .01 B B.3 .22 .41 .29 .01 B Rename groups: $ csvtk replace -f group -p \"A\" -r \"Ctrl\" otu_table2.csv \\ | csvtk replace -f group -p \"B\" -r \"Treatment\" \\ > otu_table3.csv $ csvtk pretty -s \" \" otu_table3.csv sample Proteobacteria Firmicutes Bacteroidetes Deferribacteres group A.1 .13 .42 .19 .17 Ctrl A.2 .29 .06 .62 .00 Ctrl A.3 .13 .49 .12 .24 Ctrl B.1 .16 .41 .33 .01 Treatment B.2 .13 .55 .16 .01 Treatment B.3 .22 .41 .29 .01 Treatment Sort by abundance of Proteobacteria in descending order. $ csvtk sort -k Proteobacteria:nr otu_table3.csv \\ | csvtk pretty -s \" \" sample Proteobacteria Firmicutes Bacteroidetes Deferribacteres group A.2 .29 .06 .62 .00 Ctrl B.3 .22 .41 .29 .01 Treatment B.1 .16 .41 .33 .01 Treatment B.2 .13 .55 .16 .01 Treatment A.3 .13 .49 .12 .24 Ctrl A.1 .13 .42 .19 .17 Ctrl Sort by abundance of Proteobacteria in descending order and Firmicutes in ascending order $ csvtk sort -k Proteobacteria:nr -k Firmicutes:n otu_table3.csv \\ | csvtk pretty -s \" \" sample Proteobacteria Firmicutes Bacteroidetes Deferribacteres group A.2 .29 .06 .62 .00 Ctrl B.3 .22 .41 .29 .01 Treatment B.1 .16 .41 .33 .01 Treatment A.1 .13 .42 .19 .17 Ctrl A.3 .13 .49 .12 .24 Ctrl B.2 .13 .55 .16 .01 Treatment Filter samples with abundance greater than 0 in all taxons (columns except for sample and group, you can also use -f \"2-5>0\" ). $ cat otu_table3.csv \\ | csvtk filter -f \"2-5>0\" \\ | csvtk pretty -s \" \" sample Proteobacteria Firmicutes Bacteroidetes Deferribacteres group A.1 .13 .42 .19 .17 Ctrl A.3 .13 .49 .12 .24 Ctrl B.1 .16 .41 .33 .01 Treatment B.2 .13 .55 .16 .01 Treatment B.3 .22 .41 .29 .01 Treatment Most of the time, we may want to remove samples with abundance of 0 in all taxons. $ cat otu_table3.csv \\ | csvtk filter -f \"2-5>0\" --any \\ | csvtk pretty -s \" \" sample Proteobacteria Firmicutes Bacteroidetes Deferribacteres group A.1 .13 .42 .19 .17 Ctrl A.2 .29 .06 .62 .00 Ctrl A.3 .13 .49 .12 .24 Ctrl B.1 .16 .41 .33 .01 Treatment B.2 .13 .55 .16 .01 Treatment B.3 .22 .41 .29 .01 Treatment /** * RECOMMENDED CONFIGURATION VARIABLES: EDIT AND UNCOMMENT THE SECTION BELOW TO INSERT DYNAMIC VALUES FROM YOUR PLATFORM OR CMS. * LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT: https://disqus.com/admin/universalcode/#configuration-variables */ /* var disqus_config = function () { this.page.url = PAGE_URL; // Replace PAGE_URL with your page's canonical URL variable this.page.identifier = PAGE_IDENTIFIER; // Replace PAGE_IDENTIFIER with your page's unique identifier variable }; */ (function() { // DON'T EDIT BELOW THIS LINE var d = document, s = d.createElement('script'); s.src = '//csvtk.disqus.com/embed.js'; s.setAttribute('data-timestamp', +new Date()); (d.head || d.body).appendChild(s); })(); Please enable JavaScript to view the comments powered by Disqus.","title":"Steps"},{"location":"usage/","text":"Usage and Examples Before use Attention The CSV parser requires all the lines have same number of fields/columns. Even lines with spaces will cause error. Use '-I/--ignore-illegal-row' to skip these lines if neccessary. You can also use 'csvtk fix' to fix files with different numbers of columns in rows. By default, csvtk thinks your files have header row, if not, switch flag \"-H\" on. Column names better be unique. By default, lines starting with \"#\" will be ignored, if the header row starts with \"#\", please assign flag \"-C\" another rare symbol, e.g. '$'. By default, csvtk handles CSV files, use flag \"-t\" for tab-delimited files. If double quotes exist in fields, use flag \"-l\". Do not mix use field (column) numbers and names. Table of Contents csvtk Information headers dim/nrow/ncol summary corr watch Format conversion pretty transpose space2tab csv2md csv2json csv2xlsx xlsx2csv Set operations head concat sample cut grep uniq freq inter filter filter2 join split splitxlsx comb Edit fix add-header del-header rename rename2 replace round mutate mutate2 sep gather spread unfold fold Ordering sort Ploting plot plot hist plot box plot line Misc cat genautocomplete csvtk Usage csvtk -- a cross-platform, efficient and practical CSV/TSV toolkit Version: 0.28.0 Author: Wei Shen : . e.g., \"-k name:u -L name:level.txt Examples data $ cat testdata/names.csv id,first_name,last_name,username 11,\"Rob\",\"Pike\",rob 2,Ken,Thompson,ken 4,\"Robert\",\"Griesemer\",\"gri\" 1,\"Robert\",\"Thompson\",\"abc\" NA,\"Robert\",\"Abel\",\"123\" By single column : csvtk sort -k 1 or csvtk sort -k last_name in alphabetical order $ cat testdata/names.csv \\ | csvtk sort -k first_name id,first_name,last_name,username 2,Ken,Thompson,ken 11,Rob,Pike,rob NA,Robert,Abel,123 1,Robert,Thompson,abc 4,Robert,Griesemer,gri in reversed alphabetical order ( key:r ) $ cat testdata/names.csv \\ | csvtk sort -k first_name:r id,first_name,last_name,username NA,Robert,Abel,123 1,Robert,Thompson,abc 4,Robert,Griesemer,gri 11,Rob,Pike,rob 2,Ken,Thompson,ken in numerical order ( key:n ) $ cat testdata/names.csv \\ | csvtk sort -k id:n id,first_name,last_name,username NA,Robert,Abel,123 1,Robert,Thompson,abc 2,Ken,Thompson,ken 4,Robert,Griesemer,gri 11,Rob,Pike,rob in natural order ( key:N ) $ cat testdata/names.csv | csvtk sort -k id:N id,first_name,last_name,username 1,Robert,Thompson,abc 2,Ken,Thompson,ken 4,Robert,Griesemer,gri 11,Rob,Pike,rob NA,Robert,Abel,123 in natural order ( key:N ), a bioinformatics example $ echo \"X,Y,1,10,2,M,11,1_c,Un_g,1_g\" | csvtk transpose X Y 1 10 2 M 11 1_c Un_g 1_g $ echo \"X,Y,1,10,2,M,11,1_c,Un_g,1_g\" \\ | csvtk transpose \\ | csvtk sort -H -k 1:N 1 1_c 1_g 2 10 11 M Un_g X Y By multiple columns: csvtk sort -k 1,2 or csvtk sort -k 1 -k 2 or csvtk sort -k last_name,age # by first_name and then last_name $ cat testdata/names.csv | csvtk sort -k first_name -k last_name id,first_name,last_name,username 2,Ken,Thompson,ken 11,Rob,Pike,rob NA,Robert,Abel,123 4,Robert,Griesemer,gri 1,Robert,Thompson,abc # by first_name and then ID $ cat testdata/names.csv | csvtk sort -k first_name -k id:n id,first_name,last_name,username 2,Ken,Thompson,ken 11,Rob,Pike,rob NA,Robert,Abel,123 1,Robert,Thompson,abc 4,Robert,Griesemer,gri By user-defined order # user-defined order/level $ cat testdata/size_level.txt tiny mini small medium big # original data $ cat testdata/size.csv id,size 1,Huge 2,Tiny 3,Big 4,Small 5,Medium $ csvtk sort -k 2:u -i -L 2:testdata/size_level.txt testdata/size.csv id,size 2,Tiny 4,Small 5,Medium 3,Big 1,Huge plot Usage plot common figures Notes: 1. Output file can be set by flag -o/--out-file. 2. File format is determined by the out file suffix. Supported formats: eps, jpg|jpeg, pdf, png, svg, and tif|tiff 3. If flag -o/--out-file not set (default), image is written to stdout, you can display the image by pipping to \"display\" command of Imagemagic or just redirect to file. Usage: csvtk plot [command] Available Commands: box plot boxplot hist plot histogram line line plot and scatter plot Flags: --axis-width float axis width (default 1.5) -f, --data-field string column index or column name of data (default \"1\") --format string image format for stdout when flag -o/--out-file not given. available values: eps, jpg|jpeg, pdf, png, svg, and tif|tiff. (default \"png\") -g, --group-field string column index or column name of group --height float Figure height (default 4.5) -h, --help help for plot --label-size int label font size (default 14) --na-values strings NA values, case ignored (default [,NA,N/A]) --skip-na skip NA values in --na-values --tick-width float axis tick width (default 1.5) --title string Figure title --title-size int title font size (default 16) --width float Figure width (default 6) --x-max string maximum value of X axis --x-min string minimum value of X axis --xlab string x label text --y-max string maximum value of Y axis --y-min string minimum value of Y axis --ylab string y label text Note that most of the flags of plot are global flags of the subcommands hist , box and line Notes of image output Output file can be set by flag -o/--out-file. File format is determined by the out file suffix. Supported formats: eps, jpg|jpeg, pdf, png, svg, and tif|tiff If flag -o/--out-file not set (default), image is written to stdout, you can display the image by pipping to display command of Imagemagic or just redirect to file. plot hist Usage plot histogram Notes: 1. Output file can be set by flag -o/--out-file. 2. File format is determined by the out file suffix. Supported formats: eps, jpg|jpeg, pdf, png, svg, and tif|tiff 3. If flag -o/--out-file not set (default), image is written to stdout, you can display the image by pipping to \"display\" command of Imagemagic or just redirect to file. Usage: csvtk plot hist [flags] Flags: --bins int number of bins (default 50) --color-index int color index, 1-7 (default 1) Examples example data $ zcat testdata/grouped_data.tsv.gz | head -n 5 | csvtk -t pretty Group Length GC Content Group A 97 57.73 Group A 95 49.47 Group A 97 49.48 Group A 100 51.00 plot histogram with data of the second column: $ csvtk -t plot hist testdata/grouped_data.tsv.gz -f 2 \\ --title Histogram -o histogram.png You can also write image to stdout and pipe to \"display\" command of Imagemagic: $ csvtk -t plot hist testdata/grouped_data.tsv.gz -f 2 | display plot box Usage plot boxplot Notes: 1. Output file can be set by flag -o/--out-file. 2. File format is determined by the out file suffix. Supported formats: eps, jpg|jpeg, pdf, png, svg, and tif|tiff 3. If flag -o/--out-file not set (default), image is written to stdout, you can display the image by pipping to \"display\" command of Imagemagic or just redirect to file. Usage: csvtk plot box [flags] Flags: --box-width float box width --horiz horize box plot Examples plot boxplot with data of the \"GC Content\" (third) column, group information is the \"Group\" column. csvtk -t plot box testdata/grouped_data.tsv.gz -g \"Group\" -f \"GC Content\" \\ --width 3 --title \"Box plot\" \\ > boxplot.png plot horiz boxplot with data of the \"Length\" (second) column, group information is the \"Group\" column. $ csvtk -t plot box testdata/grouped_data.tsv.gz -g \"Group\" -f \"Length\" \\ --height 3 --width 5 --horiz --title \"Horiz box plot\" \\ > boxplot2.png` plot line Usage line plot and scatter plot Notes: 1. Output file can be set by flag -o/--out-file. 2. File format is determined by the out file suffix. Supported formats: eps, jpg|jpeg, pdf, png, svg, and tif|tiff 3. If flag -o/--out-file not set (default), image is written to stdout, you can display the image by pipping to \"display\" command of Imagemagic or just redirect to file. Usage: csvtk plot line [flags] Flags: -x, --data-field-x string column index or column name of X for command line -y, --data-field-y string column index or column name of Y for command line --legend-left locate legend along the left edge of the plot --legend-top locate legend along the top edge of the plot --line-width float line width (default 1.5) --point-size float point size (default 3) --scatter only plot points Examples example data $ head -n 5 testdata/xy.tsv Group X Y A 0 1 A 1 1.3 A 1.5 1.5 A 2.0 2 plot line plot with X-Y data $ csvtk -t plot line testdata/xy.tsv -x X -y Y -g Group \\ --title \"Line plot\" \\ > lineplot.png plot scatter $ csvtk -t plot line testdata/xy.tsv -x X -y Y -g Group \\ --title \"Scatter\" --scatter \\ > lineplot.png cat Usage stream file to stdout and report progress on stderr Usage: csvtk cat [flags] Flags: -b, --buffsize int buffer size (default 8192) -h, --help help for cat -L, --lines count lines instead of bytes -p, --print-freq int print frequency (-1 for print after parsing) (default 1) -s, --total int expected total bytes/lines (default -1) Examples Stream file, report progress in bytes csvtk cat file.tsv Stream file from stdin, report progress in lines tac input.tsv | csvtk cat -L -s `wc -l < input.tsv` - genautocomplete Usage generate shell autocompletion script Supported shell: bash|zsh|fish|powershell Bash: # generate completion shell csvtk genautocomplete --shell bash # configure if never did. # install bash-completion if the \"complete\" command is not found. echo \"for bcfile in ~/.bash_completion.d/* ; do source \\$bcfile; done\" >> ~/.bash_completion echo \"source ~/.bash_completion\" >> ~/.bashrc Zsh: # generate completion shell csvtk genautocomplete --shell zsh --file ~/.zfunc/_csvtk # configure if never did echo 'fpath=( ~/.zfunc \"${fpath[@]}\" )' >> ~/.zshrc echo \"autoload -U compinit; compinit\" >> ~/.zshrc fish: csvtk genautocomplete --shell fish --file ~/.config/fish/completions/csvtk.fish Usage: csvtk genautocomplete [flags] Flags: --file string autocompletion file (default \"/home/shenwei/.bash_completion.d/csvtk.sh\") -h, --help help for genautocomplete --shell string autocompletion type (bash|zsh|fish|powershell) (default \"bash\") /** * RECOMMENDED CONFIGURATION VARIABLES: EDIT AND UNCOMMENT THE SECTION BELOW TO INSERT DYNAMIC VALUES FROM YOUR PLATFORM OR CMS. * LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT: https://disqus.com/admin/universalcode/#configuration-variables */ /* var disqus_config = function () { this.page.url = PAGE_URL; // Replace PAGE_URL with your page's canonical URL variable this.page.identifier = PAGE_IDENTIFIER; // Replace PAGE_IDENTIFIER with your page's unique identifier variable }; */ (function() { // DON'T EDIT BELOW THIS LINE var d = document, s = d.createElement('script'); s.src = '//csvtk.disqus.com/embed.js'; s.setAttribute('data-timestamp', +new Date()); (d.head || d.body).appendChild(s); })(); Please enable JavaScript to view the comments powered by Disqus.","title":"Usage"},{"location":"usage/#usage-and-examples","text":"","title":"Usage and Examples"},{"location":"usage/#before-use","text":"Attention The CSV parser requires all the lines have same number of fields/columns. Even lines with spaces will cause error. Use '-I/--ignore-illegal-row' to skip these lines if neccessary. You can also use 'csvtk fix' to fix files with different numbers of columns in rows. By default, csvtk thinks your files have header row, if not, switch flag \"-H\" on. Column names better be unique. By default, lines starting with \"#\" will be ignored, if the header row starts with \"#\", please assign flag \"-C\" another rare symbol, e.g. '$'. By default, csvtk handles CSV files, use flag \"-t\" for tab-delimited files. If double quotes exist in fields, use flag \"-l\". Do not mix use field (column) numbers and names.","title":"Before use"},{"location":"usage/#table-of-contents","text":"csvtk Information headers dim/nrow/ncol summary corr watch Format conversion pretty transpose space2tab csv2md csv2json csv2xlsx xlsx2csv Set operations head concat sample cut grep uniq freq inter filter filter2 join split splitxlsx comb Edit fix add-header del-header rename rename2 replace round mutate mutate2 sep gather spread unfold fold Ordering sort Ploting plot plot hist plot box plot line Misc cat genautocomplete","title":"Table of Contents"},{"location":"usage/#csvtk","text":"Usage csvtk -- a cross-platform, efficient and practical CSV/TSV toolkit Version: 0.28.0 Author: Wei Shen : . e.g., \"-k name:u -L name:level.txt Examples data $ cat testdata/names.csv id,first_name,last_name,username 11,\"Rob\",\"Pike\",rob 2,Ken,Thompson,ken 4,\"Robert\",\"Griesemer\",\"gri\" 1,\"Robert\",\"Thompson\",\"abc\" NA,\"Robert\",\"Abel\",\"123\" By single column : csvtk sort -k 1 or csvtk sort -k last_name in alphabetical order $ cat testdata/names.csv \\ | csvtk sort -k first_name id,first_name,last_name,username 2,Ken,Thompson,ken 11,Rob,Pike,rob NA,Robert,Abel,123 1,Robert,Thompson,abc 4,Robert,Griesemer,gri in reversed alphabetical order ( key:r ) $ cat testdata/names.csv \\ | csvtk sort -k first_name:r id,first_name,last_name,username NA,Robert,Abel,123 1,Robert,Thompson,abc 4,Robert,Griesemer,gri 11,Rob,Pike,rob 2,Ken,Thompson,ken in numerical order ( key:n ) $ cat testdata/names.csv \\ | csvtk sort -k id:n id,first_name,last_name,username NA,Robert,Abel,123 1,Robert,Thompson,abc 2,Ken,Thompson,ken 4,Robert,Griesemer,gri 11,Rob,Pike,rob in natural order ( key:N ) $ cat testdata/names.csv | csvtk sort -k id:N id,first_name,last_name,username 1,Robert,Thompson,abc 2,Ken,Thompson,ken 4,Robert,Griesemer,gri 11,Rob,Pike,rob NA,Robert,Abel,123 in natural order ( key:N ), a bioinformatics example $ echo \"X,Y,1,10,2,M,11,1_c,Un_g,1_g\" | csvtk transpose X Y 1 10 2 M 11 1_c Un_g 1_g $ echo \"X,Y,1,10,2,M,11,1_c,Un_g,1_g\" \\ | csvtk transpose \\ | csvtk sort -H -k 1:N 1 1_c 1_g 2 10 11 M Un_g X Y By multiple columns: csvtk sort -k 1,2 or csvtk sort -k 1 -k 2 or csvtk sort -k last_name,age # by first_name and then last_name $ cat testdata/names.csv | csvtk sort -k first_name -k last_name id,first_name,last_name,username 2,Ken,Thompson,ken 11,Rob,Pike,rob NA,Robert,Abel,123 4,Robert,Griesemer,gri 1,Robert,Thompson,abc # by first_name and then ID $ cat testdata/names.csv | csvtk sort -k first_name -k id:n id,first_name,last_name,username 2,Ken,Thompson,ken 11,Rob,Pike,rob NA,Robert,Abel,123 1,Robert,Thompson,abc 4,Robert,Griesemer,gri By user-defined order # user-defined order/level $ cat testdata/size_level.txt tiny mini small medium big # original data $ cat testdata/size.csv id,size 1,Huge 2,Tiny 3,Big 4,Small 5,Medium $ csvtk sort -k 2:u -i -L 2:testdata/size_level.txt testdata/size.csv id,size 2,Tiny 4,Small 5,Medium 3,Big 1,Huge","title":"sort"},{"location":"usage/#plot","text":"Usage plot common figures Notes: 1. Output file can be set by flag -o/--out-file. 2. File format is determined by the out file suffix. Supported formats: eps, jpg|jpeg, pdf, png, svg, and tif|tiff 3. If flag -o/--out-file not set (default), image is written to stdout, you can display the image by pipping to \"display\" command of Imagemagic or just redirect to file. Usage: csvtk plot [command] Available Commands: box plot boxplot hist plot histogram line line plot and scatter plot Flags: --axis-width float axis width (default 1.5) -f, --data-field string column index or column name of data (default \"1\") --format string image format for stdout when flag -o/--out-file not given. available values: eps, jpg|jpeg, pdf, png, svg, and tif|tiff. (default \"png\") -g, --group-field string column index or column name of group --height float Figure height (default 4.5) -h, --help help for plot --label-size int label font size (default 14) --na-values strings NA values, case ignored (default [,NA,N/A]) --skip-na skip NA values in --na-values --tick-width float axis tick width (default 1.5) --title string Figure title --title-size int title font size (default 16) --width float Figure width (default 6) --x-max string maximum value of X axis --x-min string minimum value of X axis --xlab string x label text --y-max string maximum value of Y axis --y-min string minimum value of Y axis --ylab string y label text Note that most of the flags of plot are global flags of the subcommands hist , box and line Notes of image output Output file can be set by flag -o/--out-file. File format is determined by the out file suffix. Supported formats: eps, jpg|jpeg, pdf, png, svg, and tif|tiff If flag -o/--out-file not set (default), image is written to stdout, you can display the image by pipping to display command of Imagemagic or just redirect to file.","title":"plot"},{"location":"usage/#plot-hist","text":"Usage plot histogram Notes: 1. Output file can be set by flag -o/--out-file. 2. File format is determined by the out file suffix. Supported formats: eps, jpg|jpeg, pdf, png, svg, and tif|tiff 3. If flag -o/--out-file not set (default), image is written to stdout, you can display the image by pipping to \"display\" command of Imagemagic or just redirect to file. Usage: csvtk plot hist [flags] Flags: --bins int number of bins (default 50) --color-index int color index, 1-7 (default 1) Examples example data $ zcat testdata/grouped_data.tsv.gz | head -n 5 | csvtk -t pretty Group Length GC Content Group A 97 57.73 Group A 95 49.47 Group A 97 49.48 Group A 100 51.00 plot histogram with data of the second column: $ csvtk -t plot hist testdata/grouped_data.tsv.gz -f 2 \\ --title Histogram -o histogram.png You can also write image to stdout and pipe to \"display\" command of Imagemagic: $ csvtk -t plot hist testdata/grouped_data.tsv.gz -f 2 | display","title":"plot hist"},{"location":"usage/#plot-box","text":"Usage plot boxplot Notes: 1. Output file can be set by flag -o/--out-file. 2. File format is determined by the out file suffix. Supported formats: eps, jpg|jpeg, pdf, png, svg, and tif|tiff 3. If flag -o/--out-file not set (default), image is written to stdout, you can display the image by pipping to \"display\" command of Imagemagic or just redirect to file. Usage: csvtk plot box [flags] Flags: --box-width float box width --horiz horize box plot Examples plot boxplot with data of the \"GC Content\" (third) column, group information is the \"Group\" column. csvtk -t plot box testdata/grouped_data.tsv.gz -g \"Group\" -f \"GC Content\" \\ --width 3 --title \"Box plot\" \\ > boxplot.png plot horiz boxplot with data of the \"Length\" (second) column, group information is the \"Group\" column. $ csvtk -t plot box testdata/grouped_data.tsv.gz -g \"Group\" -f \"Length\" \\ --height 3 --width 5 --horiz --title \"Horiz box plot\" \\ > boxplot2.png`","title":"plot box"},{"location":"usage/#plot-line","text":"Usage line plot and scatter plot Notes: 1. Output file can be set by flag -o/--out-file. 2. File format is determined by the out file suffix. Supported formats: eps, jpg|jpeg, pdf, png, svg, and tif|tiff 3. If flag -o/--out-file not set (default), image is written to stdout, you can display the image by pipping to \"display\" command of Imagemagic or just redirect to file. Usage: csvtk plot line [flags] Flags: -x, --data-field-x string column index or column name of X for command line -y, --data-field-y string column index or column name of Y for command line --legend-left locate legend along the left edge of the plot --legend-top locate legend along the top edge of the plot --line-width float line width (default 1.5) --point-size float point size (default 3) --scatter only plot points Examples example data $ head -n 5 testdata/xy.tsv Group X Y A 0 1 A 1 1.3 A 1.5 1.5 A 2.0 2 plot line plot with X-Y data $ csvtk -t plot line testdata/xy.tsv -x X -y Y -g Group \\ --title \"Line plot\" \\ > lineplot.png plot scatter $ csvtk -t plot line testdata/xy.tsv -x X -y Y -g Group \\ --title \"Scatter\" --scatter \\ > lineplot.png","title":"plot line"},{"location":"usage/#cat","text":"Usage stream file to stdout and report progress on stderr Usage: csvtk cat [flags] Flags: -b, --buffsize int buffer size (default 8192) -h, --help help for cat -L, --lines count lines instead of bytes -p, --print-freq int print frequency (-1 for print after parsing) (default 1) -s, --total int expected total bytes/lines (default -1) Examples Stream file, report progress in bytes csvtk cat file.tsv Stream file from stdin, report progress in lines tac input.tsv | csvtk cat -L -s `wc -l < input.tsv` -","title":"cat"},{"location":"usage/#genautocomplete","text":"Usage generate shell autocompletion script Supported shell: bash|zsh|fish|powershell Bash: # generate completion shell csvtk genautocomplete --shell bash # configure if never did. # install bash-completion if the \"complete\" command is not found. echo \"for bcfile in ~/.bash_completion.d/* ; do source \\$bcfile; done\" >> ~/.bash_completion echo \"source ~/.bash_completion\" >> ~/.bashrc Zsh: # generate completion shell csvtk genautocomplete --shell zsh --file ~/.zfunc/_csvtk # configure if never did echo 'fpath=( ~/.zfunc \"${fpath[@]}\" )' >> ~/.zshrc echo \"autoload -U compinit; compinit\" >> ~/.zshrc fish: csvtk genautocomplete --shell fish --file ~/.config/fish/completions/csvtk.fish Usage: csvtk genautocomplete [flags] Flags: --file string autocompletion file (default \"/home/shenwei/.bash_completion.d/csvtk.sh\") -h, --help help for genautocomplete --shell string autocompletion type (bash|zsh|fish|powershell) (default \"bash\") /** * RECOMMENDED CONFIGURATION VARIABLES: EDIT AND UNCOMMENT THE SECTION BELOW TO INSERT DYNAMIC VALUES FROM YOUR PLATFORM OR CMS. * LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT: https://disqus.com/admin/universalcode/#configuration-variables */ /* var disqus_config = function () { this.page.url = PAGE_URL; // Replace PAGE_URL with your page's canonical URL variable this.page.identifier = PAGE_IDENTIFIER; // Replace PAGE_IDENTIFIER with your page's unique identifier variable }; */ (function() { // DON'T EDIT BELOW THIS LINE var d = document, s = d.createElement('script'); s.src = '//csvtk.disqus.com/embed.js'; s.setAttribute('data-timestamp', +new Date()); (d.head || d.body).appendChild(s); })(); Please enable JavaScript to view the comments powered by Disqus.","title":"genautocomplete"}]} \ No newline at end of file diff --git a/sitemap.xml b/sitemap.xml index 9e3a073..96fb126 100644 --- a/sitemap.xml +++ b/sitemap.xml @@ -2,32 +2,32 @@ None -2023-08-23 +2023-09-22 daily None -2023-08-23 +2023-09-22 daily None -2023-08-23 +2023-09-22 daily None -2023-08-23 +2023-09-22 daily None -2023-08-23 +2023-09-22 daily None -2023-08-23 +2023-09-22 daily csvtk
Usage
csvtk -- a cross-platform, efficient and practical CSV/TSV toolkit -Version: 0.27.2 +Version: 0.28.0 Author: Wei Shen <shenwei356@gmail.com> @@ -1286,7 +1286,7 @@csvtk
-E, --ignore-empty-row ignore empty rows -I, --ignore-illegal-row ignore illegal rows. You can also use 'csvtk fix' to fix files with different numbers of columns in rows - --infile-list string file of input files list (one file per line), if given, they are appended + -X, --infile-list string file of input files list (one file per line), if given, they are appended to files from cli arguments -l, --lazy-quotes if given, a quote may appear in an unquoted field and a non-doubled quote may appear in a quoted field @@ -1666,7 +1666,19 @@pretty
Usually, the text is wrapped in space (-x/--wrap-delimiter). But if one word is longer than the -W/--max-width, it will be force split. 1b. Texts are aligned left (default), center (-m/--align-center) - or right (-r/--align-right). + or right (-r/--align-right). Users can specify columns with column names, + field indexes or ranges. + Examples: + -m A,B # column A and B + -m 1,2 # 1st and 2nd column + -m -1 # the last column (it's not unselecting in other commands) + -m 1,3-5 # 1st, from 3rd to 5th column + -m 1- # 1st and later columns (all columns) + -m -3- # the last 3 columns + -m -3--2 # the 2nd and 3rd to last columns + -m 1- -r -1 # all columns are center-aligned, except the last column + # which is right-aligned. -r overides -m. + 2. Remaining rows are read and immediately outputted, one by one, till the end. Styles: @@ -1745,11 +1757,13 @@pretty
╚════╩══════╝ Usage: - csvtk pretty [flags] + csvtk pretty [flags] Flags: - -m, --align-center strings align center/middle for selected columns (field index or column name) - -r, --align-right strings align right for selected columns (field index or column name) + -m, --align-center strings align right for selected columns (field index/range or column name, type + "csvtk pretty -h" for examples) + -r, --align-right strings align right for selected columns (field index/range or column name, type + "csvtk pretty -h" for examples) -n, --buf-rows int the number of rows to determine the min and max widths (default 128) --clip clip longer cell instead of wrapping --clip-mark string clip mark (default "...") @@ -1810,7 +1824,7 @@pretty
-
align right/center for some columns
-$ csvtk pretty testdata/names.csv -r 1,username -m first_name -w 6 -S bold +$ csvtk pretty testdata/names.csv -w 6 -S bold -r 1,username -m first_name ┏━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┓ ┃ id ┃ first_name ┃ last_name ┃ username ┃ ┣━━━━━━━━╋━━━━━━━━━━━━╋━━━━━━━━━━━╋━━━━━━━━━━┫ @@ -1824,6 +1838,21 @@pretty
┣━━━━━━━━╋━━━━━━━━━━━━╋━━━━━━━━━━━╋━━━━━━━━━━┫ ┃ NA ┃ Robert ┃ Abel ┃ 123 ┃ ┗━━━━━━━━┻━━━━━━━━━━━━┻━━━━━━━━━━━┻━━━━━━━━━━┛ + +$ csvtk pretty testdata/names.csv -w 6 -S bold -m 1- -r -1 +┏━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┓ +┃ id ┃ first_name ┃ last_name ┃ username ┃ +┣━━━━━━━━╋━━━━━━━━━━━━╋━━━━━━━━━━━╋━━━━━━━━━━┫ +┃ 11 ┃ Rob ┃ Pike ┃ rob ┃ +┣━━━━━━━━╋━━━━━━━━━━━━╋━━━━━━━━━━━╋━━━━━━━━━━┫ +┃ 2 ┃ Ken ┃ Thompson ┃ ken ┃ +┣━━━━━━━━╋━━━━━━━━━━━━╋━━━━━━━━━━━╋━━━━━━━━━━┫ +┃ 4 ┃ Robert ┃ Griesemer ┃ gri ┃ +┣━━━━━━━━╋━━━━━━━━━━━━╋━━━━━━━━━━━╋━━━━━━━━━━┫ +┃ 1 ┃ Robert ┃ Thompson ┃ abc ┃ +┣━━━━━━━━╋━━━━━━━━━━━━╋━━━━━━━━━━━╋━━━━━━━━━━┫ +┃ NA ┃ Robert ┃ Abel ┃ 123 ┃ +┗━━━━━━━━┻━━━━━━━━━━━━┻━━━━━━━━━━━┻━━━━━━━━━━┛ -
@@ -4489,10 +4518,11 @@
spread
spread, wider Flags: - -h, --help help for spread - -k, --key string field of keys. e.g -f 1,2 or -f columnA,columnB, or -f -columnA for unselect columnA - --na string content for filling NA data - -v, --value string field of values. e.g -f 1,2 or -f columnA,columnB, or -f -columnA for unselect columnA + -h, --help help for spread + -k, --key string field of the key. e.g -k 1 or -k columnA + --na string content for filling NA data + -s, --separater string separater for values that share the same key (default "; ") + -v, --value string field of the value. e.g -v 1 or -v columnAExamples:
- fix the error of