Skip to content

Commit

Permalink
Quick Save
Browse files Browse the repository at this point in the history
  • Loading branch information
@rsdoiel
rsdoiel committed Dec 8, 2017
1 parent 24bb860 commit 33ea0f4
Show file tree
Hide file tree
Showing 15 changed files with 408 additions and 108 deletions.
44 changes: 22 additions & 22 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,20 +8,20 @@ Posix shell scripting (e.g. writing scripts that run under Bash). The tools are

Command line utilities for simplifying work with CSV, JSON, Excel Workbooks and plain text files or content.

+ [csv2json](docs/csv2json.html) - a tool to take a CSV file and convert it into a JSON array or a list of JSON blobs one per line
+ [csv2mdtable](docs/csv2mdtable.html) - a tool to render CSV as a Github Flavored Markdown table
+ [csv2xlsx](docs/csv2xlsx.html) - a tool to take a CSV file and add it as a sheet to a Excel Workbook
+ [csvcleaner](docs/csvcleaner.html) - normalize a CSV file by column and row including trimming spaces and removing comments
+ [csvcols](docs/csvcols.html) - a tool for formatting command line arguments into CSV row of columns or filtering CSV rows for specific columns
+ [csvfind](docs/csvfind.html) - a tool for filtering a CSV file rows by column
+ [csvjoin](docs/csvjoin.html) - a tool to join two CSV files on common values in designated columns, writes combined CSV rows
+ [csvrows](docs/csvrows.html) - a tool for formatting command line arguments into CSV columns of rows or filtering CSV columns for specific rows
+ [jsoncols](docs/jsoncols.html) - a tool for exploring and extracting JSON values into columns
+ [jsonjoin](docs/jsonjoin.html) - a tool for joining JSON object documents
+ [jsonmunge](docs/jsonmunge.html) - a tool to transform JSON documents into something else
+ [jsonrange](docs/jsonrange.html) - a tool for iterating over JSON objects and arrays (return keys or values)
+ [xlsx2csv](docs/xlsx2csv.html) - a tool for converting Excel Workbooks sheets to CSV files
+ [xlsx2json](docs/xlsx2json.html) - a tool for converting Excel Workbooks to JSON files
+ [csv2json](docs/csv2json/) - a tool to take a CSV file and convert it into a JSON array or a list of JSON blobs one per line
+ [csv2mdtable](docs/csv2mdtable/) - a tool to render CSV as a Github Flavored Markdown table
+ [csv2xlsx](docs/csv2xlsx/) - a tool to take a CSV file and add it as a sheet to a Excel Workbook
+ [csvcleaner](docs/csvcleaner/) - normalize a CSV file by column and row including trimming spaces and removing comments
+ [csvcols](docs/csvcols/) - a tool for formatting command line arguments into CSV row of columns or filtering CSV rows for specific columns
+ [csvfind](docs/csvfind/) - a tool for filtering a CSV file rows by column
+ [csvjoin](docs/csvjoin/) - a tool to join two CSV files on common values in designated columns, writes combined CSV rows
+ [csvrows](docs/csvrows/) - a tool for formatting command line arguments into CSV columns of rows or filtering CSV columns for specific rows
+ [jsoncols](docs/jsoncols/) - a tool for exploring and extracting JSON values into columns
+ [jsonjoin](docs/jsonjoin/) - a tool for joining JSON object documents
+ [jsonmunge](docs/jsonmunge/) - a tool to transform JSON documents into something else
+ [jsonrange](docs/jsonrange/) - a tool for iterating over JSON objects and arrays (return keys or values)
+ [xlsx2csv](docs/xlsx2csv/) - a tool for converting Excel Workbooks sheets to CSV files
+ [xlsx2json](docs/xlsx2json/) - a tool for converting Excel Workbooks to JSON files


Compiled versions are provided for Linux (amd64), Mac OS X (amd64),
Expand Down Expand Up @@ -58,13 +58,13 @@ See [string](docs/string/) for full details

Various utilities for simplifying work on the command line.

+ [findfile](docs/findfile.html) - find files based on prefix, suffix or contained string
+ [finddir](docs/finddir.html) - find directories based on prefix, suffix or contained string
+ [mergepath](docs/mergepath.html) - prefix, append, clip path variables
+ [range](docs/range.html) - emit a range of integers (useful for numbered loops in Bash)
+ [reldate](docs/reldate.html) - display a relative date in YYYY-MM-DD format
+ [timefmt](docs/timefmt.html) - format a time value based on Golang's time format language
+ [urlparse](docs/urlparse.html) - split a URL into parts
+ [findfile](docs/findfile/) - find files based on prefix, suffix or contained string
+ [finddir](docs/finddir/) - find directories based on prefix, suffix or contained string
+ [mergepath](docs/mergepath/) - prefix, append, clip path variables
+ [range](docs/range/) - emit a range of integers (useful for numbered loops in Bash)
+ [reldate](docs/reldate/) - display a relative date in YYYY-MM-DD format
+ [timefmt](docs/timefmt/) - format a time value based on Golang's time format language
+ [urlparse](docs/urlparse/) - split a URL into parts

Compiled versions are provided for Linux (amd64), Mac OS X (amd64),
Windows 10 (amd64) and Raspbian (ARM7). See https://github.com/caltechlibrary/datatools/releases.
Expand All @@ -74,7 +74,7 @@ Use the utilities try "-help" option for a full list of options.

## Installation

See [INSTALL.md](install.html) for details for installing pre-compiled versions of the programs.
See [INSTALL.md](install/) for details for installing pre-compiled versions of the programs.

_datatools_ are go get-able. If you have go v1.8 (or newer) you can install with the command below.

Expand Down
47 changes: 43 additions & 4 deletions cmds/string/string.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,26 @@ string is a command line tool for transforming strings in common ways.
`

examples = `
Convert text to upper case
string toupper "one"
Convert text to lower case
string tolower "ONE"
Captialize an English phrase
string englishtitle "one more thing to know"
Split a space newline delimited list of words into a JSON array
string -i wordlist.txt split "\n"
Join a JSON array of strings into a newline delimited list
string join '\n' '["one","two","three","four","five"]'
`

// Standard Options
Expand All @@ -60,6 +80,8 @@ string is a command line tool for transforming strings in common ways.
eol string

// App Options
delimiter string
outputDelimiter string
)

//
Expand Down Expand Up @@ -161,7 +183,7 @@ func doSplit(in io.Reader, out io.Writer, eout io.Writer, args []string) int {
fmt.Fprintln(eout, "first parameter is the delimiting string")
return 1
}
delimiter := args[0]
delimiter := datatools.NormalizeDelimiter(args[0])
args = args[1:]
// Handle the case where out input is piped in or read from a file.
if inputFName != "" {
Expand All @@ -185,7 +207,7 @@ func doSplitN(in io.Reader, out io.Writer, eout io.Writer, args []string) int {
fmt.Fprintln(eout, "first parameter is the delimiting string, second is the count")
return 1
}
delimiter := args[0]
delimiter := datatools.NormalizeDelimiter(args[0])
// Now convert to cnt an integer
cnt, err := strconv.Atoi(args[1])
if err != nil {
Expand Down Expand Up @@ -216,7 +238,7 @@ func doJoin(in io.Reader, out io.Writer, eout io.Writer, args []string) int {
fmt.Fprintln(eout, "first parameter is the delimiter to join with")
return 1
}
delimiter := args[0]
delimiter := datatools.NormalizeDelimiter(args[0])
args = args[1:]

// Handle the case where out input is piped in or read from a file.
Expand Down Expand Up @@ -388,6 +410,20 @@ func doTrimRight(in io.Reader, out io.Writer, eout io.Writer, args []string) int
return 0
}

func doTrimSpace(in io.Reader, out io.Writer, eout io.Writer, args []string) int {
// Handle content coming from file
if inputFName != "" {
src, err := ioutil.ReadAll(in)
exitOnError(eout, err, quiet)
args = append(args, string(src))
}
// Handle content common from args
for _, arg := range args {
fmt.Fprintf(out, "%s%s", strings.TrimSpace(arg), eol)
}
return 0
}

func doContains(in io.Reader, out io.Writer, eout io.Writer, args []string) int {
if len(args) < 1 {
fmt.Fprintf(eout, "first parameter is the target string\n")
Expand Down Expand Up @@ -585,6 +621,8 @@ func main() {
app.BoolVar(&generateMarkdownDocs, "generate-markdown-docs", false, "output documentation in Markdown")

// App Options
app.StringVar(&delimiter, "d,delimiter", "", "set the delimiter")
app.StringVar(&outputDelimiter, "do,output-delimiter", "", "set the output delimiter")

// Add verbs and functions
app.AddAction("toupper", doToUpper, "to upper case: [STRING]")
Expand All @@ -601,6 +639,7 @@ func main() {
app.AddAction("trim", doTrim, "trim (beginning and end), CUTSET [STRING]")
app.AddAction("trimleft", doTrimLeft, "left trim: CUTSET [STRING]")
app.AddAction("trimright", doTrimRight, "right trim: CUTSET [STRING]")
app.AddAction("trimspace", doTrimSpace, "trim leading and trailing spaces: [STRING]")
app.AddAction("count", doCount, "count substrings: SUBSTRING [STRING]")
app.AddAction("contains", doContains, "has substrings: SUBSTRING [STRING]")
app.AddAction("length", doLength, "length of string: [STRING]")
Expand All @@ -623,7 +662,7 @@ func main() {
cli.ExitOnError(app.Eout, err, quiet)
defer cli.CloseFile(inputFName, app.In)

app.Out, err = cli.Create(inputFName, os.Stdout)
app.Out, err = cli.Create(outputFName, os.Stdout)
cli.ExitOnError(app.Eout, err, quiet)
defer cli.CloseFile(outputFName, app.Out)

Expand Down
26 changes: 13 additions & 13 deletions docs/csvcols/index.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,19 +35,19 @@ <h2>SYNOPSIS</h2>

<h2>OPTIONS</h2>

<pre><code> -col, -cols output specified columns (e.g. -col 1,12:14,2,4))
-d, -delimiter set the input delimiter character
-examples display example
-generate-markdown-docs generate markdown documentation
-h, -help display help
-i, -input input filename
-l, -license display license
-o, -output output filename
-od, -output-delimiter set the output delimiter character
-quiet suppress error messages
-skip-header-row skip the header row
-uuid add a prefix row with generated UUID cell
-v, -version display version
<pre><code> -col, -cols output specified columns (e.g. -col 1,12:14,2,4))
-d, -delimiter set the input delimiter character
-examples display example
-generate-markdown-docs generate markdown documentation
-h, -help display help
-i, -input input filename
-l, -license display license
-o, -output output filename
-od, -output-delimiter set the output delimiter character
-quiet suppress error messages
-skip-header-row skip the header row
-uuid add a prefix row with generated UUID cell
-v, -version display version
</code></pre>

<h2>EXAMPLES</h2>
Expand Down
26 changes: 13 additions & 13 deletions docs/csvcols/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,19 +14,19 @@ listed on the commandline (first column is 1 not 0).
## OPTIONS

```
-col, -cols output specified columns (e.g. -col 1,12:14,2,4))
-d, -delimiter set the input delimiter character
-examples display example
-generate-markdown-docs generate markdown documentation
-h, -help display help
-i, -input input filename
-l, -license display license
-o, -output output filename
-od, -output-delimiter set the output delimiter character
-quiet suppress error messages
-skip-header-row skip the header row
-uuid add a prefix row with generated UUID cell
-v, -version display version
-col, -cols output specified columns (e.g. -col 1,12:14,2,4))
-d, -delimiter set the input delimiter character
-examples display example
-generate-markdown-docs generate markdown documentation
-h, -help display help
-i, -input input filename
-l, -license display license
-o, -output output filename
-od, -output-delimiter set the output delimiter character
-quiet suppress error messages
-skip-header-row skip the header row
-uuid add a prefix row with generated UUID cell
-v, -version display version
```


Expand Down
48 changes: 38 additions & 10 deletions docs/string/index.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,15 +44,17 @@ <h2>OPTIONS</h2>

<p>Options are shared between all actions and must precede the action on the command line.</p>

<pre><code> -e, -examples display examples
-generate-markdown-docs output documentation in Markdown
-h, -help display help
-i, -input input file name
-l, -license display license
-nl, -newline if true add a trailing newline
-o, -output output file name
-quiet suppress error messages
-v, -version display version
<pre><code> -d, -delimiter set the delimiter
-do, -output-delimiter set the output delimiter
-e, -examples display examples
-generate-markdown-docs output documentation in Markdown
-h, -help display help
-i, -input input file name
-l, -license display license
-nl, -newline if true add a trailing newline
-o, -output output file name
-quiet suppress error messages
-v, -version display version
</code></pre>

<h2>ACTIONS</h2>
Expand All @@ -79,12 +81,38 @@ <h2>ACTIONS</h2>
trimleft left trim: CUTSET [STRING]
trimprefix trims prefix: PREFIX [STRING]
trimright right trim: CUTSET [STRING]
trimspace trim leading and trailing spaces: [STRING]
trimsuffix trim suffix: SUFFIX [STRING]
</code></pre>

<h2>EXAMPLES</h2>

<p>Related: <a href="contains.html">contains</a>, <a href="count.html">count</a>, <a href="englishtitle.html">englishtitle</a>, <a href="hasprefix.html">hasprefix</a>, <a href="hassuffix.html">hassuffix</a>, <a href="join.html">join</a>, <a href="length.html">length</a>, <a href="padleft.html">padleft</a>, <a href="padright.html">padright</a>, <a href="position.html">position</a>, <a href="replace.html">replace</a>, <a href="replacen.html">replacen</a>, <a href="slice.html">slice</a>, <a href="split.html">split</a>, <a href="splitn.html">splitn</a>, <a href="tolower.html">tolower</a>, <a href="totitle.html">totitle</a>, <a href="toupper.html">toupper</a>, <a href="trim.html">trim</a>, <a href="trimleft.html">trimleft</a>, <a href="trimprefix.html">trimprefix</a>, <a href="trimright.html">trimright</a>, <a href="trimsuffix.html">trimsuffix</a></p>
<p>Convert text to upper case</p>

<pre><code>string toupper &quot;one&quot;
</code></pre>

<p>Convert text to lower case</p>

<pre><code>string tolower &quot;ONE&quot;
</code></pre>

<p>Captialize an English phrase</p>

<pre><code>string englishtitle &quot;one more thing to know&quot;
</code></pre>

<p>Split a space newline delimited list of words into a JSON array</p>

<pre><code>string -i wordlist.txt split &quot;\n&quot;
</code></pre>

<p>Join a JSON array of strings into a newline delimited list</p>

<pre><code>string join '\n' '[&quot;one&quot;,&quot;two&quot;,&quot;three&quot;,&quot;four&quot;,&quot;five&quot;]'
</code></pre>

<p>Related: <a href="contains.html">contains</a>, <a href="count.html">count</a>, <a href="englishtitle.html">englishtitle</a>, <a href="hasprefix.html">hasprefix</a>, <a href="hassuffix.html">hassuffix</a>, <a href="join.html">join</a>, <a href="length.html">length</a>, <a href="padleft.html">padleft</a>, <a href="padright.html">padright</a>, <a href="position.html">position</a>, <a href="replace.html">replace</a>, <a href="replacen.html">replacen</a>, <a href="slice.html">slice</a>, <a href="split.html">split</a>, <a href="splitn.html">splitn</a>, <a href="tolower.html">tolower</a>, <a href="totitle.html">totitle</a>, <a href="toupper.html">toupper</a>, <a href="trim.html">trim</a>, <a href="trimleft.html">trimleft</a>, <a href="trimprefix.html">trimprefix</a>, <a href="trimright.html">trimright</a>, <a href="trimspace.html">trimspace</a>, <a href="trimsuffix.html">trimsuffix</a></p>

<p>string v0.0.20-pre</p>

Expand Down
43 changes: 33 additions & 10 deletions docs/string/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,15 +21,17 @@ string is a command line tool for transforming strings in common ways.
Options are shared between all actions and must precede the action on the command line.

```
-e, -examples display examples
-generate-markdown-docs output documentation in Markdown
-h, -help display help
-i, -input input file name
-l, -license display license
-nl, -newline if true add a trailing newline
-o, -output output file name
-quiet suppress error messages
-v, -version display version
-d, -delimiter set the delimiter
-do, -output-delimiter set the output delimiter
-e, -examples display examples
-generate-markdown-docs output documentation in Markdown
-h, -help display help
-i, -input input file name
-l, -license display license
-nl, -newline if true add a trailing newline
-o, -output output file name
-quiet suppress error messages
-v, -version display version
```


Expand Down Expand Up @@ -58,15 +60,36 @@ Options are shared between all actions and must precede the action on the comman
trimleft left trim: CUTSET [STRING]
trimprefix trims prefix: PREFIX [STRING]
trimright right trim: CUTSET [STRING]
trimspace trim leading and trailing spaces: [STRING]
trimsuffix trim suffix: SUFFIX [STRING]
```


## EXAMPLES


Convert text to upper case

string toupper "one"

Related: [contains](contains.html), [count](count.html), [englishtitle](englishtitle.html), [hasprefix](hasprefix.html), [hassuffix](hassuffix.html), [join](join.html), [length](length.html), [padleft](padleft.html), [padright](padright.html), [position](position.html), [replace](replace.html), [replacen](replacen.html), [slice](slice.html), [split](split.html), [splitn](splitn.html), [tolower](tolower.html), [totitle](totitle.html), [toupper](toupper.html), [trim](trim.html), [trimleft](trimleft.html), [trimprefix](trimprefix.html), [trimright](trimright.html), [trimsuffix](trimsuffix.html)
Convert text to lower case

string tolower "ONE"

Captialize an English phrase

string englishtitle "one more thing to know"

Split a space newline delimited list of words into a JSON array

string -i wordlist.txt split "\n"

Join a JSON array of strings into a newline delimited list

string join '\n' '["one","two","three","four","five"]'



Related: [contains](contains.html), [count](count.html), [englishtitle](englishtitle.html), [hasprefix](hasprefix.html), [hassuffix](hassuffix.html), [join](join.html), [length](length.html), [padleft](padleft.html), [padright](padright.html), [position](position.html), [replace](replace.html), [replacen](replacen.html), [slice](slice.html), [split](split.html), [splitn](splitn.html), [tolower](tolower.html), [totitle](totitle.html), [toupper](toupper.html), [trim](trim.html), [trimleft](trimleft.html), [trimprefix](trimprefix.html), [trimright](trimright.html), [trimspace](trimspace.html), [trimsuffix](trimsuffix.html)

string v0.0.20-pre
Loading

0 comments on commit 33ea0f4

Please sign in to comment.