Merge #36

36: Make validation of json input and output optionnal r=bidoubiwa a=bidoubiwa Validation requires to read a second time every file. If the user desire to guarantee their validity two options are introduced: ``` -I, --validate-input <file-name> Check if input JSON files are valid (default: true) -O, --validate-output <file-name> Check if output JSON is a valid JSON (default: false) ``` Co-authored-by: Charlotte Vermandel <[email protected]>
bidoubiwa · Aug 20, 2021 · dad9fd4 · dad9fd4
2 parents 0e9d18a + 2685fa5
commit dad9fd4
Show file tree

Hide file tree

Showing 7 changed files with 280 additions and 98 deletions.
diff --git a/README.MD b/README.MD
@@ -2,7 +2,7 @@
 
 <p align="center">
   <img src="assets/boat.png" width=300 />
-  
+
   <!-- <a href="https://www.npmjs.com/package/meilisearch"><img src="https://img.shields.io/npm/v/meilisearch.svg" alt="npm version"></a> -->
   <a href="https://github.com/bidoubiwa/turbo-json.js/actions"><img src="https://github.com/bidoubiwa/turbo-json.js/workflows/Tests/badge.svg" alt="Tests"></a>
   <a href="https://github.com/prettier/prettier"><img src="https://img.shields.io/badge/styled_with-prettier-ff69b4.svg" alt="Prettier"></a>
@@ -14,7 +14,7 @@
 turbo-json.js is a tool that combines all json files found in a directory into one big json file using **streaming** to avoid out-of-memory.
 
 ```bash
-npx turbo-json data/ 
+npx turbo-json data/
 # Outputs `combined.json` file containing the content of all json files found in the data/ directory
 ```
 
@@ -24,10 +24,82 @@ Both `read` and `write` actions are done using `streaming`. The maximum data sto
 
 ## Table of Contents
 
-- [How is it combined](#how-is-it-combined)
 - [Installation](#installation)
 - [Usage](#usage)
-- [Try it out](#try-it-out)
+- [How is it combined](#how-is-it-combined)
+- [Validate JSON format](#validate-json-format)
+- [Example](#example)
+
+## Installation
+
+### CLI:
+
+Global install to have the CLI accessible from everywhere on your operating system.
+```bash
+npm install -g turbo-json.js # install globaly
+```
+
+No installation needed when using `npx`.
+```bash
+npx turbo-json.js [options]
+```
+
+### Library:
+```bash
+# yarn
+yarn add turbo-json.js
+
+# npm
+npm install turbo-json.js
+```
+
+## Usage
+
+turbo-json.js takes one argument:
+
+- Input directory: the path to the directory containing all the jsons.
+
+and options:
+- Output file _optional_ : path to the output file (default: `combined.json`).
+- Validate Input JSON's: All input JSON files are validated to be a correct [JSON](https://www.json.org/json-en.html) (default: `true`).
+- Validate Output JSON's: Validate if the outputed JSON is a correct [JSON](https://www.json.org/json-en.html) (default: `false`).
+
+It accepts relative path but also fully qualified paths.
+
+### CLI usage:
+
+Usage:
+```bash
+cli [options] <input-directory>
+```
+
+Options:
+```
+  -o, --output-file <file-name>      File name in which all the json files will be merged (default: "combined.json")
+  -I, --validate-input <file-name>   Check if input JSON files are valid (default: true)
+  -O, --validate-output <file-name>  Check if output JSON is a valid JSON (default: false)
+```
+
+```bash
+turbo-json <input-directory> --output-file <file-path> (default: "combined.json")
+```
+
+**Example**
+```bash
+turbo-json /data -o combined_data.json
+```
+
+### Library usage
+
+
+```js
+const { combineJson } = require('./src');
+
+(async () => {
+  await combineJson('misc', { outputFile: 'combined_data.json', validateInput: true, validateOutput: false });
+})();
+```
+
 
 ## How is it combined
 
@@ -82,68 +154,19 @@ Output file:
 ```
 
 
-## Installation
+## Validate Json Format
 
-### CLI:
+By default the tool supposes that your JSON files are correctly formated. If you'd like to check if it is well formated before concatenation the `validateInput` options should be set to true.
+By doing so, only the well formated JSON's are concatenated but this comes at the price that every file will be stream-read two times. Once for validation and once for writing.
 
-Global install to have the CLI accessible from everywhere on your operating system.
-```bash
-npm install -g turbo-json.js # install globaly
-```
-
-No installation needed when using `npx`.
-```bash
-npx turbo-json.js [options]
-```
-
-### Library:
-```bash
-# yarn
-yarn add turbo-json.js
-
-# npm
-npm install turbo-json.js
-```
+The same is possible to validate the output file using `validateOutput`.
 
+Validity is based on the description provided by [json.org](https://www.json.org/json-en.html).
 
-## Usage
-
-turbo-json.js takes one argument:
-
-- Input directory: the path to the directory containing all the jsons.
-
-and options:
-- Output file _optional_ : path to the output file.
-
-It accepts relative path but also fully qualified paths.
-
-### CLI usage:
-
-```bash
-turbo-json <input-directory> --output-file <file-path> (default: "combined.json")
-```
-
-**Example**
-```bash
-turbo-json /data -o combined_data.json
-```
-
-### Library usage
-
-```js
-const { combineJson } = require('./src');
-
-(async () => {
-  await combineJson({ inputDir: 'misc', outputFile: 'combined_data.json' });
-})();
-```
-
-### Example
+## Example
 
 Using the JSON files present in `misc`, you can observe the outputed file [misc_output.json](./misc_output.json)
 
-## Try it out
-
 At the root of the repository use the following:
 
 ```bash

diff --git a/__tests__/combine.tests.js b/__tests__/combine.tests.js
@@ -11,9 +11,23 @@ beforeAll(() => {
 })
 
 test('Tests on 1 empty file', async () => {
-  const res = await combineJson({
-    inputDir: 'misc/one_empty',
+  const res = await combineJson('misc/one_empty', {
     outputFile: 'test-output/combine_empty.json',
+    validateInput: true,
+  })
+
+  const data = JSON.parse(
+    fs.readFileSync(`${process.cwd()}/test-output/combine_empty.json`, 'utf-8')
+  )
+  const expected = []
+  expect(res).toBe(1)
+  expect(data).toEqual(expected)
+})
+
+test('Tests on 1 empty file', async () => {
+  const res = await combineJson('misc/one_empty', {
+    outputFile: 'test-output/combine_empty.json',
+    validateInput: false,
   })
 
   const data = JSON.parse(
@@ -25,9 +39,9 @@ test('Tests on 1 empty file', async () => {
 })
 
 test('Tests on multiple empty files', async () => {
-  const res = await combineJson({
-    inputDir: 'misc/multiple_empty',
+  const res = await combineJson('misc/multiple_empty', {
     outputFile: 'test-output/combine_multiple_empty.json',
+    validateInput: true,
   })
   const data = JSON.parse(
     fs.readFileSync(
@@ -40,25 +54,98 @@ test('Tests on multiple empty files', async () => {
   expect(data).toEqual(expected)
 })
 
+test('Tests on some empty files and some valid', async () => {
+  const res = await combineJson('misc/some_empty', {
+    outputFile: 'test-output/combine_some_empty.json',
+    validateInput: true,
+  })
+  const data = JSON.parse(
+    fs.readFileSync(
+      `${process.cwd()}/test-output/combine_some_empty.json`,
+      'utf-8'
+    )
+  )
+  const expected = [
+    {
+      name: 'far away',
+    },
+  ]
+  expect(res).toBe(1)
+  expect(data).toEqual(expected)
+})
+
 test('Tests on some invalid files and some valid', async () => {
-  const res = await combineJson({
-    inputDir: 'misc/multiple_empty',
-    outputFile: 'test-output/combine_multiple_empty.json',
+  await combineJson('misc/some_non_valid', {
+    outputFile: 'test-output/combine_some_non_valid.json',
+    validateInput: false,
+  })
+
+  expect(() =>
+    JSON.parse(
+      fs.readFileSync(
+        `${process.cwd()}/test-output/combine_some_non_valid.json`,
+        'utf-8'
+      )
+    )
+  ).toThrow()
+})
+
+test('Tests on some invalid files and some valid with no validation', async () => {
+  await combineJson('misc/some_non_valid', {
+    outputFile: 'test-output/combine_some_non_valid.json',
+    validateInput: false,
+  })
+
+  expect(() =>
+    JSON.parse(
+      fs.readFileSync(
+        `${process.cwd()}/test-output/combine_some_non_valid.json`,
+        'utf-8'
+      )
+    )
+  ).toThrow()
+})
+
+test('Tests on some invalid files and some valid with no validation, but validate output file', async () => {
+  await combineJson('misc/some_non_valid', {
+    outputFile: 'test-output/combine_some_non_valid.json',
+    validateInput: false,
+    validateOutput: true,
+  })
+
+  expect(() =>
+    JSON.parse(
+      fs.readFileSync(
+        `${process.cwd()}/test-output/combine_some_non_valid.json`,
+        'utf-8'
+      )
+    )
+  ).toThrow()
+})
+
+test('Tests on some empty files and some valid, validity check disabled', async () => {
+  const res = await combineJson('misc/some_empty', {
+    outputFile: 'test-output/combine_some_empty.json',
+    validateInput: false,
   })
   const data = JSON.parse(
     fs.readFileSync(
-      `${process.cwd()}/test-output/combine_multiple_empty.json`,
+      `${process.cwd()}/test-output/combine_some_empty.json`,
       'utf-8'
     )
   )
-  const expected = []
+
+  const expected = [
+    {
+      name: 'far away',
+    },
+  ]
   expect(res).toBe(1)
   expect(data).toEqual(expected)
 })
 
 test('Tests if on 1 file containing one primitive', async () => {
-  const res = await combineJson({
-    inputDir: 'misc/one_primitive',
+  const res = await combineJson('misc/one_primitive', {
     outputFile: 'test-output/combine_a_single_primitive.json',
   })
   const data = JSON.parse(
@@ -73,8 +160,7 @@ test('Tests if on 1 file containing one primitive', async () => {
 })
 
 test('Tests if on 1 files', async () => {
-  const res = await combineJson({
-    inputDir: 'misc/one_file',
+  const res = await combineJson('misc/one_file', {
     outputFile: 'test-output/combine_one.json',
   })
   const data = JSON.parse(
@@ -86,8 +172,7 @@ test('Tests if on 1 files', async () => {
 })
 
 test('Tests on 3 array', async () => {
-  const res = await combineJson({
-    inputDir: 'misc/array_test',
+  const res = await combineJson('misc/array_test', {
     outputFile: 'test-output/combine_array.json',
   })
   const data = JSON.parse(
@@ -105,8 +190,7 @@ test('Tests on 3 array', async () => {
 })
 
 test('Tests if on all files', async () => {
-  const res = await combineJson({
-    inputDir: 'misc',
+  const res = await combineJson('misc', {
     outputFile: 'test-output/combine_all.json',
   })
   const data = JSON.parse(

diff --git a/misc/non_valid/non_valid_json.json → misc/some_non_valid/non_valid_json.json b/misc/non_valid/non_valid_json.json → misc/some_non_valid/non_valid_json.json
diff --git a/misc/some_non_valid/one_number.json b/misc/some_non_valid/one_number.json
@@ -0,0 +1 @@
+1
diff --git a/src/cli.js b/src/cli.js
@@ -17,8 +17,18 @@ program
     'File name in which all the json files will be merged',
     'combined.json'
   )
+  .option(
+    '-I, --validate-input <file-name>',
+    'Check if JSON file is valid',
+    false
+  )
+  .option(
+    '-O, --validate-output <file-name>',
+    'Check if output JSON is a valid JSON',
+    false
+  )
   .action(async (directory, options) => {
-    await combineJson({ options, inputDir: directory })
+    await combineJson(directory, options)
   })
 ;(async () => {
   try {