From 873ba6a36be04cd562b84afd7fe3d6de90879eb7 Mon Sep 17 00:00:00 2001 From: Dor Meiri <37194716+dormeiri@users.noreply.github.com> Date: Fri, 3 Mar 2023 15:32:46 +0200 Subject: [PATCH] Fix and improve docs (#18) * Fix an improve docs * Fix an improve docs * Fix an improve docs --- .github/workflows/npm-publish.yml | 2 +- CONTRIBUTING.md | 78 ++++++++++++++++++------------- README.md | 24 +++++----- packages/cli/README.md | 28 +++++------ packages/cli/package.json | 6 +-- packages/cli/src/index.ts | 37 ++++++++------- packages/cli/src/run.ts | 7 ++- packages/cli/tsconfig.json | 1 + packages/scanner/README.md | 47 +++++++------------ packages/ui/README.md | 8 ++-- tsconfig.base.json | 3 +- 11 files changed, 125 insertions(+), 116 deletions(-) diff --git a/.github/workflows/npm-publish.yml b/.github/workflows/npm-publish.yml index 3844e76..a3502fd 100644 --- a/.github/workflows/npm-publish.yml +++ b/.github/workflows/npm-publish.yml @@ -5,7 +5,7 @@ name: Node.js Package on: release: - types: [created] + types: [ created ] jobs: publish-npm: diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 64260ca..cc6e299 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,6 +2,8 @@ First, thanks for considering contributing to the project! 🤩 +If you have any question, feel free to [reach out](https://github.com/dormeiri). + - [Setup locally](#setup-locally) - [Useful commands](#useful-commands) - [Common](#common) @@ -22,6 +24,41 @@ First, thanks for considering contributing to the project! 🤩 npm install ``` +Make sure you can have the environment variables for running [automated tests](#automated-tests). + +To validate the setup, run the following commands: + +1. `nx run-many --target=test` +2. `nx run-many --target=lint` +3. `nx run-many --target=build` + +## Automated tests + +Where available, you can run only unit tests with: + +```bash +nx test:unit +``` + +And only integration tests with: + +```bash +nx test:integration +``` + +The integration tests need the following environment variables: + +- `NOODLE_GITHUB_TOKEN` -- Some GitHub token for cloning this repo when scanning. + +_NOTE:_ Code that requires integration should be separated from code that doesn't. +That will help keeping the separation in tests as well. + +All projects have a `test` command that tests both unit and integration: + +```bash +nx test +``` + ## Useful commands ### Common @@ -97,17 +134,13 @@ yarn add @noodle-graph/@local --registry=http://localhost:4873 nx build:global cli ``` -That will build and install the CLI globally on your local machine. Then you can run the CLI, for example: - -```bash -noodle run -``` +That will build and install the CLI globally on your local machine. Then you can run the CLI. Be aware that this will also `npm link @noodle-graph/scanner` the global CLI module. To avoid that, you can do: ```bash nx build cli -cd dist +cd packages/cli/dist npm i -g . ``` @@ -115,15 +148,19 @@ npm i -g . You can also `npm link` packages in the global CLI by changing the directory of the global module. -For me it is `cd /usr/local/lib/node_modules/@noodle-graph/cli`. - For example: ```bash -cd /usr/local/lib/node_modules/@noodle-graph/cli +cd $(npm root -g)/@noodle-graph/cli npm link @noodle-graph/scanner ``` +To print out linked packages you can run: + +```bash +npm list -g --depth=0 +``` + ### Testing the UI locally 1. Run a scan somewhere, for example in [the basic example](./examples/basic) @@ -132,26 +169,3 @@ npm link @noodle-graph/scanner 4. Run `nx start ui`, you should see the scan results. Changes you do in the `src` files, will change the UI. _Make sure you don't push the `scanOutput.js` file_ - -## Automated tests - -All projects have a `test` command that you can run with: - -```bash -nx test -``` - -Where available, you can run only unit tests with: - -```bash -nx test:unit -``` - -And only integration tests with: - -```bash -nx test:integration -``` - -Code that requires integration should be separated from code that doesn't. -That will help keeping the separation in tests as well. diff --git a/README.md b/README.md index c99ace9..fb3f867 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,10 @@ -# Noodle 🍜 (UNDER CONSTRUCTION 🏗️) +# Noodle 🍜 -Architecture diagram that maintains itself. +***Architecture diagram that maintains itself.*** + +Noodle is an open-source project designed to simplify the process of understanding and visualizing complex system architectures. +It offers automated architecture diagram generation, graph filtering, user-friendly UI features, and a friendly CLI, making it easy to deploy and use. +Additionally, Noodle is reliant on the best source of truth, which is the code itself, by linking to the source code of the relationships. example @@ -29,18 +33,16 @@ Architecture diagram that maintains itself. npm install --location=global @noodle-graph/cli ``` -### 2. Add [Noodle comments](./packages/scanner/README.md#noodle-comment) +### 2. Add [Noodle comments](https://github.com/noodle-graph/monorepo/packages/scanner/README.md#noodle-comment) -[Simple example](./examples/basic/someService/index.js) +[Simple example](https://github.com/noodle-graph/monorepo/examples/basic/someService/index.js) -### 3. Create [config file](./packages/cli/README.md#scan-config-file) +### 3. Create [config file](https://github.com/noodle-graph/monorepo/packages/cli/README.md#scan-config-file) -[Simple example](./examples/basic/noodle.json) +[Simple example](https://github.com/noodle-graph/monorepo/examples/basic/noodle.json) ### 4. `noodle run --open` -See more in the [CLI package](./packages/cli). - ## Commands ### `run` @@ -49,7 +51,7 @@ See more in the [CLI package](./packages/cli). noodle run ``` -Using the [scanner](../scanner) to search resources relationships and bundles a UI with the results. +Using the [scanner](https://github.com/noodle-graph/monorepo/packages/scanner) to search resources relationships and bundles a UI with the results. #### Parameters @@ -64,7 +66,7 @@ Using the [scanner](../scanner) to search resources relationships and bundles a ### Scan config file -You can find example of a config file in the [basic example](../../examples/basic/noodle.json) +You can find example of a config file in the [basic example](https://github.com/noodle-graph/monorepo/examples/basic/noodle.json) | Field | Required | Description | |-|-|-| @@ -84,4 +86,4 @@ You can find example of a config file in the [basic example](../../examples/basi ## Contributing -See [CONTRIBUTING.md](./CONTRIBUTING.md) +See [CONTRIBUTING.md](https://github.com/noodle-graph/monorepo/CONTRIBUTING.md) diff --git a/packages/cli/README.md b/packages/cli/README.md index 36dcf45..fb3f867 100644 --- a/packages/cli/README.md +++ b/packages/cli/README.md @@ -1,6 +1,10 @@ -# Noodle 🍜 (UNDER CONSTRUCTION 🏗️) +# Noodle 🍜 -Architecture diagram that maintains itself. +***Architecture diagram that maintains itself.*** + +Noodle is an open-source project designed to simplify the process of understanding and visualizing complex system architectures. +It offers automated architecture diagram generation, graph filtering, user-friendly UI features, and a friendly CLI, making it easy to deploy and use. +Additionally, Noodle is reliant on the best source of truth, which is the code itself, by linking to the source code of the relationships. example @@ -29,18 +33,16 @@ Architecture diagram that maintains itself. npm install --location=global @noodle-graph/cli ``` -### 2. Add [Noodle comments](./packages/scanner/README.md#noodle-comment) +### 2. Add [Noodle comments](https://github.com/noodle-graph/monorepo/packages/scanner/README.md#noodle-comment) -[Simple example](./examples/basic/someService/index.js) +[Simple example](https://github.com/noodle-graph/monorepo/examples/basic/someService/index.js) -### 3. Create [config file](./packages/cli/README.md#scan-config-file) +### 3. Create [config file](https://github.com/noodle-graph/monorepo/packages/cli/README.md#scan-config-file) -[Simple example](./examples/basic/noodle.json) +[Simple example](https://github.com/noodle-graph/monorepo/examples/basic/noodle.json) ### 4. `noodle run --open` -See more in the [CLI package](./packages/cli). - ## Commands ### `run` @@ -49,7 +51,7 @@ See more in the [CLI package](./packages/cli). noodle run ``` -Using the [scanner](../scanner) to search resources relationships and bundles a UI with the results. +Using the [scanner](https://github.com/noodle-graph/monorepo/packages/scanner) to search resources relationships and bundles a UI with the results. #### Parameters @@ -59,12 +61,12 @@ Using the [scanner](../scanner) to search resources relationships and bundles a | `--output`, `-o` | `NOODLE_OUTPUT` | `./noodleScanOutput` | Path to the UI bundle output folder. | | `--githubToken` | `NOODLE_GITHUB_TOKEN` | - | GitHub access token. Required for GitHub resources. | | `--open` | - | `false` | Whether to open in the browser the bundled UI when finished. | -| `--workers` | - | `8` | Amount of workers that will scan files in parallel. | -| `--verbose` | - | `false` | If true, Noodle will print more logs. | +| `--workers` | - | TODO | TODO | +| `--verbose` | - | TODO | TODO | ### Scan config file -You can find example of a config file in the [basic example](../../examples/basic/noodle.json) +You can find example of a config file in the [basic example](https://github.com/noodle-graph/monorepo/examples/basic/noodle.json) | Field | Required | Description | |-|-|-| @@ -84,4 +86,4 @@ You can find example of a config file in the [basic example](../../examples/basi ## Contributing -See [CONTRIBUTING.md](./CONTRIBUTING.md) +See [CONTRIBUTING.md](https://github.com/noodle-graph/monorepo/CONTRIBUTING.md) diff --git a/packages/cli/package.json b/packages/cli/package.json index 012554b..d82bd48 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -11,12 +11,12 @@ "type": "git", "url": "git+https://github.com/noodle-graph/monorepo.git" }, - "main": "index.js", + "main": "src/index.js", "bin": { - "noodle": "./index.js" + "noodle": "src/index.js" }, "scripts": { - "build": "tsc --project tsconfig.lib.json && cp package.json dist && cp README.md dist && node scripts/copyUi.mjs", + "build": "tsc --project tsconfig.lib.json && cp README.md dist && node scripts/copyUi.mjs", "lint": "eslint .", "test": "jest --passWithNoTests", "build:global": "nx build && cd ../scanner/dist && npm link && cd ../../cli/dist && npm i -g . && cd $(npm root -g)/@noodle-graph/cli && npm link @noodle-graph/scanner" diff --git a/packages/cli/src/index.ts b/packages/cli/src/index.ts index 050a3c7..e7d6362 100644 --- a/packages/cli/src/index.ts +++ b/packages/cli/src/index.ts @@ -1,25 +1,26 @@ #! /usr/bin/env node import { DEFAULT_FILES_WORKERS_NUM } from '@noodle-graph/scanner'; -import { Command } from 'commander'; +import { Command, Option } from 'commander'; import figlet from 'figlet'; -import { run } from './run'; - -export const program = new Command(); +import packageJson from '../package.json'; -program.name('noodle').version('0.0.5').description(figlet.textSync('Noodle')); - -program.addCommand( - new Command('run') - .description('Using the scanner to search resources relationships.') - .option('--config, -c ', 'Path to the scan configuration file.', process.env.NOODLE_CONFIG ?? 'noodle.json') - .option('--output, -o ', 'Path the the bundle output directory.', process.env.NOODLE_OUTPUT ?? './noodleScanOutput') - .option('--githubToken ', 'GitHub access token.', process.env.NOODLE_GITHUB_TOKEN) - .option('--open', 'Whether to open in the browser the bundled UI when finished.') - .option('--workers', 'Number of scanner workers', DEFAULT_FILES_WORKERS_NUM.toString()) - .option('--verbose', 'Print debug logs') - .action(run) -); +import { run } from './run'; -program.parse(); +new Command() + .name('noodle') + .version(packageJson.version) + .description(figlet.textSync('Noodle')) + .addCommand( + new Command('run') + .description('Using the scanner to search resources relationships.') + .addOption(new Option('--config, -c ', 'Path to the scan configuration file.').default('noodle.json').env('NOODLE_CONFIG')) + .addOption(new Option('--output, -o ', 'Path the the bundle output directory.').default('./noodleScanOutput').env('NOODLE_OUTPUT')) + .addOption(new Option('--githubToken ', 'GitHub access token.').env('NOODLE_GITHUB_TOKEN')) + .addOption(new Option('--open', 'Whether to open in the browser the bundled UI when finished.')) + .addOption(new Option('--workers ', 'Number of scanner workers').default(DEFAULT_FILES_WORKERS_NUM).env('NOODLE_WORKERS').argParser(parseFloat)) + .addOption(new Option('--verbose', 'Print debug logs.')) + .action(run) + ) + .parse(); diff --git a/packages/cli/src/run.ts b/packages/cli/src/run.ts index ad29016..7a543e0 100644 --- a/packages/cli/src/run.ts +++ b/packages/cli/src/run.ts @@ -5,7 +5,7 @@ import { join, isAbsolute } from 'path'; import { scan } from '@noodle-graph/scanner'; import loggerConstructor from 'pino'; -const UI_BUILD_DIR_PATH = 'ui/build'; +const UI_BUILD_DIR_PATH = '../ui/build'; const SCAN_OUTPUT_JS_FILENAME = 'scanOutput.js'; const SCAN_OUTPUT_JS_PLACEHOLDER = "'%SCAN_OUTPUT_PLACEHOLDER%'"; const INDEX_HTML_FILENAME = 'index.html'; @@ -25,7 +25,7 @@ export async function run(scanAttributes): Promise { { config: JSON.parse((await readFile(configPath)).toString()), github: { token: scanAttributes.githubToken }, - scanWorkersNum: Number(scanAttributes.workers), + scanWorkersNum: scanAttributes.workers, }, logger ); @@ -47,6 +47,9 @@ function createLogger(level: 'debug' | 'info') { level, transport: { target: 'pino-pretty', + options: { + ignore: 'pid,githubToken', + }, }, }); } diff --git a/packages/cli/tsconfig.json b/packages/cli/tsconfig.json index 93a3167..015bbc5 100644 --- a/packages/cli/tsconfig.json +++ b/packages/cli/tsconfig.json @@ -2,5 +2,6 @@ "extends": "../../tsconfig.base.json", "include": [ "src/**/*.ts", + "package.json", ] } diff --git a/packages/scanner/README.md b/packages/scanner/README.md index 5a442dd..3d175ac 100644 --- a/packages/scanner/README.md +++ b/packages/scanner/README.md @@ -2,21 +2,20 @@ - [Scanning](#scanning) - [Noodle comment](#noodle-comment) - - [Undeclared resources (TODO)](#undeclared-resources-todo) - - [Type evaluation of undeclared resources (TODO)](#type-evaluation-of-undeclared-resources-todo) + - [Undeclared resources](#undeclared-resources) - [Scan output](#scan-output) - [Relationship object](#relationship-object) ## Scanning -- Login to GitHub user. -- Clone repositories. -- Scan the files in each repository and search [Noodle comments](#noodle-comment) based on the programming language. -- [Evaluate undeclared resources types](#type-evaluation-of-undeclared-resources-todo) (TODO). +1. Login to GitHub user. +2. Clone repositories. +3. Scan the files in each repository and search [Noodle comments](#noodle-comment). +4. Enrich and return the result object. ### Noodle comment -A code comment that represents a relationship with another resource. It has the following structure: `noodle --{action}-> {resourceId} ({tags})` +A code comment that represents a relationship with another resource. It has (roughly) the following structure: `noodle --{action}-> {resourceId} ({tags})` For example: @@ -24,26 +23,14 @@ For example: - `// noodle <-receive from-- sqs-my-queue (sqs,feature2,data)` - `/* noodle --uploads to-> s3-my-bucket (s3,feature1) */` - `# noodle --publish to-> sns-topic1 (sns,feature2)` +- `// noodle --> sns-topic2 (sns,feature2)` +- `// noodle <-> sns-topic3` Every time the scanner finds a Noodle comment it will add the relationship to the results. -### Undeclared resources (TODO) +### Undeclared resources -Resources that were discovered during the scan and were not listed in the [scan config file](../cli/README.md#scan-config-file). - -#### Type evaluation of undeclared resources (TODO) - -The type of undeclared resources will be evaluated by the relationship which discovered them. - -Example of evaluation rules: - -- `id starts with "s3" => aws/s3` -- `id starts with "aws/s3" => aws/s3` -- `tags contains "sns" => aws/sns` -- `tags contains "aws/sns" => aws/sns` -- etc. - -The first match wins. +Resources that were discovered during the scan and were not listed in the [scan config file](https://github.com/noodle-graph/monorepo/packages/cli/README.md#scan-config-file). ## Scan output @@ -51,19 +38,19 @@ Here is a sample of scan output: ```json { - "resources":[ + "resources": [ { "id":"some-service", "url":"./someService", "name":"Some Service", - "tags":[ + "tags": [ "feature1", "feature2", "service" ], "type":"aws/ecs", "source":"local", - "relationships":[ + "relationships": [ { "action":"requests from", "resourceId":"another-service", @@ -81,13 +68,13 @@ Here is a sample of scan output: "id":"another-service", "url":"./anotherService", "name":"Another Service", - "tags":[ + "tags": [ "feature1", "service" ], "type":"aws/ecs", "source":"local", - "relationships":[ + "relationships": [ { "action":"query", "resourceId":"some-db", @@ -103,7 +90,7 @@ Here is a sample of scan output: { "id":"some-db", "name":"Some DB", - "tags":[ + "tags": [ "feature1", "feature2", "service" @@ -122,7 +109,7 @@ _Note that this is very similar to the [scan config file](../cli/README.md#scan- |-|-| | `resourceId` | The identifier of the related resource. | | `action` | The action declared in the relationship. | -| `tags` | The tags of the relationship, mostly used for [type evaluation](#type-evaluation-of-undeclared-resources-todo) and filtering. | +| `tags` | The tags of the relationship for filtering. | | `url` | The link to the exact line of the [Noodle comment](#noodle-comment). | | `from` | Whether arrow should be in the "from" resource. | | `to` | Whether arrow should be in the "to" resource. | diff --git a/packages/ui/README.md b/packages/ui/README.md index fa37aaa..0986475 100644 --- a/packages/ui/README.md +++ b/packages/ui/README.md @@ -1,6 +1,7 @@ # UI - [Features](#features) +- [Bundling](#bundling) The graph displayed using [vis.js](https://visjs.org/) under the hood. @@ -14,20 +15,17 @@ The graph displayed using [vis.js](https://visjs.org/) under the hood. - Tags - Name (TODO) - Type (TODO) - - Relationship tags (TODO) - Distance from filtered nodes - Resource view - Name - Type - Description - Tags - - Link to git repository (if GitHub source) - - Path (if local source) + - Link to file - Relationships - - Action - Related resource name - Tags - - Link to source code line (TODO) + - Link to source code line ## Bundling diff --git a/tsconfig.base.json b/tsconfig.base.json index 71e8f4f..47c5bcd 100644 --- a/tsconfig.base.json +++ b/tsconfig.base.json @@ -19,7 +19,8 @@ "pretty": true, "sourceMap": true, "strict": true, - "target": "ES2021" + "target": "ES2021", + "resolveJsonModule": true }, "exclude": [ "node_modules"