Merge branch 'main' of github.com:klarna-incubator/gram into main

CONTRIBUTING.md

@@ -22,5 +22,5 @@ Your contribution has to meet the following criteria:
 - All code and documentation must follow the style specified by
   the included configuration.
 - Commits should follow the [Conventional Commit convention](https://www.conventionalcommits.org/en/v1.0.0/).
-- New features and bug fixes must have accompanying unit tests.
-- All unit tests should pass.
+- New features and bug fixes should ideally have accompanying tests.
+- All tests should pass.

Development.md

@@ -0,0 +1,34 @@
+# Development
+
+## Repository Layout 🗺️
+
+- `app/` contains the frontend react application
+- `core/` contains the backend library; most data related logic lies here
+- `config/` contains the configuration.
+- `api/` contains the backend web API
+- `plugins/` contains backend plugins, which when installed into config allows for customization
+
+## Development Setup 💻
+
+These are the minimal steps for starting the project locally:
+
+1. Install the dependencies with `npm i`
+
+2. Start the test and development databases via docker compose `docker compose up -d`
+
+3. `npm run build` to build the api and any installed plugins
+
+4. `npm run dev` to start the backend API and react app frontend.
+
+5. Login using `user@localhost` as the email - check the application logs for the login link.
+
+## Tests
+
+`npm test`
+
+## Connecting to the development/test database
+
+Use the following scripts if you need to connect to your development database containers
+
+- `scripts/local-psql.sh`
+- `scripts/test-psql.sh`

Dockerfile

@@ -5,7 +5,7 @@
 # ADD app .
 
 # Build the react app
-FROM node:16-alpine
+FROM node:18-alpine
 
 WORKDIR /home/gram
 

QuickStart.md

@@ -0,0 +1,129 @@
+# Quick Start
+
+This short guide should help you get set up maintaining your own Gram deployment.
+
+## Setting up the repository
+
+The ideal setup is currently to clone the repository locally and push it to your own source control management system.
+
+Create a bare clone of the repository and mirror-push to the new repository.
+
+```sh
+git clone --bare [email protected]:klarna-incubator/gram.git
+cd gram.git
+git push --mirror <your private repo>
+```
+
+Remove the temporary local repository you created earlier.
+
+```sh
+cd ..
+rm -rf gram.git
+```
+
+Clone your own private repository and add Klarna upstream to pull updates from (more details later).
+
+```sh
+git clone <your private repo>
+git remote add github https://github.com/klarna-incubator/gram.git
+```
+
+## First local run
+
+With the repository cloned, run `npm install` to install dependencies and to set up
+the workspace.
+
+```sh
+npm install
+```
+
+Next you can try running the project a first time:
+
+```sh
+docker compose up -d
+```
+
+Then build and try to run the project
+
+```sh
+npm run build
+npm run dev # note: you'll need to run build before every dev
+```
+
+## Modify the configuration
+
+The `config/` package contains the configuration of your Gram application. This is where
+you add custom plugins and functionality.
+
+Check the `config/default.ts` for the base configuration. `config/[development|staging|production].ts` contains configuration for the different deployment environments which overload the default config. The `NODE_ENV` environment variable determines which of these is loaded at runtime.
+
+The default setup comes with a minimal configuration which you'll likely want to change for a real deployment.
+It uses static providers to feed sample data into Gram, regarding things like Systems, Teams, Users and Roles.
+
+### Plugins
+
+To install a new plugin into your config:
+
+```
+npm -w config i @gram/<pluginname>
+```
+
+Then adjust the configuration code as needed to load the new functionality.
+
+## Deploying
+
+Gram is designed to run as a single docker container with a Postgres database provided beside it. Typically the postgres database would be hosted on something like AWS RDS.
+
+### Docker Compose Example
+
+The [docker-compose.demo.yml](docker-compose.demo.yml) is a simple way you could run this application
+in a production environment using just docker compose.
+
+Ensure you set your environment variables before deployment.
+
+```yml
+environment:
+  ORIGIN: http://localhost:4726 # (or your domain)
+  POSTGRES_HOST: database
+  POSTGRES_USER: gram
+  POSTGRES_PASSWORD: somethingsecret
+  POSTGRES_DATABASE: gram
+  POSTGRES_DISABLE_SSL: "true"
+  POSTGRES_PORT: 5432
+  NODE_ENV: staging # i.e. prod/staging for non-demo purposes
+  AUTH_SECRET: <some long secret used to sign auth tokens>
+  EMAIL_HOST: ""
+  EMAIL_PORT: ""
+  EMAIL_PASSWORD: ""
+  EMAIL_USER: ""
+```
+
+Run the demo via
+
+```sh
+docker-compose -f docker-compose.demo.yml up --build
+```
+
+To run this on a server, you can either build the image on the server or use a CI/CD pipeline to build the image.
+
+### Warnings ⚠️
+
+- There are some scalability issues to be aware of - currently we only run this as a single container as the websocket server used for the
+  realtime diagram sharing only runs on the same express server. For Klarna's setup this has worked fine since we don't have that many concurrent users,
+  but it will eventually hit a bottleneck and means the application can't be scaled horizontally. This is solveable but it's not anything we're prioritizing until it actually becomes a problem.
+
+- This webapp was built with the premise that it would run on an internal-only, closed network and be accessible to all engineers. It will not withstand DoS or necessarily guarantee confidentiality if exposed
+  directly on the internet.
+
+## Pulling updates
+
+The main Gram codebase runs on a single `main` branch as a "trunk" and
+uses tag releases. The main branch should not necessarily be seen as stable, instead
+we recommend pulling in the versioned tags:
+
+```sh
+git pull github v4.4.0 # recommended
+# git pull github main # not recommended
+```
+
+After pulling new updates, you may need to re-run `npm install` to get the latest packages.

README.md

@@ -2,56 +2,16 @@
 
 [![Build Status][ci-image]][ci-url]
 [![License][license-image]][license-url]
+[![Discord][discord-image]][discord-url]
 [![Developed at Klarna][klarna-image]][klarna-url]
 
 Gram is Klarna's own [threat model][owasp-tm] diagramming tool developed internally by Klarna's Secure Development team. It is a webapp for engineers to collaboratively create threat models for their systems, providing a easy-to-understand way to document a system as a dataflow diagram with threats/controls attached.
 
 ![Screenshot](screenshot.png)
 
-## Repository Layout 🗺️
+## Getting Started 🚀
 
-- `app/` contains the frontend react application
-- `core/` contains the backend library; most data related logic lies here
-- `config/` contains the configuration.
-- `api/` contains the backend web API
-- `plugins/` contains backend plugins, which when installed into config allows for customization
-
-## Development Setup 💻
-
-These are the minimal steps for starting the project locally:
-
-1. Install the dependencies with `npm i`
-
-2. Start the test and development databases via docker compose `docker compose up -d`
-
-3. `npm run build` to build the api and any installed plugins
-
-4. `npm run dev` to start the backend API and react app frontend.
-
-5. Login using `user@localhost` as the email - check the application logs for the login link.
-
-## Deploying 🚀
-
-Gram is designed to run as a single docker container with a Postgres database provided beside it. Typically the postgres database would be hosted on something like AWS RDS.
-
-The recommended setup is to run Gram as a fork of this repository. That way you can add specific logic for your own organization, as
-you will likely want to integrate your own specific data providers.
-
-All configuration exists in the `config/` package. See [config/README.md](config/README.md) for more specific details on how to configure.
-
-### Disclaimers ⚠️
-
-- Setting up will require that you are comfortable programming a bit in typescript (since the configuration is written that way) and that
-  you know how to build and deploy a docker image.
-
-- As maintainers of this project, we will only offer limited support and are generally prioritizing features that serve our own implementation for Klarna.
-
-- There are some scalability issues to be aware of - currently we only run this as a single container as the websocket server used for the
-  realtime diagram sharing only runs on the same express server. For Klarna's setup this has worked fine since we don't have that many concurrent users,
-  but it will eventually hit a bottleneck and means the application can't be scaled horizontally. This is solveable but it's not anything we're prioritizing until it actually becomes a problem.
-
-- This webapp was built with the premise that it would run on an internal-only, closed network and be accessible to all engineers. It will not withstand DoS or necessarily guarantee confidentiality if exposed
-  directly on the internet.
+See [Quick Start](QuickStart.md).
 
 ## How to contribute 🙋
 
@@ -113,6 +73,8 @@ For license details, see the [LICENSE](LICENSE) file in the root of this project
 [ci-url]: https://github.com/klarna-incubator/gram/actions?query=branch%3Amaster
 [license-image]: https://img.shields.io/badge/license-Apache%202-blue?style=flat-square
 [license-url]: http://www.apache.org/licenses/LICENSE-2.0
+[discord-image]: https://badgen.net/badge/icon/discord?icon=discord&label
+[discord-url]: https://discord.gg/8TFdB8rUfG
 [klarna-image]: https://img.shields.io/badge/%20-Developed%20at%20Klarna-black?style=flat-square&labelColor=ffb3c7&logo=klarna&logoColor=black
 [klarna-url]: https://klarna.github.io
 [owasp-tm]: https://owasp.org/www-community/Threat_Modeling

api/package.json

@@ -5,7 +5,6 @@
   "version": "4.4.0",
   "description": "API for threat modelling",
   "scripts": {
-    "oldstart": "NODE_ENV=development nodemon --watch src --watch ../core/src --files src/index.ts",
     "start": "NODE_ENV=development node dist/index.js",
     "build": "tsc -p tsconfig.build.json",
     "clean": "rimraf ./dist",

config/default.ts

@@ -44,7 +44,7 @@ export const defaultConfig: GramConfiguration = {
     password: new EnvSecret("POSTGRES_PASSWORD"),
     database: new EnvSecret("POSTGRES_DATABASE"),
     port: new EnvSecret("POSTGRES_PORT"),
-    ssl: true,
+    ssl: process.env.POSTGRES_DISABLE_SSL === undefined ? true : false,
   },
 
   notifications: {

config/index.ts

@@ -2,11 +2,13 @@ import dotenv from "dotenv";
 import { loadConfig } from "@gram/core/dist/config/index.js";
 import { registerConfiguration } from "@gram/core/dist/config/configMap.js";
 import { developmentConfig } from "./development.js";
+import { stagingConfig } from "./staging.js";
+import { productionConfig } from "./production.js";
 
 export function initConfig() {
   dotenv.config({ path: "../.env" });
   registerConfiguration("development", developmentConfig);
-  //   registerConfiguration("staging", stagingConfig);
-  //   registerConfiguration("production", productionConfig);
+  registerConfiguration("staging", stagingConfig);
+  registerConfiguration("production", productionConfig);
   loadConfig();
 }

docker-compose.demo.yml

@@ -1,10 +1,15 @@
+version: "3.1"
+
 services:
   database:
+    container_name: gram-demo-postgres
     image: postgres:12
     restart: unless-stopped
     environment:
       POSTGRES_PASSWORD: somethingsecret
       POSTGRES_USER: gram
+    volumes:
+      - pgdata:/var/lib/postgresql/data
     # ports <dont enable this unless you change passwords to something secure>:
     #   - 127.0.0.1:5432:5432
   gram:
@@ -15,17 +20,21 @@ services:
       POSTGRES_HOST: database
       POSTGRES_USER: gram
       POSTGRES_PASSWORD: somethingsecret
-      POSTGRES_DB: gram
+      POSTGRES_DATABASE: gram
       POSTGRES_PORT: 5432
-      SERVICE_STAGE: default # i.e. prod/staging for non-demo purposes
+      POSTGRES_DISABLE_SSL: "true"
+      NODE_ENV: staging # i.e. production/staging for non-demo purposes
       AUTH_SECRET: <some long secret used to sign auth tokens>
-      GITHUB_APPID: <> # TODO: find a good way to have these optional/dynamic secrets.
-      GITHUB_CLIENTID: <>
-      GITHUB_CLIENTSECRET: <>
-      GITHUB_PRIVATEKEY: <>
+      # EMAIL_HOST: ""
+      # EMAIL_PORT: ""
+      # EMAIL_PASSWORD: ""
+      # EMAIL_USER: ""
     ports:
       - 127.0.0.1:4726:8080
 
 networks:
   default:
     name: gram-demo
+
+volumes:
+  pgdata: