Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update README for root and modules/components #45

Open
wants to merge 2 commits into
base: develop
Choose a base branch
from
Open

Update README for root and modules/components #45

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a68e365
Update README.md in root
dluudevs Oct 2, 2024
51ed102
update readme in root folder and modules/components
dluudevs Oct 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
187 changes: 3 additions & 184 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,187 +1,6 @@
<h1 align="center">Arranger</h1>
<p align="center">Generate and manage your own genomic data portal.</p>

<p align="center"><a href="http://www.overture.bio/products/arranger" target="_blank"><img alt="Release Candidate" src="http://www.overture.bio/img/progress-horizontal-RC.svg" width="320" /></a></p>

[![Slack](http://slack.overture.bio/badge.svg)](http://slack.overture.bio)

Develop (Edge): [![Build Status](https://jenkins.qa.cancercollaboratory.org/buildStatus/icon?job=Overture.bio%2Farranger%2Fdevelop)](https://jenkins.qa.cancercollaboratory.org/job/Overture.bio/job/arranger/job/develop/)

Master (Release): [![Build Status](https://jenkins.qa.cancercollaboratory.org/buildStatus/icon?job=Overture.bio%2Farranger%2Fmaster)](https://jenkins.qa.cancercollaboratory.org/job/Overture.bio/job/arranger/job/master/)

> **Note**: Arranger is undergoing refactoring work, for what will become version 3+. While we do not foresee many dramatic breaking changes, an _upgrade guide_ is already in the works. Current users of v2 may look at the `legacy` branch meanwhile, where we will continue fixing newly reported bugs until further notice.
>
> You may need to add the `--legacy-peer-deps` flag when integrating our modules into your apps. Along with the rewrite, we're steadily updating the internal dependencies so this isn't necessary.

## Documentation

This file is meant as a quick introduction, but for more in-detail documentation, you should explore Arranger's "[Read the Docs](https://arranger.readthedocs.io/en/latest)". If interested, see our Open Source [License](https://github.com/overture-stack/arranger/blob/master/LICENSE)

- [Getting Started](#getting-started)
- [Development Setup](#--development-setup)
- [Dockerized Setup (a.k.a. Quickstart)](#--dockerized-setup)
- [Motivation](#motivation)
- [What is a "Data Portal"?](#data-portal)
- [Roadmap](#roadmap)
- [Development Details](#development-details)

---
<h1 align="center">Pilot Arranger</h1>

### Getting Started

Arranger has been tested using NodeJS version: ^16.13.2, and it may not work correctly on versions <14

#### - Development Setup

Setting up the project, and prepare things to make changes

```bash
# 1. clone the repository
git clone [email protected]:overture-stack/arranger.git

# 2. enter the project's folder
cd arranger

# 3. install the dependencies
npm ci

# 4. install the module's own dependencies
npm run bootstrap

```

Now you should be able to start the following processes from the project's root folder:

```bash
# watch all modules and rebuild them when you make changes
npm run watch

# test all modules at once
npm run test

# run the server (on port 5050)
npm run server

# serve the component dashboard (on port 6060)
npm run dashboard

# serve the component portal (on port 7070)
npm run portal

# run storybook (on port 8080)
npm run storybook
```

#### - Dockerized Setup

A bit more friendly "quickstart", if you just want to get things started

```bash
# Start all services at once, using some default settings.
# This runs the following services: Elasticsearch, Kibana, and Arranger Server:
make start

# ^^^^ ^^^^^ which stands for the following command behind the scenes:
# ES_USER=elastic ES_PASS=myelasticpassword docker-compose -f docker-compose.yml up -d -build

# Alternatively, you could start the services separately, like so:
make start-es # Elasticsearch
make start-server # Arranger Server

# Note: The ES_* environment values may be customized when running your own Arranger instance. They can be found atop the `Makefile`.

---
# Afterwards, in another bash process, you may seed an example/mock file_centric index:
make seed-es

# ^^^^ ^^^^^^^ which runs the following command behind the scenes:
# ./docker/elasticsearch/load-es-data.sh ./docker/elasticsearch elastic myelasticpassword

# That script file may give you ideas on how to automate uploading indexes to your instance.


---
# Bonus: ----------------------------- #
# See other preprogrammed make targets
make help
# e.g. utilities to list the indexes, or clear the Elasticsearch; list the running docker containers, etc.
```

---

### Motivation

The Ontario Institute for Cancer Research ([OICR](https://oicr.on.ca/)) has built a few **[Data Portals](#data-portal)**.
e.g.:

- [International Cancer Genome Consortium (ICGC) Data Portal](https://dcc.icgc.org/)
- [Genomic Data Commons (GDC) Data Portal](https://portal.gdc.cancer.gov/) (joint effort with University of Chicago)

Although they are not identical in architecture, available data or overall purpose, there is tremendous amount of overlap in how they function and how users interact with them, despite being implemented differently. It's no coincidence. The GDC Data Portal was directly influenced by the ICGC Data Portal.

With new projects ahead of us, there is an opportunity to create a framework designed to act as a core library for any given data portal, similar to what Elastic's Kibana accomplishes; but based on the features of our existing portals, and the expectation of continuous improvement and domain specific customization.

There are many potential benefits:

- Reduce duplicate code
- Ability to fix bugs and add features to many projects at once
- Pool developer resources
- Increase cross-team communication
- Encourage open source contribution

---

<h3 id="data-portal">What Is A "Data Portal"?</h3>

#### Topology

![DP Topology](https://i.imgur.com/Ylm9drr.png)
_This is a simplified "birds-view" diagram of what goes on under the hood_

---

### Roadmap

#### Short Term

- cli tool for bootstrapping new projects

- Provide all necessary modules to implement searching functionality
- Dynamic GraphQL schema generation
- API Server (GraphQL endpoint)
- Query / Aggregation building middleware
- Response middleware (ie. removing null aggregations)
- UI Components
- Aggregations
- Simple view
- Advanced View
- Results Table
- SQON Display

* Provide editor interface to expose common transformations (similar to the [Babel](https://babeljs.io/repl/) or [bodybuilder](thttp://bodybuilder.js.org/) REPLs)
- Elasticsearch Mappings -> GraphQL Schema
- GraphQL Query -> Elasticsearch Queries

#### Medium Term

- Authentication
- Sets
- Analysis

#### Long Term

- Kibana Plugin
- Hosted Data Portal generating service

---

### Development Details

Arranger is a [lerna](https://github.com/lerna/lerna) flavored [monorepo](https://medium.com/@maoberlehner/monorepos-in-the-wild-33c6eb246cb9). The modules exposed by Arranger compose all of the necessary code required to build an application such as the [Genomic Data Commons](https://portal.gdc.cancer.gov/).\*

#### Releasing Instructions

- From `main` branch, run `npm run tag <version>`
- Publishing process will be run [by Jenkins](https://jenkins.qa.cancercollaboratory.org/blue/organizations/jenkins/Overture.bio%2Farranger/activity?branch=main)

_\* The GDC contains many features that are out of Arranger's scope_
Ensure you are using node version 14.19.0 and npm version 6.14.6. To manage different versions of node, we recommend using a [node version manager](https://github.com/tj/n)
Move to the branch `portal-ui` and install the dependencies. Once completed, navigate to `modules/components` and follow the README there for setup and development guides.
36 changes: 31 additions & 5 deletions modules/components/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,35 @@
# Data Portal Components
## Development Setup

## Roadmap
Make sure you are checked out in the `portal-ui` branch.

#### Aggregations
Ensure you are using node version `14.19.0` and npm version `6.14.6`. To manage different versions of node, we recommend using a [node version manager.](https://github.com/tj/n)
Please note Portal uses version node version `16.13.2` and npm version `9.1.2`

#### SQON aka "Current Filters"
Inside of `modules/components` you will need to create a file called `.npmrc`. This file is required to publish updates and properly installing the dependencies for this project.The `components` module is to be published in order for Portal to consume the latest changes. In this file you will want to add the following:

#### ResultsTable
```
@pilotdataplatform:registry=https://npm.pkg.github.com
//npm.pkg.github.com/:_authToken=<PERSONAL_ACCESS_TOKEN>
```

If you do not have a personal access token you will need to create one via github. Hover over your profile, go to settings > developer settings > personal access tokens > Tokens (classic). Set the expiry date to a year from the date of creation and ensure you have only read / write package permissions.

![Github personal access token creation](docs/images/github-personal-access-token.png)

Now we're ready to install the packages. Ensure you are in `modules/components` and input `npm install` in the terminal. If you encounter any issues with installing your packages, it is likely your token has expired. Create a new token and replace the old one in the `.npmrc` file. If this does not resolve your issue, the error message in the terminal should display a file path to inspect your logs for more detailed troubleshooting.

Upon completing the installation, input `npm run start` to start up the local server. Once it loads you will be required to login with your dev environment credentials, or if you have an instance of the dev environment opened you will be logged in automatically.

`modules/components/src/App.js` is the entry point of the local development server.

## Publishing

### Arranger

Once you have finished with your changes, the components module will have to be published. In order to do so, increment the version number in `module/components/package.json`. Now, in the terminal input `npm publish`. Using your credentials in the `.npmrc` file, this script will build the project and publish it to github's package repo.

### Portal

To get the latest updates, ensure you run `npm install` for the latest version of the arranger package. For example, to install arranger version 1.5.11 you will need to input `npm install @pilotdataplatform/arranger-components/@1.5.11.

Upon updating the version, ensure it runs correctly locally and submit the changes through a PR.
Binary file added modules/components/docs/images/github-personal-access-token.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.