Skip to content

Commit

Permalink
#30-milestone4: Rancher, K8S, and Custos setup
Browse files Browse the repository at this point in the history
  • Loading branch information
vinayakasg18 committed May 6, 2022
1 parent 4c417ee commit e3de469
Show file tree
Hide file tree
Showing 32 changed files with 2,970 additions and 143 deletions.
153 changes: 10 additions & 143 deletions deployment_scripts/README.md
Original file line number Diff line number Diff line change
@@ -1,146 +1,13 @@
## Goal: Continuous deployment using Concourse CI/CD tool
### Distributed system configurations and installation

TL;DR
This contains two components

### Pipelines
- CI/CD using concourse
- Builds and deploy to kubernetes cluster
- Cluster is running in JS2 instance
- Cluster is configured using JS2 openstack [documentation](https://docs.jetstream-cloud.org/general/k8scluster/)

```api-gateway``` | ```data-source``` | ```user-management``` | ```merra-data-source```

- Pipeline for each service resides under directory ``deployment_scripts/concourse_cd``

#### Concourse Installation

- We are using concourse as a service
- Concourse is deployed in K8S using helm chart
- ``$ helm repo add concourse https://concourse-charts.storage.googleapis.com/``
- ``$ helm install my-release concourse/concourse``
- Create K8S cluster secrets for concourse to connect to K8S cluster for deployment


#### Concourse definitions

- Define resource types and resources

: Resource Type -
- Kubernetes docker resource to deploy the applications to k8s cluster
- We used resource type {kudohn/concourse-k8s-resource}
```yaml
resource_types:
- name: k8s
type: docker-image
source:
repository: kudohn/concourse-k8s-resource
```
: Resources -
- We need explicitly define the resource types, which are declared before we can use them. so we are going to tell concourse that we want to use kubernetes external resource ``{kudohn/concourse-k8s-resource}``
- we are using ```{git}``` resource supported by concourse which will pull the code from code repositories and we are using gitHub in our case
- We are using ```{registry-image}``` resource to build and publish our docker image to docker-hub
- We are using ``` {k8s} ``` reource, as defined above in the resource type section to connect to k8s cluster and deploy applications
```yaml
resources:
- name: git-ds
type: git
icon: git
source:
uri: https://github.com/airavata-courses/DCoders.git
branch: feature/30-implement-cd
# Where we will push the image
- name: publish-image
type: registry-image
icon: docker
source:
repository: vinayakasgadag/decoders-datasource
tag: milestone-3
username: vinayakasgadag
password: ((k8s-secret.docker-hub-pwd))
- name: k8s
type: k8s
icon: kubernetes
source:
api_server_url: ((k8s-secret.k8s-cluster-url))
api_server_cert: ((k8s-secret.api_server_cert))
client_cert: ((k8s-secret.client_cert))
client_key: ((k8s-secret.client_key))
skip_tls_verify: false
```

: Jobs -
- Jobs are resposible for creating the required tasks (defined in job definition) to run the pipeline
- Jobs have stages(plan) and tasks
- Stage or plan: it can be defined to be dependent on previous stage and run only if it is successful
- Task: Useful when we want to run some scripts to achieve somehting before the job can be run
```yaml
jobs:
- name: "build-publish"
public: true
serial: true
plan:
- get: git-ds
trigger: true
- task: build-image-task
privileged: true
config:
platform: linux
image_resource:
type: registry-image
source:
repository: vito/oci-build-task
inputs:
- name: git-ds
outputs:
- name: image
params:
CONTEXT: git-ds/datasource_service
UNPACK_ROOTFS: true
run:
path: build
- put: publish-image
params:
image: image/image.tar
- name: "deploy-application"
plan:
- get: git-ds
passed: ["build-publish"]
- task: create-deploy-scripts
config:
platform: linux
image_resource:
type: docker-image
source: {repository: busybox}
inputs:
- name: git-ds
outputs:
- name: deploy-files
run:
path: git-ds/deployment_scripts/copy_files.sh
- put: k8s
params:
status_check_timeout: 60
command_timeout: 30
paths:
- deploy-files/datasource_service/deployment.yaml
- deploy-files/datasource_service/service.yaml
- deploy-files/datasource_service/autoscaler.yaml
watch_resources:
- name: datasource-deployment
kind: Deployment
```


##### Example view of concurse dashboard
<img width="1117" alt="Screen Shot 2022-04-21 at 3 24 38 PM" src="https://user-images.githubusercontent.com/52463165/164537737-15ed588b-54ac-4287-b495-29dbce79c9ed.png">


##### Why Concourse?
- The beauty of the concourse is the declarative appraoch and the flexibility to automate the builds the way we want to using the resources, jobs and task concpets
- We don't need ansible or any other tool to automate the deployments since concourse supports it through the resources
- Concourse can be run as individual standalone cluster or as a distributed system as service inside the kubernetes cluster, which will scale horizontally as the number of pipeline or the job increses, and this is a very big advantage comparing to Jenkins or any other tools


##### For more details on [concourse](https://concourse-ci.org/docs.html)
- Custos set up
- Set up rancher
- Create K8S cluster
- Follow [custos](custos) guide for more details on installation
155 changes: 155 additions & 0 deletions deployment_scripts/concourse_cd/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
## Goal: Continuous deployment using Concourse CI/CD tool

TL;DR

### Pipelines

```api-gateway``` | ```data-source``` | ```user-management``` | ```merra-data-source```

- Pipeline for each service resides under directory ``deployment_scripts/concourse_cd``

#### Concourse Installation

- We are using concourse as a service
- Concourse is deployed in K8S using helm chart
- ``$ helm repo add concourse https://concourse-charts.storage.googleapis.com/``
- Create PV for storage
``$ kubectl apply -f pv.yaml``
- ``` kubernetes helm
helm install concourse concourse/concourse \
--set persistence.worker.size=2Gi \
--set persistence.worker.storageClass=manual \
--set postgresql.persistence.size=2Gi \
--set postgresql.persistence.storageClass=manual \
--set web.service.type=NodePort
```
- Create K8S cluster secrets for concourse to connect to K8S cluster for deployment


#### Concourse definitions

- Define resource types and resources

: Resource Type -
- Kubernetes docker resource to deploy the applications to k8s cluster
- We used resource type {kudohn/concourse-k8s-resource}
```yaml
resource_types:
- name: k8s
type: docker-image
source:
repository: kudohn/concourse-k8s-resource
```
: Resources -
- We need explicitly define the resource types, which are declared before we can use them. so we are going to tell concourse that we want to use kubernetes external resource ``{kudohn/concourse-k8s-resource}``
- we are using ```{git}``` resource supported by concourse which will pull the code from code repositories and we are using gitHub in our case
- We are using ```{registry-image}``` resource to build and publish our docker image to docker-hub
- We are using ``` {k8s} ``` reource, as defined above in the resource type section to connect to k8s cluster and deploy applications
```yaml
resources:
- name: git-ds
type: git
icon: git
source:
uri: https://github.com/airavata-courses/DCoders.git
branch: feature/30-implement-cd
# Where we will push the image
- name: publish-image
type: registry-image
icon: docker
source:
repository: vinayakasgadag/decoders-datasource
tag: milestone-3
username: vinayakasgadag
password: ((k8s-secret.docker-hub-pwd))
- name: k8s
type: k8s
icon: kubernetes
source:
api_server_url: ((k8s-secret.k8s-cluster-url))
api_server_cert: ((k8s-secret.api_server_cert))
client_cert: ((k8s-secret.client_cert))
client_key: ((k8s-secret.client_key))
skip_tls_verify: false
```

: Jobs -
- Jobs are resposible for creating the required tasks (defined in job definition) to run the pipeline
- Jobs have stages(plan) and tasks
- Stage or plan: it can be defined to be dependent on previous stage and run only if it is successful
- Task: Useful when we want to run some scripts to achieve somehting before the job can be run
```yaml
jobs:
- name: "build-publish"
public: true
serial: true
plan:
- get: git-ds
trigger: true
- task: build-image-task
privileged: true
config:
platform: linux
image_resource:
type: registry-image
source:
repository: vito/oci-build-task
inputs:
- name: git-ds
outputs:
- name: image
params:
CONTEXT: git-ds/datasource_service
UNPACK_ROOTFS: true
run:
path: build
- put: publish-image
params:
image: image/image.tar
- name: "deploy-application"
plan:
- get: git-ds
passed: ["build-publish"]
- task: create-deploy-scripts
config:
platform: linux
image_resource:
type: docker-image
source: {repository: busybox}
inputs:
- name: git-ds
outputs:
- name: deploy-files
run:
path: git-ds/deployment_scripts/copy_files.sh
- put: k8s
params:
status_check_timeout: 60
command_timeout: 30
paths:
- deploy-files/datasource_service/deployment.yaml
- deploy-files/datasource_service/service.yaml
- deploy-files/datasource_service/autoscaler.yaml
watch_resources:
- name: datasource-deployment
kind: Deployment
```


##### Example view of concurse dashboard
<img width="1117" alt="Screen Shot 2022-04-21 at 3 24 38 PM" src="https://user-images.githubusercontent.com/52463165/164537737-15ed588b-54ac-4287-b495-29dbce79c9ed.png">


##### Why Concourse?
- The beauty of the concourse is the declarative appraoch and the flexibility to automate the builds the way we want to using the resources, jobs and task concpets
- We don't need ansible or any other tool to automate the deployments since concourse supports it through the resources
- Concourse can be run as individual standalone cluster or as a distributed system as service inside the kubernetes cluster, which will scale horizontally as the number of pipeline or the job increses, and this is a very big advantage comparing to Jenkins or any other tools


##### For more details on [concourse](https://concourse-ci.org/docs.html)
11 changes: 11 additions & 0 deletions deployment_scripts/concourse_cd/k8s-secrets.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
apiVersion: v1
kind: Secret
metadata:
name: k8s-secret
data:
api_server_cert: |
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQzRENDQXNTZ0F3SUJBZ0lCQWpBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwdGFXNXAKYTNWaVpVTkJNQjRYRFRJeU1EVXdNakE0TkRVeE4xb1hEVEkxTURVd01qQTRORFV4TjFvd0xERVhNQlVHQTFVRQpDaE1PYzNsemRHVnRPbTFoYzNSbGNuTXhFVEFQQmdOVkJBTVRDRzFwYm1scmRXSmxNSUlCSWpBTkJna3Foa2lHCjl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF6S0RsdS84MGZMZXA0aGtqYWkzS1pSZ3hRbEVFWGR5LzdVMHkKMUJpQkpmTGNjRC9LdHJSZFN5QkhMdnNVNDI5VHFMVEVqVEpIR3hyZEhySGJ4OWpIamUvU2VPSWhqZnJvcThBKwpPM1ExbXNtSkxBemM2UGlTMWVicFFLWEVvNlJEUGJ1S3Era2NPZnJrWjQxVmlKcjdWODZ6aGNISVRoK2xVQmZJCjdOZ0xUMW4zaUd4czhPRmlTcDQzbzEyM1JFWDhIMnhiZkE1SEtmdlNMS01CcWwxQ3pWbnJodXRhWWU0VWIxeDkKdTRibHRLb28zN1hkMlVNakNCOWUvb2wxN2YzSjFQZmRvNDNBd1RoTXMvTkZ5Z2M4bU8zck5iaG5ZU1d1QVZKaQplZUl0bVlrOFI4MmtOdmpheVp2OGR6N2JPSW45RDZOcndIekpIR2kwUFlyRmFNbVhtd0lEQVFBQm80SUJIakNDCkFSb3dEZ1lEVlIwUEFRSC9CQVFEQWdXZ01CMEdBMVVkSlFRV01CUUdDQ3NHQVFVRkJ3TUJCZ2dyQmdFRkJRY0QKQWpBTUJnTlZIUk1CQWY4RUFqQUFNQjhHQTFVZEl3UVlNQmFBRkI2VjJNVm1hNG9MYlFrNHNMRk45Qmxvdkh0dQpNSUc1QmdOVkhSRUVnYkV3Z2E2Q0NtMXBibWxyZFdKbFEwR0NIMk52Ym5SeWIyd3RjR3hoYm1VdWJXbHVhV3QxClltVXVhVzUwWlhKdVlXeUNKR3QxWW1WeWJtVjBaWE11WkdWbVlYVnNkQzV6ZG1NdVkyeDFjM1JsY2k1c2IyTmgKYklJV2EzVmlaWEp1WlhSbGN5NWtaV1poZFd4MExuTjJZNElTYTNWaVpYSnVaWFJsY3k1a1pXWmhkV3gwZ2dwcgpkV0psY201bGRHVnpnZ2xzYjJOaGJHaHZjM1NIQk1Db01RS0hCQXBnQUFHSEJIOEFBQUdIQkFvQUFBRXdEUVlKCktvWklodmNOQVFFTEJRQURnZ0VCQUxaMXIzbW5WZndiR21nS2lzSzhDekRFTDZjMFhIb2ZaenNia1c5MXpFb08KMjdlRGVMSVA2ZStLeXJndDJscllVcTAyeEgwQk11Y2NWUDR5cEJyRmZXdWgybE9rdWNtZEl2amE5SExOcEs0bwpmcHU5bjJPVHpHdVRsVnRrWUNLcEFXMURxSnlhblV4NC92MjYwWkV4OFdXbWp6aEtQbnlRakkrVnBmSzZva3pVCnQxSldZcVB1d3NtOEg3b0pOZTFUZU1rRkdJZ2haV3A4cngzTXZHK0V2WEtSTkE5OStHN3V2bEtMQ2E3eWYrMS8KSjRzcngwNzNxSHBpMTZ4RGFZcEsydFh6ek4xdVBWaGhla0YxMHV2dzIxVStGdEJIQWhoWXNZN3BVRWZ0TWpEbgpUbmJLaUFjNEM0NUsvRXpQQ0VDL2FFenBkZHI5ZWZiSkJYMTVOOHJrYkJvPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0t
client_key: |
LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBcXAzM3V5ZHRhSDZLcUpEUy8wUUJtbjc1VVBhMncxa2EzTW15WWlpQnhCZStnMFloCkNGcC9RWHpXNjRyM2o1aFBFait5NzFSUXNqRzdMakVHaDhmNzBWSE4rSTJrRDlWbndlOFUyZnNzVzh0anQ4SksKV1NQa01SYTlkVlV0WlhoVGRtSFkyZzlqNzdrLzB0L1BvUkJtbHBIMjB2ZjVUQ1BqbDFrRllyYmRpck1uMWNuRApna0YzMklWUWFDMmtHVXlzL25SVVNnZ2FzMUUxYmo4TElha3Q5Z1Zta3JEdjZvaVFNV3QxVVRhSWMwbFV5NzJwCmdFNjhBTFlwOS9ZYmE3RFdON2I2ZEpSVWxiWkdCR1RmZ1VPVkcybXdUYjl4aTgzckIwQVBFeGlBbW00bVJiNEcKem5tTGxCSFB4bjYvWGRUbXhSdFR1NUFpSVplTGxLYUJjRWIyeXdJREFRQUJBb0lCQUY3NU9ncHJJeUwwdGJpcApqVnBjLzh1QmZNVU40S1NUT0RuSTZNeHRJZmNIQkp6TWI5elhpMWpuNWpjTmowcldqTVZxd2U4cGJ4WVNTdENtCnA2enpySUJUV2lWT3F4SEpTRTJUQ1hkaHNzcVNTRHJsSXovRmsrT2pkZWtYZGdLeTNUcmJzcnVIcjZpazVSczgKVExhcWk0Q3JWY3NRRUN6TEdZaUpIRTliM2F5enJ0bWk2U0Z6b1ZadE5FZ0RBTlUzaFdWS3F0ekdtZDdoT245dgpwNS9JaHVsRjdodFRFaVdVbjgyNFdrRHhyQ2NMQkRDMXBhTmwydUVsMENwdjcvUW5Za3M3THNFeTFGaWVoMkNDCnN3Tnd5REp1N211dGpMVGN2T013Q2ppSlJjZHFxN0tGN3hycmt0MTMvNCtzdngrbVVpQU5hVGRFQzd3WVNJR3cKdVpNZVpOa0NnWUVBeWE1YkJTUEo0UGxMTW9FdTNJYWpxeS9hcUI5U0lDOE5EQlBJWkNQTU5FRHRIY0E2dEhFQQpYWmtBMGJwQkhiNllPK2V6SkdlWHVQYkJoMW5rSlpSSEVRUHY4QkJ6VGZwbkh3VW9pRzFQdUhyZ2JXTG9RTUtKClJPN2ZDNFppd2p2YmFGV2pIUHVFekIyNzI5aTNIcWFMWlVNWXZBVHplMTZ2c21PNkozWnhueDhDZ1lFQTJKSEsKREV0Tjc1SW0wall2SGI2U2dWQkptbnl0WW9Lc01PMXRRejV6SitrNlZrWElwQXJ3cnUwb01TQ3BJV2xkeVJ6SQoxNUkzaE1oZDJCdmx4cjJINHJvWnJWRTc5RjlZeDdqZlI0VDlVUlhtaGVUSTZpSHpTTXpSRHByZUU4Qnc2UllVCkg0a3ZWSXVyWWNKN0J4NE5tYS9nK0hwVXZwNk9QeU91RExBYUx0VUNnWUJuRVNlSzFPTlNpWlFZVjFSdmRvOGwKNk9yQmlHQWIrbStjZ0cra1hYYjZMVVFBTkVETC9nUEYwVzlOdnZXUUVkc1NvakkycElveENFbVd0aVdWM3RVQwpxUlJ4aHJhbVh4VmNFUExKNWJNY0FBKzVWeGFDSWVpc3hiWk8yWHFXOEtnTUJkZTU4Ly9Gb0Z4azJiZWJmbGsyCmdyZWRQcHAvcmIvMFZtckh5QXdBMlFLQmdRQzN4bmM3R1lmb0hSQ2VYMlo2Q2lhT1gxQW1MVmlBZUx5ZnhFcHMKdm9pL3ZIVkprbXdoY0RzdlpZWXVzalZ6YWRNdy95RWJkVE54bFFtMWdtN295QnFRZGpXbDBvSmE2N0lOd1Q2UApsVFhVNGcyOVh4aHpQaDRSaitSelRVM1lXdncxZnd2U2V2cFQ5eldXZm84aHlnbm1lYzRoYk1XUEFFTmJKdTdpClMybmNoUUtCZ0JHNlNVR1pSZmI1d1NUVTJFaGkybjU3TW12MGh6ZGxTdktFLzJrUUp5ZkVOQUZTU1p5bVgwMm8KRlQ0Q2hraFBBb1BXd3RLSGVLZHF2ZEJIQ3RZbEJVWDZSZk1GelJmc3c0OEdmMTVJQVU2MmgyS3MxaUdwN0Q1eQpJdFFPRUVEMk1CdENVZDYwblNsWmo4OTFvMFpmVXFrQjNvckZTRE9JbktnQllBNWxBOER2Ci0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0t
client_cert: |
LS0tLUJFR0lOQ0VSVElGSUNBVEUtLS0tLQpNSUlESVRDQ0FnbWdBd0lCQWdJQkFqQU5CZ2txaGtpRzl3MEJBUXMKRkFEQVZNUk13RVFZRFZRUURFd3B0YVc1cGEzVmlaVU5CTUI0WERUSXlNRFV3TWpBNE5EVXhObG9YRFRJMU1EVQp3TWpBNE5EVXhObG93TVRFWE1CVUdBMVVFQ2hNT2MzbHpkR1Z0T20xaGMzUmxjbk14RmpBVUJnTlZCQU1URFcxCnBibWxyZFdKbExYVnpaWEl3Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRQ3FuZmUKN0oyMW9mb3Fva05ML1JBR2FmdmxROXJiRFdScmN5YkppS0lIRUY3NkRSaUVJV245QmZOYnJpdmVQbUU4U1A3TAp2VkZDeU1ic3VNUWFIeC92UlVjMzRqYVFQMVdmQjd4VForeXhieTJPM3drcFpJK1F4RnIxMVZTMWxlRk4yWWRqCmFEMlB2dVQvUzM4K2hFR2FXa2ZiUzkvbE1JK09YV1FWaXR0MktzeWZWeWNPQ1FYZlloVkJvTGFRWlRLeitkRlIKS0NCcXpVVFZ1UHdzaHFTMzJCV2FTc08vcWlKQXhhM1ZSTm9oelNWVEx2YW1BVHJ3QXRpbjM5aHRyc05ZM3R2cAowbEZTVnRrWUVaTitCUTVVYmFiQk52M0dMemVzSFFBOFRHSUNhYmlaRnZnYk9lWXVVRWMvR2ZyOWQxT2JGRzFPCjdrQ0lobDR1VXBvRndSdmJMQWdNQkFBR2pZREJlTUE0R0ExVWREd0VCL3dRRUF3SUZvREFkQmdOVkhTVUVGakEKVUJnZ3JCZ0VGQlFjREFRWUlLd1lCQlFVSEF3SXdEQVlEVlIwVEFRSC9CQUl3QURBZkJnTlZIU01FR0RBV2dCUQplbGRqRlptdUtDMjBKT0xDeFRmUVphTHg3YmpBTkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQWhQQUlRaHRwSkxPCjdISjcvSnAySEtBM3BZWmxodmUza1k4bUw1a1Y2VjRaOVpSSk9IbGRZUnZjaUxsSlg3c0tua3R5d3JuZ3kvNWkKY0JydElkQXhoNmtqYUR0L1RLTmU3Yy84emJqaVhNZUN4TGh5YllHbVJOb3F3U3p6SmdtaWZHeXVBMmo3VEpPYgpWMkNhR0FRNjhFcHNjWHZXOFROb0VkazdDWThaRjh6ZTg5TFZFMXhGY1IvUXZUNnNRVUlZSThtMHdOMUVmOTNKCkN0VmhYQWRhQ1VXbkNhYUxSQ2xVWkdtVGo4YjNGZzYzbTFMYjgzdW9UOXJjNWdYdkRBNmw4b3BqVmZDWjFFODEKbjR3dFJTMDBrdlZ0VHhEM3RPdGNYWkx5czlBdWpSRkFoYXE3a2ZEc1REK21XNUp1ZjQyNCs0V0RxT21ud0dYWgp2MFFkTGFndXlRQT09Ci0tLS0tRU5EQ0VSVElGSUNBVEUtLS0tLQ==
File renamed without changes.
45 changes: 45 additions & 0 deletions deployment_scripts/concourse_cd/pv.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: concourse-pv-1
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/data"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: concourse-pv-2
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/concourse-work-dir"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: concourse-pv-3
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/data"
23 changes: 23 additions & 0 deletions deployment_scripts/concourse_cd/pv1.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
apiVersion: v1
kind: PersistentVolume
metadata:
name: concourse-pv
spec:
capacity:
storage: 2Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: local-storage
local:
path: /mnt/disks/ssd1
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- master
- worker1
Loading

0 comments on commit e3de469

Please sign in to comment.