Merge branch 'master' into label-ods-project-with-owner

.github/workflows/continuous-integration-workflow.yml

@@ -71,6 +71,7 @@ jobs:
     strategy:
       matrix:
         version: ['7.9', '8.2.0.32929'] # 7.9 = LTS, 8.2 = latest version
+        edition: ['community', 'enterprise']
     steps:
       -
         name: Checkout repository
@@ -82,7 +83,7 @@ jobs:
       -
         name: Run tests
         run: |
-          cd sonarqube && ./test.sh --sq-version=${{ matrix.version }}
+          cd sonarqube && ./test.sh --sq-version=${{ matrix.version }} --sq-edition=${{ matrix.edition }}
 
   nexus:
     name: Nexus tests

CHANGELOG.md

@@ -2,9 +2,33 @@
 
 ## Unreleased
 
+- Change FROM image of SonarQube to avoid build problems in the AdoptJDK11 ([994](https://github.com/opendevstack/ods-core/pull/994))
+- Fix port from 3.x for SonarQube libressl issue - change to openssl ([#996](https://github.com/opendevstack/ods-core/issues/996))
+
 ### Added
 - Assign the owner as a label to the project ([#946](https://github.com/opendevstack/ods-core/pull/946))
+- Install Aquasec scannercli on jenkins base image ([#976](https://github.com/opendevstack/ods-core/pull/976))
 - Add changelog enforcer as GitHub Action to workflow ([#891](https://github.com/opendevstack/ods-core/issues/891))
+- Narrow down system:authenticated permissions when creating new ODS project ([#942](https://github.com/opendevstack/ods-core/issues/942))
+- Added SonarQube test for commercial editions ([#978](https://github.com/opendevstack/ods-core/pull/978))
+- Added SonarQube apex plugin for enterprise and datacenter editions ([#977](https://github.com/opendevstack/ods-core/pull/977))
+- Add pub key parameter to buildbot ([#956](https://github.com/opendevstack/ods-core/pull/956))
+- Extends packer build to add a pub key as authorized key to odsbox ami image ([#953](https://github.com/opendevstack/ods-core/pull/953))
+- Add script to generate the OpenVPN client profile for the ODS in a box
+- Allow to configure database image for SonarQube ([#984](https://github.com/opendevstack/ods-core/pull/984))
+
+### Changed
+
+- ds-jupyter-notebook renamed to ds-jupyter-lab and upgrade to JupyterLab 3 ([#562](https://github.com/opendevstack/ods-quickstarters/issues/562))
+
+### Fixed
+- Preserve clusterIPs of services ([#983](https://github.com/opendevstack/ods-core/pull/983))
+- Use storageClassName instead of annotation ([#985](https://github.com/opendevstack/ods-core/pull/985))
+- Tailor detects drift in cluster IP addresses in OCP 4.7+ ([#683](https://github.com/opendevstack/ods-jenkins-shared-library/issues/683))
+
+### Removed
+
+- ds-ml-service deprecated and moved to extra-quickstarters ([#568](https://github.com/opendevstack/ods-quickstarters/issues/568))
 
 ## [3.0] - 2020-08-11
 
@@ -20,6 +44,7 @@
 - Add PHP plugin to Sonarqube ([#536](https://github.com/opendevstack/ods-core/issues/536))
 - add doc gen service and new selectors ([#515](https://github.com/opendevstack/ods-core/pull/515))
 - Add SonarQube readiness probe ([#495](https://github.com/opendevstack/ods-core/pull/495))
+- Add AWS quickstarter into the Prov-app config map ([#970](https://github.com/opendevstack/ods-core/pull/970))
 
 ### Changed
 - Updated start-and-follow-build script to wait for OpenShift build to complete sucessfully ([#939](https://github.com/opendevstack/ods-core/pull/939))

README.md

@@ -1,9 +1,9 @@
 # ODS core & infrastructure
 
 ![](https://github.com/opendevstack/ods-core/workflows/Continous%20Integration%20Tests/badge.svg?branch=master)
-![](https://327164e4f0dd.ngrok.io/images/buildStatus_master.svg)
-![](https://327164e4f0dd.ngrok.io/images/buildStatus_3.x.svg)
-![](https://327164e4f0dd.ngrok.io/images/buildStatus_feature_ods-devenv.svg)
+![](https://9659ca82cc90.ngrok.io/images/buildStatus_master.svg)
+![](https://9659ca82cc90.ngrok.io/images/buildStatus_3.x.svg)
+![](https://9659ca82cc90.ngrok.io/images/buildStatus_feature_ods-devenv.svg)
 
 ## Introduction
 OpenDevStack (ODS) Core houses all the central infrastructure components.
@@ -14,7 +14,7 @@ The extended, most up to date, user friendly documentation can be found @ [opend
 
 ## Contents
 1. [Jenkins master](jenkins/master) & base agent - the basis of the ODS build engine<br>
-The [base agent](jenkins/agent-base) provides plugins for Sonarqube, optionally Snyk, CNES, skopeo and is HTTP proxy aware.
+The [base agent](jenkins/agent-base) provides plugins for Sonarqube, optionally Snyk, AquaSec, CNES, skopeo and is HTTP proxy aware.
 Specific [quickstarters / boilerplates](https://github.com/opendevstack/ods-quickstarters/tree/master) require different technologies e.g. `gradle`, `NPM/Yarn` etc. to build, hence warrant their own `builder agents`. These `agents` are based on the ods `jenkins base agent` and are hosted in the [ods-quickstarter repository](https://github.com/opendevstack/ods-quickstarters/tree/master/common/jenkins-agents) - next to their respective [boilerplates](https://github.com/opendevstack/ods-quickstarters/tree/master). <br>During `jenkins` builds, instances/pods of those `builder / agent` images can be found within the project specific `cd` namespace.
 <br>*Deployment:* one global Jenkins instance in the central `ods` namespace
 
@@ -48,8 +48,9 @@ b) inside the [tests](tests) directory. </p> The tests can be started with `make
 1. [ODS Development Environment / ODS in a box](ods-devenv)<br>
 ODS also ships as Amazon AMI - ready to go. The scripts to create the AMI can be found in ods-devenv. These scripts can be used also be used to install a `developer` version of ODS on a plain linux vm. Simply execute [bootstrap.sh](ods-devenv/scripts/bootstrap.sh)
 
-## Current AMI build log
-[Current AMI build logs are available here.](https://327164e4f0dd.ngrok.io/images/current_log_master.tar.gz)
+## Current AMI build log (master and 3.x)
+[Master branch of current AMI build logs are available here.](https://9659ca82cc90.ngrok.io/images/current_log_master.tar.gz)<br>
+[3.x branch of current AMI build logs are available here.](https://9659ca82cc90.ngrok.io/images/current_log_3.x.tar.gz)
 
 Since the log files contain color coding, they are best viewed using a tool supporting color coding, like tail. E.g.:
 

configuration-sample/ods-core.env.sample

@@ -92,6 +92,11 @@ SONAR_CROWD_APPLICATION=sonarqube
 SONAR_CROWD_PASSWORD_B64=changeme
 
 # Postgres DB for SonarQube
+# Image to use for the PostgreSQL database. This needs to be compatible with
+# your SonarQube version, see https://docs.sonarqube.org/latest/requirements/requirements/.
+# Take care when upgrading either database or SQ version.
+# E.g. registry.redhat.io/rhel8/postgresql-12
+SONAR_DATABASE_IMAGE=docker-registry.default.svc:5000/openshift/postgresql:9.6
 # Connection string for JDBC. Typically this does not need to be changed.
 SONAR_DATABASE_JDBC_URL=jdbc:postgresql://sonarqube-postgresql:5432/sonarqube
 # Database name for SonarQube. Typically this does not need to be changed.
@@ -226,6 +231,14 @@ JENKINS_AGENT_DOCKERFILE_PATH=Dockerfile.rhel7
 # Latest tested version is v1.217.3.
 JENKINS_AGENT_BASE_SNYK_DISTRIBUTION_URL=https://github.com/snyk/snyk/releases/download/v1.217.3/snyk-linux
 
+# AquaSec CLI binary distribution url
+# Leave empty to avoid installing AquaSec.
+# Releases are published at https://download.aquasec.com/scanner
+# To Download the aquaSec scanner cli requires a valid account on aquasec.com
+# Latest tested version is 6.0.0
+# Example: https://<USER>:<PASSWORD>@download.aquasec.com/scanner/6.0.0/scannercli
+JENKINS_AGENT_BASE_AQUASEC_SCANNERCLI_URL=
+
 # Repository of shared library
 # You may also point to repository underneath REPO_BASE.
 SHARED_LIBRARY_REPOSITORY=https://github.com/opendevstack/ods-jenkins-shared-library.git

create-projects/create-projects.sh

@@ -128,9 +128,4 @@ else
   oc policy add-role-to-group view system:authenticated -n "${PROJECT_ID}-dev"
   oc policy add-role-to-group view system:authenticated -n "${PROJECT_ID}-test"
   oc policy add-role-to-group view system:authenticated -n "${PROJECT_ID}-cd"
-
-  echo "Allow all authenticated users to edit the project"
-  oc policy add-role-to-group edit system:authenticated -n "${PROJECT_ID}-dev"
-  oc policy add-role-to-group edit system:authenticated -n "${PROJECT_ID}-test"
-  oc policy add-role-to-group edit system:authenticated -n "${PROJECT_ID}-cd"
 fi

create-projects/tests/run.sh

@@ -47,10 +47,6 @@ oc mock --receive 'policy add-role-to-group view system:authenticated -n foo-dev
 oc mock --receive 'policy add-role-to-group view system:authenticated -n foo-test' --times 1
 oc mock --receive 'policy add-role-to-group view system:authenticated -n foo-cd' --times 1
 
-oc mock --receive 'policy add-role-to-group edit system:authenticated -n foo-dev' --times 1
-oc mock --receive 'policy add-role-to-group edit system:authenticated -n foo-test' --times 1
-oc mock --receive 'policy add-role-to-group edit system:authenticated -n foo-cd' --times 1
-
 ../create-projects.sh --project foo
 
 oc mock --verify
@@ -69,10 +65,6 @@ oc mock --receive 'policy add-role-to-group view system:authenticated -n foo-dev
 oc mock --receive 'policy add-role-to-group view system:authenticated -n foo-test' --times 1
 oc mock --receive 'policy add-role-to-group view system:authenticated -n foo-cd' --times 1
 
-oc mock --receive 'policy add-role-to-group edit system:authenticated -n foo-dev' --times 1
-oc mock --receive 'policy add-role-to-group edit system:authenticated -n foo-test' --times 1
-oc mock --receive 'policy add-role-to-group edit system:authenticated -n foo-cd' --times 1
-
 ../create-projects.sh --project foo --admins [email protected],[email protected] --groups=
 
 oc mock --verify
@@ -100,10 +92,6 @@ oc mock --receive 'policy add-role-to-group view system:authenticated -n foo-dev
 oc mock --receive 'policy add-role-to-group view system:authenticated -n foo-test' --times 0
 oc mock --receive 'policy add-role-to-group view system:authenticated -n foo-cd' --times 0
 
-oc mock --receive 'policy add-role-to-group edit system:authenticated -n foo-dev' --times 0
-oc mock --receive 'policy add-role-to-group edit system:authenticated -n foo-test' --times 0
-oc mock --receive 'policy add-role-to-group edit system:authenticated -n foo-cd' --times 0
-
 ../create-projects.sh --project foo --groups USERGROUP=foo,ADMINGROUP=bar,READONLYGROUP=baz
 
 oc mock --verify

docs/modules/update-guides/pages/4x.adoc

@@ -12,3 +12,7 @@ deployments (`jenkins` and `webhook-proxy`) to point to the new image tags
 (`4.x`), and trigger a deployment.
 
 include::jenkins-shared-library:partial$update-to-4x.adoc[]
+
+== Notes about quickstarters
+
+include::quickstarters:partial$notes-4x.adoc[]

jenkins/agent-base/Dockerfile.rhel7

@@ -14,6 +14,7 @@ ENV SONAR_SCANNER_VERSION=3.1.0.1141 \
 
 ARG APP_DNS
 ARG SNYK_DISTRIBUTION_URL
+ARG AQUASEC_SCANNERCLI_URL
 
 RUN yum -y install \
     openssl \
@@ -86,6 +87,16 @@ RUN if [ -z $SNYK_DISTRIBUTION_URL ] ; then echo 'Skipping snyk installation!' ;
     && echo 'Snyk installation completed!'; \
     fi
 
+# Optionally install Aquasec.
+RUN if [ -z $AQUASEC_SCANNERCLI_URL ] ; then echo 'Skipping AquaSec installation!' ; else echo 'Installing AquaSec... getting binary from' $AQUASEC_SCANNERCLI_URL \
+    && wget $AQUASEC_SCANNERCLI_URL -O aquasec \
+    && mv aquasec /usr/local/bin \
+    && chmod +rwx /usr/local/bin/aquasec \
+    && echo 'AquaSec CLI version:' \
+    && aquasec version \
+    && echo 'AquaSec installation completed!'; \
+    fi
+
 # Set java proxy var.
 COPY set_java_proxy.sh /tmp/set_java_proxy.sh
 RUN . /tmp/set_java_proxy.sh && echo $JAVA_OPTS

jenkins/agent-base/Dockerfile.ubi8

@@ -4,7 +4,7 @@ SHELL ["/bin/bash", "-o", "pipefail", "-c"]
 
 ENV SONAR_SCANNER_VERSION=3.1.0.1141 \
     CNES_REPORT_VERSION=3.2.2 \
-    TAILOR_VERSION=1.3.0 \
+    TAILOR_VERSION=1.3.2 \
     HELM_VERSION=3.4.1 \
     HELM_PLUGIN_DIFF_VERSION=3.1.3 \
     HELM_PLUGIN_SECRETS_VERSION=3.3.5 \
@@ -13,6 +13,7 @@ ENV SONAR_SCANNER_VERSION=3.1.0.1141 \
 
 ARG APP_DNS
 ARG SNYK_DISTRIBUTION_URL
+ARG AQUASEC_SCANNERCLI_URL
 
 # Add CentOS 8 repositories.
 COPY yum.repos.d/centos8.repo /etc/yum.repos.d/centos8.repo
@@ -81,6 +82,16 @@ RUN if [ -z $SNYK_DISTRIBUTION_URL ] ; then echo 'Skipping snyk installation!' ;
     && echo 'Snyk installation completed!'; \
     fi
 
+# Optionally install Aquasec.
+RUN if [ -z $AQUASEC_SCANNERCLI_URL ] ; then echo 'Skipping AquaSec installation!' ; else echo 'Installing AquaSec... getting binary from' $AQUASEC_SCANNERCLI_URL \
+    && wget $AQUASEC_SCANNERCLI_URL -O aquasec \
+    && mv aquasec /usr/local/bin \
+    && chmod +rwx /usr/local/bin/aquasec \
+    && echo 'AquaSec CLI version:' \
+    && aquasec version \
+    && echo 'AquaSec installation completed!'; \
+    fi
+
 # Set java proxy var.
 COPY set_java_proxy.sh /tmp/set_java_proxy.sh
 RUN . /tmp/set_java_proxy.sh && echo $JAVA_OPTS

jenkins/master/Dockerfile.ubi8

@@ -9,7 +9,7 @@ ARG ODS_IMAGE_TAG
 ARG SONAR_EDITION
 ARG SONAR_VERSION
 ARG APP_DNS
-ENV TAILOR_VERSION=1.3.0
+ENV TAILOR_VERSION=1.3.2
 
 USER root
 

jenkins/ocp-config/build/bc.yml

@@ -50,6 +50,8 @@ parameters:
   description: OpenShift application base dns - used for grabbing the root ca and adding into the agent
 - name: JENKINS_AGENT_BASE_SNYK_DISTRIBUTION_URL
   description: optional uri that points to the snyk binary distribution (i.e. https://github.com/snyk/snyk/releases/download/v1.180.1/snyk-linux)
+- name: JENKINS_AGENT_BASE_AQUASEC_SCANNERCLI_URL
+  description: optional uri that points to the aquasec binary distribution (i.e. https://download.aquasec.com/scanner/6.0.0/scannercli)
 - name: SONAR_EDITION
   description: SonarQube edition. One of "community", "developer", "enterprise" or "datacenter".
 - name: SONAR_VERSION
@@ -155,6 +157,8 @@ objects:
             value: ${APP_DNS}
           - name: SNYK_DISTRIBUTION_URL
             value: ${JENKINS_AGENT_BASE_SNYK_DISTRIBUTION_URL}
+          - name: AQUASEC_SCANNERCLI_URL
+            value: ${JENKINS_AGENT_BASE_AQUASEC_SCANNERCLI_URL}
         from:
           kind: DockerImage
           name: ${JENKINS_AGENT_BASE_FROM_IMAGE}

jenkins/ocp-config/deploy/Tailorfile

@@ -5,6 +5,7 @@ ignore-unknown-parameters true
 selector template=ods-jenkins-template
 
 preserve pvc:/spec/volumeMode
+preserve svc:/spec/clusterIPs
 
 svc,route,secret,pvc,dc,rolebinding,serviceaccount
 

nexus/ocp-config/Tailorfile

@@ -4,5 +4,7 @@ param-file ../../../ods-configuration/ods-core.env
 ignore-unknown-parameters true
 preserve-immutable-fields true
 preserve pvc:/spec/volumeMode
+preserve svc:/spec/clusterIPs
+preserve pvc:/metadata/annotations/volume.beta.kubernetes.io/storage-class
 
 dc,is,pvc,route,svc

nexus/ocp-config/pvc.yml

@@ -21,7 +21,6 @@ objects:
   kind: PersistentVolumeClaim
   metadata:
     annotations:
-      volume.beta.kubernetes.io/storage-class: ${STORAGE_CLASS_DATA}
       volume.beta.kubernetes.io/storage-provisioner: ${STORAGE_PROVISIONER}
     finalizers:
     - kubernetes.io/pvc-protection
@@ -34,12 +33,12 @@ objects:
     resources:
       requests:
         storage: ${NEXUS_DATA_CAPACITY}
+    storageClassName: ${STORAGE_CLASS_DATA}
     volumeMode: Filesystem
 - apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     annotations:
-      volume.beta.kubernetes.io/storage-class: ${STORAGE_CLASS_BACKUP}
       volume.beta.kubernetes.io/storage-provisioner: ${STORAGE_PROVISIONER}
     finalizers:
     - kubernetes.io/pvc-protection
@@ -52,4 +51,5 @@ objects:
     resources:
       requests:
         storage: ${NEXUS_BACKUP_CAPACITY}
+    storageClassName: ${STORAGE_CLASS_BACKUP}
     volumeMode: Filesystem

ods-devenv/buildbot/scripts/.buildbotrc

@@ -31,3 +31,7 @@ build_path="${HOME}/opendevstack/builds"
 
 # path to build result resources
 build_result_path="${HOME}/opendevstack/packer_build_result"
+
+# path to public key to be injected in AMI images
+# e.g pub-key=~/.ssh/jenkins.pub
+pub-key=

ods-devenv/buildbot/scripts/runAmiBuild.sh

@@ -1,10 +1,11 @@
 #!/usr/bin/env bash
 # Per default this script will build a new AMI on AWS for the ODS master branch.
 # The branch can be overridden to build e.g. 3.x. Caveat: the stated branch must
-# exist on each of the following repositories: 
+# exist on each of the following repositories:
 # ods-core, ods-quickstarters, ods-jenkins-shared-library, not necessarily in ods-document-generation-templates
 set -exu
-
+echo "Running runAmiBuild.sh"
+pub_key=
 targetGitRef="master"
 readonly repository=ods-core
 readonly odsCoreClonePath=https://github.com/opendevstack/ods-core
@@ -18,6 +19,7 @@ while [[ "$#" -gt 0 ]]; do
   --instance_type) instanceType="$2"; shift;;
   --aws_access_key) awsAccessKey="$2"; shift;;
   --aws_secret_access_key) awsSecretAccessKey="$2"; shift;;
+  --pub-key) pub_key="$2"; shift;;
 
 esac; shift; done
 
@@ -53,7 +55,7 @@ rm -f "${logPath}/current_${targetGitRefForPath}.log"
 ln -s "${logFile}" "${logPath}/current_${targetGitRefForPath}.log"
 
 # run packer build
-time bash 2>&1 ods-devenv/packer/create_ods_box_image.sh --target create_ods_box_ami --aws-access-key "${awsAccessKey:?}" --aws-secret-key "${awsSecretAccessKey:?}" --ods-branch "${targetGitRef}" --instance-type "${instanceType:?}" | tee "${logFile}"
+time bash 2>&1 ods-devenv/packer/create_ods_box_image.sh --target create_ods_box_ami --aws-access-key "${awsAccessKey:?}" --aws-secret-key "${awsSecretAccessKey:?}" --ods-branch "${targetGitRef}" --pub-key "${pub_key}" --instance-type "${instanceType:?}" | tee "${logFile}"
 
 # clean up after build
 cd ../..