[fix] hadolint warnings to base and rstudio Dockerfiles

These changes shouldn't have any functional impact. [fix] CI for the images checks based on recent updates [fix] This fixes an inconsistency with the kustomize params Inconsistency for codeserver notebook parameters. There was upstream change recently that probably not got properly backported to downstream, see [1,2]. * [1] opendatahub-io#524 * [2] red-hat-data-services@ceb3dc8 Set the rstudio builds with the branch rhoai-2.10 Signed-off-by: Harshad Reddy Nalla <[email protected]> Update image commits for release N via digest-updater-9215094498 GitHub action Update images for release N via digest-updater-9215094498 GitHub action Update file via digest-updater-9213110410 GitHub action Allow runtime script to cp the package from bin to Rpackage default path Update codeflare-sdk version on imagestreams annotations (opendatahub-io#235) * Update codeflare-sdk version on imagestreams annotations * fix kfp version in the annotation for tensorflow Co-authored-by: Jan Stourac <[email protected]> Update images for release N and N-1 with 2024a commit db8bd76 Update file via digest-updater-8806399693 GitHub action Update annotations for kfp (opendatahub-io#229) Update image commits for release N via digest-updater-8665769109 GitHub action Update images for release N via digest-updater-8665769109 GitHub action Update manifest for code-freeze 2.9 Signed-off-by: Harshad Reddy Nalla <[email protected]> Update image commits for release N-1 via digest-updater-8581586298 GitHub action Update images for release N-1 via digest-updater-8581586298 GitHub action Update image commits for release N via digest-updater-8581586298 GitHub action Update images for release N via digest-updater-8581586298 GitHub action Update file via digest-updater-8577545330 GitHub action Fix the runtime updater github action branch 2024a Signed-off-by: Harshad Reddy Nalla <[email protected]> Fix the runtime updater github action Signed-off-by: Harshad Reddy Nalla <[email protected]> Remove the intel based image from the overlay as its ODH only - Fix the typo in the datascience notebook Signed-off-by: Harshad Reddy Nalla <[email protected]> Revert nginx version to 1.22 since 1.24 is not available on rhel yet update cuda layer for RHEL to 12.1 Add runtimes workflow updater Update digest updater workflow Fix check-params-env test with the new changes (opendatahub-io#196) Update Imagesteam for habana 1.13 Update runtime images with e1aee40 build commit Update the manifests to retain old image in shadow state Signed-off-by: Harshad Reddy Nalla <[email protected]> Update image commits for release N via digest-updater-8319475892 GitHub action Update images for release N via digest-updater-8319475892 GitHub action Update Codeserver ImageStream for the 2024a release inclusion (opendatahub-io#173) * Update Codeserver imagestream with the 2024a release Co-authored-by: Harshad Reddy Nalla <[email protected]> Fix test file for the trustyai image I don't really understand how and why this file was broken by this commit aac0662 . Our CI check notifies that something is broken in the file. Update Imagestreams with in favor of the new release 2024.1 (opendatahub-io#175) Co-authored-by: Harshad Reddy Nalla <[email protected]> Update digest updater workflow in favor 2024a release Remove opendatahub.io/dashboard: 'true' label from rstudio ImageSteams Create sync workflow for the release-2024a Format yaml and json files to statisfy code-quality - Fix validation of the params-env Signed-off-by: Harshad Reddy Nalla <[email protected]> Update RStudio-server Dockefile for RHEL version Fix library path version on rsession.conf file hot fix: bump cuda resources HotFix: Remove the annotation notebook-images=true from RStudio imagestreams Signed-off-by: Harshad Reddy Nalla <[email protected]> Fix user R library path version Update image commits for release N via digest-updater-7846262944 GitHub action Update images for release N via digest-updater-7846262944 GitHub action Remove the R-package install from workbench Co-authored-by: Diamond Bryant <[email protected]> Signed-off-by: Harshad Reddy Nalla <[email protected]> Fix naming for RStudio Server on rhel flavor Increase build resources for R Studio buildconfigs Mount the secret on the buildConfig instead of using ENVs to avoid their expose on the logs Adjust the imagestream annotation for codeflare-sdk upgrade Signed-off-by: Harshad Reddy Nalla <[email protected]> Update image commits for release N via digest-updater-7761501425 GitHub action Update images for release N via digest-updater-7761501425 GitHub action Add optional: true option for the base and server url envs Add BuildConfiguration objects to build RStudio and CUDA RStudio images on OCP cluster Fixes on the CUDA Dockerfile setup r-studio based with rhel9 base image (opendatahub-io#125) * Content of R Studio switched to the rhel based image. Add rhel9 base image [Fix] typo in logging of the `notebook-digest-updater.yaml` Update image commits for release N via digest-updater-7533330854 GitHub action Update images for release N-1 via digest-updater-7533330854 GitHub action Update images for release N via digest-updater-7533330854 GitHub action Fix: update the code-server and annotation Signed-off-by: Harshad Reddy Nalla <[email protected]> Co-authored-by: aTheo <[email protected]> Incorporate VSCode on Downstream (opendatahub-io#105) Co-authored-by: Harshad Reddy Nalla <[email protected]> hot-fix: Fix the tensorflow imagestream by removing the trailing space Signed-off-by: Harshad Reddy Nalla <[email protected]> hot-fix: Fix the imagestream minimal-cuda sha Signed-off-by: Harshad Reddy Nalla <[email protected]> hot-fix: Fixed imagestream with CVE 44487 changes Signed-off-by: Harshad Reddy Nalla <[email protected]> hot-fix: update the base ubi9 images for cve 44487 fix Signed-off-by: Harshad Reddy Nalla <[email protected]> hot-fix: CVE 44487 fix with libnghttp2 Signed-off-by: Harshad Reddy Nalla <[email protected]> Update the pipfile.lock via the weekly workflow action chores: Update the runtime image with the commit: 8bda2fa Signed-off-by: Harshad Reddy Nalla <[email protected]> Patch the imagestream by removing habana 1.11.0 Signed-off-by: Harshad Reddy Nalla <[email protected]> Update the runtime image with the commit: 8bda2fa on main Update images for release N via digest-updater-6655629712 GitHub action Fix the annotation and additional recommended-true Signed-off-by: Harshad Reddy Nalla <[email protected]> Patch the imagestream to have same name as in odh-manifests Signed-off-by: Harshad Reddy Nalla <[email protected]> Fix digest updater from failing if there are no updates on the image streams Fix the path to the params.env file Several fixes Upgrade the notebook images with 2023b and 2023a images Signed-off-by: Harshad Reddy Nalla <[email protected]> Include only sync github workflow on the main branch Signed-off-by: Harshad Reddy Nalla <[email protected]>
jiridanek · Jun 6, 2024 · f7d2b08 · f7d2b08
1 parent 66050e6
commit f7d2b08
Show file tree

Hide file tree

Showing 81 changed files with 4,556 additions and 3,218 deletions.
diff --git a/.github/workflows/notebook-digest-updater.yaml b/.github/workflows/notebook-digest-updater.yaml
@@ -0,0 +1,214 @@
+---
+# The aim of this GitHub workflow is to update the params.env file with the latest digest.
+name: Update notebook image build commit hashes
+on:  # yamllint disable-line rule:truthy
+  workflow_dispatch:
+    inputs:
+      branch:
+        required: true
+        description: "Provide branch name: "
+# Put the scheduler on comment until automate the full release procedure
+#   schedule:
+#     - cron:  "0 0 * * 5" #Scheduled every Friday
+env:
+  DIGEST_UPDATER_BRANCH: digest-updater-${{ github.run_id }}
+  BRANCH_NAME: ${{ github.event.inputs.branch || 'master' }}
+  RELEASE_VERSION_N: 2024a
+  RELEASE_VERSION_N_1: 2023b
+jobs:
+  initialize:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+    steps:
+      - name: Install Skopeo CLI
+        shell: bash
+        run: |
+          sudo apt-get -y update
+          sudo apt-get -y install skopeo
+
+      # Checkout the branch
+      - name: Checkout branch
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ env.BRANCH_NAME }}
+
+      # Create a new branch
+      - name: Create a new branch
+        run: |
+         echo ${{ env.DIGEST_UPDATER_BRANCH }}
+         git checkout -b ${{ env.DIGEST_UPDATER_BRANCH }}
+         git push --set-upstream origin ${{ env.DIGEST_UPDATER_BRANCH }}
+
+  update-n-version:
+    needs: [initialize]
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+    steps:
+      - name: Configure Git
+        run: |
+         git config --global user.email "github-actions[bot]@users.noreply.github.com"
+         git config --global user.name "GitHub Actions"
+
+        # Get latest build commit from the https://github.com/red-hat-data-services/notebooks/${release_branch} using this as identifier for the latest tag name
+      - name: Retrive latest commit hash from the release branch
+        id: hash-n
+        shell: bash
+        run: |
+          PAYLOAD=$(curl --silent -H 'Accept: application/vnd.github.v4.raw' https://api.github.com/repos/red-hat-data-services/notebooks/commits?sha=release-$RELEASE_VERSION_N&per_page=1)
+          echo "HASH_N=$(echo $PAYLOAD | jq -r '.[0].sha[0:7]')" >> ${GITHUB_OUTPUT}
+
+      # Checkout the release branch to apply the updates
+      - name: Checkout release branch
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ env.DIGEST_UPDATER_BRANCH }}
+
+      - name: Fetch digest, and update the params.env file
+        shell: bash
+        run: |
+              echo Latest commit is: ${{ steps.hash-n.outputs.HASH_N }} on ${{ env.RELEASE_VERSION_N}}
+              IMAGES=("odh-minimal-notebook-image-n" "odh-minimal-gpu-notebook-image-n" "odh-pytorch-gpu-notebook-image-n" "odh-generic-data-science-notebook-image-n" "odh-tensorflow-gpu-notebook-image-n" "odh-trustyai-notebook-image-n" "odh-codeserver-notebook-image-n")
+              REGEXES=("v2-${{ env.RELEASE_VERSION_N }}-\d{8}+-${{ steps.hash-n.outputs.HASH_N }}" "cuda-[a-z]+-minimal-[a-z0-9]+-[a-z]+-3.9-${{ env.RELEASE_VERSION_N }}-\d{8}-${{ steps.hash-n.outputs.HASH_N }}" "v2-${{ env.RELEASE_VERSION_N }}-\d{8}+-${{ steps.hash-n.outputs.HASH_N }}" \
+                       "v2-${{ env.RELEASE_VERSION_N }}-\d{8}+-${{ steps.hash-n.outputs.HASH_N }}" "cuda-[a-z]+-tensorflow-[a-z0-9]+-[a-z]+-3.9-${{ env.RELEASE_VERSION_N }}-\d{8}-${{ steps.hash-n.outputs.HASH_N }}" "v2-${{ env.RELEASE_VERSION_N }}-\d{8}+-${{ steps.hash-n.outputs.HASH_N }}" \
+                       "codeserver-[a-z0-9]+-[a-z]+-3.9-${{ env.RELEASE_VERSION_N }}-\d{8}-${{ steps.hash-n.outputs.HASH_N }}")
+
+              for ((i=0;i<${#IMAGES[@]};++i)); do
+                image=${IMAGES[$i]}
+                echo "CHECKING: " $image
+                regex=${REGEXES[$i]}
+                img=$(cat manifests/base/params.env | grep -E "${image}=" | cut -d '=' -f2)
+                registry=$(echo $img | cut -d '@' -f1)
+                latest_tag=$(skopeo inspect docker://$img | jq -r --arg regex "$regex" '.RepoTags | map(select(. | test($regex))) | .[0]')
+                digest=$(skopeo inspect docker://$registry:$latest_tag | jq .Digest | tr -d '"')
+                output=$registry@$digest
+                echo "NEW: " $output
+                sed -i "s|${image}=.*|${image}=$output|" manifests/base/params.env
+              done
+              if [[ $(git status --porcelain | wc -l) -gt 0 ]]; then
+                git fetch origin ${{ env.DIGEST_UPDATER_BRANCH }} && git pull origin ${{ env.DIGEST_UPDATER_BRANCH }} && git add manifests/base/params.env && git commit -m "Update images for release N via ${{ env.DIGEST_UPDATER_BRANCH }} GitHub action" && git push origin ${{ env.DIGEST_UPDATER_BRANCH }}
+              else
+                echo "There were no changes detected in the images for the ${{ env.RELEASE_VERSION_N}}"
+              fi
+
+      - name: Fetch digest, and update the commit.env file
+        run: |
+            echo Latest commit is: ${{ steps.hash-n.outputs.HASH_N }} on ${{ env.RELEASE_VERSION_N}}
+            COMMIT=("odh-minimal-notebook-image-commit-n"
+             "odh-minimal-gpu-notebook-image-commit-n"
+             "odh-pytorch-gpu-notebook-image-commit-n"
+             "odh-generic-data-science-notebook-image-commit-n"
+             "odh-tensorflow-gpu-notebook-image-commit-n"
+             "odh-trustyai-notebook-image-commit-n"
+             "odh-codeserver-notebook-image-commit-n")
+
+            for val in "${COMMIT[@]}"; do
+              echo $val
+              sed -i "s|${val}=.*|${val}=${{ steps.hash-n.outputs.HASH_N }}|" manifests/base/commit.env
+            done
+            if [[ $(git status --porcelain | wc -l) -gt 0 ]]; then
+              git fetch origin  ${{ env.DIGEST_UPDATER_BRANCH }} && git pull origin  ${{ env.DIGEST_UPDATER_BRANCH }} && git add manifests/base/commit.env && git commit -m "Update image commits for release N via ${{ env.DIGEST_UPDATER_BRANCH }} GitHub action" && git push origin  ${{ env.DIGEST_UPDATER_BRANCH }}
+            else
+              echo "There were no changes detected in the images for the ${{ env.RELEASE_VERSION_N}}"
+            fi
+
+  update-n-1-version:
+    needs: [initialize, update-n-version]
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+    steps:
+      - name: Configure Git
+        run: |
+         git config --global user.email "github-actions[bot]@users.noreply.github.com"
+         git config --global user.name "GitHub Actions"
+
+      # Get latest build commit from the https://github.com/red-hat-data-services/notebooks/${release_branch} using this as identifier for the latest tag name
+      - name: Retrive latest commit hash from the release branch
+        id: hash-n-1
+        shell: bash
+        run: |
+          PAYLOAD=$(curl --silent -H 'Accept: application/vnd.github.v4.raw' https://api.github.com/repos/red-hat-data-services/notebooks/commits?sha=release-$RELEASE_VERSION_N_1&per_page=1)
+          echo "HASH_N_1=$(echo $PAYLOAD | jq -r '.[0].sha[0:7]')" >> ${GITHUB_OUTPUT}
+
+      # Checkout the release branch to apply the updates
+      - name: Checkout release branch
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ env.DIGEST_UPDATER_BRANCH }}
+
+      - name: Fetch digest, and update the params.env file
+        shell: bash
+        run: |
+              echo Latest commit is: ${{ steps.hash-n-1.outputs.HASH_N_1 }} on ${{ env.RELEASE_VERSION_N_1}}
+              IMAGES=("odh-minimal-notebook-image-n-1" "odh-minimal-gpu-notebook-image-n-1" "odh-pytorch-gpu-notebook-image-n-1" "odh-generic-data-science-notebook-image-n-1" "odh-tensorflow-gpu-notebook-image-n-1" "odh-trustyai-notebook-image-n-1" "odh-codeserver-notebook-image-n-1")
+              REGEXES=("v2-${{ env.RELEASE_VERSION_N_1 }}-\d{8}+-${{ steps.hash-n-1.outputs.HASH_N_1 }}" "cuda-[a-z]+-minimal-[a-z0-9]+-[a-z]+-3.9-${{ env.RELEASE_VERSION_N_1 }}-\d{8}-${{ steps.hash-n-1.outputs.HASH_N_1 }}" "v2-${{ env.RELEASE_VERSION_N_1 }}-\d{8}+-${{ steps.hash-n-1.outputs.HASH_N_1 }}" \
+                       "v2-${{ env.RELEASE_VERSION_N_1 }}-\d{8}+-${{ steps.hash-n-1.outputs.HASH_N_1 }}" "cuda-[a-z]+-tensorflow-[a-z0-9]+-[a-z]+-3.9-${{ env.RELEASE_VERSION_N_1 }}-\d{8}-${{ steps.hash-n-1.outputs.HASH_N_1 }}" "v2-${{ env.RELEASE_VERSION_N_1 }}-\d{8}+-${{ steps.hash-n-1.outputs.HASH_N_1 }}" \
+                       "codeserver-[a-z0-9]+-[a-z]+-3.9-${{ env.RELEASE_VERSION_N_1 }}-\d{8}-${{ steps.hash-n-1.outputs.HASH_N_1 }}")
+
+              for ((i=0;i<${#IMAGES[@]};++i)); do
+                image=${IMAGES[$i]}
+                echo "CHECKING: " $image
+                regex=${REGEXES[$i]}
+                img=$(cat manifests/base/params.env | grep -E "${image}=" | cut -d '=' -f2)
+                registry=$(echo $img | cut -d '@' -f1)
+                latest_tag=$(skopeo inspect docker://$img | jq -r --arg regex "$regex" '.RepoTags | map(select(. | test($regex))) | .[0]')
+                digest=$(skopeo inspect docker://$registry:$latest_tag | jq .Digest | tr -d '"')
+                output=$registry@$digest
+                echo "NEW: " $output
+                sed -i "s|${image}=.*|${image}=$output|" manifests/base/params.env
+              done
+              if [[ $(git status --porcelain | wc -l) -gt 0 ]]; then
+                git fetch origin  ${{ env.DIGEST_UPDATER_BRANCH }} && git pull origin  ${{ env.DIGEST_UPDATER_BRANCH }} && git add manifests/base/params.env && git commit -m "Update images for release N-1 via ${{ env.DIGEST_UPDATER_BRANCH }} GitHub action" && git push origin  ${{ env.DIGEST_UPDATER_BRANCH }}
+              else
+                echo "There were no changes detected in the images for the ${{ env.RELEASE_VERSION_N}}"
+              fi
+
+      - name: Fetch digest, and update the commit.env file
+        run: |
+              echo Latest commit is: ${{ steps.hash-n-1.outputs.HASH_N_1 }} on ${{ env.RELEASE_VERSION_N_1}}
+              COMMIT=("odh-minimal-notebook-image-commit-n-1"
+               "odh-minimal-gpu-notebook-image-commit-n-1"
+               "odh-pytorch-gpu-notebook-image-commit-n-1"
+               "odh-generic-data-science-notebook-image-commit-n-1"
+               "odh-tensorflow-gpu-notebook-image-commit-n-1"
+               "odh-trustyai-notebook-image-commit-n-1"
+               "odh-codeserver-notebook-image-commit-n-1")
+
+              for val in "${COMMIT[@]}"; do
+                echo $val
+                sed -i "s|${val}=.*|${val}=${{ steps.hash-n-1.outputs.HASH_N_1 }}|" manifests/base/commit.env
+              done
+              if [[ $(git status --porcelain | wc -l) -gt 0 ]]; then
+                git fetch origin  ${{ env.DIGEST_UPDATER_BRANCH }} && git pull origin  ${{ env.DIGEST_UPDATER_BRANCH }} && git add manifests/base/commit.env && git commit -m "Update image commits for release N-1 via ${{ env.DIGEST_UPDATER_BRANCH }} GitHub action" && git push origin  ${{ env.DIGEST_UPDATER_BRANCH }}
+              else
+                echo "There were no changes detected in the images for the ${{ env.RELEASE_VERSION_N}}"
+              fi
+
+  open-pull-request:
+    needs: [update-n-version, update-n-1-version]
+    runs-on: ubuntu-latest
+    permissions:
+      pull-requests: write
+    steps:
+      - name: Checkout repo
+        uses: actions/checkout@v4
+
+      - name: pull-request
+        uses: repo-sync/pull-request@v2
+        with:
+          source_branch: ${{ env.DIGEST_UPDATER_BRANCH }}
+          destination_branch: ${{ env.BRANCH_NAME}}
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          pr_label: "automated pr"
+          pr_title: "[Digest Updater Action] Update Notebook Images"
+          pr_body: |
+            :rocket: This is an automated Pull Request.
+            Created by `/.github/workflows/notebooks-digest-updater-upstream.yaml`
+
+            This PR updates the following files:
+            - `manifests/base/params.env` file with the latest updated SHA digests of the notebooks (N & N-1).
+            - `manifests/base/commit.env` file with the latest commit (N & N-1).
+
+            :exclamation: **IMPORTANT NOTE**: Remember to delete the ` ${{ env.DIGEST_UPDATER_BRANCH }}` branch after merging the changes
diff --git a/.github/workflows/runtimes-digest-updater-upstream.yaml b/.github/workflows/runtimes-digest-updater-upstream.yaml