Bugfix: DB refresh - python version

- Python 3.9-3.10 got obsoleted for macos, so we switched the DB refresh (objects API) runner to use ubuntu instead. It hasn't needed MacOS in a long time or ever. - For the DB refresh (datasets API), we changed to 3.11. It still remains on the macos runner. - Update: Deleted some obsolete comments. General - Codestyle, documentation, and comment / todo tweaks in dataset_upload.py
jhu-bids · Apr 24, 2024 · 6331ef2 · 6331ef2
1 parent 207fe9c
commit 6331ef2
Show file tree

Hide file tree

Showing 10 changed files with 1,377 additions and 1,391 deletions.
diff --git a/.github/workflows/backend_dev.yml b/.github/workflows/backend_dev.yml
@@ -1,88 +1,87 @@
-# Docs for the Azure Web Apps Deploy action: https://github.com/Azure/webapps-deploy
-# More GitHub Actions for Azure: https://github.com/Azure/actions
-# More info on Python, GitHub Actions, and Azure App Service: https://aka.ms/python-webapps-actions
-# A good guide for Python Azure action: https://azure.github.io/AppService/2020/12/11/cicd-for-python-apps.html
-# The actual command that runs to initiate our servers on dev/prod isn't shown in the GH action. Instead, go to the following URL, and then click the "General Settings" tab:
-# Dev: https://portal.azure.com/#@live.johnshopkins.edu/resource/subscriptions/fe24df19-d251-4821-9a6f-f037c93d7e47/resourceGroups/jh-termhub-webapp-rg/providers/Microsoft.Web/sites/termhub/slots/dev/configuration
-# Prod: https://portal.azure.com/#@live.johnshopkins.edu/resource/subscriptions/fe24df19-d251-4821-9a6f-f037c93d7e47/resourceGroups/JH-TERMHUB-WEBAPP-RG/providers/Microsoft.Web/sites/termhub/configuration
-name: Backend dev - build and deploy
-
-on:
-#  push:
-#    branches:
-#      - develop
-#  pull_request:
-#    types: [opened, synchronize, reopened, closed]
-#    branches:
-#      - main
-#      - develop
-  workflow_dispatch:
-
-jobs:
-  build:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v2
-        with:
-          submodules: true
-      - name: Set up Python version
-        uses: actions/setup-python@v2
-        with:
-          # Consider '3.10' or 'v3.10.0': https://github.com/actions/setup-python/issues/160
-          python-version: '3.9'
-      - name: Print commit hash & branch for rollbacks & troubleshooting
-        run: |
-          echo "Commit hash: ${{ github.sha }}"
-          echo "Branch: ${{ github.ref }}"
-      - name: 'Create env file'
-        run: |
-          mkdir env
-          echo "${{ secrets.ENV_FILE }}" > env/.env
-          echo "HOSTENV=dev" >> env/.env
-      - name: Create and start virtual environment
-        run: |
-          python3 -m venv venv
-          source venv/bin/activate
-      - name: Git Large File Store
-        run: |
-          git lfs install
-          cd termhub-vocab
-          git lfs pull
-      - name: Install dependencies
-        run: |
-          python -m pip install --upgrade pip
-          pip install --upgrade wheel
-          pip install --upgrade setuptools
-          pip install -r requirements.txt
-# todo: optional: run tests
-#      - name: Run tests
-#        run: python -m unittest discover -v
-      # About: https://azure.github.io/AppService/2020/12/11/cicd-for-python-apps.html
-      - name: Upload artifact for deployment jobs
-        uses: actions/upload-artifact@v2
-        with:
-          name: python-app
-          path: |
-            .
-            !venv/
-
-  deploy:
-    runs-on: ubuntu-latest
-    needs: build
-    environment:
-      name: 'Backend: Development'
-      url: ${{ steps.deploy-to-webapp.outputs.webapp-url }}
-    steps:
-      # About: https://azure.github.io/AppService/2020/12/11/cicd-for-python-apps.html
-      - name: Download artifact from build job
-        uses: actions/download-artifact@v2
-        with:
-          name: python-app
-          path: .
-      - name: 'Deploy to Azure Web App'
-        uses: azure/webapps-deploy@v2
-        id: deploy-to-webapp
-        with:
-          app-name: 'termhub'
-          slot-name: 'dev'
+# Docs for the Azure Web Apps Deploy action: https://github.com/Azure/webapps-deploy
+# More GitHub Actions for Azure: https://github.com/Azure/actions
+# More info on Python, GitHub Actions, and Azure App Service: https://aka.ms/python-webapps-actions
+# A good guide for Python Azure action: https://azure.github.io/AppService/2020/12/11/cicd-for-python-apps.html
+# The actual command that runs to initiate our servers on dev/prod isn't shown in the GH action. Instead, go to the following URL, and then click the "General Settings" tab:
+# Dev: https://portal.azure.com/#@live.johnshopkins.edu/resource/subscriptions/fe24df19-d251-4821-9a6f-f037c93d7e47/resourceGroups/jh-termhub-webapp-rg/providers/Microsoft.Web/sites/termhub/slots/dev/configuration
+# Prod: https://portal.azure.com/#@live.johnshopkins.edu/resource/subscriptions/fe24df19-d251-4821-9a6f-f037c93d7e47/resourceGroups/JH-TERMHUB-WEBAPP-RG/providers/Microsoft.Web/sites/termhub/configuration
+name: Backend dev - build and deploy
+
+on:
+#  push:
+#    branches:
+#      - develop
+#  pull_request:
+#    types: [opened, synchronize, reopened, closed]
+#    branches:
+#      - main
+#      - develop
+  workflow_dispatch:
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          submodules: true
+      - name: Set up Python version
+        uses: actions/setup-python@v2
+        with:
+          python-version: '3.9'
+      - name: Print commit hash & branch for rollbacks & troubleshooting
+        run: |
+          echo "Commit hash: ${{ github.sha }}"
+          echo "Branch: ${{ github.ref }}"
+      - name: 'Create env file'
+        run: |
+          mkdir env
+          echo "${{ secrets.ENV_FILE }}" > env/.env
+          echo "HOSTENV=dev" >> env/.env
+      - name: Create and start virtual environment
+        run: |
+          python3 -m venv venv
+          source venv/bin/activate
+      - name: Git Large File Store
+        run: |
+          git lfs install
+          cd termhub-vocab
+          git lfs pull
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install --upgrade wheel
+          pip install --upgrade setuptools
+          pip install -r requirements.txt
+# todo: optional: run tests
+#      - name: Run tests
+#        run: python -m unittest discover -v
+      # About: https://azure.github.io/AppService/2020/12/11/cicd-for-python-apps.html
+      - name: Upload artifact for deployment jobs
+        uses: actions/upload-artifact@v2
+        with:
+          name: python-app
+          path: |
+            .
+            !venv/
+
+  deploy:
+    runs-on: ubuntu-latest
+    needs: build
+    environment:
+      name: 'Backend: Development'
+      url: ${{ steps.deploy-to-webapp.outputs.webapp-url }}
+    steps:
+      # About: https://azure.github.io/AppService/2020/12/11/cicd-for-python-apps.html
+      - name: Download artifact from build job
+        uses: actions/download-artifact@v2
+        with:
+          name: python-app
+          path: .
+      - name: 'Deploy to Azure Web App'
+        uses: azure/webapps-deploy@v2
+        id: deploy-to-webapp
+        with:
+          app-name: 'termhub'
+          slot-name: 'dev'
           publish-profile: ${{ secrets.AZUREAPPSERVICE_PUBLISHPROFILE_DEV }}
diff --git a/.github/workflows/backend_prod.yml b/.github/workflows/backend_prod.yml
@@ -1,88 +1,87 @@
-# Docs for the Azure Web Apps Deploy action: https://github.com/Azure/webapps-deploy
-# More GitHub Actions for Azure: https://github.com/Azure/actions
-# More info on Python, GitHub Actions, and Azure App Service: https://aka.ms/python-webapps-actions
-# A good guide for Python Azure action: https://azure.github.io/AppService/2020/12/11/cicd-for-python-apps.html
-# The actual command that runs to initiate our servers on dev/prod isn't shown in the GH action. Instead, go to the following URL, and then click the "General Settings" tab:
-# Dev: https://portal.azure.com/#@live.johnshopkins.edu/resource/subscriptions/fe24df19-d251-4821-9a6f-f037c93d7e47/resourceGroups/jh-termhub-webapp-rg/providers/Microsoft.Web/sites/termhub/slots/dev/configuration
-# Prod: https://portal.azure.com/#@live.johnshopkins.edu/resource/subscriptions/fe24df19-d251-4821-9a6f-f037c93d7e47/resourceGroups/JH-TERMHUB-WEBAPP-RG/providers/Microsoft.Web/sites/termhub/configuration
-name: Backend prod - build and deploy
-
-# I believe this will do the deploy when merged. We don't want it to deploy prod when PR opened
-on:
-  push:
-    branches:
-      - main
-#  pull_request:
-#    types: [opened, synchronize, reopened, closed]
-#    branches:
-#      - main
-  workflow_dispatch:
-
-jobs:
-  build:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v2
-        with:
-          submodules: true
-      - name: Set up Python version
-        uses: actions/setup-python@v2
-        with:
-          # Consider '3.10' or 'v3.10.0': https://github.com/actions/setup-python/issues/160
-          python-version: '3.9'
-      - name: Print commit hash & branch for rollbacks & troubleshooting
-        run: |
-          echo "Commit hash: ${{ github.sha }}"
-          echo "Branch: ${{ github.ref }}"
-      - name: 'Create env file'
-        run: |
-          mkdir env
-          echo "${{ secrets.ENV_FILE }}" > env/.env
-          echo "HOSTENV=prod" >> env/.env
-      - name: Create and start virtual environment
-        run: |
-          python3 -m venv venv
-          source venv/bin/activate
-      - name: Git Large File Store
-        run: |
-          git lfs install
-          cd termhub-vocab
-          git lfs pull
-      - name: Install dependencies
-        run: |
-          python -m pip install --upgrade pip
-          pip install --upgrade wheel
-          pip install --upgrade setuptools
-          pip install -r requirements.txt
-# todo: optional: run tests
-#      - name: Run tests
-#        run: python -m unittest discover -v
-      # About: https://azure.github.io/AppService/2020/12/11/cicd-for-python-apps.html
-      - name: Upload artifact for deployment jobs
-        uses: actions/upload-artifact@v2
-        with:
-          name: python-app
-          path: |
-            .
-            !venv/
-
-  deploy:
-    runs-on: ubuntu-latest
-    needs: build
-    environment:
-      name: 'Backend: Production'
-      url: ${{ steps.deploy-to-webapp.outputs.webapp-url }}
-    steps:
-      # About: https://azure.github.io/AppService/2020/12/11/cicd-for-python-apps.html
-      - name: Download artifact from build job
-        uses: actions/download-artifact@v2
-        with:
-          name: python-app
-          path: .
-      - name: 'Deploy to Azure Web App'
-        uses: azure/webapps-deploy@v2
-        id: deploy-to-webapp
-        with:
-          app-name: 'termhub'
-          slot-name: 'production'
-          publish-profile: ${{ secrets.AZUREAPPSERVICE_PUBLISHPROFILE_01B978E0A1074AF5B9757FB5907ED5D2 }}
+# Docs for the Azure Web Apps Deploy action: https://github.com/Azure/webapps-deploy
+# More GitHub Actions for Azure: https://github.com/Azure/actions
+# More info on Python, GitHub Actions, and Azure App Service: https://aka.ms/python-webapps-actions
+# A good guide for Python Azure action: https://azure.github.io/AppService/2020/12/11/cicd-for-python-apps.html
+# The actual command that runs to initiate our servers on dev/prod isn't shown in the GH action. Instead, go to the following URL, and then click the "General Settings" tab:
+# Dev: https://portal.azure.com/#@live.johnshopkins.edu/resource/subscriptions/fe24df19-d251-4821-9a6f-f037c93d7e47/resourceGroups/jh-termhub-webapp-rg/providers/Microsoft.Web/sites/termhub/slots/dev/configuration
+# Prod: https://portal.azure.com/#@live.johnshopkins.edu/resource/subscriptions/fe24df19-d251-4821-9a6f-f037c93d7e47/resourceGroups/JH-TERMHUB-WEBAPP-RG/providers/Microsoft.Web/sites/termhub/configuration
+name: Backend prod - build and deploy
+
+# I believe this will do the deploy when merged. We don't want it to deploy prod when PR opened
+on:
+  push:
+    branches:
+      - main
+#  pull_request:
+#    types: [opened, synchronize, reopened, closed]
+#    branches:
+#      - main
+  workflow_dispatch:
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          submodules: true
+      - name: Set up Python version
+        uses: actions/setup-python@v2
+        with:
+          python-version: '3.9'
+      - name: Print commit hash & branch for rollbacks & troubleshooting
+        run: |
+          echo "Commit hash: ${{ github.sha }}"
+          echo "Branch: ${{ github.ref }}"
+      - name: 'Create env file'
+        run: |
+          mkdir env
+          echo "${{ secrets.ENV_FILE }}" > env/.env
+          echo "HOSTENV=prod" >> env/.env
+      - name: Create and start virtual environment
+        run: |
+          python3 -m venv venv
+          source venv/bin/activate
+      - name: Git Large File Store
+        run: |
+          git lfs install
+          cd termhub-vocab
+          git lfs pull
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install --upgrade wheel
+          pip install --upgrade setuptools
+          pip install -r requirements.txt
+# todo: optional: run tests
+#      - name: Run tests
+#        run: python -m unittest discover -v
+      # About: https://azure.github.io/AppService/2020/12/11/cicd-for-python-apps.html
+      - name: Upload artifact for deployment jobs
+        uses: actions/upload-artifact@v2
+        with:
+          name: python-app
+          path: |
+            .
+            !venv/
+
+  deploy:
+    runs-on: ubuntu-latest
+    needs: build
+    environment:
+      name: 'Backend: Production'
+      url: ${{ steps.deploy-to-webapp.outputs.webapp-url }}
+    steps:
+      # About: https://azure.github.io/AppService/2020/12/11/cicd-for-python-apps.html
+      - name: Download artifact from build job
+        uses: actions/download-artifact@v2
+        with:
+          name: python-app
+          path: .
+      - name: 'Deploy to Azure Web App'
+        uses: azure/webapps-deploy@v2
+        id: deploy-to-webapp
+        with:
+          app-name: 'termhub'
+          slot-name: 'production'
+          publish-profile: ${{ secrets.AZUREAPPSERVICE_PUBLISHPROFILE_01B978E0A1074AF5B9757FB5907ED5D2 }}
diff --git a/.github/workflows/kill_idle_cons.yml b/.github/workflows/kill_idle_cons.yml
@@ -16,9 +16,7 @@ jobs:
       - name: Set up Python version
         uses: actions/setup-python@v2
         with:
-          # Consider '3.10' or 'v3.10.0': https://github.com/actions/setup-python/issues/160
-          python-version: '3.9'  # Not sure why 3.9.7 here and 3.9 elsewhere. the .7 works on Mac, but not on Ubuntu
-
+          python-version: '3.9'
       - name: 'Create env file'
         run: |
           mkdir env

diff --git a/.github/workflows/refresh_counts.yml b/.github/workflows/refresh_counts.yml
@@ -21,7 +21,6 @@ jobs:
       - name: Set up Python version
         uses: actions/setup-python@v2
         with:
-          # Consider '3.10' or 'v3.10.0': https://github.com/actions/setup-python/issues/160
           python-version: '3.9'
       - name: Print commit hash & branch for rollbacks & troubleshooting
         run: |

diff --git a/.github/workflows/refresh_db.yml b/.github/workflows/refresh_db.yml
@@ -20,18 +20,15 @@ on:
 jobs:
   refresh-db:
 #    runs-on: ubuntu-18.04
-#    runs-on: ubuntu-latest   # not sure which is more current when resolving merge conflicts
-    # todo: macos-latest: I think was just needed for datasets API because of high memory. should be able to switch back to ubuntu
-    runs-on: macos-latest
+    runs-on: ubuntu-latest
     steps:
       # Set up
       - name: Checkout repository and submodules
         uses: actions/checkout@v2
       - name: Set up Python version
         uses: actions/setup-python@v2
         with:
-          # Consider '3.10' or 'v3.10.0': https://github.com/actions/setup-python/issues/160
-          python-version: '3.9.7'  # Not sure why 3.9.7 here and 3.9 elsewhere. the .7 works on mac, but not on ubuntu
+          python-version: '3.9'
       - name: Print commit hash & branch for rollbacks & troubleshooting
         run: |
           echo "Commit hash: ${{ github.sha }}"