feat(Windows): refactor build process

[lemeurherve]

Refactoring of the Windows image build like https://github.com/jenkinsci/docker-agent, https://github.com/jenkinsci/docker-inbound-agent and https://github.com/jenkinsci/docker-ssh (this last one is WIP until I found out how to fix some recalcitrant tests).

This PR allows to:

• Keep Windows image updated (for windowsservercore-2019 image not updated #1736 & Update Jenkins version for Windows Image #1490)

• Publish Windows images for each supported JDKs and not only JDK11 (for Publish a Windows image for each supported Java version #1774)

  • Only JDK17 (default JDK since breaking(Windows): set Java 17 as default #1773) and JDK11 (to continue providing an image as still supported), JDK21 will have to wait until tests can be parallelized as builds take between 35 and 40 minutes for both JDKs
  • Building and publishing JDK11 & JDK17 Windows images take around 10 minutes (compared to around 20 minutes for longest Linux images)

• Get a Windows base image matching the controller image tag (for windowsservercore-2019 images use windowsservercore-1809 as base image #1775)

  • This PR switches from a windowsservercore-1809 Temurin base image to a windowsservercore-ltsc2019 Microsoft base image

    • According to https://learn.microsoft.com/en-us/windows/release-health/status-windows-10-1809-and-windows-server-2019 the 1809 version have reached end of servicing since May 11, 2021
    • I don't think we should continue providing a 1809 image, but if we do, it's easy as adding a windowsservercore-1809 image type in the Jenkinsfile, see below

  • Container image size reduced from 2.31Gio to 2.16Gio

• Publish a proper set of Windows image tags (for Windows Tags #1489)

  • Current tags for reference (On the 3rd of November 2023):

    
  • Tags from this pull request:

    jdk17-hotspot-windowsservercore-ltsc2019 (default JDK)

    • windowsservercore-ltsc2019
    • 2.430-windowsservercore-ltsc2019
    • jdk17-hotspot-windowsservercore-ltsc2019
    • 2.430-jdk17-hotspot-windowsservercore-ltsc2019

    jdk11-hotspot-windowsservercore-ltsc2019

    • jdk11-hotspot-windowsservercore-ltsc2019,
    • 2.430-jdk11-hotspot-windowsservercore-ltsc2019

  • Windows container image tags are now properly reflecting the Windows version (see previous point), and have changed from windowsservercore-2019 to windowsservercore-ltsc2019

    • While the current tags contain "windowsservercore-2019", the current image is a 1809, not a ltsc2019:

      docker/17/windows/windowsservercore-2019/hotspot/Dockerfile

      Line 3 in c7ae4b8

      FROM eclipse-temurin:"${JAVA_VERSION}"-jdk-windowsservercore-1809

    • This can understandably considered as a breaking change, let me know if you think it requires a pull request renaming and the addition of corresponding label

• Use docker compose file to declaratively define tags and docker compose build for building, tagging and testing images, reducing script complexity

  • Ease transition to docker bake when it will be available for Windows (cf this comment: cleanup(Windows): remove unused make.ps1 parameters #1769 (comment))
  • I tried publishing images and tags with docker compose push but unfortunately this command pushes only the image of the builds, not its additional tags defined in the docker-compose file.

• Have only one Dockerfile for every JDK versions

• The possibility to integrate additional images for new Windows version like Windows Server Core ltsc2022 if wanted, with only a Jenkinsfile change and no time penalty (parallel builds)

  https://ci.jenkins.io/job/Packaging/job/docker/job/PR-1770/154/:

  • Combination of Windows Server Core ltsc2019 & ltsc2022 and JDK11 & JDK17 images
  • 12 minutes, skipping tests like on trusted.ci.jenkins.io, no Linux image
  • Jenkinsfile change for the additional image type:

  --- old/Jenkinsfile
  +++ new/Jenkinsfile
  @@ -1,4 +1,13 @@
   #!/usr/bin/env groovy
  +def agentSelector(String imageType) {
  +    // Image types running on a Windows Server Core 2022 agent
  +    if (imageType.contains('2022')) {
  +        return 'windows-2022'
  +    }
  +    // Remaining image types running on a Windows Server Core 2019 agent: (nanoserver|windowservercore)-(1809|2019)
  +    return 'windows-2019'
  +}
   
   def listOfProperties = []
   listOfProperties << buildDiscarder(logRotator(numToKeepStr: '50', artifactNumToKeepStr: '5'))
  @@ -15,11 +24,15 @@
       def builds = [:]
       def windowsImageTypes = [
           'windowsservercore-ltsc2019',
  +        'windowsservercore-ltsc2022',
       ]
      for (anImageType in windowsImageTypes) {
           def imageType = anImageType
           builds[imageType] = {
  -            nodeWithTimeout('windows-2019') {
  +            nodeWithTimeout(agentSelector(imageType)) {

• The possibility to integrate additional images for new Windows flavor like Nanoserver if wanted, with only a Jenkinsfile change, a dedicated Dockerfile and no time penalty (parallel builds)
• Align build process between Jenkins images

Testing done

• Executed dry-run before and after

  Before:

  $ pwsh make.ps1 publish -DryRun  
  = PREPARE: List of jenkins4eval/jenkins images and tags to be processed:
  {
    "jdk17-hotspot-windowsservercore-2019": {
      "Tags": [
        "jdk17-hotspot-windowsservercore-2019",
        "windowsservercore-2019"
      ],
      "Folder": "17/windows/windowsservercore-2019/hotspot"
    }
  }
  Building jdk17-hotspot-windowsservercore-2019 => tag=jdk17-hotspot-windowsservercore-2019
  (dry-run) docker build -t jenkins4eval/jenkins:jdk17-hotspot-windowsservercore-2019 17/windows/windowsservercore-2019/hotspot
  Building jdk17-hotspot-windowsservercore-2019 => tag=windowsservercore-2019
  (dry-run) docker build -t jenkins4eval/jenkins:windowsservercore-2019 17/windows/windowsservercore-2019/hotspot
  Publishing jdk17-hotspot-windowsservercore-2019 => tag=jdk17-hotspot-windowsservercore-2019
  (dry-run) docker push jenkins4eval/jenkins:jdk17-hotspot-windowsservercore-2019
  Publishing jdk17-hotspot-windowsservercore-2019 => tag=windowsservercore-2019
  (dry-run) docker push jenkins4eval/jenkins:windowsservercore-2019
  Build finished successfully

  After:

  $ pwsh make.ps1 publish -DryRun
  = PREPARE: List of jenkins4eval/jenkins images and tags to be processed:
  name: jenkinsci-docker
  services:
    jdk11:
      build:
        context: /Users/veve/j-infra/jenkinsci-docker
        dockerfile: ./windows/windowsservercore/hotspot/Dockerfile
        args:
          COMMIT_SHA: 73a6176f2a18f9b69fc7a1c42fb338c43c9ef328
          JAVA_HOME: C:/openjdk-11
          JAVA_VERSION: 11.0.20.1_1
          JENKINS_SHA: C0FE49DBD53F7CC52D90810CECDEC6AB9B54665F4137977C2E7D7F642145E734
          JENKINS_VERSION: "2.431"
          TOOLS_WINDOWS_VERSION: "1809"
          WINDOWS_VERSION: ltsc2019
        tags:
        - jenkins4eval/jenkins:2.431-jdk11-hotspot-windowsservercore-ltsc2019
      image: jenkins4eval/jenkins:jdk11-hotspot-windowsservercore-ltsc2019
      networks:
        default: null
    jdk17:
      build:
        context: /Users/veve/j-infra/jenkinsci-docker
        dockerfile: ./windows/windowsservercore/hotspot/Dockerfile
        args:
          COMMIT_SHA: 73a6176f2a18f9b69fc7a1c42fb338c43c9ef328
          JAVA_HOME: C:/openjdk-17
          JAVA_VERSION: 17.0.9_9
          JENKINS_SHA: C0FE49DBD53F7CC52D90810CECDEC6AB9B54665F4137977C2E7D7F642145E734
          JENKINS_VERSION: "2.431"
          TOOLS_WINDOWS_VERSION: "1809"
          WINDOWS_VERSION: ltsc2019
        tags:
        - jenkins4eval/jenkins:2.431-jdk17-hotspot-windowsservercore-ltsc2019
        - jenkins4eval/jenkins:2.431-windowsservercore-ltsc2019
        - jenkins4eval/jenkins:windowsservercore-ltsc2019
      image: jenkins4eval/jenkins:jdk17-hotspot-windowsservercore-ltsc2019
      networks:
        default: null
  networks:
    default:
      name: jenkinsci-docker_default
  = BUILD: Building all images...
  (dry-run) docker-compose --file=build-windows.yaml build --parallel --pull
  = BUILD: Finished building all images.
  Publishing jenkins4eval/jenkins:jdk17-hotspot-windowsservercore-ltsc2019 => tag=jenkins4eval/jenkins:jdk17-hotspot-windowsservercore-ltsc2019
  (dry-run) docker push jenkins4eval/jenkins:jdk17-hotspot-windowsservercore-ltsc2019
  Publishing jenkins4eval/jenkins:jdk17-hotspot-windowsservercore-ltsc2019 => tag=jenkins4eval/jenkins:2.431-jdk17-hotspot-windowsservercore-ltsc2019
  (dry-run) docker push jenkins4eval/jenkins:2.431-jdk17-hotspot-windowsservercore-ltsc2019
  Publishing jenkins4eval/jenkins:jdk17-hotspot-windowsservercore-ltsc2019 => tag=jenkins4eval/jenkins:2.431-windowsservercore-ltsc2019
  (dry-run) docker push jenkins4eval/jenkins:2.431-windowsservercore-ltsc2019
  Publishing jenkins4eval/jenkins:jdk17-hotspot-windowsservercore-ltsc2019 => tag=jenkins4eval/jenkins:windowsservercore-ltsc2019
  (dry-run) docker push jenkins4eval/jenkins:windowsservercore-ltsc2019
  Publishing jenkins4eval/jenkins:jdk11-hotspot-windowsservercore-ltsc2019 => tag=jenkins4eval/jenkins:jdk11-hotspot-windowsservercore-ltsc2019
  (dry-run) docker push jenkins4eval/jenkins:jdk11-hotspot-windowsservercore-ltsc2019
  Publishing jenkins4eval/jenkins:jdk11-hotspot-windowsservercore-ltsc2019 => tag=jenkins4eval/jenkins:2.431-jdk11-hotspot-windowsservercore-ltsc2019
  (dry-run) docker push jenkins4eval/jenkins:2.431-jdk11-hotspot-windowsservercore-ltsc2019
  Build finished successfully

• Ran make.ps1 test and make.ps1 publish from a Windows Server Core 2019 VM with $env:DOCKERHUB_ORGANISATION="hlemeur", based on the "prod-packer-images/providers/Microsoft.Compute/images/jenkins-agent-windows-2019" image used by ci.jenkins.io Azure VM agents

• Full CI job with build and tests: https://ci.jenkins.io/job/Packaging/job/docker/job/PR-1770/155/

• Test CI publication of Windows Server Core ltsc2019 & 2022 images to jenkins4eval/jenkins

  • https://ci.jenkins.io/job/Packaging/job/docker/job/PR-1770/154/

  • https://hub.docker.com/r/jenkins4eval/jenkins/tags

  • Jenkinsfile change for images publication test:

    --- old/Jenkinsfile
    +++ new/Jenkinsfile
    @@ -1,4 +1,13 @@
    #!/usr/bin/env groovy
                            // disable until we get the parallel changes merged in
                            // def branchName = "${env.BRANCH_NAME}"
                            // if (branchName ==~ 'master'){
    -                        //    stage('Publish Experimental') {
    -                        //        infra.withDockerCredentials {
    -                        //            withEnv(['DOCKERHUB_ORGANISATION=jenkins4eval','DOCKERHUB_REPO=jenkins']) {
    -                        //                powershell './make.ps1 publish'
    -                        //            }
    -                        //        }
    -                        //    }
    +                           stage('Publish Experimental') {
    +                               infra.withDockerCredentials {
    +                                   withEnv(['DOCKERHUB_ORGANISATION=jenkins4eval','DOCKERHUB_REPO=jenkins']) {
    +                                       powershell './make.ps1 publish'
    +                                   }
    +                               }
    +                           }

Submitter checklist

Beta Give feedback

1. Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
2. Ensure that the pull request title represents the desired changelog entry
3. Please describe what you did
4. Link to relevant issues in GitHub or Jira
5. Link to relevant pull requests, esp. upstream and downstream changes
6. Ensure you have provided tests - that demonstrates feature works or fixes the issue

[lemeurherve]

@slide I'd appreciate a lot if you could take a look at this change and tell me what you think about it.

[timja]

nice work

[dduportal]

Nice work 👏

Let's validate this PR with the upcoming 2.432 weekly release and the next LTS (in 2 days).

Then we'll be able to check the issues related to Windows image!

[Jonny-vb]

Thanks for doing something about this! Means a lot to the few of us out here using Windows 😀