chore: align with docker-ssh-agent build process #827

lemeurherve · 2024-06-22T21:31:24Z

This PR aligns this repository with what have been implemented in jenkinsci/docker-ssh-agent#415, all extracted changes mentioned in jenkinsci/docker-ssh-agent#415 (comment) and additional cleanup permitted by this change.

List of commits, notes and references to docker-ssh-agent issues and pull requests

chore: define jdks to build at a single place in docker bake (`974969e`)

Cf The list of JDK to build is duplicated in docker-bake.hcl docker-ssh-agent#418
Cf chore: define jdks to build at a single place in docker bake docker-ssh-agent#419
Also added missing jdk in "debian" bake target name for consistency

chore: add `TestsDebug` parameter (`22934df`)

Cf chore: improve Windows tests debug and small code cleanup docker-ssh-agent#396
Supersede chore(Windows): add -TestsDebug parameter to build.ps1 #828

Detailed description from above closed pull request:

This PR introduces -TestsDebug parameter to build.ps1.

With it we can use -TestsDebug parameter or $env:TESTS_DEBUG to improve Windows tests debugging:

As it can be passed in the Jenkinsfile, it's easy now to activate it on a build replay instead of having to add a commit to switch debug on or off.

This parameter can take the following values:

empty/undefined: no additional debug (default value).

debug: output every test command and stderr on top of every test. (Equivalent of setting $quiet=$true in the test_helpers "Run-Program" function currently)

verbose: same as debug + output of stdout.

Output (from docker-ssh-agent pull request):

With .\build.ps1 test -TestsDebug 'debug':

With .\build.ps1 test -TestsDebug 'verbose':

Similar to:

chore: improve Windows tests debug and small code cleanup docker-ssh-agent#396

Testing done

Local tests + CI

Replay with -TestsDebug set to "verbose" and only one Windows image type to validate the functionality:
diff
--- old/Jenkinsfile
+++ new/Jenkinsfile
@@ -24,7 +24,7 @@
                 axes {
                     axis {
                         name 'IMAGE_TYPE'
-                        values 'linux', 'nanoserver-1809', 'nanoserver-ltsc2019', 'nanoserver-ltsc2022', 'windowsservercore-1809', 'windowsservercore-ltsc2019', 'windowsservercore-ltsc2022'
+                        values 'nanoserver-1809'
                     }
                 }
                 stages {
@@ -63,7 +63,7 @@
                                             // If the tests are passing for Linux AMD64, then we can build all the CPU architectures
                                             sh 'docker buildx bake --file docker-bake.hcl linux'
                                         } else {
-                                            powershell '& ./build.ps1 test'
+                                            powershell '& ./build.ps1 test -TestsDebug verbose'
                                         }
                                     }
                                 }

chore: lint *.ps1 (`19cce30`)

Single quotes instead of double quotes
Add space after ifs

chore: add log output to Windows tests (`57d5a57`)

Small QoL improvement

fix: implement `docker` calls as `make` targets (`f96d882`)

chore: shortens `agentSelector` comments (`af17f85`)

fix: remove interpolation in last sh and powershell calls (`f48ef34`)

chore: set bats formatter to junit (`2bc7774`)

Adapted from docker-ssh-agent Makefile to get rid of the call to nodejs (and associated docker image pull) to convert bats output to junit
Cf chore(tests) use bats junit direct exporter to get rid of NodeJS and NPM docker-ssh-agent#344

chore: use docker bake to generate docker compose file (`4cc014e`)

Cf Use only docker-bake.hcl for both Linux and Windows image definitions docker-ssh-agent#426
Cf chore: use docker bake to generate docker compose file docker-ssh-agent#415
Addition features added here:
- Archive the docker compose files
- New windowsagenttypes bake function to take care of the agent type ("agent" or "inbound-agent")
- ~~New windowsversiondigest bake function to take care of hotfix(Windows Server Core LTSC2019) pin 2019 images to April version #816~~
  (Removed in 1086ab1 as follow-up of Revert "hotfix(Windows Server Core LTSC2019) pin 2019 images to April version" #833)
This is one of the main commits to review

chore: get rid of build.sh and env.props (`52b4455`)

All environment variables defined in build.sh are now defined in docker-bake.hcl
env.props doesn't add any functionality that are used anymore AFAIK
Only apply to this repository, no equivalent files in docker-ssh-agent

doc: adapt docker-ssh-agent's 'building' section to this repository (`affb8e4`)

Commit importing original docker-ssh-agent "Building" section: 3ebdb13
Should satisfy Add building documentation #830

(Docker images content remains untouched hence the "chore" label.)

Testing done

CI

Submitter checklist

Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
Ensure that the pull request title represents the desired changelog entry
Please describe what you did
Link to relevant issues in GitHub or Jira
Link to relevant pull requests, esp. upstream and downstream changes
Ensure you have provided tests - that demonstrates feature works or fixes the issue

Ref: https://github.com/jenkinsci/docker-ssh-agent/blob/d1bfd814232ea509e555d50c55c972a5f24fb941/README.md

Ref: - jenkinsci#830

dduportal

👏

dduportal · 2024-09-24T14:27:28Z

Putting on draft as Hervé requested a few more changes

lemeurherveCB · 2024-09-24T14:37:36Z

Putting on draft as Hervé requested a few more changes

Was about to list all changes induced by the UBI9 image introduction, but I think they'll be better in another distinct pull request, this one is good as it is.

dduportal · 2024-09-24T16:19:23Z

For info, this PR fails to execute on trusted.ci with the following error:

17:51:01  Also:   org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 5b9ce4dc-da33-4851-a2ad-9ef8f9dc7c9e
17:51:01  Also:   	Also:   org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 0f351b2d-ac36-4fc8-8e10-495df4d0e685
17:51:01  java.lang.IllegalArgumentException: Expected named arguments but got [[REMOTING_VERSION=3261.v9c670a_4748a_9], [BUILD_NUMBER=6]]
17:51:01  Also:   	Also:   org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 9950e6ac-9089-4116-8edc-c33725d1f486
17:51:01  java.lang.IllegalArgumentException: Expected named arguments but got [[REMOTING_VERSION=3261.v9c670a_4748a_9], [BUILD_NUMBER=6]]
17:51:01  Also:   	Also:   org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 087332a6-a659-4b54-b500-59c19f650107
17:51:01  java.lang.IllegalArgumentException: Expected named arguments but got [[REMOTING_VERSION=3261.v9c670a_4748a_9], [BUILD_NUMBER=6]]
17:51:01  Also:   	Also:   org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 5cd68db8-cac0-49bc-83d9-5da7bb5d3835
17:51:01  java.lang.IllegalArgumentException: Expected named arguments but got [[REMOTING_VERSION=3261.v9c670a_4748a_9], [BUILD_NUMBER=6]]
17:51:01  Also:   	Also:   org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 3c52d8e6-b717-4372-bf83-104f9c901e68
17:51:01  java.lang.IllegalArgumentException: Expected named arguments but got [[REMOTING_VERSION=3261.v9c670a_4748a_9], [BUILD_NUMBER=6]]
17:51:01  Also:   	Also:   org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 75f14500-69e1-49e1-8347-1795c12c53f4
17:51:01  java.lang.IllegalArgumentException: Expected named arguments but got [[REMOTING_VERSION=3261.v9c670a_4748a_9], [BUILD_NUMBER=6]]
17:51:01  java.lang.IllegalArgumentException: Expected named arguments but got [[REMOTING_VERSION=3261.v9c670a_4748a_9], [BUILD_NUMBER=6]]
17:51:01  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.DSL.parseArgs(DSL.java:713)
17:51:01  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.DSL.parseArgs(DSL.java:640)
17:51:01  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:238)
17:51:01  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:196)
17:51:01  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:124)
17:51:01  	at jdk.internal.reflect.GeneratedMethodAccessor198.invoke(Unknown Source)
17:51:01  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
17:51:01  	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
17:51:01  	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
17:51:01  	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
17:51:01  	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1225)
17:51:01  	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
17:51:01  	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:41)
17:51:01  	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
17:51:01  	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
17:51:01  	at PluginClassLoader for script-security//org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:180)
17:51:01  	at PluginClassLoader for script-security//org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
17:51:01  	at PluginClassLoader for script-security//org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:163)
17:51:01  	at PluginClassLoader for script-security//org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:178)
17:51:01  	at PluginClassLoader for script-security//org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:182)
17:51:01  	at PluginClassLoader for script-security//org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:152)
17:51:01  	at PluginClassLoader for script-security//org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:152)
17:51:01  	at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
17:51:01  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.LoggingInvoker.methodCall(LoggingInvoker.java:117)
17:51:01  	at WorkflowScript.__model__Declaration_1_1624640462__(WorkflowScript:84)
17:51:01  	at ___cps.transform___(Native Method)
17:51:01  	at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:90)
17:51:01  	at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:116)
17:51:01  	at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:85)
17:51:01  	at jdk.internal.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
17:51:01  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
17:51:01  	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
17:51:01  	at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
17:51:01  	at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
17:51:01  	at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.Next.step(Next.java:83)
17:51:01  	at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:147)
17:51:01  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:17)
17:51:01  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:49)
17:51:01  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:180)
17:51:01  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:423)
17:51:01  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:331)
17:51:01  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:295)
17:51:01  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService.lambda$wrap$4(CpsVmExecutorService.java:140)
17:51:01  	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
17:51:01  	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
17:51:01  	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
17:51:01  	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
17:51:01  	at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
17:51:01  	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
17:51:01  	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
17:51:01  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
17:51:01  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
17:51:01  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.call(CpsVmExecutorService.java:53)
17:51:01  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.call(CpsVmExecutorService.java:50)
17:51:01  	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136)
17:51:01  	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275)
17:51:01  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService.lambda$categoryThreadFactory$0(CpsVmExecutorService.java:50)
17:51:01  	at java.base/java.lang.Thread.run(Unknown Source)

dduportal · 2024-09-24T16:47:54Z

Jenkinsfile

+                                            withEnv(
+                                                ["REMOTING_VERSION=${tagItems[0]}"],
+                                                ["BUILD_NUMBER=${tagItems[1]}"]
+                                            ) {


Suggested change

withEnv(

["REMOTING_VERSION=${tagItems[0]}"],

["BUILD_NUMBER=${tagItems[1]}"]

) {

withEnv([

"REMOTING_VERSION=${tagItems[0]}",

"BUILD_NUMBER=${tagItems[1]}",

]) {

lemeurherve changed the title ~~feat: align with docker-ssh-agent process~~ feat: align with docker-ssh-agent build process Jun 22, 2024

lemeurherve changed the title ~~feat: align with docker-ssh-agent build process~~ chore: align with docker-ssh-agent build process Jun 22, 2024

lemeurherve added enhancement chore labels Jun 22, 2024

lemeurherve force-pushed the align-with-docker-ssh-agent-process branch 4 times, most recently from a956d29 to fdbaacb Compare June 23, 2024 12:21

lemeurherve marked this pull request as ready for review June 23, 2024 12:24

lemeurherve requested a review from a team as a code owner June 23, 2024 12:24

This was referenced Jun 23, 2024

chore(Windows): add -TestsDebug parameter to build.ps1 #828

Closed

Add building documentation #830

Closed

lemeurherve force-pushed the align-with-docker-ssh-agent-process branch from cd389b3 to 1283c9f Compare June 23, 2024 22:54

lemeurherve added 12 commits June 24, 2024 01:05

chore: define jdks to build at a single place in docker bake

974969e

chore: add TestsDebug parameter

22934df

chore: lint *.ps1

19cce30

chore: add log output to Windows tests

57d5a57

fix: implement docker calls as make targets

f96d882

chore: shortens agentSelector comments

af17f85

fix: remove interpolation in last sh and powershell calls

f48ef34

chore: set bats formatter to junit

2bc7774

chore: use docker bake to generate docker compose file

4cc014e

chore: get rid of build.sh and env.props

52b4455

chore: import 'building' README section from jenkinsci/docker-ssh-agent

3ebdb13

Ref: https://github.com/jenkinsci/docker-ssh-agent/blob/d1bfd814232ea509e555d50c55c972a5f24fb941/README.md

doc: adapt docker-ssh-agent's 'building' section to this repository

affb8e4

Ref: - jenkinsci#830

lemeurherve force-pushed the align-with-docker-ssh-agent-process branch from 1283c9f to affb8e4 Compare June 23, 2024 23:05

lemeurherve added the documentation label Jun 29, 2024

lemeurherve and others added 3 commits July 8, 2024 10:11

Merge branch 'master' into align-with-docker-ssh-agent-process

2e77995

remove Windows Server Core version pinning workaround

1086ab1

fix merge conflicts

c4bcede

lemeurherve and others added 6 commits July 18, 2024 00:07

Merge branch 'master' into align-with-docker-ssh-agent-process

da1e5ed

Merge branch 'master' into align-with-docker-ssh-agent-process

404c748

Merge branch 'master' into align-with-docker-ssh-agent-process

9b667c8

fix merge conflicts

a886418

fix merge conflict

850fd7b

Merge branch 'master' into align-with-docker-ssh-agent-process

7bfdeb4

dduportal mentioned this pull request Sep 24, 2024

Bump Git version on Windows to 2.46.2.windows.1 #879

Closed

dduportal self-assigned this Sep 24, 2024

Merge branch 'master' into align-with-docker-ssh-agent-process

ffed12e

dduportal approved these changes Sep 24, 2024

View reviewed changes

dduportal enabled auto-merge September 24, 2024 14:09

dduportal removed their assignment Sep 24, 2024

dduportal disabled auto-merge September 24, 2024 14:26

dduportal marked this pull request as draft September 24, 2024 14:27

dduportal marked this pull request as ready for review September 24, 2024 15:43

dduportal merged commit 5b26913 into jenkinsci:master Sep 24, 2024
10 checks passed

dduportal removed enhancement documentation labels Sep 24, 2024

dduportal reviewed Sep 24, 2024

View reviewed changes

dduportal mentioned this pull request Sep 24, 2024

chore(Jenkinsfile) fix withEnv syntax error #881

Merged

4 tasks

This was referenced Sep 24, 2024

Add doc how to build the different images #748

Closed

chore(Jenkinsfile): set ON_TAG to true to publish tagged versions of the images #884

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: align with docker-ssh-agent build process #827

chore: align with docker-ssh-agent build process #827

lemeurherve commented Jun 22, 2024 •

edited

Loading

Testing done

dduportal left a comment

dduportal commented Sep 24, 2024

lemeurherveCB commented Sep 24, 2024

dduportal commented Sep 24, 2024

dduportal Sep 24, 2024

chore: align with docker-ssh-agent build process #827

chore: align with docker-ssh-agent build process #827

Conversation

lemeurherve commented Jun 22, 2024 • edited Loading

List of commits, notes and references to docker-ssh-agent issues and pull requests

chore: define jdks to build at a single place in docker bake (974969e)

chore: add TestsDebug parameter (22934df)

Testing done

chore: lint *.ps1 (19cce30)

chore: add log output to Windows tests (57d5a57)

fix: implement docker calls as make targets (f96d882)

chore: shortens agentSelector comments (af17f85)

fix: remove interpolation in last sh and powershell calls (f48ef34)

chore: set bats formatter to junit (2bc7774)

chore: use docker bake to generate docker compose file (4cc014e)

chore: get rid of build.sh and env.props (52b4455)

doc: adapt docker-ssh-agent's 'building' section to this repository (affb8e4)

Testing done

Submitter checklist

dduportal left a comment

Choose a reason for hiding this comment

dduportal commented Sep 24, 2024

lemeurherveCB commented Sep 24, 2024

dduportal commented Sep 24, 2024

dduportal Sep 24, 2024

Choose a reason for hiding this comment

lemeurherve commented Jun 22, 2024 •

edited

Loading

chore: define jdks to build at a single place in docker bake (`974969e`)

chore: add `TestsDebug` parameter (`22934df`)

chore: lint *.ps1 (`19cce30`)

chore: add log output to Windows tests (`57d5a57`)

fix: implement `docker` calls as `make` targets (`f96d882`)

chore: shortens `agentSelector` comments (`af17f85`)

fix: remove interpolation in last sh and powershell calls (`f48ef34`)

chore: set bats formatter to junit (`2bc7774`)

chore: use docker bake to generate docker compose file (`4cc014e`)

chore: get rid of build.sh and env.props (`52b4455`)

doc: adapt docker-ssh-agent's 'building' section to this repository (`affb8e4`)