Bump kubernetes-client-bom from 6.8.1 to 6.9.2 #36312

manusa · 2023-10-05T16:39:51Z

~~Kubernetes Client 6.9.0 was just released: https://github.com/fabric8io/kubernetes-client/releases/tag/v6.9.0~~
~~Kubernetes Client 6.9.1 was just released: https://github.com/fabric8io/kubernetes-client/releases/tag/v6.9.1~~
Kubernetes Client 6.9.2 was just released: https://github.com/fabric8io/kubernetes-client/releases/tag/v6.9.2
/cc @metacosm

gsmet · 2023-10-05T16:48:10Z

@metacosm let us know if it's all good for you.

manusa · 2023-10-06T04:46:19Z

...quarkus-integration-test-kubernetes-client-999-SNAPSHOT-runner.jar
2023-10-05T18:46:17.4004088Z ========================================================================================================================
2023-10-05T18:46:17.4027720Z GraalVM Native Image: Generating 'quarkus-integration-test-kubernetes-client-999-SNAPSHOT-runner' (executable)...
2023-10-05T18:46:17.4033873Z ========================================================================================================================
2023-10-05T18:46:17.4039943Z For detailed information and explanations on the build output, visit:
2023-10-05T18:46:17.4046045Z https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md
2023-10-05T18:46:17.4052182Z ------------------------------------------------------------------------------------------------------------------------
2023-10-05T18:46:23.5542193Z [1/8] Initializing...                                                                                   (19.2s @ 0.31GB)
2023-10-05T18:46:23.5547264Z  Java version: 17.0.8+7, vendor version: Mandrel-23.0.1.2-Final
2023-10-05T18:46:23.5548189Z  Graal compiler: optimization level: 2, target machine: x86-64-v3
2023-10-05T18:46:23.5548800Z  C compiler: gcc (redhat, x86_64, 8.5.0)
2023-10-05T18:46:23.5568395Z  Garbage collector: Serial GC (max heap size: 80% of RAM)
2023-10-05T18:46:23.5579655Z  3 user-specific feature(s)
2023-10-05T18:46:23.5587928Z  - io.quarkus.runner.Feature: Auto-generated class by Quarkus from the existing extensions
2023-10-05T18:46:23.5589002Z  - io.quarkus.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase
2023-10-05T18:46:23.5589806Z  - org.eclipse.angus.activation.nativeimage.AngusActivationFeature
2023-10-05T18:46:26.6780858Z os.name
2023-10-05T18:58:51.5995309Z Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
2023-10-05T18:58:52.3209333Z The Native Image build process ran out of memory. (The maximum heap size of the process was set with '-Xmx5g'.)
2023-10-05T18:58:52.3214684Z Please make sure your build system has more memory available or increase the maximum heap size using the '-Xmx' option.

... quarkus-integration-test-openshift-client-999-SNAPSHOT-runner.jar
2023-10-05T19:05:44.4828590Z ========================================================================================================================
2023-10-05T19:05:44.4848101Z GraalVM Native Image: Generating 'quarkus-integration-test-openshift-client-999-SNAPSHOT-runner' (executable)...
2023-10-05T19:05:44.4849244Z ========================================================================================================================
2023-10-05T19:05:44.4852410Z For detailed information and explanations on the build output, visit:
2023-10-05T19:05:44.4855091Z https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md
2023-10-05T19:05:44.4857508Z ------------------------------------------------------------------------------------------------------------------------
2023-10-05T19:05:49.3010847Z [1/8] Initializing...                                                                                   (15.7s @ 0.37GB)
2023-10-05T19:05:49.3024948Z  Java version: 17.0.8+7, vendor version: Mandrel-23.0.1.2-Final
2023-10-05T19:05:49.3035161Z  Graal compiler: optimization level: 2, target machine: x86-64-v3
2023-10-05T19:05:49.3042377Z  C compiler: gcc (redhat, x86_64, 8.5.0)
2023-10-05T19:05:49.3055473Z  Garbage collector: Serial GC (max heap size: 80% of RAM)
2023-10-05T19:05:49.3074476Z  3 user-specific feature(s)
2023-10-05T19:05:49.3085265Z  - io.quarkus.runner.Feature: Auto-generated class by Quarkus from the existing extensions
2023-10-05T19:05:49.3090867Z  - io.quarkus.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase
2023-10-05T19:05:49.3094678Z  - org.eclipse.angus.activation.nativeimage.AngusActivationFeature
2023-10-05T19:25:06.6318479Z Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded

manusa · 2023-10-06T05:10:54Z

Local build and test passed for kubernetes-client

This is the GraalVM output:

========================================================================================================================
GraalVM Native Image: Generating 'quarkus-integration-test-kubernetes-client-999-SNAPSHOT-runner' (executable)...
========================================================================================================================
For detailed information and explanations on the build output, visit:
https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md
------------------------------------------------------------------------------------------------------------------------
[1/8] Initializing...                                                                                    (6.6s @ 0.39GB)
 Java version: 17.0.8+7, vendor version: Mandrel-23.0.1.2-Final
 Graal compiler: optimization level: 2, target machine: x86-64-v3
 C compiler: gcc (redhat, x86_64, 8.5.0)
 Garbage collector: Serial GC (max heap size: 80% of RAM)
 3 user-specific feature(s)
 - io.quarkus.runner.Feature: Auto-generated class by Quarkus from the existing extensions
 - io.quarkus.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase
 - org.eclipse.angus.activation.nativeimage.AngusActivationFeature
os.name
[2/8] Performing analysis...  [******]                                                                 (330.9s @ 7.34GB)
  34,129 (94.89%) of 35,968 types reachable
  72,469 (84.26%) of 86,007 fields reachable
 335,426 (86.29%) of 388,698 methods reachable
  24,160 types, 51,546 fields, and 214,821 methods registered for reflection
      63 types,    68 fields, and    55 methods registered for JNI access
       4 native libraries: dl, pthread, rt, z
[3/8] Building universe...                                                                              (11.3s @ 7.27GB)
[4/8] Parsing methods...      [***]                                                                     (11.8s @ 8.09GB)
[5/8] Inlining methods...     [***]                                                                      (5.8s @ 7.79GB)
[6/8] Compiling methods...    [********]                                                                (71.8s @ 7.16GB)
[7/8] Layouting methods...    [****]                                                                    (16.2s @ 7.52GB)
[8/8] Creating image...       [*****]                                                                   (21.9s @ 6.98GB)
 126.30MB (50.46%) for code area:   300,487 compilation units
 123.64MB (49.40%) for image heap:  993,581 objects and 12 resources
 346.79kB ( 0.14%) for other data
 250.28MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area:                                Top 10 object types in image heap:
  18.33MB io.fabric8.openshift-model-operator-6.9.0.jar       25.77MB byte[] for code metadata
  15.20MB svm.jar (Native Image)                              20.08MB java.lang.Class
  14.33MB io.fabric8.openshift-model-6.9.0.jar                19.13MB byte[] for java.lang.String
  12.81MB java.base                                           16.95MB byte[] for reflection metadata
   9.06MB io.fabric8.kubernetes-model-core-6.9.0.jar           9.78MB c.oracle.svm.core.reflect.SubstrateMethodAccessor
   4.73MB io.fabric8.openshift-model-monitoring-6.9.0.jar      8.82MB java.lang.String
   4.71MB io.fabric8.openshift-model-hive-6.9.0.jar            6.89MB byte[] for general heap data
   2.91MB i.f.k.9.0.jar                                        2.86MB com.oracle.svm.core.hub.DynamicHubCompanion
   2.72MB io.fabric8.openshift-model-operatorhub-6.9.0.jar     2.32MB java.lang.Object[]
   2.44MB io.fabric8.kubernetes-model-flowcontrol-6.9.0.jar    1.83MB java.lang.String[]
  36.76MB for 137 more packages                                8.07MB for 3549 more object types
------------------------------------------------------------------------------------------------------------------------
Recommendations:
 HEAP: Set max heap for improved and more predictable memory usage.
 CPU:  Enable more CPU features with '-march=native' for improved performance.
------------------------------------------------------------------------------------------------------------------------
                      35.6s (7.4% of total time) in 143 GCs | Peak RSS: 13.19GB | CPU load: 11.22
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
 /project/quarkus-integration-test-kubernetes-client-999-SNAPSHOT-runner (executable)
 /project/quarkus-integration-test-kubernetes-client-999-SNAPSHOT-runner-build-output-stats.json (build_info)
========================================================================================================================
Finished generating 'quarkus-integration-test-kubernetes-client-999-SNAPSHOT-runner' in 7m 59s.

This release of the client includes new Kubernetes v1.28 model types for the kubernetes-client and also OpenShift v4.13 model types. The increase in the number of models might be the cause for the surge in the memory usage.

gsmet · 2023-10-06T07:35:48Z

Unfortunately, I'm not sure we can go much higher than 5g on GitHub Actions without having things failing.

Maybe try to set -Dquarkus.native.native-image-xmx=6g here: https://github.com/quarkusio/quarkus/blob/main/.github/workflows/ci-actions-incremental.yml#L46C65-L46C101 in your PRand see how it goes.

metacosm · 2023-10-09T10:59:28Z

@metacosm let us know if it's all good for you.

No, it's not good for us. Applications don't compile anymore with errors similar to:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.11.0:compile (default-compile) on project sample-webpage-operator: Compilation failure: Compilation failure: 
[ERROR] /Volumes/CaseSensitiveDev/java-operator-sdk/sample-operators/webpage/src/main/java/io/javaoperatorsdk/operator/sample/WebPageStandaloneDependentsReconciler.java: accept(java.util.List<java.util.Map.Entry<java.lang.String,java.lang.Object>>,java.lang.String,io.fabric8.kubernetes.api.builder.Visitor...) in io.fabric8.kubernetes.api.builder.BaseFluent cannot implement accept(java.util.List<java.util.Map.Entry<java.lang.String,java.lang.Object>>,java.lang.String,io.fabric8.kubernetes.api.builder.Visitor...) in io.fabric8.kubernetes.api.builder.Visitable
[ERROR]   return type capture#1 of ? is not compatible with capture#2 of ?
[ERROR] /Volumes/CaseSensitiveDev/java-operator-sdk/sample-operators/webpage/src/main/java/io/javaoperatorsdk/operator/sample/WebPageDependentsWorkflowReconciler.java: accept(java.util.List<java.util.Map.Entry<java.lang.String,java.lang.Object>>,java.lang.String,io.fabric8.kubernetes.api.builder.Visitor...) in io.fabric8.kubernetes.api.builder.BaseFluent cannot implement accept(java.util.List<java.util.Map.Entry<java.lang.String,java.lang.Object>>,java.lang.String,io.fabric8.kubernetes.api.builder.Visitor...) in io.fabric8.kubernetes.api.builder.Visitable
[ERROR]   return type capture#3 of ? is not compatible with capture#4 of ?

Note that this occurs in code that doesn't use this method so there's nothing that can be done on our side as the error occurs outside of our code. The same code works fine with 6.8.1.

gsmet · 2023-10-16T14:49:22Z

@manusa see comment above ^

manusa · 2023-10-16T15:52:25Z

Yes, sorry for not commenting here.

There's some work being done, I created fabric8io/kubernetes-client#5522 to keep track of it.

I'm not sure if @metacosm has been able to try this weekend's snapshots to see if there's any difference.

I tried to create a simple reproducer in the main Fabric8 repository, but was unable to.

@iocanel was doing some research on the subject, but I'm not sure if he found out anything else.

manusa · 2023-10-26T16:39:14Z

Hi @metacosm

I've updated the PR to bump Kubernetes Client to 6.9.1 which contains the fix for the type inference issue. Please check if it solves your issue.

manusa · 2023-10-27T03:03:19Z

2023-10-26T20:48:17.7071483Z 	[error]: Build step io.quarkus.kubernetes.deployment.KubernetesProcessor#build threw an exception: java.lang.NoSuchMethodError: 'io.fabric8.kubernetes.api.builder.Fluent io.fabric8.kubernetes.api.model.KubernetesListBuilder.accept(io.fabric8.kubernetes.api.builder.Visitor[])'
2023-10-26T20:48:17.7074491Z 	at io.dekorate.ResourceRegistry.lambda$generate$2(ResourceRegistry.java:181)
2023-10-26T20:48:17.7075473Z 	at java.base/java.util.HashMap.forEach(HashMap.java:1421)
2023-10-26T20:48:17.7077453Z 	at io.dekorate.ResourceRegistry.generate(ResourceRegistry.java:173)
2023-10-26T20:48:17.7078342Z 	at io.dekorate.Session.generate(Session.java:293)
2023-10-26T20:48:17.7079051Z 	at io.dekorate.Session.close(Session.java:256)
2023-10-26T20:48:17.7080150Z 	at io.quarkus.kubernetes.deployment.KubernetesProcessor.lambda$build$2(KubernetesProcessor.java:194)
2023-10-26T20:48:17.7081376Z 	at java.base/java.util.Optional.ifPresent(Optional.java:178)
2023-10-26T20:48:17.7082524Z 	at io.quarkus.kubernetes.deployment.KubernetesProcessor.build(KubernetesProcessor.java:142)
2023-10-26T20:48:17.7088949Z 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2023-10-26T20:48:17.7090423Z 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
2023-10-26T20:48:17.7094883Z 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2023-10-26T20:48:17.7159281Z 	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
2023-10-26T20:48:17.7160219Z 	at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:849)
2023-10-26T20:48:17.7161481Z 	at io.quarkus.builder.BuildContext.run(BuildContext.java:256)
2023-10-26T20:48:17.7162701Z 	at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
2023-10-26T20:48:17.7163765Z 	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
2023-10-26T20:48:17.7165540Z 	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
2023-10-26T20:48:17.7166656Z 	at java.base/java.lang.Thread.run(Thread.java:833)
2023-10-26T20:48:17.7167370Z 	at org.jboss.threads.JBossThread.run(JBossThread.java:501)

It seems we need to bump Dekorate's dependency on Sundrio too

metacosm · 2023-11-06T08:42:31Z

~~Update looks OK on our side.~~
Spoke too soon: things look OK on the JOSDK side but fail to compile in QOSDK with:
[ERROR] /Volumes/CaseSensitiveDev/quarkus-operator-sdk/core/deployment/src/main/java/io/quarkiverse/operatorsdk/deployment/CRDGeneration.java:[161,59] incompatible types: java.lang.Class<capture#1 of ? extends io.fabric8.kubernetes.client.CustomResource> cannot be converted to java.lang.Class>

Looking into it, seems like type inference is stricter now.

metacosm · 2023-11-06T09:11:14Z

Moving past that issue, I get other errors similar to:
Build step io.quarkus.kubernetes.deployment.KubernetesProcessor#build threw an exception: java.lang.NoSuchMethodError: 'io.fabric8.kubernetes.api.builder.Fluent io.fabric8.kubernetes.api.model.KubernetesListBuilder.accept(io.fabric8.kubernetes.api.builder.Visitor[])'

metacosm · 2023-11-06T09:30:15Z

The error I'm seeing is consistent with CI here. I guess we need a dekorate upgrade as well.

manusa · 2023-11-06T10:35:37Z

I updated the PR to include Dekorate's updated version that Ioannis just released.

manusa · 2023-11-10T14:41:45Z

I've got something working, but I'll need some feedback on this.

The main issue with the GraalVM out of memory error is regarding the presence of a really big model module (io.fabric8:openshift-model-operator).

This model will only be required by a fraction of users and for sure is not part of the integration test suite. An easy way to overcome the out of memory problem is to exclude this module in the pom.xml. However, if we exclude it we hit #25526 and the GraalVM build is errored.

My proposed solution is to add a config property: quarkus.kubernetes-client.allow-incomplete-model that when set to true, disables the addition of the --link-at-build-time flag. You can see this in the last commit feat(kubernetes-client): allow incomplete model classpath.

I guess this needs a wider consensus though. Or maybe there's a more suitable solution that I didn't think of. @geoand @gastaldi @gsmet

geoand · 2023-11-13T09:26:30Z

My proposed solution is to add a config property: quarkus.kubernetes-client.allow-incomplete-model that when set to true, disables the addition of the --link-at-build-time flag. You can see this in the last commit feat(kubernetes-client): allow incomplete model classpath.

I don't think this is the way to go, as we almost never want extensions to be adding global GraalVM flags.

manusa · 2023-11-13T09:33:57Z

I don't think this is the way to go, as we almost never want extensions to be adding global GraalVM flags.

Well, in this case is not adding, but providing a configuration option to prevent Quarkus from adding a flag automatically.

I'd rather see #25526 fixed instead too.

geoand · 2023-11-13T09:39:23Z

Well, in this case is not adding, but providing a configuration option to prevent Quarkus from adding a flag automatically.

My point is that the an extension is making changes to GraalVMs's globak configuration

manusa · 2023-11-13T09:46:27Z

My point is that the an extension is making changes to GraalVMs's globak configuration

https://github.com/quarkusio/quarkus/blob/93bab5ae065274f910a444bc940ca6349379f008/extensions/jdbc/jdbc-oracle/deployment/src/main/java/io/quarkus/jdbc/oracle/deployment/OracleMetadataOverrides.java#L148C1-L151

It's basically the same as this, but with an option to opt-in.

geoand · 2023-11-13T09:48:11Z

Yeah, that's a very very special case that we try very hard to avoid

metacosm · 2023-11-13T10:09:53Z

So what is the proposed solution, @geoand? We need an update to the client in Quarkus.

geoand · 2023-11-13T10:13:24Z

@manusa is working on it after a chat we had

manusa · 2023-11-13T10:14:18Z

So what is the proposed solution, @geoand? We need an update to the client in Quarkus.

I'm currently working on one. Since this only affects GH actions environment, I'm adding some hack extension to the kubernetes/openshift-client integration test modules that prevents the --link-at-build-time from being passed on to GraalVM only for those two test projects.

Prevent Quarkus from adding --link-at-build-time GraalVM flag Signed-off-by: Marc Nuri <[email protected]>

quarkus-bot · 2023-11-13T20:06:21Z

Failing Jobs - Building `26713db`

Status	Name	Step	Failures	Logs	Raw logs
✔️	Gradle Tests - JDK 11
✖	Gradle Tests - JDK 11 Windows	`Build`	Failures	Logs	Raw logs
✔️	Maven Tests - JDK 11
✖	Maven Tests - JDK 11 Windows	`Build`	Failures	Logs	Raw logs

Full information is available in the Build summary check run.

Failures

⚙️ Gradle Tests - JDK 11 Windows #

- Failing: integration-tests/gradle

📦 integration-tests/gradle

✖ io.quarkus.gradle.CustomManifestArgumentsTest.shouldContainsSpecificManifestProperty line 37 - More details - Source on GitHub

java.lang.AssertionError: 

Expecting actual not to be null

⚙️ Maven Tests - JDK 11 Windows #

📦 integration-tests/maven

✖ io.quarkus.maven.it.DevMojoIT.testConfigFactoryInAppModuleBannedInCodeGen line 80 - More details - Source on GitHub

org.awaitility.core.ConditionTimeoutException: Condition with lambda expression in io.quarkus.test.devmode.util.DevModeClient was not fulfilled within 1 minutes.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:78)

✖ io.quarkus.maven.it.DevMojoIT.testConfigFactoryInAppModuleBannedInCodeGen line 80 - More details - Source on GitHub

org.awaitility.core.ConditionTimeoutException: Condition with lambda expression in io.quarkus.test.devmode.util.DevModeClient was not fulfilled within 1 minutes.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:78)

metacosm · 2023-11-14T07:04:13Z

Woot! 💪

quarkus-bot bot added the area/kubernetes label Oct 5, 2023

manusa mentioned this pull request Oct 5, 2023

Release Fabric8 Kubernetes Client 6.9.0 + 6.9.1 + 6.9.2 fabric8io/kubernetes-client#5479

Closed

6 tasks