Prometheus Plugin Update prevents Jenkins from startup #647
Comments
|
Sounds stange. I didn't have any issues in my systems (multiple platforms). I'll check if I can provide a quick fix |
|
I've just updated 2 of our production Jenkins instances to 2.5.2 in the company without any issues. Are you sure your Jenkins instance is healthy? |
|
Hi, I also have issue with new version. Full stacktrace is Not sure if it's same issue or not. But started to appear with 2.5.2. Instance is healthy and latest LTS 2.440.2. All plugins are up-to-date |
|
But have you been able to update and does the 2.5.2 version produce results? That's kind of strange. I have multiple Jenkins instances. Windows, Docker... also colleagues tried that and nobody had any issues yet. Are we using the |
|
Since I cannot reproduce the issue I've implemented a "shot in the dark" in PR #648. @jonesbusy @Alphayeeeet Could anybody of you test the .hpi file which comes out of https://ci.jenkins.io/job/Plugins/job/prometheus-plugin/job/PR-648/ ? |
|
Sure thanks, I will test the incrementals tomorrow. I would be curious if the jenkins bom would be able to discover such issue |
|
Yesterday it happened to me too. I manually remove recently updated plugins one by one and after deleting prometheus, it could start normally. Jenkins 2.440.2 More details: Actually I was not upgrading "Prometheus Plugin" to "2.5.2". I done that earlier. When "Prometheus Plugin, 2.5.2" already installed, I was trying to upgrade other plugins. After upgrading them and restart, it hangs and timeouts start of the jenkins. Maybe helpful logs: |
|
After investigating logs, i remembered more details and edited my comment. I hope it will help. |
|
Yes my feeling is also similar that is related with restart or reload. When starting Jenkins inside a container (lets say from a fresh start) it works. But when restarting a controller or upgrading plugin and then restart it doesn't work. I think it has do to with init phase, plugin reload or even class path ? It could explain why test are not discovering such issue. Perhaps it can be reproduced using RealJenkinsRule 🤔 |
|
I think the root cause was the introduction of #637 that |
|
I never used Initializer on any of my plugin so I'm so sure about the behavior. Perhaps worth to ask on the dev list. Thanks in any case! |
|
Guess I'll revert things. Takes some time - currently low on time. Hopefully done at the weekend |
|
@jonesbusy I've created a new PR at #650. I've tested it explicitly on a Windows hosted machine with update and let Jenkins restart over the plugins page. Didn't find any issues. Could you maybe check if it also works on your instances? |
|
@Alphayeeeet @jonesbusy @nilbek . I've reverted the commit which might have caused the issue and published 2.5.3. Hopefully this release works for you all. Sorry for the trouble! |
|
Hi, Thanks, no problem it can happen. I will test the 2.5.3 next week after holidays. Best regards, |
|
We were seeing the same issue described here, updating to |
|
I also just tested the 2.5.3 this morning. This is also fixed on my side. Thanks again for the fix |
|
This issue is back after upgrading from I will open a new issue to track it |
|
New issue #683 |
Jenkins and plugins versions report
Jenkins: 2.440.2
OS: Linux - 4.18.0-513.18.1.el8_9.x86_64
Java: 11.0.22 - Red Hat, Inc. (OpenJDK 64-Bit Server VM)
analysis-model-api:12.1.0
ansicolor:1.0.4
ant:497.v94e7d9fffa_b_9
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
asm-api:9.6-3.v2e1fa_b_338cd7
authentication-tokens:1.53.v1c90fd9191a_b_
badge:1.9.1
basic-branch-build-strategies:81.v05e333931c7d
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9
branch-api:2.1152.v6f101e97dd77
build-timeout:1.32
build-user-vars-plugin:1.9
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.0.2
cloudbees-bitbucket-branch-source:880.vcf4056c5a_71f
cloudbees-folder:6.901.vb_4c7a_da_75da_3
cobertura:1.17
code-coverage-api:4.99.0
command-launcher:107.v773860566e2e
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.11.0-95.v22a_d30ee5d36
conditional-buildstep:1.4.3
config-file-provider:968.ve1ca_eb_913f8c
content-replace:1.8.2
copyartifact:722.v0662a_9b_e22a_c
coverage:1.13.0
credentials:1337.v60b_d7b_c7b_c9f
credentials-binding:657.v2b_19db_7d6e6d
custom-tools-plugin:0.8
data-tables-api:2.0.2-1
dependency-track:4.3.1
display-url-api:2.200.vb_9327d658781
docker-commons:439.va_3cb_0a_6a_fb_29
docker-workflow:572.v950f58993843
durable-task:550.v0930093c4b_a_6
echarts-api:5.5.0-1
email-ext:2.105
exclusive-execution:0.8
extended-choice-parameter:381.v360a_25ea_017c
extended-read-permission:53.v6499940139e5
extra-tool-installers:139.v723fee51b_7f2
favorite:2.208.v91d65b_7792a_c
file-operations:214.v2e7dc7f25757
file-parameters:316.va_83a_1221db_a_7
font-awesome-api:6.5.1-3
forensics-api:2.4.0
generic-webhook-trigger:2.1.0
git:5.2.1
git-client:4.7.0
git-parameter:0.9.19
git-server:114.v068a_c7cc2574
gitlab-plugin:1.8.0
gradle:2.10
groovy:457.v99900cb_85593
groovy-postbuild:228.vcdb_cf7265066
gson-api:2.10.1-15.v0d99f670e0a_7
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
hidden-parameter:237.v4b_df26c7a_f0e
htmlpublisher:1.33
http_request:1.18
instance-identity:185.v303dc7c645f9
ionicons-api:56.v1b_1c8c49374e
jackson2-api:2.17.0-379.v02de8ec9f64c
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javadoc:243.vb_b_503b_b_45537
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.9-1
jdk-tool:73.vddf737284550
jersey2-api:2.42-147.va_28a_44603b_d5
jjwt-api:0.11.5-77.v646c772fddb_0
job-dsl:1.87
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_
jquery:1.12.4-1
jquery3-api:3.7.1-2
jsch:0.2.16-86.v42e010d9484b_
json-api:20240303-41.v94e11e6de726
json-path-api:2.9.0-58.v62e3e85b_a_655
junit:1259.v65ffcef24a_88
ldap:719.vcb_d039b_77d0d
mailer:472.vf7c289a_4b_420
mapdb-api:1.0.9-28.vf251ce40855d
matrix-auth:3.2.2
matrix-project:822.824.v14451b_c0fd42
maven-plugin:3.23
metrics:4.2.21-449.v6960d7c54c69
mina-sshd-api-common:2.12.0-90.v9f7fb_9fa_3d3b_
mina-sshd-api-core:2.12.0-90.v9f7fb_9fa_3d3b_
msbuild:1.30
nodejs:1.6.1
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
openshift-client:1.1.0.424.v829cb_ccf8798
pam-auth:1.10
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-groovy-lib:704.vc58b_8890a_384
pipeline-input-step:491.vb_07d21da_1a_fb_
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2184.v0b_358b_953e69
pipeline-model-definition:2.2184.v0b_358b_953e69
pipeline-model-extensions:2.2184.v0b_358b_953e69
pipeline-rest-api:2.34
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2184.v0b_358b_953e69
pipeline-stage-view:2.34
pipeline-utility-steps:2.16.2
plain-credentials:179.vc5cb_98f6db_38
plugin-util-api:4.1.0
powershell:2.1
prism-api:1.29.0-13
prometheus:2.5.1 --> Upgrade to 2.5.2 not possible needed to rollback to backup
publish-over:0.22
publish-over-ssh:1.25
pubsub-light:1.18
purge-build-queue-plugin:88.v23b_97b_f2c7a_d
purge-job-history:1.6
rebuild:330.v645b_7df10e2a_
resource-disposer:0.23
role-strategy:713.vb_3837801b_8cc
run-condition:1.7
scm-api:689.v237b_6d3a_ef7f
script-security:1326.vdb_c154de8669
slave-setup:1.16
snakeyaml-api:2.2-111.vc6598e30cc65
sonar:2.17.2
sse-gateway:1.26
ssh-agent:346.vda_a_c4f2c8e50
ssh-credentials:326.v7fcb_a_ef6194b_
ssh-slaves:2.948.vb_8050d697fec
sshd:3.322.v159e91f6a_550
strict-crumb-issuer:2.1.1
structs:337.v1b_04ea_4df7c8
throttle-concurrents:2.14
timestamper:1.26
token-macro:400.v35420b_922dcb_
trilead-api:2.142.v748523a_76693
uno-choice:2.8.1
variant:60.v7290fc0eb_b_cd
warnings-ng:11.2.2
workflow-aggregator:596.v8c21c963d92d
workflow-api:1291.v51fd2a_625da_7
workflow-basic-steps:1049.v257a_e6b_30fb_d
workflow-cps:3889.v937e0b_3412d3
workflow-durable-task-step:1331.vc8c2fed35334
workflow-job:1400.v7fd111b_ec82f
workflow-multibranch:773.vc4fe1378f1d5
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:657.v03b_e8115821b_
workflow-support:881.v7663695646cf
ws-cleanup:0.45
What Operating System are you using (both controller, and any agents involved in the problem)?
Controller Node is Running RHEL 8
Reproduction steps
Updating Prometheus Plugin to 2.5.2
Jenkins version is 2.440.2
Expected Results
Jenkins should startup normally.
Actual Results
Jenkins was stuck in startup process and throws errors in log.
Anything else?
Log Messages attached:
2024-03-26 14:08:26.920+0000 [id=43] WARNING o.j.p.prometheus.JobCollector#lambda$collect$0: Caught error when processing job [JOB] error: java.lang.IllegalStateException: Jenkins.instance is missing. Read the documentation of Jenkins.getInstanceOrNull to see what you are doing wrong. at jenkins.model.Jenkins.get(Jenkins.java:819) at org.jenkinsci.plugins.prometheus.config.PrometheusConfiguration.get(PrometheusConfiguration.java:79) at org.jenkinsci.plugins.prometheus.JobCollector.appendJobMetrics(JobCollector.java:215) at org.jenkinsci.plugins.prometheus.JobCollector.lambda$collect$0(JobCollector.java:162) at org.jenkinsci.plugins.prometheus.util.Jobs.forEachJob(Jobs.java:19) at org.jenkinsci.plugins.prometheus.JobCollector.collect(JobCollector.java:155) at io.prometheus.client.CollectorRegistry.collectorNames(CollectorRegistry.java:113) at io.prometheus.client.CollectorRegistry.register(CollectorRegistry.java:50) at org.jenkinsci.plugins.prometheus.service.DefaultPrometheusMetrics.registerCollectors(DefaultPrometheusMetrics.java:46) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:109) at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:185) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1170) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-03-26 14:08:26.910+0000 [id=23] SEVERE h.i.i.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler#uncaughtException: A thread (Jenkins initialization thread/23) died unexpectedly due to an uncaught exception. This may leave your server corrupted and usually indicates a software bug. java.lang.ClassNotFoundException: hudson.util.HudsonFailedToLoad at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:594) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527) at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:511) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527) Caused: java.lang.NoClassDefFoundError: hudson/util/HudsonFailedToLoad at hudson.WebAppMain$3.run(WebAppMain.java:276)Are you interested in contributing a fix?
No response
The text was updated successfully, but these errors were encountered: