Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configuration of Solr MultiAuthplugin with JWT and basic auth gives the error of PKI authentication on creating cores. #719

Open
khandnb opened this issue Aug 22, 2024 · 12 comments

Comments

@khandnb
Copy link

khandnb commented Aug 22, 2024

No description provided.

@khandnb
Copy link
Author

khandnb commented Aug 22, 2024

The solr admin UI is successfully logged in with token received from IDP and is able to access security, list collections etc. but the core creation fails with invalid PKI header. The Solr is deployed on GKE with istio proxy.
{
"textPayload": "MESSAGE:Could not validate PKI header.",
"insertId": "xabmc3lmfdshg1akrcw",
"resource": {
"type": "k8s_container",
"labels": {
"location": "us",
"pod_name": "podname",
"cluster_name": "cluster_name",
"container_name": "solrcloud-node",
"namespace_name": "solr",
"project_id": "xxx"
}
},

@khandnb
Copy link
Author

khandnb commented Aug 22, 2024

"authentication": {
"class": "solr.MultiAuthPlugin",
"schemes": [{
"scheme": "bearer",
"blockUnknown":false,
"class":"solr.JWTAuthPlugin",
"adminUiScope": "api://ttt/admin",
"principalClaim":"unique_name",
"iss":"https://sts.windows.net/abc/",
"aud":"api://xyz",
"wellKnownUrl":"https://login.microsoftonline.com/abc/v2.0/.well-known/openid-configuration",
"redirectUris": "https://localhost:8983/solr/",
"clientId":"xyz",
"authorizationFlow":"code_pkce",
"trustedCertsFile":"/path/to/certificate",
"jwkCacheDur":"60",
},{
"scheme": "basic",
"blockUnknown": false,
"class": "solr.BasicAuthPlugin",
"realm":"Solr Basic Auth",
"credentials": {
"solr":"bfjbf",
},
"forwardCredentials": false
}]
},

@khandnb khandnb changed the title Hi, When I have configured Solr MultiAuthplugin with JWT and basic auth, I start getting the error of PKI authentication on creating cores. Can you please suggest what is the issue here. Configuration of Solr MultiAuthplugin with JWT and basic auth gives the error of PKI authentication on creating cores. Aug 22, 2024
@khandnb
Copy link
Author

khandnb commented Sep 4, 2024

hi @janhoy @HoustonPutman Can you please suggest the miss here or fix that I can make to resolve this issue.

@janhoy
Copy link
Contributor

janhoy commented Sep 5, 2024

You’re using istio. I guess Istio proxy may be swallowing the SolrAuth http header, can you check?

@HoustonPutman
Copy link
Contributor

What version of Solr are you running? Also what logs is solr printing? It will likely give some reasoning behind why the PKIAuth could not be verified.

@khandnb
Copy link
Author

khandnb commented Sep 6, 2024

I am using solr version 9.6.0 and solr operator 0.8.1 . Below are the logs:


2024-09-06 07:25:56.992 INFO (qtp1212191909-53-solr-solrcloud-0.solr-solrcloud-headless.solr-343) [c: s: r: x: t:solr-solrcloud-0.solr-solrcloud-headless.solr-343] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/health params={} status=0 QTime=0
2024-09-06 07:25:57.503 INFO (OverseerThreadFactory-19-thread-1) [c:testing s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection testing
2024-09-06 07:25:57.710 INFO (OverseerStateUpdate-72062713179013124-solr-solrcloud-0.solr-solrcloud-headless.solr:8983_solr-n_0000000002) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() {
"core":"testing_shard1_replica_n1",
"node_name":"solr-solrcloud-0.solr-solrcloud-headless.solr:8983_solr",
"base_url":"http://solr-solrcloud-0.solr-solrcloud-headless.solr:8983/solr",
"collection":"testing",
"shard":"shard1",
"state":"down",
"type":"NRT",
"operation":"ADDREPLICA",
"waitForFinalState":"false"}
2024-09-06 07:25:57.837 INFO (zkCallback-13-thread-4) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testing/state.json zxid: -1] for collection [testing] has occurred - updating... (live nodes size: [1])
2024-09-06 07:25:58.139 ERROR (qtp1212191909-54-solr-solrcloud-0.solr-solrcloud-headless.solr-345) [c: s: r: x: t:solr-solrcloud-0.solr-solrcloud-headless.solr-345] o.a.s.s.PKIAuthenticationPlugin Exception trying to get public key from: http://solr-solrcloud-0.solr-solrcloud-headless.solr:8983/solr => org.noggit.JSONParser$ParseException: JSON Parse Error: char=<,position=0 AFTER='<' BEFORE='!DOCTYPE html PUBLIC "-//W3C//DTD XHTML'
at org.noggit.JSONParser.err(JSONParser.java:447)
org.noggit.JSONParser$ParseException: JSON Parse Error: char=<,position=0 AFTER='<' BEFORE='!DOCTYPE html PUBLIC "-//W3C//DTD XHTML'
at org.noggit.JSONParser.err(JSONParser.java:447) ~[?:?]
at org.noggit.JSONParser.handleNonDoubleQuoteString(JSONParser.java:808) ~[?:?]
at org.noggit.JSONParser.next(JSONParser.java:1013) ~[?:?]
at org.noggit.JSONParser.nextEvent(JSONParser.java:1059) ~[?:?]
at org.noggit.ObjectBuilder.(ObjectBuilder.java:85) ~[?:?]
at org.apache.solr.common.util.Utils.lambda$static$1(Utils.java:331) ~[?:?]
at org.apache.solr.common.util.Utils.fromJSON(Utils.java:283) ~[?:?]
at org.apache.solr.common.util.Utils.fromJSON(Utils.java:264) ~[?:?]
at org.apache.solr.common.util.Utils.fromJSON(Utils.java:260) ~[?:?]
at org.apache.solr.security.PKIAuthenticationPlugin.fetchPublicKeyFromRemote(PKIAuthenticationPlugin.java:367) ~[?:?]
at org.apache.solr.security.PKIAuthenticationPlugin.getOrFetchPublicKey(PKIAuthenticationPlugin.java:227) ~[?:?]
at org.apache.solr.security.PKIAuthenticationPlugin.decipherHeaderV2(PKIAuthenticationPlugin.java:235) ~[?:?]
at org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:166) ~[?:?]
at org.apache.solr.security.AuthenticationPlugin.authenticate(AuthenticationPlugin.java:91) ~[?:?]
at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:366) ~[?:?]
at org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:240) ~[?:?]
at org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:219) ~[?:?]
at org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:249) ~[?:?]
at org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:215) ~[?:?]
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:213) ~[?:?]
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195) ~[?:?]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210) ~[jetty-servlet-10.0.20.jar:10.0.20]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.20.jar:10.0.20]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) ~[jetty-servlet-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598) ~[jetty-security-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) ~[jetty-servlet-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:228) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301) ~[jetty-rewrite-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.Server.handle(Server.java:563) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:208) ~[http2-common-10.0.20.jar:10.0.20]
at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:155) ~[http2-common-10.0.20.jar:10.0.20]
at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:450) ~[http2-common-10.0.20.jar:10.0.20]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) ~[jetty-io-10.0.20.jar:10.0.20]
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) ~[jetty-io-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) ~[jetty-util-10.0.20.jar:10.0.20]
at java.base/java.lang.Thread.run(Unknown Source) [?:?]
2024-09-06 07:25:58.139 WARN (qtp1212191909-54-solr-solrcloud-0.solr-solrcloud-headless.solr-345) [c: s: r: x: t:solr-solrcloud-0.solr-solrcloud-headless.solr-345] o.a.s.s.PKIAuthenticationPlugin Key is null when attempting to validate signature; skipping...
2024-09-06 07:25:58.139 WARN (qtp1212191909-54-solr-solrcloud-0.solr-solrcloud-headless.solr-345) [c: s: r: x: t:solr-solrcloud-0.solr-solrcloud-headless.solr-345] o.a.s.s.PKIAuthenticationPlugin Failed to verify signature, trying after refreshing the key
2024-09-06 07:25:58.166 ERROR (qtp1212191909-54-solr-solrcloud-0.solr-solrcloud-headless.solr-345) [c: s: r: x: t:solr-solrcloud-0.solr-solrcloud-headless.solr-345] o.a.s.s.PKIAuthenticationPlugin Exception trying to get public key from: http://solr-solrcloud-0.solr-solrcloud-headless.solr:8983/solr => org.noggit.JSONParser$ParseException: JSON Parse Error: char=<,position=0 AFTER='<' BEFORE='!DOCTYPE html PUBLIC "-//W3C//DTD XHTML'
at org.noggit.JSONParser.err(JSONParser.java:447)
org.noggit.JSONParser$ParseException: JSON Parse Error: char=<,position=0 AFTER='<' BEFORE='!DOCTYPE html PUBLIC "-//W3C//DTD XHTML'
at org.noggit.JSONParser.err(JSONParser.java:447) ~[?:?]
at org.noggit.JSONParser.handleNonDoubleQuoteString(JSONParser.java:808) ~[?:?]
at org.noggit.JSONParser.next(JSONParser.java:1013) ~[?:?]
at org.noggit.JSONParser.nextEvent(JSONParser.java:1059) ~[?:?]
at org.noggit.ObjectBuilder.(ObjectBuilder.java:85) ~[?:?]
at org.apache.solr.common.util.Utils.lambda$static$1(Utils.java:331) ~[?:?]
at org.apache.solr.common.util.Utils.fromJSON(Utils.java:283) ~[?:?]
at org.apache.solr.common.util.Utils.fromJSON(Utils.java:264) ~[?:?]
at org.apache.solr.common.util.Utils.fromJSON(Utils.java:260) ~[?:?]
at org.apache.solr.security.PKIAuthenticationPlugin.fetchPublicKeyFromRemote(PKIAuthenticationPlugin.java:367) ~[?:?]
at org.apache.solr.security.PKIAuthenticationPlugin.decipherHeaderV2(PKIAuthenticationPlugin.java:244) ~[?:?]
at org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:166) ~[?:?]
at org.apache.solr.security.AuthenticationPlugin.authenticate(AuthenticationPlugin.java:91) ~[?:?]
at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:366) ~[?:?]
at org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:240) ~[?:?]
at org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:219) ~[?:?]
at org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:249) ~[?:?]
at org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:215) ~[?:?]
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:213) ~[?:?]
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195) ~[?:?]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210) ~[jetty-servlet-10.0.20.jar:10.0.20]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.20.jar:10.0.20]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) ~[jetty-servlet-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598) ~[jetty-security-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) ~[jetty-servlet-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:228) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301) ~[jetty-rewrite-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.Server.handle(Server.java:563) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461) ~[jetty-server-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:208) ~[http2-common-10.0.20.jar:10.0.20]
at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:155) ~[http2-common-10.0.20.jar:10.0.20]
at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:450) ~[http2-common-10.0.20.jar:10.0.20]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) ~[jetty-io-10.0.20.jar:10.0.20]
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) ~[jetty-io-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) ~[jetty-util-10.0.20.jar:10.0.20]
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) ~[jetty-util-10.0.20.jar:10.0.20]
at java.base/java.lang.Thread.run(Unknown Source) [?:?]
2024-09-06 07:25:58.166 WARN (qtp1212191909-54-solr-solrcloud-0.solr-solrcloud-headless.solr-345) [c: s: r: x: t:solr-solrcloud-0.solr-solrcloud-headless.solr-345] o.a.s.s.PKIAuthenticationPlugin Key is null when attempting to validate signature; skipping...
2024-09-06 07:25:58.167 ERROR (qtp1212191909-54-solr-solrcloud-0.solr-solrcloud-headless.solr-345) [c: s: r: x: t:solr-solrcloud-0.solr-solrcloud-headless.solr-345] o.a.s.s.PKIAuthenticationPlugin Could not validate PKI header.
2024-09-06 07:25:58.188 ERROR (OverseerThreadFactory-19-thread-1) [c:testing s: r: x: t:] o.a.s.c.a.c.CollectionHandlingUtils Error from shard: http://solr-solrcloud-0.solr-solrcloud-headless.solr:8983/solr => org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error from server at http://solr-solrcloud-0.solr-solrcloud-headless.solr:8983/solr/admin/cores: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html.

org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error from server at http://solr-solrcloud-0.solr-solrcloud-headless.solr:8983/solr/admin/cores: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <title>Error 401 Could not validate PKI header.</title>

HTTP ERROR 401 Could not validate PKI header.

URI:/solr/admin/cores
STATUS:401
MESSAGE:Could not validate PKI header.
SERVLET:default

@khandnb
Copy link
Author

khandnb commented Sep 19, 2024

hi @janhoy @HoustonPutman any idea here?

@VitalijusDBentley
Copy link

VitalijusDBentley commented Nov 27, 2024

Knowledge sharing, because documentation does not disclose this

Solr version: 9.5
source code
When enabling authentication you need to allow /admin/info/key public access, otherwise Solr will not be able to get keys from incoming request node during RequestForwarding process. Why Public Keys are not saved in zookeeper?

Also looks like solr does not pass JWT Principal with its full claims and it's looses scopes/roles(looks like a bug).
You need to add all your principals into Basic authentication access matrix so RequestForwarding will work.

Configuration MultiAuthRuleBasedAuthorizationPlugin that works for me

{
    "authorization": {
        "class": "solr.MultiAuthRuleBasedAuthorizationPlugin",
        "permissions": [{
                "collection": null,
                "name": "inner-node-comm",
                "path": "/admin/info/key",
                "role": null
            }, {
                "collection": null,
                "method": ["HEAD", "GET"],
                "name": "k8s-probe-0",
                "path": "/admin/info/system",
                "role": null
            }, {
                "collection": null,
                "method": ["HEAD", "GET"],
                "name": "k8s-probe-1",
                "path": "/admin/info/health",
                "role": null
            }, {
                "collection": null,
                "method": ["HEAD", "GET"],
                "name": "k8s-metrics",
                "path": "/admin/metrics",
                "role": null
            }, {
                "collection": null,
                "method": ["HEAD", "GET"],
                "name": "k8s-zk",
                "path": "/admin/zookeeper/status",
                "role": null
            }, {
                "collection": "*",
                "method": ["HEAD", "GET"],
                "name": "k8s-ping",
                "path": "/admin/ping",
                "role": null
            }, {
                "collection": null,
                "method": ["HEAD", "GET"],
                "name": "k8s-collection",
                "params": {
                    "action": ["LIST", "CLUSTERSTATUS"]
                },
                "path": "/admin/collections",
                "role": null
            }, {
                "name": "health",
                "role": null
            }, {
                "name": "metrics-read",
                "role": null
            }, {
                "name": "security-read",
                "role": ["admin", "k8s"]
            }, {
                "name": "security-edit",
                "role": ["admin", "k8s"]
            }, {
                "name": "schema-edit",
                "role": ["admin", "k8s"]
            }, {
                "name": "schema-read",
                "role": ["admin", "k8s"]
            }, {
                "name": "config-read",
                "role": ["admin", "k8s"]
            }, {
                "name": "config-edit",
                "role": ["admin", "k8s"]
            }, {
                "name": "core-admin-edit",
                "role": null
            }, {
                "name": "core-admin-read",
                "role": null
            }, {
                "name": "collection-admin-read",
                "role": ["admin", "k8s"]
            }, {
                "name": "collection-admin-edit",
                "role": ["admin", "k8s"]
            }, {
                "name": "update",
                "role": ["admin", "k8s"]
            }, {
                "name": "read",
                "role": ["admin", "k8s"]
            }, {
                "name": "zk-read",
                "role": ["admin", "k8s"]
            }, {
                "name": "all",
                "role": ["admin", "k8s"]
            }
        ],
        "schemes": [{
                "class": "solr.ExternalRoleRuleBasedAuthorizationPlugin",
                "scheme": "Bearer"
            }, {
                "class": "solr.RuleBasedAuthorizationPlugin",
                "scheme": "Basic",
                "user-role": {
                    "admin": ["admin", "k8s"],
                    "k8s-oper": ["k8s"],
                    "{{JWT.(clientId|name|email|...)}}": ["admin"]
                }
            }
        ]
    }
}

MultiAuthPlugin configuration

{
    "authentication": {
        "class": "solr.MultiAuthPlugin",
        "schemes": [{
                "adminUiScope": "{{AdminScope}}",
                "blockUnknown": true,
                "class": "solr.JWTAuthPlugin",
                "issuers": [{
                        "clientId": "{{UIClientId}}",
                        "name": "admin-ui",
                        "wellKnownUrl": "https://{{IDP.Url}}/.well-known/openid-configuration"
                    }, {
                        "aud": "{{ApiAudience}}",
                        "name": "api",
                        "wellKnownUrl": "https://{{IDP.Url}}/.well-known/openid-configuration"
                    }
                ],
                "principalClaim": "client_id",
                "redirectUris": "https://localhost:8983/solr/",
                "rolesClaim": "scope",
                "scheme": "Bearer",
                "scope": "{{Scopes that will act as roles}}"
            }, {
                "blockUnknown": false,
                "class": "solr.BasicAuthPlugin",
                "credentials": {
                    "admin": "...",
                    "k8s-oper": "..."
                },
                "forwardCredentials": false,
                "realm": "Solr",
                "scheme": "Basic"
            }
        ]
    },
}

P.s. "blockUnknown": false, should be true, but Prometheus Exporter does NOT send basic authentication configured with basicauth variable.

@VitalijusDBentley
Copy link

Anyone looking for SolrPrometheusExporter 9.6 or older and having problems with Basic Auth and "blockUnknown": true upgrade exporter to 9.7 and change SolrPrometheusExporter CRD:

  customKubeOptions:
    podOptions:
      envVars:
        - name: CREDENTIALS
          valueFrom:
            secretKeyRef:
              key: {{keyName with combined `username:password` format }}
              name: {{secret name}}
  image:
    repository: library/solr
    tag: '9.7'

Command source
Solr Exporter source

@VitalijusDBentley
Copy link

@janhoy @HoustonPutman forgot to ping you. Hope this will help :)

@janhoy
Copy link
Contributor

janhoy commented Nov 27, 2024

/admin/info/key is always public no matter security.json. It’s a special case in the code. So must be a bug causing it to not be open for you?

@gerlowskija
Copy link
Contributor

Yeah, something smells "off" here. Our dispatch code has a pretty explicit exception for /admin/info/key 🤔

In any case, it sounds like some folks are able to get JWT+basic multi-auth working without issue in Kubernetes. So I suspect we can close this out?


Why Public Keys are not saved in zookeeper?

Others might easily see good reasons not to do this that I'm missing. Perhaps theres some security implication?

But it's an interesting idea. It kindof makes sense conceptually to have nodes put their public-key as the "content" of their live-node entry. And it'd save a lot of "fetch the public key" requests to boot. Maybe worth raising on the dev@ list to see what people think...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants