Facing trouble while restoring solr in 8.11.3

[pritamdas99]

Hello.
I have deployed solr in my linode cluster.

apiVersion: solr.apache.org/v1beta1
kind: SolrCloud
metadata:
  name: solr
  namespace: demo
spec:
  replicas: 2
  solrImage:
    tag: 8.11.3
  solrModules:
  - s3-repository
  backupRepositories:
  - name: "linode-s3"
    s3:
      region: "us-east-1"
      bucket: "backuprepo"
      endpoint: "https://ap-south-1.linodeobjects.com"
      baseLocation: "/yyy/ton"
      credentials:
        accessKeyIdSecret:
            name: data
            key: accessKeyIdSecret
        secretAccessKeySecret:
            name: data
            key: secretAccessKeySecret

Now, I wanted to take backup manually using curl command.

~ $ curl -XPOST -k   "http://localhost:8983/solr/admin/collections?action=BACKUP&collection=book&location=yyy/ton&name=book-backup-opo&repository=linode-s3"

{
  "responseHeader":{
    "status":0,
    "QTime":17180},
  "success":{
    "solr-solrcloud-1.solr-solrcloud-headless.demo:8983_solr":{
      "responseHeader":{
        "status":0,
        "QTime":2351},
      "response":[
        "startTime","2024-07-01T08:39:56.144912Z",
        "indexFileCount",1,
        "uploadedIndexFileCount",1,
        "indexSizeMB",0.0,
        "uploadedIndexFileMB",0.0,
        "shard","shard2",
        "endTime","2024-07-01T08:39:57.202791Z",
        "shardBackupId","md_shard2_0"]},
    "solr-solrcloud-0.solr-solrcloud-headless.demo:8983_solr":{
      "responseHeader":{
        "status":0,
        "QTime":4842},
      "response":[
        "startTime","2024-07-01T08:39:54.938802Z",
        "indexFileCount",15,
        "uploadedIndexFileCount",15,
        "indexSizeMB",0.003,
        "uploadedIndexFileMB",0.003,
        "shard","shard1",
        "endTime","2024-07-01T08:39:59.686387Z",
        "shardBackupId","md_shard1_0"]}},
  "response":[
    "collection","book",
    "numShards",2,
    "backupId",0,
    "indexVersion","8.11.3",
    "startTime","2024-07-01T08:39:53.517537Z",
    "indexSizeMB",0.003]}

So, backup went find there.

But I am facing an error in case of restore.

Make sure to delete the collection before restoring.

curl -XPOST "http://localhost:8983/solr/admin/collections?action=RESTORE&collection=book&location=yyy/ton&name=book-backup-opo&repository=linode-s3"
{
  "responseHeader":{
    "status":500,
    "QTime":1016},
  "Operation restore caused exception:":"org.apache.solr.s3.S3NotFoundException:org.apache.solr.s3.S3NotFoundException: An AmazonServiceException was thrown! [serviceName=S3] [awsRequestId=tx00000a4cbadee5ce3c705-0066826b92-7b276391-default] [httpStatus=404] [s3ErrorCode=NoSuchKey] [message=The specified key does not exist.]",
  "exception":{
    "msg":"An AmazonServiceException was thrown! [serviceName=S3] [awsRequestId=tx00000a4cbadee5ce3c705-0066826b92-7b276391-default] [httpStatus=404] [s3ErrorCode=NoSuchKey] [message=The specified key does not exist.]",
    "rspCode":-1},
  "error":{
    "metadata":[
      "error-class","org.apache.solr.common.SolrException",
      "root-error-class","org.apache.solr.common.SolrException"],
    "msg":"An AmazonServiceException was thrown! [serviceName=S3] [awsRequestId=tx00000a4cbadee5ce3c705-0066826b92-7b276391-default] [httpStatus=404] [s3ErrorCode=NoSuchKey] [message=The specified key does not exist.]",
    "trace":"org.apache.solr.common.SolrException: An AmazonServiceException was thrown! [serviceName=S3] [awsRequestId=tx00000a4cbadee5ce3c705-0066826b92-7b276391-default] [httpStatus=404] [s3ErrorCode=NoSuchKey] [message=The specified key does not exist.]\n\tat org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:65)\n\tat org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:301)\n\tat org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:257)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:216)\n\tat org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:843)\n\tat org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:807)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:545)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:437)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:367)\n\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)\n\tat org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)\n\tat org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)\n\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)\n\tat org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:516)\n\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)\n\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)\n\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)\n\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)\n\tat java.base/java.lang.Thread.run(Unknown Source)\n",
    "code":500}}

[jpholtmeyer]

I also received this issue. It seems that there is an extra forward slash being placed at the end of the S3 file path when Solr is trying to upload missing configsets to ZooKeeper.

In my case, Solr is trying to find dictionary.txt/ rather than dictionary.txt.

The workaround was to manually upload the configsets to ZooKeeper.
If the configsets already exist in ZooKeeper then Solr won't attempt to upload the configsets.

https://github.com/apache/lucene-solr/tree/branch_8_11/solr/contrib/s3-repository/src/java/org/apache/solr/s3