egov-filestore pod gives 400 error when i try to upload any document #266

mannansiddiqui · 2023-11-01T07:31:36Z

mannansiddiqui
Nov 1, 2023

Hi Everyone,
I have create digit infra on AWS using https://github.com/egovernments/DIGIT-DevOps/tree/kubernetes-1.22-changes branch. Also, all modules are deployed using same branch.

When i try to upload any document while registering any complaint it gives 400 error in egov-filestore logs.

What version of DIGIT are you using?
---> v2.7
Are you attempting quickstart (single node deploy) or full installation of Digit?
---> full installation
Is this installed using a laptop or AWS/GCP/Azure or State Data Centre?
---> AWS
Please provide link to the documentation used while installing.
---> https://core.digit.org/guides/installation-guide/production-setup/aws
Please provide specific step number where error was encountered.
---> All steps done and all pods up and running but when i upload any document while registering any complaint it give 400 error in pod logs
What is the OS? Provide version as well. For eg. Ubuntu 20.04
---> For running terraform script and deploying helm charts is used ubuntu 22.04 and EKS worked nodes are using amazon-eks-node-1.28-v20231002 AMI
Number of cores/vCPUs and RAM available?
---> 4 vCPU and 16GB RAM per node and total 4 nodes running
Free RAM currently available.(How to find free memory - )
---> sufficient RAM is available i checked with kubectl describe nodes
Description of the error along with a screenshot. Please paste the exact error encountered on the screen.
Kubernetes logs (if applicable) or other relevant logs.
---> Logs of egov-filestore pod

2023-11-01 05:41:44.731+0000  INFO --- [080-exec-4] o.e.t.h.filters.TracerFilter             -- CORRELATION_ID=99b4ea00-6f30-48ad-9589-39f5707eed81 : Received request URI: https://egov-filestore.egov/filestore/v1/files  
2023-11-01 05:41:44.736+0000  INFO --- [080-exec-4] o.e.f.d.service.StorageService           -- CORRELATION_ID=99b4ea00-6f30-48ad-9589-39f5707eed81 : Received upload request for jurisdiction: %s, module: %s, tag: %s with file count: %s 
2023-11-01 05:41:44.869+0000  INFO --- [080-exec-4] o.e.f.r.impl.CloudFileMgrUtils           -- CORRELATION_ID=99b4ea00-6f30-48ad-9589-39f5707eed81 : Different versions of the image created! 
2023-11-01 05:41:45.867+0000 ERROR --- [080-exec-4] o.e.f.r.i.m.MinioRepository              -- CORRELATION_ID=99b4ea00-6f30-48ad-9589-39f5707eed81 : Error occurred:  

io.minio.errors.XmlParserException: null
	at io.minio.Xml.unmarshal(Xml.java:68)
	at io.minio.MinioClient.execute(MinioClient.java:1134)
	at io.minio.MinioClient.execute(MinioClient.java:1236)
	at io.minio.MinioClient.executePost(MinioClient.java:1388)
	at io.minio.MinioClient.createMultipartUpload(MinioClient.java:5092)
	at io.minio.MinioClient.putObject(MinioClient.java:3708)
	at io.minio.MinioClient.putObject(MinioClient.java:3870)
	at org.egov.filestore.repository.impl.minio.MinioRepository.push(MinioRepository.java:93)
	at org.egov.filestore.repository.impl.minio.MinioRepository.lambda$saveFiles$0(MinioRepository.java:74)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at org.egov.filestore.repository.impl.minio.MinioRepository.saveFiles(MinioRepository.java:69)
	at org.egov.filestore.persistence.repository.ArtifactRepository.save(ArtifactRepository.java:42)
	at org.egov.filestore.domain.service.StorageService.save(StorageService.java:86)
	at org.egov.filestore.web.controller.StorageController.storeFiles(StorageController.java:105)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:879)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at io.opentracing.contrib.web.servlet.filter.TracingFilter.doFilter(TracingFilter.java:165)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.egov.tracer.http.filters.TracerFilter.doFilter(TracerFilter.java:95)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:109)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
	at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1594)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

2023-11-01 05:41:45.868+0000 ERROR --- [080-exec-4] o.egov.tracer.ExceptionAdvise            -- CORRELATION_ID=99b4ea00-6f30-48ad-9589-39f5707eed81 : Exception caught in tracer  

java.lang.RuntimeException: Error in Configuration
	at org.egov.filestore.repository.impl.minio.MinioRepository.push(MinioRepository.java:99)
	at org.egov.filestore.repository.impl.minio.MinioRepository.lambda$saveFiles$0(MinioRepository.java:74)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at org.egov.filestore.repository.impl.minio.MinioRepository.saveFiles(MinioRepository.java:69)
	at org.egov.filestore.persistence.repository.ArtifactRepository.save(ArtifactRepository.java:42)
	at org.egov.filestore.domain.service.StorageService.save(StorageService.java:86)
	at org.egov.filestore.web.controller.StorageController.storeFiles(StorageController.java:105)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:879)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at io.opentracing.contrib.web.servlet.filter.TracingFilter.doFilter(TracingFilter.java:165)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.egov.tracer.http.filters.TracerFilter.doFilter(TracerFilter.java:95)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:109)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
	at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1594)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

2023-11-01 05:41:45.870+0000  INFO --- [080-exec-4] o.e.t.h.filters.TracerFilter             -- CORRELATION_ID=99b4ea00-6f30-48ad-9589-39f5707eed81 : Response code sent: 400

Below is the egov-demo.yml for egov-filestore

I tried adding minio.source: "minio" and changing minio.url from "http://minio.backbone:9000/" to "https://s3.amazonaws.com" as mentioned in document i.e. https://core.digit.org/platform/core-services/filestore-service

But 400 error is still there. So i removed minio.source: "minio" and revert minio.url to "http://minio.backbone:9000/" as it's not working

All pods are ruunning. I thing some issue in config of egov-filestore

Sharing the screenshot of kubectl describe pod egov-filestore -n egov for Environment variables:

Thanks in advance...

mannansiddiqui · 2023-11-02T05:56:47Z

mannansiddiqui
Nov 2, 2023
Author

This issue is resolved only i changed the region of s3 bucket from me-central-1 to us-east-1.
Don't know why it works only for us-east-1. It will be great if anyone from egovernments team can tell the reason @nikhilmulinti-egov @NeileGov @Karthik-egov @subhashini-egov @naresh-egov @shashwat-egov

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

egov-filestore pod gives 400 error when i try to upload any document #266

{{title}}

Replies: 1 comment

{{title}}

Select a reply

egov-filestore pod gives 400 error when i try to upload any document #266

mannansiddiqui Nov 1, 2023

Replies: 1 comment

mannansiddiqui Nov 2, 2023 Author

mannansiddiqui
Nov 1, 2023

mannansiddiqui
Nov 2, 2023
Author