Access denied (anonymous user) error when using AsynqMuxServer running on Google Kubernetes Engine(GKE) to access Google Cloud Storage(GCS)

[ns-singhh]

Hello,

Background

I have a simple a service written in Go that receives a request and makes a read request to the Google Cloud Storage(GCS) bucket. This service is deployed in Google Kubernetes Engine(GKE) in the same project as the GCS bucket. GKE is using Workload Identity User to make requests to the storage bucket. This workload identity user is bound to one of the Service Accounts which has the storage admin permissions on the bucket that the service wants to read from.

All in all, I have verified, by using a Python script that uses the aforementioned Workload Identity User to make read/write requests to the storage bucket, that permissions and access are working correctly. My service leverages Asynq to accept the incoming requests asynchronously. Asynq mux server is started as part of the service's constructor using a go routine:

go func() {
		_, err = StartAsynqMuxServer(...)
                if err != nil {
                    // Panic
                }
}()

Problem

I am able to enqueue requests. Requests get picked up by the AsynqMuxServer using its HandleFunc function. However, when a call is made to the storage bucket, I get the following error:
"error":"googleapi: got HTTP response code 403 with body: <?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource (or it may not exist).</Details></Error>"

Given that all permissions and access are otherwise working, my suspicion is that the problem is with the AsynqMuxServer not utilizing the Workload Identity User which GCS recognizes.

Questions

• Has anyone tried using AsynqMuxServer where ultimately call is made to the other objects in infrastructure?
• Has anyone done this with GCP, in particular using Workload Identity User?
• Is there a need or a way to bind Workload Identity User to the AsynqMuxServer so that it doesn't think of the incoming requests being sent by 'anonymous user'?
  FYI @hibiken

Thank you!

[ns-singhh]

I was able to verify that Asynq isn't the issue here. Please feel free to remove this post, if needed.