Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Uploading of files to Google Cloud Storage throwing errors #2939

Open
adesso-david opened this issue Dec 23, 2024 · 0 comments
Open

Uploading of files to Google Cloud Storage throwing errors #2939

adesso-david opened this issue Dec 23, 2024 · 0 comments
Assignees
Labels
priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@adesso-david
Copy link

Environment details

  • Programming language: Go
  • OS: Mac
  • Language runtime version: 1.23
  • Package version: google.golang.org/api v0.214.0

Steps to reproduce

The following code throws an error, indicating that the endpoint is not supporting the upload type "resumable"
Error 400: Unsupported upload type: resumable, badRequest

func (s *SearchService) UploadAsset(item *Item, content []byte) (*cloudsearch.UploadItemRef, error) {
	// Check if content is empty
	if len(content) == 0 {
		return nil, errors.New("empty content slice")
	}

	itemName := GenSha256(item.URL)
	fullItemName := fmt.Sprintf("datasources/%s/items/%s", item.Datasource, itemName)

	// Start an upload session
	uploadRequest := &cloudsearch.StartUploadItemRequest{}
	uploadItemRef, err := s.Client.Indexing.Datasources.Items.Upload(fullItemName, uploadRequest).Do()
	if err != nil {
		return nil, fmt.Errorf("failed to start upload session: %w", err)
	}

	// Create a Media content
	media := &cloudsearch.Media{
		ResourceName: uploadItemRef.Name,
	}

	chunkSize := googleapi.ChunkSize(256 * 1024)

	// Upload the content
	call := s.Client.Media.Upload(uploadItemRef.Name, media).Media(
		bytes.NewReader(content),
		chunkSize,
	)

	_, err = call.Do()
	if err != nil {
		if googleapi.IsNotModified(err) {
			s.Logger.LogInfo(fmt.Sprintf("%s already uploaded", item.URL))
			return uploadItemRef, nil
		}
		return nil, fmt.Errorf("failed to upload content: %w", err)
	}

	return uploadItemRef, nil
}

I tracked down the bug to this code snippet of the corresponding library

func (c *MediaUploadCall) doRequest(alt string) (*http.Response, error) {
	reqHeaders := gensupport.SetHeaders(c.s.userAgent(), "application/json", c.header_)
	body, err := googleapi.WithoutDataWrapper.JSONBuffer(c.media)
	if err != nil {
		return nil, err
	}
	c.urlParams_.Set("alt", alt)
	c.urlParams_.Set("prettyPrint", "false")
	urls := googleapi.ResolveRelative(c.s.BasePath, "v1/media/{+resourceName}")
	if c.mediaInfo_ != nil {
		urls = googleapi.ResolveRelative(c.s.BasePath, "/upload/v1/media/{+resourceName}")
		c.urlParams_.Set("uploadType", c.mediaInfo_.UploadType())
	}
	newBody, getBody, cleanup := c.mediaInfo_.UploadRequest(reqHeaders, body)
	defer cleanup()
	urls += "?" + c.urlParams_.Encode()
	req, err := http.NewRequest("POST", urls, newBody)
	if err != nil {
		return nil, err
	}
	req.Header = reqHeaders
	req.GetBody = getBody
	googleapi.Expand(req.URL, map[string]string{
		"resourceName": c.resourceName,
	})
	c.s.logger.DebugContext(c.ctx_, "api request", "serviceName", apiName, "rpcName", "cloudsearch.media.upload", "request", internallog.HTTPRequest(req, body.Bytes()))
	return gensupport.SendRequest(c.ctx_, c.s.client, req)
}

It seems to me, that there is currently no possible way to upload files to the Google Cloud Search using the provided api, since the upload in a single chunk is also not working as expected. The same code is executed, but the response body is empty, even though the code expects it to be not empty (as far as I understand).

Sadly the little documentation on these GCS Endpoints is not very helpful, because it is nowhere described what query parameters are available and how to use them (e.g. "uploadType")

@adesso-david adesso-david added priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Dec 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Development

No branches or pull requests

2 participants