Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support StripPrefix on upload for TransferManager #2573

Open
sydney-munro opened this issue Jun 4, 2024 · 1 comment
Open

Support StripPrefix on upload for TransferManager #2573

sydney-munro opened this issue Jun 4, 2024 · 1 comment
Labels
api: storage Issues related to the googleapis/java-storage API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@sydney-munro
Copy link
Collaborator

Currently transfer manager supports stripping a customer supplied prefix on an object on object download, the same functionality is not supported on upload and the name of the object maintains the full File name as the object name.

@product-auto-label product-auto-label bot added the api: storage Issues related to the googleapis/java-storage API. label Jun 4, 2024
@sydney-munro sydney-munro added the type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. label Jun 4, 2024
@chalmagr
Copy link

Adding scenarios to show some context - need some traction on this

Environment details

OS type and version: MacOS 14.6.1 / Windows 11 (issue happens on both)
Java version: Temurin-11.0.19+7 (build 11.0.19+7)
Version(s): 2.42.0

Steps to reproduce

  1. Implement the shared code in the documentation to upload multiple files (https://cloud.google.com/storage/docs/uploading-objects#storage-upload-object-java)
  2. Run the code trying to upload one directory (e.g. /Users/me/source/dir) to some location in a bucket: e.g. gs://my-bucket-name/destination (directory contents are file1.txt, file2.txt) (run the code with arguments

Code example

package org.example;

import com.google.cloud.storage.transfermanager.ParallelUploadConfig;
import com.google.cloud.storage.transfermanager.TransferManager;
import com.google.cloud.storage.transfermanager.TransferManagerConfig;
import com.google.cloud.storage.transfermanager.UploadResult;

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
import java.util.stream.Stream;

public class Main {
    public static void main(String[] args) throws IOException {
        uploadDirectoryContents(args[0], Paths.get(args[1]), args[2]);
    }

    private static void uploadDirectoryContents(String bucketName, Path sourceDirectory, String remotePath) throws IOException {
        TransferManager transferManager = TransferManagerConfig.newBuilder().build().getService();
        ParallelUploadConfig parallelUploadConfig = ParallelUploadConfig.newBuilder()
                .setBucketName(bucketName)
                .setPrefix(remotePath)
                .build();

        List<Path> filePaths = new ArrayList<>();
        try (Stream<Path> pathStream = Files.walk(sourceDirectory)) {
            pathStream.filter(Files::isRegularFile).forEach(filePaths::add);
        }
        List<UploadResult> results = transferManager.uploadFiles(filePaths, parallelUploadConfig).getUploadResults();

        for (UploadResult result : results) {
            System.out.println(
                    "Upload for "
                            + result.getInput().getName()
                            + " completed with status "
                            + result.getStatus());
        }
    }
}

Run the above example using arguments
my-bucket-name /Users/me/source/dir destination

I would expect that the bucket contains the following files now

gs://my-bucket-name/destination/file1.txt
gs://my-bucket-name/destination/file2.txt

However, the bucket contains the following (for a MacOS scenario)

gs://my-bucket-name/destination/Users/me/source/dir/file1.txt
gs://my-bucket-name/destination/Users/me/source/dir/file2.txt

For Windows, the /Users/me/source/dir/file1.txt gets changed to C:\Users\me\source\dir\file1.txt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the googleapis/java-storage API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
2 participants