Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tencent cloud cos(s3) upload failed #605

Closed
dispensable opened this issue Jul 28, 2023 · 2 comments · May be fixed by #790
Closed

Tencent cloud cos(s3) upload failed #605

dispensable opened this issue Jul 28, 2023 · 2 comments · May be fixed by #790
Labels
done Issues in the state 'done' stale

Comments

@dispensable
Copy link

dispensable commented Jul 28, 2023 •
edited
Loading

Project board link

I am using tencent cloud(cloud.tencent.com) cos (s3 compat service) for cassandra backup. It failed with uploadId None.

image

After some investigations, I find libcloud will use default aws NAMESPACE for xml response parse. But tencent cloud has their own xmlns field or in same case there is just no xmlns field exists. I can't find ns settings in ini file. So I fixed ns setting with this patch. But I am wondering if there is a option to setting xmlns field ? Or should i open a issue for libcloud lib and tencent maby ?

--- /usr/local/lib/python3.9/dist-packages/libcloud/storage/drivers/s3.py	2023-07-28 15:05:18.919248455 +0800
+++ s3_fixed.py	2023-07-28 15:04:26.435089224 +0800
@@ -108,7 +108,7 @@
 }
 
 API_VERSION = '2006-03-01'
-NAMESPACE = 'http://s3.amazonaws.com/doc/%s/' % (API_VERSION)
+NAMESPACE = "http://www.qcloud.com/document/product/436/7751"
 
 # AWS multi-part chunks must be minimum 5MB
 CHUNK_SIZE = 5 * 1024 * 1024
@@ -576,8 +576,13 @@
             raise LibcloudError('Error initiating multipart upload',
                                 driver=self)
 
-        return findtext(element=response.object, xpath='UploadId',
-                        namespace=self.namespace)
+        upload_id = findtext(element=response.object, xpath='UploadId',
+                             namespace=self.namespace)
+        if upload_id is None:
+            # tencent cos xml has no xmlns tag
+            return findtext(element=response.object, xpath='UploadId')
+        else:
+            return upload_id
 
     def _upload_multipart_chunks(self, container, object_name, upload_id,
                                  stream, calculate_hash=True):
@adejanovski adejanovski added the new Issues requiring triage label Jul 28, 2023
@rzvoncek
Copy link
Contributor

rzvoncek commented Mar 5, 2024

Hi @dispensable. If this is still relevant, can you please try with a recent medusa (0.19.1)? It uses different libraries to talk to the storage backends.

But in general, this looks like a very low-level thing where a S3-compatible vendor is not exactly compatible. It might be necessary to write a brand new storage implementation for this, but it's hard to tell.

@rzvoncek rzvoncek added help wanted Extra attention is needed stale and removed new Issues requiring triage to-groom labels Mar 5, 2024
@rzvoncek rzvoncek added help-wanted Issues in the state 'help-wanted' and removed help wanted Extra attention is needed labels Apr 8, 2024
@dispensable dispensable mentioned this issue Jul 11, 2024
Open
@dispensable
Copy link
Author

@rzvoncek hi, sorry for the delay. I have tried new s3 lib. it works pretty well except the tencent cloud need to force s3={"addressing_style": "virtual"} to the boto3 Config. I just submitted a pr to allow user config this opt.

thanks again

@adejanovski adejanovski added done Issues in the state 'done' and removed help-wanted Issues in the state 'help-wanted' labels Jul 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
done Issues in the state 'done' stale
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Support s3 address style config dispensable/cassandra-medusa
3 participants
@rzvoncek @dispensable @adejanovski