[wip] add support for Azure appendable blobs

[vilterp]

See https://learn.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs#about-append-blobs

[vilterp]

Locally I'm getting a Bad Request back from Azurite, sadly with no error message 😕

 caused by: HTTP.Exceptions.StatusError(400, "PUT", "/devstoreaccount1/jl-azurite-9952/my_append_blob?comp=block", HTTP.Messages.Response:
  """
  HTTP/1.1 400 Bad Request
  Server: Azurite-Blob/3.24.0
  Date: Tue, 12 Sep 2023 22:42:54 GMT
  Connection: keep-alive
  Keep-Alive: timeout=5
  Content-Length: 0
  
  """)
  Stacktrace:
   [1] (::HTTP.ExceptionRequest.var"#exceptions#2"{HTTP.ExceptionRequest.var"#exceptions#1#3"{CloudBase.var"#cloudsign#64"{CloudBase.var"#cloudsign#63#65"{typeof(HTTP.StreamRequest.streamlayer)}}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Response, HTTP.Connections.Connection{OpenSSL.SSLStream}}; status_exception::Bool, timedout::ConcurrentUtilities.TimedOut{HTTP.Messages.Response}, logerrors::Bool, logtag::Nothing, kw::Base.Pairs{Symbol, Any, NTuple{8, Symbol}, NamedTuple{(:iofunction, :decompress, :azure, :aws, :awsv2, :require_ssl_verification, :verbose, :credentials), Tuple{Nothing, Nothing, Bool, Bool, Bool, Bool, Int64, CloudBase.AzureCredentials}}})
     @ HTTP.ExceptionRequest ~/.julia/packages/HTTP/Y2JKB/src/clientlayers/ExceptionRequest.jl:19

Maybe it's missing a request header like Content-Length or something? I was hoping Azure.put would take care of that… 🤔

[codecov]

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.14% 🎉

Comparison is base (328b427) 83.13% compared to head (18d0dc3) 83.27%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #45      +/-   ##
==========================================
+ Coverage   83.13%   83.27%   +0.14%     
==========================================
  Files           7        7              
  Lines         587      592       +5     
==========================================
+ Hits          488      493       +5     
  Misses         99       99              

Files Changed	Coverage Δ
src/blobs.jl	90.19% <100.00%> (+1.06%)	⬆️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Drvi]

Thanks Pete, I like the idea! A couple of questions:

• What should happen if you put without appending to appendable blob? Would be nice to mention in a comment / docstring.
• How can the user know that a blob is appendable? Is that something a head would tell you?

[Drvi]

So ideally, we'd utilize the parallel multipart upload for larger files, as we do in API.putObjectImpl. This would need some tweaking because currently, API.uploadPart sets "comp" => "block" (couple of lines above in this file), so we'd probably want to propagate the query to it.

[vilterp]

I think the main use case for appendable blobs vs block blobs (and certainly my use case) is to essentially use blobs as log storage for an ongoing process — I.e. you're appending events every couple seconds, over the course of minutes or hours, while being able to read back the whole thing at any time.

This could be used to build log storage for e.g. a cloud CI system, or (in our case) a cloud database system 😛

So anyway, I don't think we'd really benefit from parallelism here — since if you already have all the data up front, you can just use parallel multipart upload for a normal block blob.

[Drvi]

But if you need to append every couple of seconds or minutes say 32MiB chunks of data at a time, then you'd still benefit from multipart uploads, no?

[vilterp]

I suppose there could be big spikes in the log stream rate, but in my use case I haven't seen spikes big enough to necessitate uploading in parallel.

Looking at the Azure docs, I'm not sure append blobs support parallel appending: https://learn.microsoft.com/en-us/rest/api/storageservices/append-block?tabs=azure-ad#remarks As far as I can tell, once you make an append blob you can only append one at a time. Otherwise there'd have to be some scheme to tell it what block id you're appending; I don't see that in these docs.

[Drvi]

How about we make this functionality available via API.putObject(...; append=true) instead of create_append_blob / API.put(...; append=true) instead of append_block? It would keep similar functionality under similar API, but I'm open to alternatives, especially if there is a precedent.

[vilterp]

I basically mimicked the Python SDK

• see usage here: https://github.com/Azure/azure-sdk-for-python/blob/91f5a8bc5fb41273b348ee45023a715f26fb79ac/sdk/storage/azure-storage-blob/tests/test_append_blob.py#L63-L65
• docs here: https://learn.microsoft.com/en-us/python/api/azure-storage-blob/azure.storage.blob.appendblobservice.appendblobservice?view=azure-python-previous#azure-storage-blob-appendblobservice-appendblobservice-append-block

I kind of prefer the explicit names, but idk, it's a matter of taste 🤷‍♂️ The style of the repo does skew toward kwargs

[vilterp]

No longer planning on using this; will resurrect if needed