From 113335e537a85e1e129cf762c95c26e0802faa60 Mon Sep 17 00:00:00 2001 From: meteorgan Date: Thu, 16 Jan 2025 17:54:41 +0800 Subject: [PATCH] rfc: write returns metadata --- .../docs/rfcs/0000_write_returns_metadata.md | 119 ++++++++++++++++++ 1 file changed, 119 insertions(+) create mode 100644 core/src/docs/rfcs/0000_write_returns_metadata.md diff --git a/core/src/docs/rfcs/0000_write_returns_metadata.md b/core/src/docs/rfcs/0000_write_returns_metadata.md new file mode 100644 index 000000000000..3d37e21c18c2 --- /dev/null +++ b/core/src/docs/rfcs/0000_write_returns_metadata.md @@ -0,0 +1,119 @@ +- Proposal Name: `write_returns_metadata` +- Start Date: 2025-01-16 +- RFC PR: [apache/opendal#0000](https://github.com/apache/opendal/pull/0000) +- Tracking Issue: [apache/opendal#0000](https://github.com/apache/opendal/issues/0000) + +# Summary + +Enhance write operations by returning metadata after successful writes. + +# Motivation + +Currently, write operations (`write`, `write_with`, `writer`, `writer_with`) only return `Result<()>` or `Result`. +Users who need metadata after writing (like `ETag` or `version_id`) must make an additional `stat()` call. This is inefficient +and can lead to race conditions if the file is modified between the write and stat operations. + +Many storage services (like S3, GCS, Azure Blob) return metadata in their write responses. We should expose this information +to users directly after write operations. + +# Guide-level explanation + +The write operations will be enhanced to return metadata: + +```rust +// Before +op.write("path/to/file", data).await?; +let meta = op.stat("path/to/file").await?; +if Some(etag) = meta.etag() { + println!("File ETag: {}", etag); +} + +// After +let meta = op.write("path/to/file", data).await?; +if Some(etag) = meta.etag() { + println!("File ETag: {}", etag); +} +``` + +For writer operations: + +```rust +// Before +let mut writer = op.writer("path/to/file").await?; +writer.write(data).await?; +writer.close().await?; +let meta = op.stat("path/to/file").await?; +if Some(etag) = meta.etag() { + println!("File ETag: {}", etag); +} + +// After +let mut writer = op.writer("path/to/file").await?; +writer.write(data).await?; +let meta = writer.close().await?; +if Some(etag) = meta.etag() { + println!("File ETag: {}", etag); +} +``` + +The behavior remains unchanged if users don't need the metadata - they can simply ignore the return value. + +# Reference-level explanation + +## Changes to `Operator` API + +The following functions will be modified to return `Result` instead of `Result<()>`: + +- `write()` +- `write_with()` + +The `writer()` and `writer_with()` return types remain unchanged as they return `Result`. + +## Changes to struct `types::write::writer::Writer` + +The `Writer` struct will be modified to return `Result` instead of `Result<()>` for the `close()` function. + +## Changes to trait `oio::Write` and trait `oio::MultipartWrite` + +The `Write` trait will be modified to return `Result` instead of `Result<()>` for the `close()` function. + +The `MultipartWrite` trait will be modified to return `Result` instead of `Result<()>` for the `complete_part()` +and `write_once` function. + +## Implementation Details + +For services that return metadata in their write responses: +- The metadata will be captured from the service response +- All available fields (etag, version_id, etc.) will be populated + +For services that don't return metadata in write responses: +- A default metadata object will be returned + + +# Drawbacks + +- Minor breaking change for users who explicitly type the return value of write operations +- Additional complexity in the Writer implementation + +# Rationale and alternatives + +- Provides a clean, consistent API +- Maintains backward compatibility for users who ignore the return value +- Improves performance by avoiding additional stat calls when possible + +# Prior art + +Similar patterns exist in other storage SDKs: + +- `object_store` crate returns metadata in `PutResult` after calling `put_opts` +- AWS SDK returns metadata in `PutObjectOutput` +- Azure SDK returns `UploadFileResponse` after uploads + +# Unresolved questions + +- None + + +# Future possibilities + +- None \ No newline at end of file