You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When tf.io writes a file, it checks whether HasAtomicMove is true and if so, simulates atomic writes by first writing to a temporary file, then renaming the file to the correct name. This is great on a local filesystem. But for S3, this is undesirable behavior for a few reasons:
S3 writes are already atomic, so there is no need to simulate one.
In the AWS S3 SDK, which tf.io uses, moving a file is implemented as a copy+delete, which increases the load on S3 compared to a direct write.
The text was updated successfully, but these errors were encountered:
When
tf.io
writes a file, it checks whetherHasAtomicMove
is true and if so, simulates atomic writes by first writing to a temporary file, then renaming the file to the correct name. This is great on a local filesystem. But for S3, this is undesirable behavior for a few reasons:tf.io
uses, moving a file is implemented as a copy+delete, which increases the load on S3 compared to a direct write.The text was updated successfully, but these errors were encountered: