Why does saving_lib/_load_model_from_fileobj unpack the archive?

[mmtrebuchet]

I came across a nasty race condition in _load_model_from_fileobj (which was recently patched by bba7b1a, kudos to @james77777778 for the fix) where two processes loading the model would each create a file named model_weights.h5 and then both would try to remove it. My workaround was to just raise an OSError manually so that the 'fallback' path is always used, but it got me wondering: Why unpack the zip file at all if we can extract the weights directly with H5IOStore(_VARS_NAME_H5, zf)? Is there a performance boost to extracting the weights beforehand? I'm on a NFS filesystem and it's really awful at working with many small files (or lots of seeks and small reads in large files), so if there's a way to reduce the number of file creation and deletion events, I'd be very happy.

For reference, loading up a 256KiB .keras file takes about five seconds on our system with the (buggy) 3.4.1 release.

Related, what's the timeline for releasing 3.4.2? I can't currently advise people to use my code on our institute cluster since they could encounter the race condition.

Cheers,
Charles.

[abhaskumarsinha]

Hello Charles, If possible can you try, backend based model saving/loading for the model and check if that is unique to Keras only or just some condition in the backend is causing it?

If possible prefer attaching a minimal Google Colab gist, that makes job easier to spot the bug and suggest a fix.

Thank You.

[mmtrebuchet]

Howdy, Abhas. The behavior is unique to Keras due to a logic error on lines 329-331 and 366-367 in saving_lib.py. The issue is that a file is created on disk and then removed, but another process could create and delete a file with the same name at the same time, causing the race. The bug was fixed by bba7b1a and I'm more curious about the reason for the behavior. Why are the weights extracted at all? It seems like an unnecessary filesystem operation.
Cheers,
Charles.

[james77777778]

Hi @mmtrebuchet
I have submitted a PR #19989 to improve saving/loading performance for small models.
Feel free to share your thoughts on it. Thanks.

[mmtrebuchet]

That looks great. The use of a tempfile should clear up the race, and getting a performance boost for small models is icing on the cake! Thanks for your hard work!
Cheers,
Charles.