Skip to content

Commit

Permalink
Increase oracle client timeout from 5 to 60 minutes.
Browse files Browse the repository at this point in the history
The timeout is so high to prevent a rare race condition from happening.
We need clients to wait till chief oracle server starts. This normally takes
a few minutes, but sometimes might take longer.
See keras-team#990 for more details.
Initially we didn't have any timeout. It was introduced to avoid tuner jobs
hanging forever if chief oracle stops responding.
See keras-team#957.
  • Loading branch information
onponomarev committed Mar 4, 2024
1 parent 7430e6c commit c21e192
Showing 1 changed file with 8 additions and 1 deletion.
9 changes: 8 additions & 1 deletion keras_tuner/distribute/oracle_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,14 @@
from keras_tuner.engine import hyperparameters as hp_module
from keras_tuner.engine import trial as trial_module

TIMEOUT = 5 * 60 # 5 mins
# The timeout is so high to prevent a rare race condition from happening.
# We need clients to wait till chief oracle server starts. This normally takes
# a few minutes, but sometimes might take longer.
# See https://github.com/keras-team/keras-tuner/issues/990 for more details.
# Initially we didn't have any timeout. It was introduced to avoid tuner jobs
# hanging forever if chief oracle stops responding.
# See https://github.com/keras-team/keras-tuner/pull/957.
TIMEOUT = 60 * 60 # 60 mins


class OracleClient:
Expand Down

0 comments on commit c21e192

Please sign in to comment.