Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(config): make dask autoscaler configurable #604

Merged
merged 1 commit into from
Nov 11, 2024

Conversation

Alputer
Copy link
Member

@Alputer Alputer commented Sep 20, 2024

Copy link

codecov bot commented Sep 20, 2024

Codecov Report

Attention: Patch coverage is 92.00000% with 2 lines in your changes missing coverage. Please review.

Project coverage is 76.61%. Comparing base (1515865) to head (db9ac05).
Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
reana_workflow_controller/consumer.py 33.33% 2 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #604      +/-   ##
==========================================
+ Coverage   76.46%   76.61%   +0.14%     
==========================================
  Files          17       17              
  Lines        1831     1847      +16     
==========================================
+ Hits         1400     1415      +15     
- Misses        431      432       +1     
Files with missing lines Coverage Δ
reana_workflow_controller/config.py 93.95% <100.00%> (+0.08%) ⬆️
reana_workflow_controller/dask.py 76.36% <100.00%> (+1.85%) ⬆️
reana_workflow_controller/consumer.py 55.21% <33.33%> (-0.07%) ⬇️

@Alputer Alputer changed the title Dask autoscaler config feat(dask): dask autoscaler config (#604) Sep 20, 2024
Alputer added a commit to Alputer/reana-workflow-controller that referenced this pull request Oct 31, 2024
@Alputer Alputer changed the title feat(dask): dask autoscaler config (#604) feat(dask): make dask autoscaler configurable Oct 31, 2024
Alputer added a commit to Alputer/reana-workflow-controller that referenced this pull request Nov 1, 2024
Alputer added a commit to Alputer/reana-workflow-controller that referenced this pull request Nov 4, 2024
@@ -290,6 +290,9 @@ def _parse_interactive_sessions_environments(env_var):
DASK_ENABLED = strtobool(os.getenv("DASK_ENABLED", "true"))
"""Whether Dask is enabled in the cluster or not"""

DASK_AUTOSCALER_ENABLED = strtobool(os.getenv("DASK_AUTOSCALER_ENABLED", "true"))
"""Whether dask autoscaler is enabled in the cluster or not"""
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cosmetics: s/dask/Dask/g in comments.

@@ -290,6 +290,9 @@ def _parse_interactive_sessions_environments(env_var):
DASK_ENABLED = strtobool(os.getenv("DASK_ENABLED", "true"))
"""Whether Dask is enabled in the cluster or not"""

DASK_AUTOSCALER_ENABLED = strtobool(os.getenv("DASK_AUTOSCALER_ENABLED", "true"))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding "true" vs "True", how things work if people use uppercase or lowercase Helm values?

We may want to be ready to support two scenarios, so do some lower() gymnastics here, as we do elsewhere, for example:

os.getenv("K8S_USE_SECURITY_CONTEXT", "True").lower() == "true"

P.S. We could amend other occurrences of this boolean handling, see:

$ grep -Ei '"(true|false)' */*/config.py
reana-commons/reana_commons/config.py:    os.getenv("K8S_USE_SECURITY_CONTEXT", "True").lower() == "true"
reana-db/reana_db/config.py:    os.getenv("SQLALCHEMY_POOL_PRE_PING", "False").lower() == "true"
reana-db/reana_db/config.py:    os.getenv("REANA_PERIODIC_RESOURCE_QUOTA_UPDATE_POLICY", "false")
reana-job-controller/reana_job_controller/config.py:    strtobool(os.getenv("REANA_DB_CLOSE_POOL_CONNECTIONS", "false"))
reana-server/reana_server/config.py:DASK_ENABLED = strtobool(os.getenv("DASK_ENABLED", "true"))
reana-server/reana_server/config.py:DASK_AUTOSCALER_ENABLED = strtobool(os.getenv("DASK_AUTOSCALER_ENABLED", "true"))
reana-server/reana_server/config.py:    os.getenv("REANA_USER_EMAIL_CONFIRMATION", "true")
reana-workflow-controller/reana_workflow_controller/config.py:    {"name": "WDB_NO_BROWSER_AUTO_OPEN", "value": "True"},
reana-workflow-controller/reana_workflow_controller/config.py:    os.getenv("REANA_OPENSEARCH_ENABLED", "false").lower() == "true"
reana-workflow-controller/reana_workflow_controller/config.py:    os.getenv("REANA_OPENSEARCH_USE_SSL", "false").lower() == "true"
reana-workflow-controller/reana_workflow_controller/config.py:DASK_ENABLED = strtobool(os.getenv("DASK_ENABLED", "true"))
reana-workflow-controller/reana_workflow_controller/config.py:DASK_AUTOSCALER_ENABLED = strtobool(os.getenv("DASK_AUTOSCALER_ENABLED", "true"))
reana-workflow-engine-cwl/reana_workflow_engine_cwl/config.py:MOUNT_CVMFS = os.getenv("REANA_MOUNT_CVMFS", "false")
reana-workflow-engine-cwl/reana_workflow_engine_cwl/config.py:WORKFLOW_KERBEROS = bool(strtobool(os.getenv("REANA_WORKFLOW_KERBEROS", "false")))
reana-workflow-engine-serial/reana_workflow_engine_serial/config.py:MOUNT_CVMFS = os.getenv("REANA_MOUNT_CVMFS", "false")
reana-workflow-engine-serial/reana_workflow_engine_serial/config.py:WORKFLOW_KERBEROS = bool(strtobool(os.getenv("REANA_WORKFLOW_KERBEROS", "false")))
reana-workflow-engine-snakemake/reana_workflow_engine_snakemake/config.py:MOUNT_CVMFS = os.getenv("REANA_MOUNT_CVMFS", "false")
reana-workflow-engine-snakemake/reana_workflow_engine_snakemake/config.py:WORKFLOW_KERBEROS = bool(strtobool(os.getenv("REANA_WORKFLOW_KERBEROS", "false")))
reana-workflow-engine-yadage/reana_workflow_engine_yadage/config.py:MOUNT_CVMFS = os.getenv("REANA_MOUNT_CVMFS", "false")
reana-workflow-engine-yadage/reana_workflow_engine_yadage/config.py:WORKFLOW_KERBEROS = bool(strtobool(os.getenv("REANA_WORKFLOW_KERBEROS", "false")))

@Alputer Alputer changed the title feat(dask): make dask autoscaler configurable feat(config): make dask autoscaler configurable Nov 11, 2024
@tiborsimko tiborsimko merged commit db9ac05 into reanahub:master Nov 11, 2024
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

feat(dask): make dask autoscaler optional
2 participants