Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Jobs] Managed jobs database use WAL mode #4283

Merged
merged 4 commits into from
Nov 8, 2024
Merged

Conversation

Michaelvll
Copy link
Collaborator

@Michaelvll Michaelvll commented Nov 7, 2024

WAL is more robust for more parallel access, as we have done for #3863 #3923

Tested (run the relevant ones):

  • Code formatting: bash format.sh
  • Any manual or new tests for this PR (please specify below)
    • Launch 1000 jobs with a 64 core CPU controller.
      import subprocess
      from multiprocessing.pool import ThreadPool
      
      def run_task(task):
          print(f'Running task {task}')
          subprocess.run(
              f'sky jobs launch -n job-{task} -dy --fast --cloud aws -t t3.medium --use-spot "echo hi {task}; sleep 3600"',
              shell=True
          )
      
      with ThreadPool(8) as pool:
          pool.map(run_task, range(1000))
      
      
  • All smoke tests: pytest tests/test_smoke.py
  • Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
  • Backward compatibility tests: conda deactivate; bash -i tests/backward_compatibility_tests.sh

@@ -34,10 +35,6 @@ def _get_db_path() -> str:

_DB_PATH = _get_db_path()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move this closer to use site?

(Technically, googlestyle defines the ordering of members, funcs, etc.; but we haven't been following that ordering consistently.)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these formatting changes from the recent pylint hook?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we recently updated pylint to use sky/ instead of sky which includes those files without __init__.py in the parent folder. #4184

@Michaelvll Michaelvll added this pull request to the merge queue Nov 8, 2024
Merged via the queue into master with commit 4420fd5 Nov 8, 2024
18 of 20 checks passed
@Michaelvll Michaelvll deleted the make-jobs-launch-robust branch November 8, 2024 03:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants