[Cylon-RP] Cylon scaling test with radical-pilot #661
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Arup Sarker <[email protected]>
arupcsedu
requested review from
laszewsk,
mstaylor,
nirandaperera and
kaiyingshan
Signed-off-by: Arup Sarker <[email protected]>
Signed-off-by: Arup Sarker <[email protected]>
Signed-off-by: Arup Sarker <[email protected]>
Signed-off-by: Arup Sarker <[email protected]>
Signed-off-by: Arup Sarker <[email protected]>
Signed-off-by: Arup Sarker <[email protected]>
Signed-off-by: Arup Sarker <[email protected]>
Signed-off-by: Arup Sarker <[email protected]>
Signed-off-by: Arup Sarker <[email protected]>
Signed-off-by: Arup Sarker <[email protected]>
Signed-off-by: Arup Sarker <[email protected]>
nirandaperera
requested changes
Aug 3, 2023
.github/workflows/c-cpp.yml
Outdated
| @@ -22,7 +22,7 @@ jobs: | |||
| - name: Installing UCX | |||
| run: wget https://github.com/openucx/ucx/releases/download/v1.13.1/ucx-1.13.1.tar.gz -P $HOME/ucx && (cd $HOME/ucx && tar xzf ucx-1.13.1.tar.gz) && (cd $HOME/ucx/ucx-1.13.1 && ./contrib/configure-release --prefix=$PWD/install --with-go=no && make -j8 install) | |||
| - name: Build and test | |||
| run: python3 -m venv ENV && ./build.sh -pyenv $(pwd)/ENV -bpath $(pwd)/build --cpp --test --python --pytest --cmake-flags "-DCYLON_UCX=ON -DUCX_LIBDIR=$HOME/ucx/ucx-1.13.1/install/lib -DUCX_INCLUDEDIR=$HOME/ucx/ucx-1.13.1/install/include" | |||
| run: python3 -m venv ENV && ./build.sh -pyenv $(pwd)/ENV -bpath $(pwd)/build --cpp --test --cpp --python --pytest --cmake-flags "-DCYLON_UCX=ON -DUCX_LIBDIR=$HOME/ucx/ucx-1.13.1/install/lib -DUCX_INCLUDEDIR=$HOME/ucx/ucx-1.13.1/install/include" | |||
There was a problem hiding this comment.
NIT. --cpp is already there
Co-authored-by: niranda perera <[email protected]>
nirandaperera
approved these changes
Aug 3, 2023
cylon-manager
added a commit
that referenced
this pull request
Aug 3, 2023
* add redis * able to run without mpi * remove useless file * separate oob logic from ucx/ucc communicators * re-enable tests * code placements * minor fixes * added python script to run cylon ucx/ucc without mpirun * mimic gather with allgather * Cylon Release Version 0.6.0 (#650) Signed-off-by: Arup Sarker <[email protected]> * Fixes missing MPI_Comm in UCXConfig * [Cylon] Rivanna documentation issues #652 (#653) * [Cylon] Rivanna documentation issues #652 Signed-off-by: Arup Sarker <[email protected]> * Update README.md Improvements by Gregor * [Cylon] Updated Python version for Cylon version Signed-off-by: Arup Sarker <[email protected]> --------- Signed-off-by: Arup Sarker <[email protected]> Co-authored-by: Gregor von Laszewski <[email protected]> * Adds CYLON_USE_REDIS flag to allow UCC/UCX builds that don't require redis * Changes ucc_operations to reflect code in cylondata/cylon main * Changes adds defaults for CYLON_UCX, CYLON_UCC and CYLON_USE_REDIS * Changes adds defaults for CYLON_UCX, CYLON_UCC and CYLON_USE_REDIS * Changes adds defaults for CYLON_UCX, CYLON_UCC and CYLON_USE_REDIS * Changes adds defaults for CYLON_UCX, CYLON_UCC and CYLON_USE_REDIS * Adds missing constructor for UCXUCCCommunicator from main * Further resolution of differences between main and branch * Fix merge issue where CommType Type() was private and results in a conflict in python tests * Fix issue with UCX build where CreateChannel in UCX Communicator is not implemented * Fixes issue with UCX (non-UCC) tests * Adds support for redis build git workflow * fix hiredis workflow * Adds support for redis build git workflow * Adds support for redis build git workflow * Adds support for redis build git workflow - root install via sudo * moves OOBType to separate hpp + cython support * adds oob_context cython + updates to build.py and setup.py in support of includes and linking libraries in python * adds oob_context cython + updates to build.py and setup.py in support of includes and linking libraries in python * adds oob_context cython + updates to build.py and setup.py in support of includes and linking libraries in python * adds oob_context cython + updates to build.py and setup.py in support of includes and linking libraries in python * adds oob_context cython + updates to build.py and setup.py in support of includes and linking libraries in python * separates redis oob contexts in separate source files to facilitate cython processing * refactoring related to non-redis environments + introducing UCXRedisOOBContext * refactoring related to non-redis environments + introducing UCXRedisOOBContext - part 2 (bug fixes) * introduces UCCRedisOOBContext and adds calls to wrapper class * adds UCC Config * adds UCC Config (removes redis hard dependency) * adds necessary hooks in lib.pxd, lib.pyx and context for initDistributed * includes aws cf scripts for redis and minor change for operator example to execute for a redis environment * minor change * minor changes to oob contexts to support Cython initialization + updated cylon-redis to reflect new cloudbank account * adds redis example and minor changes related to redis oob context * Cylon scaling operation on Rivanna (#657) * Cylon scaling on Rivanna Signed-off-by: Arup Sarker <[email protected]> * [Cylon] Fix scaling scripts and dependencies Signed-off-by: Arup Sarker <[email protected]> --------- Signed-off-by: Arup Sarker <[email protected]> * fixes for running redis example * updates to redis_example to take argument for world size, redis host and port and demo_join example * [Cylon-RP] Cylon scaling test with radical-pilot (#661) * [Cylon-RP] Cylon scaling test with radical-pilot Signed-off-by: Arup Sarker <[email protected]> * [Cylon-RP] Cylon scaling test with radical-pilot Signed-off-by: Arup Sarker <[email protected]> * [Cylon] Fix panda build error Signed-off-by: Arup Sarker <[email protected]> * [Cylon] Update numpy and panda version Signed-off-by: Arup Sarker <[email protected]> * [Cylon] Update panda version Signed-off-by: Arup Sarker <[email protected]> * [Cylon-radical pilot] Fix updated radical pilot script for scaling test Signed-off-by: Arup Sarker <[email protected]> * [Cylon] Fix Cylon CI system and add slice operation on scaling test Signed-off-by: Arup Sarker <[email protected]> * [Cylon] Fix cython version for CI build Signed-off-by: Arup Sarker <[email protected]> * [Cylon] Update new version Signed-off-by: Arup Sarker <[email protected]> * fix native build * [Cylon] Fix numpy and pyarrow dependencies Signed-off-by: Arup Sarker <[email protected]> * [Pyarrow] Update workspace file for CI build Signed-off-by: Arup Sarker <[email protected]> * update native script Signed-off-by: Arup Sarker <[email protected]> * Update build.sh Co-authored-by: niranda perera <[email protected]> * Update c-cpp.yml --------- Signed-off-by: Arup Sarker <[email protected]> Co-authored-by: niranda perera <[email protected]> * Update conda-cpp-redis.yml * Revert "Update conda-cpp-redis.yml" This reverts commit ba05f99. * minor changes --------- Signed-off-by: Arup Sarker <[email protected]> Co-authored-by: kaiyingshan <[email protected]> Co-authored-by: arupcsedu <[email protected]> Co-authored-by: Gregor von Laszewski <[email protected]> Co-authored-by: niranda perera <[email protected]> Co-authored-by: cylon-manager <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.