Add support to clear-buffer-cache.py for checking SPARK_HOME for slaves #342
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…spark-ec2/slaves . This is not consistent with the Spark setup guide. I added the ability to find the slaves file relative to the SPARK_HOME environmental variable, if set. If the slaves file cannot be found in either location, an error is raised.
|
Can one of the admins verify this patch? |
| machinesFile = spark_home + "/conf/slaves" | ||
| if not os.path.exists(machinesFile): | ||
| print "Could not find Spark slaves file. Based on SPARK_HOME, expected file to be located at %s" % machinesFile | ||
| sys.exit(1) |
There was a problem hiding this comment.
suggested refactor - lift the test out of the if spark_home is None block, use a generic message
…anged variable names to camel case for consistency, and prioritized EC2 location to maintain compatibility with original script.
|
Incorporated mattf's suggestion. I also updated the variables to use camel case to be consistent with the rest of the script and prioritized the EC2 file location to maintain compatibility with original version of script. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
clear-buffer-cache.py assumes the slaves file is located under /root/spark-ec2/slaves . This is not consistent with the Spark setup guide for custom deployments.
I added the ability to find the slaves file relative to the SPARK_HOME environmental variable, if set. If the slaves file cannot be found in either location, an error is raised.