Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add github CLI to global-workflow-env #1270

Open
wants to merge 4 commits into
base: develop
Choose a base branch
from
Open

Conversation

eap
Copy link
Collaborator

@eap eap commented Aug 23, 2024

Add the github-cli to global-workflow-env

Testing

  • Using an unsubmitted comparison shell tool (coming soon) I compared the spack lock files of the unified-dev environment before and after addition of the gh command line tool to the template. No dependences are added or changed (other than gh) in order to include this CLI tool
  • I built and ran this code on an ubuntu 22.04 host in the unified-dev environment.

Issue(s) addressed

Fixes: #1105

Checklist

  • This PR addresses one issue/problem/enhancement, or has a very good reason for not doing so.
  • These changes have been tested on the affected systems and applications.
  • All dependency PRs/issues have been resolved and this PR can be merged.

@eap eap marked this pull request as ready for review August 23, 2024 04:21
Copy link
Collaborator

@climbfuji climbfuji left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed CI is failing, can you check please?

Also, I am torn. We should check the original issue, and if the users requesting gh cli want it for a specific application (e.g. global workflow), then it should either go in that virtual package or higher up in the tree if more applications want it.

The goal is to be able to load a single module (after the stack-* metamodules) such as module load global-workflow-env and having access to everything that is needed. Users shouldn't have to know all dependencies themselves, that's our job.

configs/templates/unified-dev/spack.yaml Outdated Show resolved Hide resolved
@climbfuji
Copy link
Collaborator

It seems like the permissions in the extracted gh cli source code are causing other PR's github actions to fail:

https://github.com/JCSDA/spack-stack/actions/runs/10526384237/job/29167119508?pr=1257

I'll remove the build caches manually on the systems for now so that #1257 can proceed, please fix this before pushing to the branch again:

rm: cache/build_stage/spack-stage-gh-2.50.0-gmwttfxsdsroeb4d6wsgtl66ydnxo26q/pkg/mod/github.com/cyberphone: Permission denied
rm: cache/build_stage/spack-stage-gh-2.50.0-gmwttfxsdsroeb4d6wsgtl66ydnxo26q/pkg/mod/github.com/pkg/[email protected]/example_test.go: Permission denied
rm: cache/build_stage/spack-stage-gh-2.50.0-gmwttfxsdsroeb4d6wsgtl66ydnxo26q/pkg/mod/github.com/pkg/[email protected]/errors_test.go: Permission denied
rm: cache/build_stage/spack-stage-gh-2.50.0-gmwttfxsdsroeb4d6wsgtl66ydnxo26q/pkg/mod/github.com/pkg/[email protected]/LICENSE: Permission denied
rm: cache/build_stage/spack-stage-gh-2.50.0-gmwttfxsdsroeb4d6wsgtl66ydnxo26q/pkg/mod/github.com/pkg/[email protected]/format_test.go: Permission denied
rm: cache/build_stage/spack-stage-gh-2.50.0-gmwttfxsdsroeb4d6wsgtl66ydnxo26q/pkg/mod/github.com/pkg/[email protected]/Makefile: Permission denied
rm: cache/build_stage/spack-stage-gh-2.50.0-gmwttfxsdsroeb4d6wsgtl66ydnxo26q/pkg/mod/github.com/pkg/[email protected]/go113_test.go: Permission denied
rm: cache/build_stage/spack-stage-gh-2.50.0-gmwttfxsdsroeb4d6wsgtl66ydnxo26q/pkg/mod/github.com/pkg/[email protected]/README.md: Permission denied
rm: cache/build_stage/spack-stage-gh-2.50.0-gmwttfxsdsroeb4d6wsgtl66ydnxo26q/pkg/mod/github.com/pkg/[email protected]/json_test.go: Permission denied
rm: cache/build_stage/spack-stage-gh-2.50.0-gmwttfxsdsroeb4d6wsgtl66ydnxo26q/pkg/mod/github.com/pkg/[email protected]/appveyor.yml: Permission denied
rm: cache/build_stage/spack-stage-gh-2.50.0-gmwttfxsdsroeb4d6wsgtl66ydnxo26q/pkg/mod/github.com/pkg/[email protected]/.gitignore: Permission denied
rm: cache/build_stage/spack-stage-gh-2.50.0-gmwttfxsdsroeb4d6wsgtl66ydnxo26q/pkg/mod/github.com/pkg/[email protected]/stack.go: Permission denied
rm: cache/build_stage/spack-stage-gh-2.50.0-gmwttfxsdsroeb4d6wsgtl66ydnxo26q/pkg/mod/github.com/pkg/[email protected]/stack_test.go: Permission denied
rm: cache/build_stage/spack-stage-gh-2.50.0-gmwttfxsdsroeb4d6wsgtl66ydnxo26q/pkg/mod/github.com/pkg/[email protected]/go113.go: Permission denied
rm: cache/build_stage/spack-stage-gh-2.50.0-gmwttfxsdsroeb4d6wsgtl66ydnxo26q/pkg/mod/github.com/pkg/[email protected]/bench_test.go: Permission denied
rm: cache/build_stage/spack-stage-gh-2.50.0-gmwttfxsdsroeb4d6wsgtl66ydnxo26q/pkg/mod/github.com/pkg/[email protected]/.travis.yml: Permission denied
rm: cache/build_stage/spack-stage-gh-2.50.0-gmwttfxsdsroeb4d6wsgtl66ydnxo26q/pkg/mod/github.com/pkg/[email protected]/errors.go: Permission denied
rm: cache/build_stage/spack-stage-gh-2.50.0-gmwttfxsdsroeb4d6wsgtl66ydnxo26q/pkg/mod/github.com/pkg/[email protected]: Permission denied
rm: cache/build_stage/spack-stage-gh-2.50.0-gmwttfxsdsroeb4d6wsgtl66ydnxo26q/pkg/mod/github.com/pkg: Permission denied
rm: cache/build_stage/spack-stage-gh-2.50.0-gmwttfxsdsroeb4d6wsgtl66ydnxo26q/pkg/mod/github.com: Permission denied
rm: cache/build_stage/spack-stage-gh-2.50.0-gmwttfxsdsroeb4d6wsgtl66ydnxo26q/pkg/mod: Permission denied
rm: cache/build_stage/spack-stage-gh-2.50.0-gmwttfxsdsroeb4d6wsgtl66ydnxo26q/pkg: Permission denied
rm: cache/build_stage/spack-stage-gh-2.50.0-gmwttfxsdsroeb4d6wsgtl66ydnxo26q: Permission denied

@climbfuji
Copy link
Collaborator

Looks like all the offending files are missing the write permission:

ip-10-0-0-117:spack-stack ec2-user$ ls -la cache/build_stage/spack-stage-gh-2.50.0-gmwttfxsdsroeb4d6wsgtl66ydnxo26q/pkg/mod/github.com/cyberphone
total 0
drwxr-xr-x   3 ec2-user  staff    96 Aug 23 04:33 .
drwxr-xr-x  79 ec2-user  staff  2528 Aug 23 04:33 ..
dr-xr-xr-x  16 ec2-user  staff   512 Aug 23 04:33 [email protected]

climbfuji pushed a commit that referenced this pull request Aug 23, 2024
A change in #1270 uncovered a behavior where go mod derived package install modules cannot be removed from our scratch directory due to undesirable folder permissions. This has been observed before and golang maintainers marked the behavior as WAI suggesting that people maintain these directory trees with go-language tools. In our case this is not highly feasible due to spack's intervention in the project structure meaning we must find a workaround.

The workaround here uses find to update directory permissions to ensure that write/execute is set on all directories in the tree. This has been tested locally but due to environment carryover we cannot ensure it has the desired effect for go packages in our CI system until it is submitted. (Ie; testing would require re-running an affected workflow and all other pull requests would still be subject to the issue and would break).
@eap eap changed the title Add github CLI to unified-dev and skylab-dev Add github CLI to global-workfflow-env Aug 23, 2024
@eap eap changed the title Add github CLI to global-workfflow-env Add github CLI to global-workflow-env Aug 23, 2024
@eap
Copy link
Collaborator Author

eap commented Aug 23, 2024

Thanks for the review - I've updated this per your suggestion; I'm still learning the various enviornment modules we manage. I'll watch the workflow results here to see if that CI fix we merged works as expected.

climbfuji
climbfuji previously approved these changes Aug 23, 2024
Copy link
Collaborator

@climbfuji climbfuji left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, thanks for making the updates. I'll go ahead and approve. Auto-merge will take care of the rest, assuming that the tests all pass.

@climbfuji climbfuji enabled auto-merge (squash) August 23, 2024 21:00
@climbfuji
Copy link
Collaborator

I see that the ubuntu intel CI test fails because of go. We should stay away from go. That's the reason why git-lfs is always an external package. go is difficult to build, if we can avoid it then let's do it. If not, we need to look into building go with GNU (along the lines of #1257

@climbfuji climbfuji dismissed their stale review August 26, 2024 02:57

Withdrawing my approval because I didn't realize that gh cli depends on "go" - need to take a closer look at the package

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

[INSTALL]: GitHub CLI
2 participants