Use temp .netrc file for integration tests and support NETRC environment variable #808

chuckwondo · 2024-09-16T21:00:06Z

This primary intent for the PR was to address #806. Because the changes required for #806 also produced at least partial completion of #743 and #480, it also includes the full set of changes for them as well. All 3 issues are very closely related, so it didn't make much sense to attempt to separate them out.

As a result,

contributors may now run integration tests locally without clobbering their existing ~/.netrc file, if they have one
because the bullet above required a means for configuring integration tests to use a temporary .netrc file location, it made sense to do so by setting a NETRC environment variable for this purpose, so users may now also set this environment variable, if they need to specify a different location for their .netrc file
to better enable local execution of integration tests, some refactoring was required, which led to simply folding in the changes required to fix Update integration tests to use pytest conventions #743 as well

Fixes #806
Fixes #743
Fixes #480

Pull Request (PR) draft checklist - click to expand

Please review our
contributing documentation
before getting started.
Ensure an issue exists representing the problem being solved in this PR.
Populate a descriptive title. For example, instead of "Updated README.md", use a
title such as "Add testing details to the contributor section of the README".
Populate the body of the pull request with:
- A clear description of the change you are proposing.
- Links to any issues resolved by this PR with text in the PR description, for
  example closes #1. See
  GitHub docs - Linking a pull request to an issue.
Update CHANGELOG.md with details about your change in a section titled
## Unreleased. If such a section does not exist, please create one. Follow
Common Changelog for your additions.
Update the documentation and/or the README.md with details of changes to the
earthaccess interface, if any. Consider new environment variables, function names,
decorators, etc.

Click the "Ready for review" button at the bottom of the "Conversation" tab in GitHub
once these requirements are fulfilled. Don't worry if you see any test failures in
GitHub at this point!

Pull Request (PR) merge checklist - click to expand

Please do your best to complete these requirements! If you need help with any of these
requirements, you can ping the @nsidc/earthaccess-support team in a comment and we
will help you out!

Add unit tests for any new features.
Apply formatting and linting autofixes. You can add a GitHub comment in this Pull
Request containing "pre-commit.ci autofix" to automate this.
Ensure all automated PR checks (seen at the bottom of the "conversation" tab) pass.
Get at least one approving review.

📚 Documentation preview 📚: https://earthaccess--808.org.readthedocs.build/en/808/

codecov-commenter · 2024-09-17T18:39:51Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 63.04348% with 34 lines in your changes missing coverage. Please review.

Please upload report for BASE (main@2939f80). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
earthaccess/auth.py	39.13%	14 Missing ⚠️
earthaccess/kerchunk.py	13.33%	13 Missing ⚠️
tests/integration/test_kerchunk.py	0.00%	4 Missing ⚠️
tests/integration/test_cloud_open.py	80.00%	1 Missing ⚠️
tests/integration/test_onprem_download.py	88.88%	1 Missing ⚠️
tests/integration/test_onprem_open.py	66.66%	1 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #808   +/-   ##
=======================================
  Coverage        ?   69.88%           
=======================================
  Files           ?       31           
  Lines           ?     2115           
  Branches        ?        0           
=======================================
  Hits            ?     1478           
  Misses          ?      637           
  Partials        ?        0

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Fixes nsidc#806 Fixes nsidc#743 Fixes nsidc#480

chuckwondo · 2024-09-23T10:23:56Z

tests/integration/test_onprem_download.py

+    #
+    # ORNLDAAC no longer has any on-prem collections.  This returns 0 collections:
+    # https://cmr.earthdata.nasa.gov/search/collections?data_center=ORNL_DAAC&cloud_hosted=false
+    # The following is commented out because the test in this file will now always fail
+    # because there are no longer any on-prem collections.
+    #
+    # {
+    #     "short_name": "ORNLDAAC",
+    #     "collections_count": 100,
+    #     "collections_sample_size": 3,
+    #     "granules_count": 100,
+    #     "granules_sample_size": 2,
+    #     "granules_max_size_mb": 50,
+    # },


Do we want to remove this completely?

chuckwondo · 2024-09-23T10:24:13Z

tests/integration/test_onprem_open.py

+    #
+    # ORNLDAAC no longer has any on-prem collections.  This returns 0 collections:
+    # https://cmr.earthdata.nasa.gov/search/collections?data_center=ORNL_DAAC&cloud_hosted=false
+    # The following is commented out because the test in this file will now always fail
+    # because there are no longer any on-prem collections.
+    #
+    # {
+    #     "short_name": "ORNLDAAC",
+    #     "collections_count": 100,
+    #     "collections_sample_size": 2,
+    #     "granules_count": 100,
+    #     "granules_sample_size": 2,
+    #     "granules_max_size_mb": 50,
+    # },


Do we want to remove this completely?

chuckwondo · 2024-09-23T10:35:54Z

Apologies for the relatively large PR. I could perhaps split out the changes for #743, if anybody feels this is too large.

I would at least like to keep the changes addressing both #480 and #806 together since the changes for #806 effectively necessitate the changes for #480.

This also now fixes #815. It made sense for me to add the changes for that issue so that I could actually run integration tests locally since this issue is precisely for handling integration tests.

Regardless, integration tests now run successfully on PRs from forks. For PRs from forks owned by those of use who are maintainers of this repo, the integration tests run successfully (barring actual failing tests, of course) without intervention, as we have write permission.

For PRs from forks owned by anybody without write access to this repo, the integration tests job will now fail due to a permission check, not due to failure to read the necessary secrets, but now (once this PR is merged) it will be possible for a maintainer to check the PR, and if all looks safe, re-run the failed job. When this happens, the job re-runs with the maintainer's permissions, and thus the integration tests will run (and ideally succeed as well).

This was referenced Sep 17, 2024

Use temp .netrc file for integration tests and support NETRC environment variable #809

Closed

Allow maintainer to re-run integration tests for PRs from forks #811

Closed

chuckwondo force-pushed the netrc-issue-480 branch 2 times, most recently from 2f1bd8a to 98ba9bb Compare September 17, 2024 18:38

chuckwondo added 10 commits September 21, 2024 06:49

Use temp .netrc file for integration tests

be0da17

Fixes nsidc#806 Fixes nsidc#743 Fixes nsidc#480

Suppress pip warning about running as root user

e40e07e

Check permission only for pull_request_target event

633f7ea

Remove unnecessary dup env vars

4b4edc5

Move project urls to correct section of pyproject.toml

10934dc

Correct Luis's email address

87f6a43

Move and complete kerchunk dependencies

9fef6cb

Add missing type annotation

f805cc1

Include kerchunk extra within test extra

7d50892

Add nox integration-tests session

9fea543

chuckwondo force-pushed the netrc-issue-480 branch from 98ba9bb to 9fea543 Compare September 22, 2024 23:10

chuckwondo added 2 commits September 22, 2024 20:29

Mindeps requires python 3.9, h5py 3.0+

510a330

Remove noise from CHANGELOG

04c6dcc

chuckwondo commented Sep 23, 2024

View reviewed changes

chuckwondo marked this pull request as ready for review September 23, 2024 10:24

chuckwondo requested review from mfisher87, betolink and jhkennedy September 23, 2024 10:24

Update uv.lock

6f645d3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use temp .netrc file for integration tests and support NETRC environment variable #808

Use temp .netrc file for integration tests and support NETRC environment variable #808

chuckwondo commented Sep 16, 2024 •

edited by github-actions bot

Loading

codecov-commenter commented Sep 17, 2024 •

edited

Loading

chuckwondo Sep 23, 2024

chuckwondo Sep 23, 2024

chuckwondo commented Sep 23, 2024

Use temp .netrc file for integration tests and support NETRC environment variable #808

Are you sure you want to change the base?

Use temp .netrc file for integration tests and support NETRC environment variable #808

Conversation

chuckwondo commented Sep 16, 2024 • edited by github-actions bot Loading

codecov-commenter commented Sep 17, 2024 • edited Loading

Codecov Report

chuckwondo Sep 23, 2024

Choose a reason for hiding this comment

chuckwondo Sep 23, 2024

Choose a reason for hiding this comment

chuckwondo commented Sep 23, 2024

chuckwondo commented Sep 16, 2024 •

edited by github-actions bot

Loading

codecov-commenter commented Sep 17, 2024 •

edited

Loading