-
Notifications
You must be signed in to change notification settings - Fork 84
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Deprecated get_s3fs_session #768
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Sherwin-14, please read through the conversation again: #766
tl;dr: (1) you must deprecate get_s3fs_session
, (2) the new name should be get_s3_filesystem
Oops I missed that! There are two methods of deprecation(if I am not wrong) which one should we choose here? Should we make a change in docstrings or should we add a |
For full details on use of the In summary, here's what you need to do (with an example afterward):
Example: In def get_s3fs_session(
self,
daac: Optional[str] = None,
concept_id: Optional[str] = None,
provider: Optional[str] = None,
endpoint: Optional[str] = None,
) -> s3fs.S3FileSystem:
# Existing implementation Here's how we want to change things: @deprecated("Use get_s3_filesystem instead")
def get_s3fs_session(
self,
daac: Optional[str] = None,
concept_id: Optional[str] = None,
provider: Optional[str] = None,
endpoint: Optional[str] = None,
) -> s3fs.S3FileSystem:
# COPY OVER THE DOCSTRING HERE
return self.get_s3_filesystem(daac, concept_id, provider, endpoint)
def get_s3_filesystem(
self,
daac: Optional[str] = None,
concept_id: Optional[str] = None,
provider: Optional[str] = None,
endpoint: Optional[str] = None,
) -> s3fs.S3FileSystem:
# Existing implementation You also need to add this import to from typing_extensions import deprecated |
Neat, TIL! But I don't think EDIT: Nevermind :) Should have read this part:
|
ddcfb58
to
1a57e19
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Sherwin-14, there's a bit more to do than only deprecating and renaming the Store.get_s3fs_session
method.
Please do a search throughout the entire codebase for occurrences of get_s3fs_session
and determine the appropriate changes as follows:
- Change all references to the
Store.get_s3fs_session
method toget_s3_filesystem
so that within the codebase we're not using the newly deprecated method, but are instead using the newly introduced method that replaces it. - There is at least one other place where you must do the same thing you did in
store.py
. For example, inapi.py
, there is a top-levelget_s3fs_session
function. You must do the same thing with that function: deprecate it, copy it to a new function namedget_s3_filesystem
, and modify all of its references toearthaccess.__store__.get_s3fs_session
toearthaccess.__store__.get_s3_filesystem
. - Update references in
kerchunk.py
,test_auth.py
, andtest_store.py
(and see if there are any other places that I may have missed listing here)
Please also add an appropriate entry to CHANGELOG.md, but before doing so, please make sure you have merged the latest changes from the main
branch, as Luis just released v0.10.0, so he has updated CHANGELOG.md, so you want to pull in those changes before adding your own entry.
Once you have merged in the change made by Luis, you should add an entry above the new v0.10.0 heading that reads like so:
### Changed
- Deprecate `earthaccess.get_s3fs_session` and `Store.get_s3fs_session`. Use
`earthaccess.get_s3_filesystem` and `Store.get_s3_filesystem`, respectively,
instead ([#766](https://github.com/nsidc/earthaccess/issues/766))
In general, before opening a PR, you should locally run unit tests to make sure they pass. Further, after pushing up your work to your fork, but before opening a PR, you should check the success of the automated build. See https://github.com/Sherwin-14/earthaccess/actions/workflows/test.yml. If the build fails (because unit tests fail, for example), you should attempt to fix the failure.
Once you get things to work, or if you cannot determine how to fix things, you can then open a PR. In the case of opening the PR even when you cannot determine how to fix the broken build, you can ask for help to get things working.
@chuckwondo Oh I guess there is some confusion here. Maybe I wasn't able to communicate what I felt after making a PR, all I was asking was that should we replace all the occurrences of |
No worries. I'm happy to help you through the process. I'm also happy for us to collaborate on this during the next earthaccess hackday, which is tomorrow, if you're able to attend. If you're not able to attend tomorrow, I'm happy to schedule a pairing session with you some other time this week. |
@Sherwin-14, we'll also want to add a unit test to make sure the deprecation warning is actually emitted. For reference: https://docs.python.org/3/library/warnings.html#testing-warnings |
Did all the changes in the respective files. I am also working on coming up with the unit test for emitting deprecation warning as well, would probably amend in the same commit. |
This is something I came up with for the unit test. import warnings
import earthaccess
from earthaccess import api
from earthaccess.store import Store
def test_deprecation_warning():
store = earthaccess.store.Store()
with warnings.catch_warnings(record=True) as w:
# Cause all warnings to always be triggered.
warnings.simplefilter("always")
# Trigger a warning.
api.get_s3fs_session()
store.get_s3fs_session()
# Verify some things
assert len(w) == 2
assert issubclass(w[0].category, DeprecationWarning)
assert "Use get_s3_filesystem instead" in str(w[0].message)
assert issubclass(w[1].category, DeprecationWarning)
assert "Use get_s3_filesystem instead" in str(w[1].message) I ran into an error while running this test on my local machine which goes something like this |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking good! Just a couple of suggestions for adjusting the docstrings for the new functions. I'll reply separately to your question about the unit test you're working on.
@Sherwin-14, nice work so far! The unit test all pass now. @mfisher87, any idea why the integration tests are failing? I see that authentication is failing:
The workflow pulls secrets from the github environment, but that first line of output above shows that the username is an empty string. I don't have access to the environments in the repo, and I don't know whose EDL credentials are being used in the secrets. Do you know? |
@Sherwin-14, nice job on coming up with the initial unit test for this. Will you be at today's hackday? If so, we can continue with this then. |
Yes I would be there :) |
@Sherwin-14, thanks for joining today's hackday! Here's a summary of what we covered today:
Within the same file where you put your new test code shown above, also add this: @pytest.fixture(scope="session")
@responses.activate
@mock.patch("getpass.getpass")
@mock.patch("builtins.input")
def auth(user_input, user_password):
user_input.return_value = "user"
user_password.return_value = "password"
json_response = [
{"access_token": "EDL-token-1", "expiration_date": "12/15/2021"},
{"access_token": "EDL-token-2", "expiration_date": "12/16/2021"},
]
responses.add(
responses.GET,
"https://urs.earthdata.nasa.gov/api/users/tokens",
json=json_response,
status=200,
)
responses.add(
responses.GET,
"https://urs.earthdata.nasa.gov/profile",
json={"email_address": "[email protected]"},
status=200,
)
earthdata.login(strategy="interactive")
return earthdata.__auth__ Then you should be able to modify your test like so: def test_deprecation_warning(auth):
store = earthaccess.store.Store(auth)
... I think that should work, or at least get you much closer. |
Co-authored-by: Chuck Daniels <[email protected]>
Co-authored-by: Chuck Daniels <[email protected]>
@mfisher87, can you tell if the failing integration tests are related to the recent integration test troubles you've been working on recently? |
Hey just a small doubt. What is earthdata in the above code for the authentication. I am running into an attribute error and I am not sure what is earthdata in this case? |
It's a typo. Apologies. It should be |
Thanks this helps! But then should'nt it be |
Not necessary, as the functions in api.py are exported at the top level. |
I ran into this error while using earthacess instead of earthdata This something different now where can we access daac and provider? |
Just closing the loop -- we think we need to set up credentials in the fork repo. More details in #773 #736 |
Ah, okay. I think we can ignore the ValueError and still get the deprecation warning we expect. So instead of this line:
do this to suppress the ValueError:
and at the top of the file, add this import:
|
Just linking this PR to it's issue #766 |
60fe156
to
40edbc4
Compare
Co-authored-by: Chuck Daniels <[email protected]>
I have added unit test for deprecations |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Sherwin-14, nicely done!
@mfisher87, regarding the failing integration tests, do you think that once this PR is merged that perhaps they will succeed because they should end up pulling the creds from the nsidc/earthaccess repo? Any harm in merging and trying?
Awesome job @Sherwin-14 ! I do think they should pass on the main branch after a merge, because our repository credentials will be used. Would you be able to keep an eye on the tests after merge, and if they don't pass, open an issue? Remember, the "10% rule" is still in place, so even if the integration tests say they passed, up to 10% may have failed. We really need to get on top of fixing that! |
So was my BASE repo theory right? I am kind of still confused this is quite complicated |
I agree it is quite complicated. I think at this point we really don't know the cause, but we expect things to work after merging to main because the complexity of the fork rules are out of the picture :) |
@mfisher87, shall I merge? @Sherwin-14 are you okay if I do a squash merge? |
I'm good with a merge, but for whoever clicks merge: can you please keep an eye on the tests on the main branch to ensure they do pass? I am concerned that if they fail we may not notice right away. |
Yeah I am okay with that |
Fixes #766 |
I have made the changes. Do we update all the instances of get_s3fs_session now or is that something to be decided later?
📚 Documentation preview 📚: https://earthaccess--768.org.readthedocs.build/en/768/