Thread-local Session Management and Cookie Reuse to Address EDL DSE issue

[hailiangzhang]

This pull request addresses a DSE issue reported by Earthdata Login (EDL) during data download requests involving large amount of granules through EarthAccess. One issue was traced to EarthAccess creating a new session for each download instead of reusing the existing session cookie, which is intended to reduce the number of requests to EDL by the DAAC data distribution endpoint (TEA). This PR introduces changes to enhance session cookie handling within each thread, avoiding EDL requests from TEA in subsequent authenticated data downloads.

Key modifications in this PR include:

• Thread-local Session Management: Added _clone_session_in_local_thread method to clone and store sessions in thread-local storage.
• Cookie Reuse in Downloads: Adapted _download_file method to utilize thread-local session cookies and reduce redundant session creation within each thread.
• Session Handling: Updated get_requests_session to retrieve an existing session or raise an error if none exists.

Notes:

• This PR does not address the cookie reuse issue for same-region access. I can create a separate PR for that later.
• I did not add or adjust unit tests because the current test coverage is limited and does not include the modified methods.
• I have issues running integration tests, so I am uncertain about their status with these changes.
• I didn't adjust the following PR template since I am not if my PR is fully applicable to it. Please advise if adjustments or compliance with the template are required.

Pull Request (PR) draft checklist - click to expand

• Please review our
  contributing documentation
  before getting started.
• Populate a descriptive title. For example, instead of "Updated README.md", use a
  title such as "Add testing details to the contributor section of the README".
  Example PRs: #763
• Populate the body of the pull request with:
  • A clear description of the change you are proposing.
  • Links to any issues resolved by this PR with text in the PR description, for
    example closes #1. See
    GitHub docs - Linking a pull request to an issue.
• Update CHANGELOG.md with details about your change in a section titled
  ## Unreleased. If such a section does not exist, please create one. Follow
  Common Changelog for your additions.
  Example PRs: #763
• Update the documentation and/or the README.md with details of changes to the
  earthaccess interface, if any. Consider new environment variables, function names,
  decorators, etc.

Click the "Ready for review" button at the bottom of the "Conversation" tab in GitHub
once these requirements are fulfilled. Don't worry if you see any test failures in
GitHub at this point!

Pull Request (PR) merge checklist - click to expand

Please do your best to complete these requirements! If you need help with any of these
requirements, you can ping the @nsidc/earthaccess-support team in a comment and we
will help you out!

• Add unit tests for any new features.
• Apply formatting and linting autofixes. You can add a GitHub comment in this Pull
  Request containing "pre-commit.ci autofix" to automate this.
• Ensure all automated PR checks (seen at the bottom of the "conversation" tab) pass.
• Get at least one approving review.

---

📚 Documentation preview 📚: https://earthaccess--909.org.readthedocs.build/en/909/

[github-actions]

👈 Launch a binder notebook on this branch for commit 9930d84

I will automatically update this comment whenever this PR is modified

[hailiangzhang]

Following my earlier note, This PR does not address the cookie reuse issue for same-region access. I can create a separate PR for that later., after looking closer into the code, for same-region direct S3 access, it appears we are not using multi-threading, and authentication occurs only once. Therefore, it should not trigger my EDL DSE (unless I am overlooking something).

[betolink]

Hi @hailiangzhang, I'm looking at the PR now. I think it would be useful to create an issue since it will be more visible to other users and this PR will be the fix to that. For integration tests, we need to export our EDL credentials and follow these instructions: https://earthaccess.readthedocs.io/en/latest/contributing/development/#running-tests let us know if they are clear otherwise we can work on improving them!

I'm going to start some unit tests for this. I think this is a major change in the way the download methods will work so we better test them extensively. Thanks for working on this!!

[hailiangzhang]

Hi @hailiangzhang, I'm looking at the PR now. I think it would be useful to create an issue since it will be more visible to other users and this PR will be the fix to that. For integration tests, we need to export our EDL credentials and follow these instructions: https://earthaccess.readthedocs.io/en/latest/contributing/development/#running-tests let us know if they are clear otherwise we can work on improving them!

I'm going to start some unit tests for this. I think this is a major change in the way the download methods will work so we better test them extensively. Thanks for working on this!!

Thanks @luzpaz for your suggestions -- I just created an issue report here. Please let me know if it was not created under the right category.

For the integration tests, the main issue is really coming from the laptop system setup I have. I can try to follow the instructions and will ping you if I have problems. Thanks!