AttributeError: 'DataGranules' object has no attribute 'headers' with python-cmr==0.7.0

[aldotapia]

With version 0.8.0, I'm getting AttributeError after querying.

import earthaccess

earthaccess.login()

results = earthaccess.granule_query().short_name('MOD13Q1').bounding_box(-73.73,-55.01,-67.05,-17.63).version('061').temporal('2023-01-01','2023-01-31').get_all()

The error is AttributeError: 'DataGranules' object has no attribute 'headers':

AttributeError                            Traceback (most recent call last)
Input In [7], in <cell line: 1>()
----> 1 results = earthaccess.granule_query().short_name('MOD13Q1').bounding_box(-73.73,-55.01,-67.05,-17.63).version('061').temporal('2023-01-01','2023-01-31').get_all()

File ~/miniforge3/envs/hidrocl/lib/python3.10/site-packages/cmr/queries.py:104, in Query.get_all(self)
     95 def get_all(self):
     96     """
     97     Returns all of the results for the query. This will call hits() first to determine how many
     98     results their are, and then calls get() with that number. This method could take quite
   (...)
    101     :returns: query results as a list
    102     """
--> 104     return self.get(self.hits())

File ~/miniforge3/envs/hidrocl/lib/python3.10/site-packages/earthaccess/search.py:331, in DataGranules.hits(self)
    322 """
    323 Returns the number of hits the current query will return. This is done by
    324 making a lightweight query to CMR and inspecting the returned headers.
    325 
    326 :returns: number of results reported by CMR
    327 """
    329 url = self._build_url()
--> 331 response = self.session.get(url, headers=self.headers, params={"page_size": 0})
    333 try:
    334     response.raise_for_status()

I tested past versions and I got the same problem with 0.7.0 and 0.7.1. I rolled back to 0.6.1 and it's working now.

Any hints?

[mfisher87]

Thanks for the report, especially for documenting all the versions you tested! ❤️

[MattF-NSIDC]

We tried to reproduce and had no luck. Is this still occurring for you?

If this was an intermittent issue with an external system, we should show a better error message.

[jrbourbeau]

For reference, I tried with 0.8.0 and 0.8.1, but both times wasn't able to reproduce any error

In [1]: import earthaccess

In [2]: results = earthaccess.granule_query().short_name('MOD13Q1').bounding_box(-73.73,-55.01,-67.05,-17.63).version('061').temporal('2023-01-01','2023-01-31').get_all()

In [3]: len(results)
Out[3]: 30

[aldotapia]

I tried to reproduce it in a fresh environment:

conda create -n "test" python=3.10
conda activate test
pip install earthaccess
python

import earthaccess
earthaccess.login()
results = earthaccess.granule_query().short_name('MOD13Q1').bounding_box(-73.73,-55.01,-67.05,-17.63).version('061').temporal('2023-01-01','2023-01-31').get_all()

And it worked well. I tested again in the original environment where I found the error and got the same result:

results = earthaccess.granule_query().short_name('MOD13Q1').bounding_box(-73.73,-55.01,-67.05,-17.63).version('061').temporal('2023-01-01','2023-01-31').get_all()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/aldotapia/miniforge3/envs/hidrocl/lib/python3.10/site-packages/cmr/queries.py", line 104, in get_all
    return self.get(self.hits())
  File "/Users/aldotapia/miniforge3/envs/hidrocl/lib/python3.10/site-packages/earthaccess/search.py", line 331, in hits
    response = self.session.get(url, headers=self.headers, params={"page_size": 0})
AttributeError: 'DataGranules' object has no attribute 'headers'

With this error message, I upgraded python-cmr from version 0.7.0 to 0.9.0, and everything is working well now.

Although the setup requires v0.9.0, the installation using pip didn't upgrade the package version:

Requirement already satisfied: python-cmr>=0.7 in ./miniforge3/envs/hidrocl/lib/python3.10/site-packages (from earthaccess) (0.7.0)

I've been using this library since it was called earthdata, so I have old dependencies. I'll try to upgrade all of them

Thanks for following up on the case

Cheers

[MattF-NSIDC]

Thank you for this extra information, I think I understand the problem much better now!

v0.9.0 is actually only our locked dependency on python-cmr. In pyproject.toml, the current pin is >=0.7.0, and that's what your local dependency manager will see when it goes to install earthaccess. If that version is installed, it will correctly report already satisfied.

Is our pin of >=0.7.0 actually incompatible with earthaccess, and we simply haven't noticed because our dependency managers have been picking up newer versions implicitly?

My bandwidth is so saturated this week... sorry... @jrbourbeau @betolink @jhkennedy would you mind considering whether this should be addressed before AGU? We could have participants in workshops with older environments that technically satisfy our dependency spec, but fail at runtime.

Bonus thought:
I have never done this, but could we try and set up our automated tests to run (in addition to the current testing setup) in a environment that satisfies our spec with minimum versions? Then we'd have caught this.

[jrbourbeau]

I have never done this, but could we try and set up our automated tests to run (in addition to the current testing setup) in a environment that satisfies our spec with minimum versions? Then we'd have caught this.

We do this over in Dask and Xarray. Those projects use conda environment files for managing their dependencies (like this). I could quickly push something up that follows that pattern, but it would be different than our main testing setup here. Do folks know if there's an easy way to tell poetry to "install the minimum supported version of my dependencies"?

[MattF-NSIDC]

We do this over in Dask and Xarray. Those projects use conda environment files for managing their dependencies (like this).

Sick! How did you generate these mindeps files? I use conda for pretty much all my projects and would love to implement this technique in at least one.

EDIT: I copied from your post, and GitHub changed which "this" has the hyperlink. That is weird as heck.

[jrbourbeau]

How did you generate these mindeps files?

In Dask, just manually making sure the versions match what we have in project.toml. Xarray has some extra infrastructure around checking minimum dependencies