Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[sentinel-2]: Retry on urllib exception #260

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

TomAugspurger
Copy link
Contributor

@TomAugspurger TomAugspurger commented Oct 3, 2023

Observed an error:

[INFO] 2023-10-03 20:02:10,671 -  === PCTasks: Task Failed! ===
[ERROR] 2023-10-03 20:02:10,671 - Failed to create item from blob://sentinel2l2a01/sentinel2-l2/14/R/LS/2022/01/27/S2A_MSIL2A_20220127T172601_N0400_R012_T14RLS_20220212T210659.SAFE/
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/pystac/stac_io.py", line 300, in read_text_from_href
    with urlopen(req) as f:
  File "/opt/conda/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/opt/conda/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/opt/conda/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/opt/conda/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/opt/conda/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/opt/conda/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 503: Operations per second is over the account limit.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/pctasks/dataset/items/task.py", line 211, in create_items
    result = self._create_item(asset_uri, storage_factory)
  File "/mnt/batch/tasks/workitems/sentinel-2-process-chunk-09f7d19-cc-8752-599c767e2a06-tsk_gen_bl/job-1/create-items-12/wd/_code/sentinel2.py", line 95, in create_item
    item = with_backoff(
  File "/opt/conda/lib/python3.8/site-packages/pctasks/core/utils/backoff.py", line 149, in with_backoff
    return fn()
  File "/mnt/batch/tasks/workitems/sentinel-2-process-chunk-09f7d19-cc-8752-599c767e2a06-tsk_gen_bl/job-1/create-items-12/wd/_code/sentinel2.py", line 87, in get_item
    item: pystac.Item = stac.create_item(
  File "/opt/conda/lib/python3.8/site-packages/stactools/sentinel2/stac.py", line 57, in create_item
    granule_metadata = GranuleMetadata(safe_manifest.granule_metadata_href,
  File "/opt/conda/lib/python3.8/site-packages/stactools/sentinel2/granule_metadata.py", line 22, in __init__
    self._root = XmlElement.from_file(href, read_href_modifier)
  File "/opt/conda/lib/python3.8/site-packages/stactools/core/io/xml.py", line 74, in from_file
    text = read_text(href, read_href_modifier)
  File "/opt/conda/lib/python3.8/site-packages/stactools/core/io/__init__.py", line 20, in read_text
    return StacIO.default().read_text(read_href_modifier(href))
  File "/opt/conda/lib/python3.8/site-packages/pystac/stac_io.py", line 282, in read_text
    return self.read_text_from_href(href)
  File "/opt/conda/lib/python3.8/site-packages/pystac/stac_io.py", line 303, in read_text_from_href
    raise Exception("Could not read uri {}".format(href)) from e
Exception: Could not read uri https://sentinel2l2a01.blob.core.windows.net/senti..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant