[BUG] Pin fsspec<2024.5.0

[eapolinario]

Describe the bug

We're seeing errors in gcs like this:

[3/3] currentAttempt done. Last Error: SYSTEM::Traceback (most recent call last):
  File "/opt/venv/lib/python3.11/site-packages/flytekit/core/data_persistence.py", line 324, in get_data
    self.get(remote_path, to_path=local_path, recursive=is_multipart, **kwargs)
  File "/opt/venv/lib/python3.11/site-packages/flytekit/core/data_persistence.py", line 222, in get
    return file_system.get(from_path, to_path, recursive=recursive, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/fsspec/asyn.py", line 118, in wrapper
    return sync(self.loop, func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/fsspec/asyn.py", line 103, in sync
    raise return_result
  File "/opt/venv/lib/python3.11/site-packages/fsspec/asyn.py", line 56, in _runner
    result[0] = await coro
                ^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/fsspec/asyn.py", line 665, in _get
    return await _run_coros_in_chunks(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/fsspec/asyn.py", line 268, in _run_coros_in_chunks
    result, k = await done.pop()
                ^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/fsspec/asyn.py", line 245, in _run_coro
    return await asyncio.wait_for(coro, timeout=timeout), i
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/tasks.py", line 442, in wait_for
    return await fut
           ^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/fsspec/callbacks.py", line 81, in func
    return await fn(path1, path2, callback=child, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/gcsfs/core.py", line 1215, in _get_file
    async with self.session.get(
  File "/opt/venv/lib/python3.11/site-packages/aiohttp/client.py", line 1197, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/aiohttp/client.py", line 548, in _request
    req = self._request_class(
          ^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 304, in __init__
    url2 = url.with_query(params)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/yarl/_url.py", line 1010, in with_query
    new_query = self._get_str_query(*args, **kwargs) or ""
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/yarl/_url.py", line 971, in _get_str_query
    query = "&".join(self._query_seq_pairs(quoter, query.items()))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/yarl/_url.py", line 934, in _query_seq_pairs
    yield quoter(key) + "=" + quoter(cls._query_var(val))
                                     ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/yarl/_url.py", line 949, in _query_var
    raise TypeError(
TypeError: Invalid variable type: value should be str, int or float, got <fsspec.callbacks.NoOpCallback object at 0x7fd0ac965110> of type <class 'fsspec.callbacks.NoOpCallback'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.11/site-packages/flytekit/bin/entrypoint.py", line 88, in _dispatch_execute
    ctx.file_access.get_data(inputs_path, local_inputs_file)
  File "/opt/venv/lib/python3.11/site-packages/flytekit/core/data_persistence.py", line 326, in get_data
    raise FlyteAssertion(
flytekit.exceptions.user.FlyteAssertion: Failed to get data from gs://opta-gcp-demo-gcp/metadata/propeller/union-health-monitoring-development-aw2s7pttxr9wk4k96kft/n0/data/inputs.pb to /tmp/flyte-s_sb2t95/sandbox/local_flytekit/inputs.pb (recursive=False).

Original exception: Invalid variable type: value should be str, int or float, got <fsspec.callbacks.NoOpCallback object at 0x7fd0ac965110> of type <class 'fsspec.callbacks.NoOpCallback'>

fsspec released 2024.5.0 today, but no corresponding release of gcsfs went out (yet?). This works with s3fs but not gcsfs. We'll very likely have to pin this in flytekit as well.

Expected behavior

Published versions of fsspec should be backwards compatible, unless otherwise stated. In order to unblock our builds we're going to pin the version of fsspec.

Additional context to reproduce

No response

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[runllm]

To address the issue with fsspec and gcsfs, you can pin the fsspec version to <2024.5.0 in your Flyte setup. This will ensure compatibility until a corresponding gcsfs release is available. Here's how you can do it:

1. Update your requirements.txt or setup.py:

   fsspec<2024.5.0
   

2. Reinstall dependencies:

   pip install -r requirements.txt

Would you like more details on how to implement this in a Flyte project?

Sources:

• Flyte Community Discussion
• Flyte Community Discussion
• Flyte Community Discussion

---

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @RunLLM to ask follow-up questions. See runllm.com for more!

[eapolinario]

gcsfs just released 2024.5.0: https://github.com/fsspec/gcsfs/releases/tag/2024.5.0. We should revert flyteorg/flytesnacks#1680.