Update torchx for fsspec 2023.10

[alanhdu]

This updates the code to the latest version of fsspec, updating the test to work around fsspec/filesystem_spec#1394.

I have written the test so that it should actually be compatible with both the old and new versions of fsspec, but for testing purposes updated the requirements.txt as well.

Supersedes #783 (so it can be merged separately from the internal diff).

[codecov]

Codecov Report

Merging #784 (e232ceb) into main (1fe0607) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #784   +/-   ##
=======================================
  Coverage   92.88%   92.88%           
=======================================
  Files          96       96           
  Lines        6099     6099           
=======================================
  Hits         5665     5665           
  Misses        434      434           

Flag	Coverage Δ
unittests	92.88% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[kiukchung]

LGTM. Any idea why the unittest: torchx/schedulers/test/local_scheduler_test.py::LocalDirectorySchedulerTest::test_no_orphan_process_function is failing?

[alanhdu]

I have no idea -- when I tried to run it locally, everything seemed to pass. It looks like when someone also retried the github job and it's passing now?

[kiukchung]

I have no idea -- when I tried to run it locally, everything seemed to pass. It looks like when someone also retried the github job and it's passing now?

@kurman (can you help taking a look at the error logs for kfp?) I retried KFP Integration Tests to see if it was something flaky causing the failure, but the retry failed as well.

[kiukchung]

merged since KFP Integ tests seem broken in other PRs as well (#781, #784). Since we just bumped to 0.6.0dev, we can forward fix the test if these PRs continue breaking KFP tests in main.