bug: Unable to install ibis-framework on windows via poetry

[Prototype4988]

What happened?

Hi Team,

I tried to install ibis-framework via poetry and got the following error

Installing C:\Users\Solom\AppData\Local\pypoetry\Cache\virtualenvs\wren-engine-Z5cLiW44-py3.11\Lib\site-packages\ibis\expr\operations\tes - Installing ibis-framework (9.5.0): Failed

FileNotFoundError

[Errno 2] No such file or directory: 'C:\Users\Solom\AppData\Local\pypoetry\Cache\virtualenvs\wren-engine-Z5cLiW44-py3.11\Lib\site-packages\ibis\expr\operations\tests\snapshots\test_generic\test_error_message_when_constructing_literal\call0-missing_a_required_argument\missing_a_required_argument.txt'

at ~\AppData\Local\Programs\Python\Python311\Lib\pathlib.py:1044 in open
1040│ the built-in open() function does.
1041│ """
1042│ if "b" not in mode:
1043│ encoding = io.text_encoding(encoding)
→ 1044│ return io.open(self, mode, buffering, encoding, errors, newline)
1045│
1046│ def read_bytes(self):
1047│ """
1048│ Open the file in bytes mode, read it, and close the file.

Cannot install ibis-framework.

error: Recipe install failed on line 18 with exit code 1

What version of ibis are you using?

9.5.0

What backend(s) are you using, if any?

No response

Relevant log output

Installing C:\Users\Solom\AppData\Local\pypoetry\Cache\virtualenvs\wren-engine-Z5cLiW44-py3.11\Lib\site-packages\ibis\expr\operations\tes  - Installing ibis-framework (9.5.0): Failed

  FileNotFoundError

  [Errno 2] No such file or directory: 'C:\\Users\\Solom\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\wren-engine-Z5cLiW44-py3.11\\Lib\\site-packages\\ibis\\expr\\operations\\tests\\snapshots\\test_generic\\test_error_message_when_constructing_literal\\call0-missing_a_required_argument\\missing_a_required_argument.txt'

  at ~\AppData\Local\Programs\Python\Python311\Lib\pathlib.py:1044 in open
      1040│         the built-in open() function does.
      1041│         """
      1042│         if "b" not in mode:
      1043│             encoding = io.text_encoding(encoding)
    → 1044│         return io.open(self, mode, buffering, encoding, errors, newline)
      1045│
      1046│     def read_bytes(self):
      1047│         """
      1048│         Open the file in bytes mode, read it, and close the file.

Cannot install ibis-framework.

error: Recipe `install` failed on line 18 with exit code 1

Code of Conduct

• I agree to follow this project's Code of Conduct

[hershelm]

in your pyproject.toml, i've included the following line for ibis-framework, and it is working for me on windows with poetry:

ibis-framework = {extras=["pyspark", "snowflake"], version = "^9.4.0"}

you can replace with your backends of choice (e.g., "duckdb")

if you're installing from command line, do you get the error if you run the following:

poetry install ibis-framework[duckdb]

[Prototype4988]

in my pyproject.toml it is written as below

ibis-framework = { version = "9.5.0", extras = [
  "bigquery",
  "clickhouse",
  "mssql",
  "mysql",
  "postgres",
  "snowflake",
  "trino",
] }

still the error occures

[cpcloud]

Please show all of the code that is involved here.

I don't see how you're invoking poetry, and I see this line

error: Recipe install failed on line 18 with exit code 1

which suggests you're using some other tool to invoke poetry.

[Prototype4988]

we are using casey/just to invoke poetry
PFA the souce code of the justfile
https://github.com/Prototype4988/wren-engine/blob/7e84d56f4c4cf02de7e600f6d00ccecdc5ddc54a/ibis-server/justfile#L18

[gforsyth]

Hey @Prototype4988 -- none of us have windows machines available to troubleshoot this. It would be a big help if you can try to narrow down which backend extras are causing installation issues so we can tackle the problem appropriately.

You should be able to install the extras one-by-one and that should inform which backend's dependencies are the culprit here.

Thanks!

[Prototype4988]

i have tried different variations. every single one is giving the error. Could it be that the issue is not with the particular dependancy but as ibis-framework on the whole?

[gforsyth]

That seems unlikely as we're installing it on Windows in our CI, and we have other Windows users who aren't having this issue.

Can you create a clean environment/virtualenv/whatever and try to pip install 'ibis-framework[duckdb] and paste the output of that command?

[grieve54706]

This issue seems the same with #9184.

[gforsyth]

Ahh, thanks for pointing that out @grieve54706 ! It does seem to be related to the same snapshot test files.

[gforsyth]

UGGH WINDOWS.

[ins] In [2]: len(r'C:\Users\Solom\AppData\Local\pypoetry\Cache\virtualenvs\wren-engi
         ...: ne-Z5cLiW44-py3.11\Lib\site-packages\ibis\expr\operations\tests\snapsho
         ...: ts\test_generic\test_error_message_when_constructing_literal\call0-miss
         ...: ing_a_required_argument\missing_a_required_argument.txt')
Out[2]: 262

Windows MAX_PATH for a path+filename is 256 or 260 characters, so windows can't find this file because the path is too deep. I can't believe this is a problem in 2024, but it clearly is.

[cpcloud]

And I guess we're below this limit in CI...

[cpcloud]

This seems like poetry's problem, not ours.

We can of course remove these files, but that's not a robust solution since files that aren't test files and actually critical for ibis to work at all could easily hit this problem.

[cpcloud]

It's awful, but the problem is not in poetry, it's in virtualenv, and the "solution" appears to be "enable long path support if you're on windows >= 10": pypa/virtualenv#1769

[cpcloud]

Another option might be to try using the PR where I'm moving away from poetry to uv, but if that's also using the virtualenv library then it'll have the same problem.