feat: Overload return type for from_arrow

[deanm0000]

While pyarrow doesn't have types built in there is pyarrow-stubs. It works well

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 59.90%. Comparing base (900dc3b) to head (912ca0e).
Report is 1 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main   #19231       +/-   ##
===========================================
- Coverage   79.99%   59.90%   -20.10%     
===========================================
  Files        1527     1527               
  Lines      209203   209203               
  Branches     2415     2415               
===========================================
- Hits       167352   125313    -42039     
- Misses      41303    83342    +42039     
  Partials      548      548               

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ritchie46]

Hmm.. I don't understand I think. How does your IDE know arrow type? Asking @MarcoGorelli @stinodego here.

[MarcoGorelli]

I think vscode has its own extra stubs for some popular libraries which are missing them (e.g. IIRC they wrote pandas-type-stubs originally, which was then moved into the pandas org)

[stinodego]

This is nice for people that have pyarrow-stubs installed, but I believe that for everyone else, from_arrow will then mistakenly infer the type as the return type of the first overload (DataFrame in this case), which may be wrong.
If that's true, I don't think this is a good idea.

Could you confirm what types are inferred for people without stubs installed for various inputs?

[alexander-beedie]

This is nice for people that have pyarrow-stubs installed, but I believe that for everyone else, from_arrow will then mistakenly infer the type as the return type of the first overload (DataFrame in this case), which may be wrong. If that's true, I don't think this is a good idea.

I recall writing that comment about not type-overloading this function after trying to type it a second time (having forgotten that it didn't go well the first time). Perhaps the previous thread will jog some memories as to the exact nature of the downside, because it's definitely a shame not to be able to type it more exactly 🤣 #8011 (comment)

[deanm0000]

If pyarrow-stubs isn't installed then it reverts to Any so the only downside is the extra code. It doesn't revert to the first overload.

Here's a little bit more on how python and IDEs load types https://peps.python.org/pep-0561/

To make it work all someone needs to do is pip install pyarrow-stubs and it just works with the caveat that after installing you have to close and reopen your IDE to get it to look for type definitions.

[deanm0000]

I put in a draft for the full adoption of pyarrow-stubs #19259