Make faiss-cpu an optional dependency in v3.0.0

[dlqqq]

Problem

Listing faiss-cpu as a required dependency has caused numerous issues for users.

• Meta does not distribute any builds of FAISS. FAISS distributions on PyPI and Conda Forge are unofficially maintained by third-party contributors.
• On PyPI, we can only install FAISS via faiss-cpu, therefore we have faiss-cpu listed as a required dependency.
• However, installing faiss-cpu from Conda Forge does not install a Python package named faiss-cpu, but instead a Python package named faiss. This causes pip check to fail, since faiss-cpu is not installed. Jupyter AI still works however, since both packages provide the faiss module that we import.
• FAISS releases on PyPI and Conda Forge may lag behind the latest releases of FAISS on GitHub, as they are maintained by third-party contributors. This will limit the rate at which we can upgrade to new releases of Python, e.g. Python 3.13.

Proposed Solution

• Make faiss-cpu an optional dependency.
• Allow other vector databases to be used by /learn.
• Ideally, users should be able to install support for other vector databases at runtime, without requiring a restart of the Jupyter Server.

Additional context

• jupyter-ai v2.28.1 (removes post-link script) conda-forge/jupyter-ai-feedstock#47

[asadoughi]

Hi! Official FAISS builds are available via the conda pytorch channel: https://github.com/facebookresearch/faiss/blob/main/INSTALL.md

[krassowski]

Thanks for the link @asadoughi! However, asking all conda users of jupyter-ai to include pytorch channel might be a high bar. Some organizations only allow conda-forge (due to licensing concerns) and sometimes having multiple channels enabled will lead to conflicts in the binaries. I think it would be much easier to reconsider if the official distributions were mirrored in the conda-forge channel.

[krassowski]

Also, I understand that the pytorch channel is deprecated anyways, right?

• [Announcement] Deprecating PyTorch’s official Anaconda channel pytorch/pytorch#138506