Update to Pydantic 2 and FastAPI 0.100.0

[mattt]

From https://fastapi.tiangolo.com/release-notes/#01000:

Pydantic version 2 has the core re-written in Rust and includes a lot of improvements and features... In some cases, for pure data validation and processing, you can get performance improvements of 20x or more.

[nickstenning]

Cross-linking this comment from the dependabot PR so we don't lose it: #1187 (comment)

[mattt]

Related to #1216

[mattkindy]

Until this is implemented, this prevents use of packages (e.g. https://github.com/outlines-dev/outlines) that have dependencies on pydantic and have upgraded to v2 in their most recent updates.

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/local/lib/python3.11/site-packages/cog/server/http.py", line 16, in <module>
    from fastapi import Body, FastAPI, Header, HTTPException, Path, Response
  File "/usr/local/lib/python3.11/site-packages/fastapi/__init__.py", line 7, in <module>
    from .applications import FastAPI as FastAPI
  File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 16, in <module>
    from fastapi import routing
  File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 22, in <module>
    from fastapi import params
  File "/usr/local/lib/python3.11/site-packages/fastapi/params.py", line 4, in <module>
    from pydantic.fields import FieldInfo, Undefined
ImportError: cannot import name 'Undefined' from 'pydantic.fields' (/usr/local/lib/python3.11/site-packages/pydantic/fields.py)
ⅹ Failed to get container status: exit status 1

[sinopec]

really need it ,specially when some llm inference engine requires pydantic 2....
annoying conflicts.....

[dkhokhlov]

really need it ,specially when some llm inference engine requires pydantic 2.... annoying conflicts.....

@sinopec - could you give model examples for testing?

[jd7h]

I deployed a model today that required pydantic==2.5.3. While we're waiting for the update to newer versions of pydantic and fastAPI, here are some possible workarounds. Choose one depending on the speed you require and how tightly integrated your code is with the incompatible versions of pydantic and fastAPI:

Supplemental venv

Create a virtual environment with venv to supplement the packages installed from cog.yaml

$ cog run bash
# python -m venv --system-site-packages venv_pydantic
# source venv_pydantic/bin/activate
# pip install pydantic==2.5.3

predict.py:

import subprocess
subprocess.check_call(["venv_pydantic/bin/python", "this_script_requires_pydantic2"])

Drawback: You can't easily pre-load your modelweights in setup().

Poetry

2. in cog.yaml, git clone your code and install it with poetry
   cog.yaml:

python_packages:
  - poetry

run:
  - git clone myrepo.git && cd myrepo && poetry install

in predict.py:

import subprocess
subprocess.check_call("cd /myrepo && python -m poetry run python -m script.py".split())

Drawback: You can't easily pre-load your modelweights in setup().

Monkeypatching

3. You can Mock() or otherwise monkeypatch parts of incompatible libraries.
   predict.py:

from unittest.mock import Mock

sys.modules['pydantic.computed_field'] = Mock()
sys.modules['pydantic_settings'] = Mock()
sys.modules['pydantic_settings'].BaseSettings = MyCustomSettings

This hack works best if the pydantic/fastAPI modules are imported but not essential for your code. The advantage is that you can use setup() to preload model weights.
For a more elaborate example see the texify cog demo. I used this method for texify because this way I didn't need to modify the original model code to make the cog demo work.

[yorickvP]

Summarizing here:

• Upgrading pydantic also requires a fastAPI upgrade, which will bump the generated OpenAPI schemas from 3.0 to 3.1.
• kin-openapi, which we use in cog and registry-proxy (potentially other places), does not support this: OpenAPI 3.1.0 is coming getkin/kin-openapi#230, and has asked for sponsorship to update it
• I attempted to switch to libopenapi: Switch kin-openapi to libopenapi #1539, but concluded its validation is not as good as kin-openapi

cc @nickstenning for the sponsorship
cc @dkhokhlov for coordination
Another issue for this: #1336

[nickstenning]

I chatted to @yorickvP on Slack. Here's a quick summary of what we talked about:

• I'm very happy in principle for us to sponsor kin-openapi but we'd need to contact the maintainer and understand whether that would actually get us what we need. The linked issue about sponsorship is several years old.
• In practice we use kin-openapi with Replicate's own 3.1 spec without issue. It seems likely we can get away with it for most if not all Cog models.
• We have a decent little library of test models/fixtures in this repo that we can use to test that hypothesis.

[micos7]

Same issue wtih pydantic 2, bumping this issue magically expecting to be fixed.. :)

[yorickvP]

@micos7 the current effort is at #1858