Document developer API surface in v3

[dlqqq]

Problem

It is sometimes ambiguous on what changes are breaking and non-breaking. Renaming a public method is certainly a breaking change, but what about public attributes? Should we allow ourselves to change function names if they're not intended for developer use, even if they are importable from the root module?

I've outlined a proposal below. Contributors, please feel free to offer feedback in the comments below.

Proposed Solution

• Establish a new convention where only documented functions, classes, and other objects are part of the official developer API.
• Everything not covered in the documentation should be considered for internal use, even if its name lacks an underscore prefix. These may change in any release so their use should be discouraged.
• Attributes are never public (i.e. they can change in any release).
• Properties may be public, if they lack an underscore. By only publishing properties (whose definition can be modified), we will have much more flexibility in maintaining backwards compatibility for future releases.

Additional context

There still remain open questions regarding:

• Dependency upgrades. We treated pydantic v1=>v2 as a breaking change given its pervasiveness throughout the Python ecosystem, but haven't done so for any other dependency. It would be nice if we could establish objective boundaries for when dependency upgrades are breaking. Alternatively (and preferably), we could design Jupyter AI in a way that allows for dependency upgrades that don't affect the user's env (e.g. always running in a venv / Conda env).

• Server logs. Can we change/modify server logs in non-major releases? I think we have to, simply because it would be too restrictive/impractical otherwise. However, I know admins may use regexes on logs to identify issues with deployments. Is there a solution that works for both contributors & admins?

  • Perhaps we could ensure each log message has some sort of "log ID" present in the message. However it's unclear how we can enforce this in our code for other contributors to follow.