Question: how to handle UnionType annotations

[tpgillam]

Testing with pyright 1.1.378

Consider the following minimal example. The real-world context is a function load that will deserialise an object somehow, and then ensure it is of some desired type or uniontype - so long as they're compatible with a bound.

With the following approach, pyright flags an error where indicated:

import typing
from pathlib import Path
from typing import TypeVar, overload


class Base:
    pass


class A(Base):
    pass


class B(Base):
    pass


class C(Base):
    pass


T = TypeVar("T", bound=Base)
S = TypeVar("S", bound=Base)


@overload
def load(path: Path, type_: type[T]) -> T: ...
@overload
def load(path: Path, type_: type[T | S]) -> T | S: ...  # ERROR: reportOverlappingOverload
def load(path, type_) -> Base:
    del path, type_
    raise NotImplementedError


path = Path("moo")
x = load(path, A)
typing.assert_type(x, A)
y = load(path, A | B)
typing.assert_type(y, A | B)
z = load(path, A | B | C)
typing.assert_type(z, A | B | C)

However if I remove overload 2 then errors appear below:

import typing
from pathlib import Path
from typing import TypeVar


class Base:
    pass


class A(Base):
    pass


class B(Base):
    pass


class C(Base):
    pass


T = TypeVar("T", bound=Base)


def load(path: Path, type_: type[T]) -> T:
    del path, type_
    raise NotImplementedError


path = Path("moo")
x = load(path, A)
typing.assert_type(x, A)
y = load(path, A | B)  # ERROR: (see below)
typing.assert_type(y, A | B)
z = load(path, A | B | C)
typing.assert_type(z, A | B | C)

where the ERROR is:

Diagnostics:
1. Argument of type "UnionType" cannot be assigned to parameter "type_" of type "type[T@load]" in function "load"
     Type "UnionType" is not assignable to type "type[T@load]" [reportArgumentType]

What's the correct way to handle this?

I'm also suspecting that there might be a bug in here, given that removing the second overload wasn't actually a no-op.

[erictraut]

There's currently no way in the type system to handle this. The typing spec is clear that a union is not assignable to type[T]. That makes sense because most code that accepts a type expects that it can call this object to instantiate an instance of the type, but this would fail for a union. There is a draft PEP 747 that addresses this use case, but it is incomplete and requires additional work before it will be incorporated into the typing spec.

If you're able to change the interface to this function, I recommend that you change it from accepting a UnionType to a tuple of type objects. A "union" is intended to describe a set-theoretical concept in type theory. It is not meant to describe a list of classes. So this load method is abusing the concept of a union.

Here's how I'd recommend coding this:

from pathlib import Path
from typing import assert_type, overload

class Base:
    pass

class A(Base):
    pass

class B(Base):
    pass

class C(Base):
    pass

@overload
def load[T: Base](path: Path, type_: type[T]) -> T: ...
@overload
def load[T: Base](path: Path, type_: tuple[type[T], ...]) -> T: ...
def load[T](path, type_: type[T] | tuple[type[T], ...]) -> Base:
    del path, type_
    raise NotImplementedError

path = Path("moo")
x = load(path, A)
assert_type(x, A)
y = load(path, (A, B))
assert_type(y, A | B)
z = load(path, (A, B, C))
assert_type(z, A | B | C)

[tpgillam]

Thank you for the explanation; yes PEP 747 seems to be exactly what I was hoping existed.

So this load method is abusing the concept of a union.

How come? I intended that type_ would represent some set of allowed object types, and I wanted to guarantee that we return an instance of one the types in that set. Is your point that this set is finite and (at runtime) of known size in the example above?

Interestingly, using the tuple approach it seems we can still pass in UnionTypes. This is great, although surprises me a bit. i.e. the following works fine:

T = TypeVar("T", bound=Base)


@overload
def load[T: Base](path: Path, type_: type[T]) -> T: ...
@overload
def load[T: Base](path: Path, type_: tuple[type[T], ...]) -> T: ...
def load[T](path, type_: type[T] | tuple[type[T], ...]) -> Base:
    del path, type_
    raise NotImplementedError


path = Path("moo")
x = load(path, A)
typing.assert_type(x, A)
y = load(path, A | B)
typing.assert_type(y, A | B)
z = load(path, A | B | C)
typing.assert_type(z, A | B | C)

does that make sense to you?

[erictraut]

How come?

A type expression in the Python type system describes a set of potential runtime values. For example, the type expression A means all objects that are instantiated from class A or a subclass thereof. A union of two types represents the union of those two sets of values. These are abstract set-theoretic concepts. Some type systems also support intersection types (something that is not currently supported in Python but could be in the future), and those represent intersections of two sets.

In your example, you're not describing abstract types. You're describing a collection of concrete classes, and a "union" operator doesn't make sense in this case. You're effectively (ab)using the UnionType (which is not part of the type system abstraction; you won't see it mentioned even once in the Python typing spec) as a way to hold two or more class objects. That's why I suggested a tuple as a more appropriate container in this case. A set or list would also work fine.

[tpgillam]

a "union" operator doesn't make sense in this case

I think we're talking at cross-purposes slightly. It's valid to write A | B for concrete A and B, and the result is a runtime UnionType. And for users the syntax is a natural fit for our case, since it's conceptually similar to isinstance(x, A | B) which has been supported syntax since 3.10ish I think?

You're describing a collection of concrete classes

Yep - although ideally they might not always have been concrete. So far there's only one protocol that we might constrain on, but for the time being I special cased that.

A set or list would also work fine.

Interesting. It seems as though pyright will still happily conform an expression-that-would-evaluate-to-a-UnionType to a set or list subtype (as per the second question above).

[arcusmaximus]

I guess the builtin cast function enjoys a magic feature that custom code doesn't have access to then:

from typing import Any, TypeVar, cast

T = TypeVar("T")
def my_cast(type_: type[T], value: Any) -> T:
    return cast(type_, value)

x =    cast(str | None, "a")  # Works
y = my_cast(str | None, "b")  # Error, even though it has the same signature

It's not just unions either - even just my_cast(None, "b") will fail with a similar "can't be assigned to type" error.

Personally I reverted to Pylance 2023.12.1 (not sure which Pyright version that corresponds to). That one doesn't raise the error, instead allowing my_cast to behave (almost) just like cast. Maybe it's abuse, but I'll gladly do that if the correct way isn't available yet.

[erictraut]

I guess the builtin cast function enjoys a magic feature

Yes, cast is special. It effectively tells the type checker to ignore all typing rules. In that regard, it's can make your code less robust because it masks type violations.

We've made some good progress recently on PEP 747, which is still in draft form. It proposes to add a new typing.TypeForm symbol to the type system. Pyright has provisional support for this feature (and can be used by setting enableExperimentalFeatures to true), but there is currently no runtime support in typing_extensions.

Code sample in pyright playground

[arcusmaximus]

In that regard, it's can make your code less robust because it masks type violations.

I know - I don't really like to use it either, but sometimes you have no choice.

We've made some good progress recently on PEP 747

Good to hear, thank you!