Interpretation of type `Any & T`

[mark-todd]

Following on from the lengthy discussion here #1, it was concluded that Any & T is the best resultant type for intersecting Any. This is symmetric with the way union works Any | T -> Any | T. Conversation on this new thread will investigate the types of the attributes and methods of this intersection type. Just a note, inspired by @CarliJoy we can number and summarise different interpretations. We should also seek to again create a real world example that explores the new types, and compares different interpretations.

Also if a summary of each new interpretation could be succinctly provided alongside a usage example that would be grand :)

[CarliJoy]

Assume this simple but verbose example.
I will handle only accessing properties at the moment.

Module a_untyped

class A:
    common: str
    a: int
    a_str: str

    def fn_a(self, a: int) -> int:
        return a

    def fn_mix(self, mix: int) -> int:

    ab_mix: "A"

Module b_typed

class B:
    common: str
    b: str

    def fn_b(self, b: str) -> str:
        return b

    def fn_mix(self, mix: str) -> str:
        return mix

    ab_mix: "B"

main.py

import typing as t
from a_untyped import A
from b_typed import B

def needs_int(x: int) -> None:
    """Example function that needs to be called with integer to work"""

def needs_str(x: str) -> None:
    """Example function that needs to be called with string to work"""

def can_handle_any(x: t.Any) -> None:
    """Example function that is able to handle any type"""

def use_ab_intersected(x: A & B) -> None:
    # All this examples should work, as they work in runtime
    needs_str(x.common)

    needs_int(x.a)
    needs_int(x.fn_a(1))
    needs_int(x.fn_mix(1))  # should be detected as the correct call
    needs_int(len(x.a_str.split()[0]))
    can_handle_any(x.a)

    needs_str(x.b)
    needs_str(x.fn_b("b"))
    needs_str(x.fn_mix("b"))
    can_handle_any(x.b)

    use_ab_intersected(x.ab_mix)  # Intersect the property is valid
    needs_int(x.ab_mix.b)
    can_handle_any(x.ab_mix.a)

    # This should raise an error detected by the type checker (if both types known)
    invalid1 = x.not_existing
    invalid2 = x.fn_mix(object)

In general even so a variable is Any we can assume that runtime implementation of the object hold by this variable has a concrete implementation.

But if package is not marked as typed, everything is imported as Any, so let's assume A is typed as Any.

As an user I would want the following behaviour. This is probably not possible, but we should check rules against this example.

def use_ab_intersected(x: A & B) -> None:
    # All this examples should work, as they work in runtime
    needs_str(x.common)

    needs_int(x.a) # type: ignore - required
    needs_int(x.fn_a(1))  # type: ignore - required
    needs_int(x.fn_mix(1)) # type: ignore - required
    needs_int(len(t.cast(str, x.a_str).split()[0]))  # this needs a cast
    can_handle_any(x.a) # should be fine, because it is `Any` already

    needs_str(x.b)  # Hard requirement, this *must* work
    needs_str(x.fn_b("b"))  # Hard requirement, this *must* work
    needs_str(x.fn_mix("b"))  # Hard requirement, this *must* work
    can_handle_any(x.b) # Hard requirement, this *must* work

    use_ab_intersected(x.ab_mix)  # Hard requirement, this *must* work
    needs_int(x.ab_mix.b) # Hard requirement, this *must* work
    can_handle_any(x.ab_mix.a) # Hard requirement, this *must* work

    # This should raise an error detected by the type checker (if both types known)
    invalid1 = x.not_existing  # type checker should complain
    invalid2 = x.fn_mix(object) # type checker should complain

[CarliJoy]

Hmm looking at the example it seems that we have to decided between allowing can_handle_any(x.a) and raising type errors on invalid accesses.

As for the most A related call we have to # type: ignore or cast things anyway, for accessing properties I would tend for the T & Any -> T.

This is in contract to the example given by @ippeiukai here.

Copied and modified below for easier reference:

By the way, in terms of gradual typing, I feel Intersection[T, Any] = T is awful.

Module 1

some_fn(some_obj)

Module 2

def wrap_obj(val: T) -> T & SomeAddedTrait:
  …

a = wrap_obj(some_obj)
some_fn(a)

Here, let’s assume some_fn and some_obj are provided by two different libraries both currently untyped. What if some_fn got typed before some_obj gets typed? I expect I can do anything to a that I can do to some_obj. I don’t expect a to be reduced to mere SomeAddedTrait.

TODO: formulate example more concrete, I don't get it atm,@ippeiukai could you formulate it with proper files and imports like I did above

[mikeshardmind]

By the way, in terms of gradual typing, I feel Intersection[T, Any] = T is awful.

I'll type up some examples when I have more time, hopefully later today, but both of the following are true here:

• In terms of gradual typing, T & Any = T is awful
• In terms of gradual typing T & Any = Any or otherwise unchecked (one interpretation of Any & T) is awful

I think @DiscordLiz had a good point in the prior thread about Any & T never being an intentional construct. When is this going to come up? Do people intending on having Any have a use case for intersections, or is this sufficiently advanced that people should be expected to have typed their bases better and we should err on the side of safety for users on this? I think we need to start from where we expect this to occur before we can evaluate the impact of any choice. It's clear it can only happen at the boundary between typed and either untyped or gradually typed things, but can we get more specific than that and determine if there's ever an intentional use for T & Any which isn't better served by other existing things?

[mark-todd]

@CarliJoy Thanks for this example! It's very thorough - I've reduced it down a bit to consider a particular subcase, not because the others aren't relevant, but just because I think it might be easier to discuss in pieces:

import typing
from intersection_examples import Intersection
class A:
    common: str

# A effectively gets cast to Any, so nobody knows the type common even exists

class B:
    common: int

def needs_int(x: int) -> None:
    """Example function that needs to be called with integer to work"""

def needs_str(x: str) -> None:
    """Example function that needs to be called with string to work"""

def use_ab_intersected(x: Intersection[A, B]) -> None:
    # Note: A&B is an LSP violation, but we don't know that it is, because
    # we don't know the type of A
    needs_int(x.common) # should be fine - this is a property of B
    needs_str(x.common) # This should also be fine - if A is used at runtime
    # It could be string? Or could it?
    
    # Follow up idea - do we have to assume the intersection created with Any
    # has not created any LSP violations?

My conclusion from this is that similarly to other types we intersect, perhaps we have to assume LSP has not been violated by the intersection with the Any type - although we have no way to be sure

[gvanrossum]

I am still groping my way around this subject. I am unsure of the technical meaning of "gradual typing" that seems to be used by some participants.

In the meantime, I have an example, which has actually convinced me that this proposal is fine:

def f(a: Any & int):
    x = a + 1
    reveal_type(x)

Given that a: Any & T means thata has all the attributes of T and they have the types specified by T, in this example, x has __add__(self, arg: int) -> int so the type of x should be revealed as int. That matches my intuition.

[DiscordLiz]

My conclusion from this is that similarly to other types we intersect, perhaps we have to assume LSP has not been violated by the intersection with the Any type - although we have no way to be sure

Which makes it intersections all the way down and no different than being Any, cause we have to assume LSP wasn't violated.

If that were intentional, someone could skip the intersection and just write Any, and that works today.

I think @DiscordLiz had a good point in the prior thread about Any & T never being an intentional construct. When is this going to come up? Do people intending on having Any have a use case for intersections, or is this sufficiently advanced that people should be expected to have typed their bases better and we should err on the side of safety for users on this?

I don't think it's intentional, but I do think it can happen. The middle ground (that was option 5? and we're only considering options 4 and 5 now? Sorry, I seem to have missed the reason for splitting this to a new thread.) seems to avoid the accidental issues possible in 4, but I'd like more examples of it from @mikeshardmind

Given that a: Any & T means thata has all the attributes of T and they have the types specified by T, in this example, x has add(self, arg: int) -> int so the type of x should be revealed as int. That matches my intuition.

That would be the behavior in option 5. Option 4, the return type can only be int & Any, it's recursive with that option, and more pervasive than I think is beneficial.

[NeilGirdhar]

Given that a: Any & T means thata has all the attributes of T and they have the types specified by T, in this example, x has __add__(self, arg: int) -> int so the type of x should be revealed as int. That matches my intuition.

Just something for you to consider. Let's just look at an example that doesn't involve Any:

class C:
  def f(self) -> None: pass
class D:
  def g(self) -> None: pass
class E(C, D): pass

def f(x: C):
  if isinstance(x, D):
    reveal_type(x)  # C & D
    x.f()  # should be okay since x inherits from C
    x.g()  # should be okay since x inherits from D

f(E())

This is why the interface of C&D is the union of the interfaces of C and D.

Now, consider:

• what is the interface of Any?
• what is the interface of Any & C?
• So if x has type Any & C, what is the type of x.f(), x.g(), and x.h()?

It's really worth reading Eric's elucidating comment.

For reference, there are two proposals being considered here. I'll call them 4 and 5 (relating to names from #1).
Proposal 4 says: x.f(), x.g(), x.h() are all Any.
Proposal 5 says: x.g(), x.h() are both Any, but x.f() is None.

[NeilGirdhar]

Even though I'm not yet in favor of proposal 5, I have a way to illustrate the problem it may be trying to solve:

Consider what isinstance does:

x: Any
assert isinstance(x, U)
y: T
assert isinstance(y, U)

The general case illustrated by y gives y type T & U. But there's a special case: if we start with something of type Any, then we don't want to end up with Any & U, and we instead set x's type to U. The reason we don't want Any & U is because that would cripple type checking: You would not be able to check most things related to the interface of x.

Essentially, there are two isinstance operators. One acts is an intersection that acts on ordinary types and type variables, and the other is a type replacement that acts replaces an Any type with whatever type is provided. Currently, type checkers do a lot of type replacement, and only a rudimentary intersection in some special cases.

I think the people in camp 5 are worried that this pernicious behavior that we avoid using the type replacement will creep in when we enable intersections generally.

I considered a 6th proposal: encourage to choose replacement if they want to protect against the spread of Any. This should only be relevant in some special cases involving things like unbounded generics. This can be done like this:

def f[T](x: T):
     assert isinstance(x, U)
     # Normally, this would have type T & U, but you can force replacement by casting.
     x_as_u = cast(x, U)
     # The over-broad interface of T & U is avoided.

This would allow us to leave the ordinary meaning of T & Any alone without any special cases, but potentially eliminate the problems associated with it.

I think we need to see some concrete examples to test if this could work.

[mikeshardmind]

If I need a cast everywhere I use an intersection to avoid issues with Any, then I don't need an intersection, I can cast to a protocol that matches my use.

You're just replacing one wart with another worse one that disables other checks here, and requires runtime checkable protocols and a runtime cast use.

The reason we don't want Any & U is because that would cripple type checking: You would not be able to check most things related to the interface of x.

That's exactly the issue with option 4. It cripples type checking for the person with fully typed info on their side of the intersection and is no better for the person with Any on their yet untyped object than not using an intersection and using Any.

[NeilGirdhar]

That's exactly the issue with option 4. It cripples type checking for the person with fully typed info

So we're on the same page, and I think I understand your point of view.

I see how the deference proposal prevents false negatives, but it does so at the cost of false positives. Historically, the Python typing community has preferred solutions that avoid false positives even if they cause false negatives. For example, the adoption of the sequence protocol into the typeshed causes false negatives in order to avoid false positives.

I think we just need to see at least a few common examples to get a better handle of what the tradeoffs are.

If I need a cast everywhere I use an intersection

I don't think it would be "everywhere you use an intersection", but as I stated in the previous comment, it would only be in the cases of intersections with unbound type variables, and similar. Like I said, this was just something I thought of. We need to see some examples to know what it would look like.

[mark-todd]

I am still groping my way around this subject. I am unsure of the technical meaning of "gradual typing" that seems to be used by some participants.

In the meantime, I have an example, which has actually convinced me that this proposal is fine:

def f(a: Any & int):
    x = a + 1
    reveal_type(x)

Given that a: Any & T means thata has all the attributes of T and they have the types specified by T, in this example, x has __add__(self, arg: int) -> int so the type of x should be revealed as int. That matches my intuition.

@gvanrossum I think your example makes sense, but the controversial part comes from the other side of the intersection - as you describe the result must have all the properties of int so using it with __add__ makes sense here. But on the other side we have Any, and therefore x must also have all the properties of Any. What does it mean to have all the properties of Any? If it encompasses everything, can we say it says any property we like? What if Any also has a method __add__ that receives str?

My conclusion from this is that perhaps it has to be some subset of every available option, e.g. Any type that fulfils this LSP condition. Or perhaps no methods that "clash" with those from the other half of the intersection.

@DiscordLiz

Which makes it intersections all the way down and no different than being Any, cause we have to assume LSP wasn't violated.

Perhaps we should restrict our examples to those that fulfil this condition? Even in the case we're untyped, it seems we have to assume that this method clash hasn't occurred.

@NeilGirdhar @mikeshardmind Specifics aside, I think whatever solution we come up with, it shouldn't require cast or type:ignore to pass through the type checker successfully. I mentioned the below relating to an earlier debate, but I think the same philosophy applies here - given the choice between:

• Being given a type that's too broad for the attributes available - there won't be any errors raised when we do some things that are invalid.
• Being given a type that's too narrow for the attributes available - there will be errors raised when we do things that are valid

The second one just seems much worse to me than the first, but I guess that is personal preference somewhat.

I guess this is maybe another way of phrasing @NeilGirdhar 's previous point about false positives vs false negatives.

[mikeshardmind]

The False positives case is preferable because we still give the user a way to handle it without a type ignore and without additional runtime costs. The False negative case cannot be remedied except by removing Any, and the whole reason we can't ban Any from intersections is that type checkers have stated this could happen in a way users don't have a way to remedy or even a clear way to inform them.

Consider this:

from untyped_library import SomeParserAugmentor  # user may not realize this is untyped at first
from typed_library import SomeParser  # generic over T, provides .parse(self, arg: str) -> T

def my_func(input_: str, parser: SomeParser[int] & SomeParserAugmenter ) -> None:
    """ Typed permissively so that other people can use the function """

   # This errors, we restrict the interface of Any when it clashes with what is typed
   # This also means the person writing the intersection is the one who gets the error
    x = parser.parse(input_, fix_mojibake=True) 


class _Parser(SomeParser, SomeParserAugmentor):
    """ Internal fallback where we create only the baseline functionality """
    pass
...


# this is fine, subclassing from Any means the attributes needed are assumed there. 
# This is the case currently with subclassing Any, not even just with this proposal
my_func(some_user_content, _Parser(int))  

In this case, the user actually has the tools to fix this without a type ignore in multiple ways.

1. Because they are informed that something isn't meeting their expectations, they may contribute upstream types
2. They can also add a protocol stating that the attribute exists:

from untyped_library import SomeParserAugmentor  # user may not realize this is untyped at first
from typed_library import SomeParser  # generic over T, provides .parse(self, arg: str) -> T
from typing import Protocol

class FixesMojiBake[T](Protocol[T]):
    def parse(self, input_: str, fix_mojibake: bool=False) -> T: 
        ...
    

def my_func(input_: str, parser: SomeParser[int] & SomeParserAugmenter & FixesMojiBake) -> None:
    x = parser.parse(input_, fix_mojibake=True)  # This no longer errors

class _Parser(SomeParser, SomeParserAugmentor):
    """ Internal fallback where we create only the baseline functionality """
    pass

# this is still fine
my_func(some_user_content, _Parser(int))  

With this, they've now explicitly stated they expect this compatible method to exist. This gives users the information they need to type things without compromising on safety for the portions of code that are already typed, and while being easy to accommodate situations outside of their control with untyped libraries. Note that this is safer thana type ignore, it would rely on the nature of Any for the thing Assigned to it, but it requires explicitly escaping out for things that could otherwise be accidentally hidden in terms of misuse by restricting the interface on usage to what is explicitly stated. This also automatically continues working should the untyped library become typed in the future without something lurking around for compatibility that could be silencing errors.

I think if we pair this approach with a note that type checkers should inform users when Any is present in an intersection that is causing an error and point them to a reference of how to handle this, we can make this easy to learn and ergonomic as well.

[gvanrossum]

A few observations after having read Eric's comment and trying to reason various things out in my head.

1. It is very easy to confuse yourself by mixing up the meaning of & with that of | (or vice versa). This happened to myself a few times. I think it's due to the fact that after x: T1 | T2, we consider valid properties of x those that are valid for T1 and for T2, while after x: T1 & T2, intuitively, the set of valid properties is the union of the valid properties of T1 and T2. (It's a similar confusion as often happens around contravariance.)
2. If is easy to reason things out for proper types, including edge cases like object and Never.(top and bottom, respectively), by thinking of types as sets of objects. The intersection and union operators work sensibly in this case, and all the rules you can derive this way (e.g. distributivity, subtyping, and attribute type evaluation, as shown in Eric's comment). However, Any is *not* just another edge case. It is the defining building block of gradual typing, and behaves like top or bottom depending on which side of the subtyping operator (<:`) it appears.
3. Using assert isinstance(x, T) in examples is not all that helpful, because at runtime T is constrained to concrete types. We need to go back to presenting examples in terms of whether x = y is allowable or not according to the type system given specific types for x and y (and possibly what the type of x is afterwards, in situations where narrowing happens).
4. Generics present a whole nother can of worms. I haven't spent enough time thinking about these.
5. There are probably also traps due to the difference between structural subtyping (Protocol) and nominal subtyping.
6. And perhaps others due to the pragmatic treatment of TypedDict.
7. Folks, please proofread your messages.

I'l write more later.

[gvanrossum]

[me]

Given that a: Any & T means thata has all the attributes of T and they have the types specified by T, in this example, x has __add__(self, arg: int) -> int so the type of x should be revealed as int. That matches my intuition.

@NeilGirdhar

Just something for you to consider. Let's just look at an example that doesn't involve Any:

Great, it should be possible to reason this out by considering types as sets of objects.

class C:
  def f(self) -> None: pass
class D:
  def g(self) -> None: pass
class E(C, D): pass

def f(x: C):
  if isinstance(x, D):
    reveal_type(x)  # C & D
    x.f()  # should be okay since x inherits from C
    x.g()  # should be okay since x inherits from D

f(E())

This is why the interface of C&D is the union of the interfaces of C and D.

But why introduce a new term, "interface"? Python's type system never uses this word -- we talk about types. I feel it's misleading to introduce this term -- probably because of my point (2). We should only consider questions of the form "if x has type T1 and y has type T2, is x = y acceptable?" And in addition, questions of the form "given x has type T and OP(a) is some operation (such as getting an attribute of a specific name, or calling it with specific arguments), is OP(x) valid?"

Now, consider:

* what is the interface of `Any`?

I feel this is not helping clear reasoning, given that in x = y, the rules when x has type Any are totally different from the rules when y has type Any. (And different again for OP().)

* what is the interface of `Any & C`?

Should be rephrase in terms of what happens to x = y or perhaps OP(x), separately for the case where x: Any & C and y: Any & C.

* So if `x` has type `Any & C`, what is the type of `x.f()`, `x.g()`, and `x.h()`?

That's a valid question. I think it should be answered by considering separately what the type of an expression is when x: Any and when x: C, and then doing an & over the resulting types.

Interestingly, since the answer might well be "an error occurs" for one of the branches of the & operator, maybe we should consider what the result should be of Error & T. Thinking about some examples, it seems that the Error type should disappear and the answer should be T -- IOW, it is not an error unless all branches of the & operator have type Error. This is nicely the dual of the error behavior for |, where the rule is that it's an error if it is an error for any branch of the | operator.

But now we get to think about when an error should be reported to the user -- surely not as soon as it occurs, in the case of &, or else x.attr where x: C & D would report an error if C supports attr but D doesn't -- and the intersection is meant to not report an error in that case.

Also, it seems that whatever rules we come up with for Error should also apply to Any? Except they don't -- It would seem that Error | T reduces to Error, but Any | T is irreducible. (I now understand that word. :-) It seems that we perhaps might regret that choice, but even those who regret it seem to agree that it's too late to change -- and I'm with Eric in that I think we should make sure that & is the dual of | in this respect, like it is everywhere else.

It's really worth reading Eric's elucidating comment.

For reference, there are two proposals being considered here. I'll call them 4 and 5 (relating to names from #1). Proposal 4 says: x.f(), x.g(), x.h() are all Any. Proposal 5 says: x.g(), x.h() are both Any, but x.f() is None.

I don't know what the type of x is here, but it can't be from the code snippet you gave at the top of your comment (which has neither Any nor h()). Could you provide the whole context?