Quo vadis, higher order function type signatures

[martinholters]

A remark by @vtjnash, "it would be better not to use ::Function in a signature, now that everything it callable, that distinction can be a bit fuzzy" during the functor -> function rewrite (originally here: #15696 (comment)) made me look at the situation a little closer, and the result did not make me exactly happy. Currently we have many higher order functions with ::Function in their signature, some without any type restriction, some with ::Callable, where Callable is an alias for Union{Function,DataType}. I would have loved to open a PR that changes the definition of Callable to Any and then use ::Callable everywhere (to mark the points where possibly traits might be used in the future).

Unfortunately, this would lead to a great many ambiguities. The first problematic candidate I found was, ehem, findfirst:

findfirst(A, v) # look for value v in array-like A
findfirst(testf::Function, A) # look for element in A where testf returns true

Obviously, without the ::Function, this would be totally ambiguous. Furthermore, traits won't save the day, as conceivably, one could make arrays callable (coming directly from the little shop of call overload horrors).

The number of complaints that non-Function callables cannot be passed to findfirst seems to be relatively low, so I'm not suggesting any immediate action. However, settling on a guideline where to go in the long run might be a good idea. I see four main options:

1. Use ::Function everywhere. This is overly restrictive, but I still wonder how often one wants to pass non-Function callables to higher-order functions in practice.
2. Use ::Callable (with its present definition) everywhere. Somewhat less restrictive, and probably takes care of most cases where 1. would hurt in practice. (But actually Callable is a misnomer, so it might be replaced with something better.)
3. Make the signatures as liberal as possible without introducing ambiguities. Obviously, this is least disruptive, but at the cost of inconsistency.
4. Use Any for Callable and use it everywhere, reworking the API if necessary to avoid ambiguities. Obviously, this would be quite disruptive, but once done, the API would be consistent and powerful.

I'm still undecided what I'd prefer. Please weigh in!

[DNF2]

A question about alternative 1:

If someone wants to pass a non-Function callable, can that not be fixed by passing x->MyCallable(x) instead? Especially now that anonymous functions are cheap?

[martinholters]

@okvs Yes. I'd then vote for adding convert(::Type{Function}, f) = (args...) -> f(args...) though, which would allow writing Function(MyCallable), which is longer, but clarifies the intent. (x->f(x) looks pretty redundant until you realize that f is not a function; Function(f) implies this to a certain extent.)

[nalimilan]

conceivably, one could make arrays callable (coming directly from the little shop of call overload horrors).

I guess we could decide that AbstractArrays should not be callable, or at least not if you want to pass them to findfirst.

I think we should wait for traits support, and then make Callable a trait and use it everywhere. Until then, I'm not sure the situation can really be improved.

[vtjnash]

The original function (findfirst(A, v)) is gone. So quick reply to the main points before closing this (additional speculation, if desired, may continue on discourse):

1. Use ::Function everywhere. This is overly restrictive, but I still wonder how often one wants to pass non-Function callables to higher-order functions in practice.

This is generally the suggestion, when needed to resolve dispatch ambiguity. It, for example, is how we replaced findfirst. Any value can be converted to a function fairly easily (any -> (args...) -> any(args...), so this isn't much of a problem for users.

2. Use ::Callable (with its present definition) everywhere. Somewhat less restrictive, and probably takes care of most cases where 1. would hurt in practice. (But actually Callable is a misnomer, so it might be replaced with something better.)

basically the same as 1: we only use this for AbstractDict and Set (nowhere in stdlib, for example)

3. Make the signatures as liberal as possible without introducing ambiguities. Obviously, this is least disruptive, but at the cost of inconsistency.

Of course, this is most preferable, when applicable. It's often good advice well beyond just ::Function too.

4. Use Any for Callable and use it everywhere, reworking the API if necessary to avoid ambiguities. Obviously, this would be quite disruptive, but once done, the API would be consistent and powerful.

largely a non-starter, as it is used because the alternative (Any) would be ambiguous