the future of Random.gentype

[tpapp]

Random.gentype was introduced in #27756, so that custom random samplers would not have to use Base.eltype for returning element types, but the former falls back to the latter.

In the discussion of the recent docs cleanup PR #31787 it was unclear whether to make Random.gentype part of the API.

It would be great to decide

• whether Random.gentype(sampler) or Base.eltype(sampler) should be the API for querying typeof(rand(sampler)), especially if sampler is not a collection.
• whether Random.gentype is needed at all, or should be removed from the internal code, too.

@rfourquet, please comment.

[rfourquet]

Feedback in #27756 have been very helpful to avoid the initially proposed gentype API. The result is that there is not much substance left to gentype, which behaves essentially like eltype. The takeaway is that:

1. rand(T), when T is a type, is hardcoded to return an object of type T.
2. gentype(x) == gentype(typeof(x))
3. gentype(x) defaults to eltype(x)

Even if 1. looks like a restriction, it makes it simpler to reason about what type you get, and it doesn't prevent accomplishing anything.

The main question is: for collections, do we want to allow gentype(x) != eltype(x) ?
For the same reason as 1., I believe now that we shouldn't: it's simpler to have the restrictive rule "rand on a collection returns an element from this collection". If we do this, then only one of gentype or eltype should be defined, which is difficult to enforce. Then the simplest would be to simply remove gentype, and to extend the definition of eltype so that it applies to non-collection objects on which rand can be called.

[tpapp]

Now that I have studied the code in detail, I think that removing Random.gentype and just using Base.eltype for this purpose would be innocuous and simplify the code.

An objection could be that this is punning on eltype to a certain extent.

I am fine with either solution though, but a decision should be made so that it can be documented as part of the Random API.

[rfourquet]

Now that I have studied the code in detail

Yay, I feel less pressure to not disappear under a bus now 😜 (not that I want to)

[tpapp]

I wonder whether I should redo #31988 once #31787 is merged, to demonstrate the simplifications in the code, or wait for a decision here.

My understanding is that Random.gentype was not part of the existing API, so no one should have relied on it and we can remove it without technically breaking anything. Of course, https://github.com/rfourquet/RandomExtensions.jl is an exception but I trust that fixing that won't be a problem.

However, I think that custom samplers in Random is a very nice interface now and more packages should be using it (in particular, Distributions.jl), which is why I want to help with this. But I recognize that clarifying this may not end up in 1.2 and we could have to wait for 1.3 for that.

[rfourquet]

I wonder whether I should redo #31988 once #31787 is merged, to demonstrate the simplifications in the code, or wait for a decision here.

I wil merge the doc change in #31990 when CI finishes, and you can then update #31988, or wait for a decision. But I don't think that code simplification are important here, they are quite small and negligible compared to the conceptual simplification for the user.

My understanding is that Random.gentype was not part of the existing API, so no one should have relied on it and we can remove it without technically breaking anything

You are correct, that's why I insisted that you don't introduce gentype in your doc changes. I believe I'm the only one having a package using this function, so this is indeed not a problem.

and more packages should be using it (in particular, Distributions.jl)

🙏 I have been meaning to tackle this, it's great to have you on board!

clarifying this may not end up in 1.2 and we could have to wait for 1.3 for that

There is not a big problem waiting for 1.3 for that. As of now, the official interface is using eltype, so we can just start with this. If gentype becomes official, it will be a matter of changing eltype to gentype, which is not urgent. I agree though that it would be great to sort out this question fast.

[tpapp]

@rfourquet: I am wondering what would be the best way to move forward with this minor issue. If you can think of core devs who would want to comment on this but may have missed it, please mention them.

I have the feeling that relatively few people follow the Random API closely, and also that the support for #27756 was lukewarm at best and no one would object to reverting it, ie just removing Random.gentype as in #32008.

[rfourquet]

I believe that we don't really need more support to remove gentype per se, but that we do to "upgrade" the eltype meaning/documentation (which I find important if we remove gentype). So for me the best way forward is to update the eltype docstring in your PR and to ask for feedback on this specific point.

[rfourquet]

I realised that there is a PR (#28704) wanting to add rand(::Pair), i.e. rand(1=>2) would be equivalent to rand((1, 2)). But there is opposition to this change: Pair is more than a 2-tuple semantically, so we may reserve this special case for a possible future use-case. I just added a comment there with such a possible use-case.
As a consequence, we may want rand(p::Pair) to return something of a different type than eltype(p)!
I.e. eltype(p) != gentype(p).
This is what I did in JuliaRandom/RandomExtensions.jl#4 (I do even "worse" there, by allowing gentype(p) to be a subtype of gentype(typeof(p)), to allow more tightly typed collections).

The take-away for me is that, if the API for rand(::Pair) is reserved, we should keep gentype internally. Of course, we could delete gentype for now and re-introduce it later if needed, but I don't see deleting gentype urgent enough to warrant this code-churn.

For now, my prefered solution is to keep gentype entirely internal, and to disallow having gentype(x) != eltype(x) except for Pair!

[tpapp]

Sorry, but I don't see the advantage of the syntax introduced in that PR, and for me it is not worth special-casing the Random API for this.

[rfourquet]

I'm not totally sold on this alternative API (from this PR), but just wanted to illustrate a possible meaning to rand(::Pair), which requires a distinct gentype function from eltype.
(One advantage is brevity and readability, which becomes apparent when you nest calls, but it's not the place to discuss it).

[tpapp]

FWIW, I think one should not become overenthusiastic about defining samplers for types. It should only be done where the choices are pretty standard an unambiguous, which can then be used as building blocks. I imagine most (all?) of these are already in Base or Random.

For anything beyond generating uniform random values of some T <: Real, I think that the right choice is to encapsulate the distribution information in a value, and define a sampler on that.

From this perspective, Random.gentype should not be needed.