Return basic statistical functionality to Base

[ararslan]

It seems incredibly unfortunate to me, and indeed almost actively user-hostile, to remove basic functionality such as std from Base. @sbromberger summarized it well in Slack: "if the function is generally understood by a layperson, it should not be removed." You don't have to be a statistics expert to understand what a standard deviation is. The collection of basic statistics functionality in Base in previous versions of Julia was fairly lean and I think struck the perfect balance of "here are the basics, now load a package if you want to get more advanced." Indeed, in the release-0.6 branch, base/statistics.jl is a couple hundred lines of code, including whitespace and extensive documentation. Who was it hurting to have in Base?

[sbromberger]

I note that mean and median are still in Base. What are the current criteria for moving things out?

The collection of basic statistics functionality in Base in previous versions of Julia was fairly lean and I think struck the perfect balance of "here are the basics, now load a package if you want to get more advanced."

This, 100%. I will fight the "npm-ification" of Julia as long as I'm a participating member of the community. Having things ready to go out of the box was a huge selling point for me.

Moving large chunks of "basic" functionality out of Base to other packages (especially third-party packages; I'm willing to consider stdlib packages like Random and Iterators special cases, though I haven't really made up my mind as to whether this is a good thing yet) also makes the library ecosystem that much more fragile – while Pkg.add("StatsBase") isn't so much of a problem for an end user, it introduces a dependence fragility on any libraries that need a function like std now, which then propagates to other libraries / end users.

[pablosanjose]

FWIW I agree.

Is there an official decision of where precisely the line is drawn between Base, stdlib and third-party packages?

I think this is a crucial definition that can totally affect the character and feel of a language, from Julia-the-language to Julia-the-platform. With the current strategy, the development of Julia-the-language is fast and nimble, but at the potential cost of a fragile/fragmented Julia-the-platform, as nicely pointed out above.

[fredrikekre]

if the function is generally understood by a layperson, it should not be removed.

How about matrix multiplication? Or solving a linear system? These are things a layperson understand even better than std! Should we move LinearAlgebra back too?

Moving large chunks of "basic" functionality out of Base to other packages (especially third-party packages; I'm willing to consider stdlib packages like Random and Iterators special cases, though I haven't really made up my mind as to whether this is a good thing yet) also makes the library ecosystem that much more fragile – while Pkg.add("StatsBase") isn't so much of a problem for an end user, it introduces a dependence fragility on any libraries that need a function like std now, which then propagates to other libraries / end users.

I thought the plan was that StatsBase will be a stdlib (perhaps named Statistics?). It just felt unnecessary to move StatsBase into this repo, only to move it out soon again.

[sbromberger]

How about matrix multiplication?

Matrix multiplication is in stdlib, which is "close enough to Base" right now, I guess. I'm still not really sold on "small base, large stdlib" but as long as the functionality is being distributed and made available via a standard Julia install, I guess it's better than going third-party.

I thought the plan was that StatsBase will be a stdlib (perhaps named Statistics?).

That would perhaps change things somewhat, but currently, it's not that way, and in order to use std in 0.7, we now have to rely on a third-party package. That doesn't seem right.

[StefanKarpinski]

Yes, the plan is to move StatsBase to stdlib/Statistics. I had proposed moving these functions into a new stdlib/Statistics module which would ship with Julia and then move the appropriate parts of StatsBase later but people preferred doing it this way.

[sbromberger]

@StefanKarpinski until that's done, can we please keep std (and any others) in Base?

[StefanKarpinski]

No, because we can't move things out of Base later but we can move things into stdlib later.

[sbromberger]

so until then the burden falls on the library developers to change where they source their functions from? That seems ... wrong, somehow.

This wasn't an issue for Random, or any of the others (disregard Iterators for the moment). Why is it an issue now?

[ararslan]

Why do these need to be removed from Base at all?

[StefanKarpinski]

We already had this discussion and came to a pretty clear consensus at the time, I'm not inclined to rehash the whole thing.

[ararslan]

Seems like there's fairly broad support now for undoing it.