[SPARK-15761] [MLlib] [PySpark] Load ipython when default python is Python3

[MechCoder]

What changes were proposed in this pull request?

I would like to use IPython with Python 3.5. It is annoying when it fails with IPython requires Python 2.7+; please install python2.7 or set PYSPARK_PYTHON when I have a version greater than 2.7

How was this patch tested

It now works with IPython and Python3

[MLnick]

LGTM

[SparkQA]

Test build #59970 has finished for PR 13503 at commit a051953.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

cc @JoshRosen

[MechCoder]

Merge?

[MechCoder]

bump?

[JoshRosen]

If the default Python executable is named python but actually points to a Python 2.7+ or Python 3 executable then this is going to return a false positive because the string python is < python2.7.

Rather than relying on the name of the executable, it might make more sense to shell out to Python and ask it its version. For example, I think the following will work:

$DEFAULT_PYTHON -c 'import os; import sys; sys.exit(int(sys.version_info <= (2, 7, 0)))'

If the Python version is greater than or equal to 2.7.0, then this will exit with 0; otherwise, this will exit with 1. This lets you check the exit status and not rely on the executable name.

[JoshRosen]

That said, the code in this PR is a strict improvement over the old overly-stringent check, so it's okay to merge it in its current form but I'd prefer to just make this super robust so we fix this once and for all.

[MechCoder]

@JoshRosen fixed, thanks! let me know if you need any other changes.

[MechCoder]

I would also like to change

if hash python2.7 2>/dev/null; then
  # Attempt to use Python 2.7, if installed:
  DEFAULT_PYTHON="python2.7"
else
  DEFAULT_PYTHON="python"
fi

to just

DEFAULT_PYTHON="python"

I'm not sure if it is a great assumption that python2.7 is used by default, when python points to something else.

[jodersky]

lgtm

[SparkQA]

Test build #61578 has finished for PR 13503 at commit 85ce78d.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[MechCoder]

retest this please

[SparkQA]

Test build #61592 has finished for PR 13503 at commit 85ce78d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Merged to master/2.0/1.6