[SPARK-8528] Expose SparkContext.applicationId in PySpark

[smartkiwi]

Use case - we want to log applicationId (YARN in hour case) to request help with troubleshooting from the DevOps

[JoshRosen]

The motivation for this change seems fine to me, although I have some nits to pick with the format of the docstrings.

[JoshRosen]

I would try to make this docstring match the style of our other docstrings a bit better. Specifically, there's not a need to say applicationId - here, since that's clear from the method's name.

Instead, I'd have this first line just read something like this:

A unique identifier for the Spark application.
Its format depends on the scheduler implementation.

I'm not sure whether we need to show the examples here, but I guess it doesn't necessarily hurt.

[JoshRosen]

Also, do you mind updating the PR title so that it's not truncated, e.g. "[SPARK-8528] Expose SparkContext.applicationId in PySpark"?

[smartkiwi]

Done updating docstrings. Thank you for suggestions.

[JoshRosen]

Jenkins, this is ok to test.

[SparkQA]

Test build #35561 timed out for PR 6936 at commit b4ebfc5 after a configured wait of 175m.

[smartkiwi]

Interesting build results - not tests failed, but whole build timeouted

[smartkiwi]

Jenkins, please test it again

[JoshRosen]

Jenkins, retest this please.

[SparkQA]

Test build #35815 has finished for PR 6936 at commit b4ebfc5.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[smartkiwi]

Hm, looks like last test haven't honored ELLIPSIS. Anyway this is doctest is not particularly useful. The contents of the applicationId is set outside of the test - and can be changed anytime, i.e. if tests would run in SparkContext in yarn or remote cluster - the result will be different.

So I'll simplify the test - it will just check that such property exists and returns something

__main__.SparkContext.applicationId
Failed example:
    sc.applicationId  # doctest: +ELLIPSIS
Expected:
    u'local-...'
Got:
    'local-1435285070164'

[smartkiwi]

Please test updated version - I made the doctest stable

[SparkQA]

Test build #35861 has finished for PR 6936 at commit 14d77a8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Minor nit: the u' is going to be confusing to Scala users. I'd just go with regular quoted strings here.

[JoshRosen]

BTW, I think that you can get this to work with doctest.ELLIPSIS; the problem is likely Python 2.x vs 3.x differences. You can fix this by using the ignore_unicode_prefix decorator defined in rdd.py (see that file or grep the python/pyspark/ subdirectory for examples).

[smartkiwi]

Thanks for hints
Retesting locally with python3 and python2. I will push update in ~30 minutes

[JoshRosen]

Yeah, no rush; sorry for the hassle. At some point we should update the contributing guide to include tips on this stuff, such as the doctest trick.

[smartkiwi]

Python doctests are rather limited in expressiveness, but efficient in keeping examples bug free in documentation.
Anywhere - here is another patch that works in both py2.6/2.7 and py3.4

[SparkQA]

Test build #35905 has finished for PR 6936 at commit 870338b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

LGTM, so I'm going to merge this to master. Thanks!