[SPARK-1550] [PySpark] Allow SparkContext creation after failed attempts

[JoshRosen]

This addresses a PySpark issue where a failed attempt to construct SparkContext would prevent any future SparkContext creation.

[SparkQA]

QA tests have started for PR 1606. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17232/consoleFull

[JoshRosen]

Because we keep a reference to this object in SparkContext._active_spark_context, this method never gets called except when cleaning up after a SparkContext creation attempt that failed because another context was already running. In that case, the call to sc.stop() clears SparkContext._active_spark_context and we lose track of the active context, which can allow the creation of multiple running contexts.

[SparkQA]

QA results for PR 1606:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17232/consoleFull

[mattf]

+1 lgtm

i've been hitting this issue repeatedly, but assumed it was a corner case that wouldn't get much attention. my assumption is that use of spark-submit and pyspark are the primary ways to get a context.

[mateiz]

Thanks Josh, merged this.