[SPARK-7018][BUILD] Refactor dev/run-tests-jenkins into Python

[JoshRosen]

This commit refactors the run-tests-jenkins script into Python. This refactoring was done by @brennonyork in #7401; this PR contains a few minor edits from @JoshRosen in order to bring it up to date with other recent changes.

From the original PR description (by @brennonyork):

Currently a few things are left out that, could and I think should, be smaller JIRA's after this.

1. There are still a few areas where we use environment variables where we don't need to (like CURRENT_BLOCK). I might get around to fixing this one in lieu of everything else, but wanted to point that out.
2. The PR tests are still written in bash. I opted to not change those and just rewrite the runner into Python. This is a great follow-on JIRA IMO.
3. All of the linting scripts are still in bash as well and would likely do to just add those in as follow-on JIRA's as well.

Closes #7401.

[JoshRosen]

When I tried launching this via "NewSparkPullRequestBuilder", it failed because it wasn't passed the PR title via an environment variable:

Traceback (most recent call last):
  File "./dev/run-tests-jenkins.py", line 228, in <module>
    main()
  File "./dev/run-tests-jenkins.py", line 156, in main
    ghprb_pull_title = os.environ["ghprbPullTitle"]
  File "/home/anaconda/lib/python2.7/UserDict.py", line 23, in __getitem__
    raise KeyError(key)
KeyError: 'ghprbPullTitle'

The fix here is simple: pass the title when sending the RPC from spark-prs. I'll fix this on the spark-pr-dashboard side shortly.

[JoshRosen]

Fixed spark-prs here: databricks/spark-pr-dashboard@a060947

[JoshRosen]

Jenkins, retest this please.

[JoshRosen]

Jenkins, retest this please.

[SparkQA]

Test build #43905 has finished for PR 9161 at commit 43123d7.

• This patch fails R style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #43903 has finished for PR 9161 at commit ca9f49b.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #43902 has finished for PR 9161 at commit ca9f49b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #1925 has finished for PR 9161 at commit ca9f49b.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #43907 has finished for PR 9161 at commit c6ca012.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Whoops, realized that I also need to port #6664 so that we link the full console logs instead of the previews.

[SparkQA]

Test build #43910 has finished for PR 9161 at commit f93a62a.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Jenkins, retest this please.

[JoshRosen]

Aright, this looks good to me so I'm going to merge it now and will triage / hotfix any issues that we discover.

[SparkQA]

Test build #43911 has finished for PR 9161 at commit f93a62a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.