[SPARK-11315] [YARN] Add YARN extension service to publish Spark events to YARN timeline service

[steveloughran]

SPARK-11315 is the publisher side of the SPARK-1537 Yarn timeline server integration: the event listener to asynchronously publish events to ATS; the basiccoreunit and mock tests to verify functionality.

[SparkQA]

Test build #42424 has finished for PR 8744 at commit 34c0283.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • trait YarnExtensionService extends Closeable

[steveloughran]

Failing test was org.apache.spark.network.sasl.SaslIntegrationSuite.testNoSaslClient in spark-network-shuffle; this is not downstream of the artifacts modified in this patch so presumably unrelated.

[vanzin]

nit: ordering

[SparkQA]

Test build #46890 has finished for PR 8744 at commit c47f4d6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[steveloughran]

Test run had 0 failing tests and a lot of stack related to connection refused in jenkins ... possibly an"environmental" failure

[SparkQA]

Test build #47080 has finished for PR 8744 at commit e687612.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #47268 has finished for PR 8744 at commit c2add7b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[steveloughran]

Now that it's dependency PR is in, I welcome comments and reviews on this. The latest change just adds a version counter to every entity publishing, so that when checking for incomplete app updates via #6935, there's none of the timestamp comparison issues which surfaced in FsHistoryProvider (and which forced a switch to file length checks there).

[SparkQA]

Test build #47345 has finished for PR 8744 at commit 5acaebb.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #47496 has finished for PR 8744 at commit d13b48e.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #47595 has finished for PR 8744 at commit fe0c7b7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #48250 has finished for PR 8744 at commit 6d79d77.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #48566 has finished for PR 8744 at commit 8403920.

• This patch fails build dependency tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #48769 has finished for PR 8744 at commit 90a91c9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #49242 has finished for PR 8744 at commit 7a08ac7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #50203 has finished for PR 8744 at commit 99161a6.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[steveloughran]

I'm closing this for now as there's enough in Spark 2 to let me hook it up independently; I'm also switching to the ATS1.5 publisher, which pushes out via HDFS.