[SPARK-8657] [YARN] Fail to upload conf archive to viewfs

[litao-buptsse]

Fail to upload conf archive to viewfs in spark-1.4
JIRA Link: https://issues.apache.org/jira/browse/SPARK-8657

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

OK to test

[SparkQA]

Test build #977 has started for PR 7055 at commit cbae84e.

[SparkQA]

Test build #977 has finished for PR 7055 at commit cbae84e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

@litao-buptsse would you mind opening this against the master branch instead so future Spark releases also benefit from the fix? This is currently opened against branch-1.4.

[srowen]

There is also #7053 for master; the discussion at https://issues.apache.org/jira/browse/SPARK-8657 and the third PR has a little more about why.

[andrewor14]

Hm, I'm pretty confused. @litao-buptsse Which is your proposed fix, this one or #7053? Did you intend to have different fixes merged into different branches?

[srowen]

I think the code paths are a little different; see discussion at #7042

[litao-buptsse]

@andrewor14 @srowen #7125 is for master and #7055 is for branch-1.4

[tgravescs]

why did you change the name of this? please revert as previous name was more descriptive and the change is unnecessary.

[litao-buptsse]

OK I got it

[srowen]

@litao-buptsse this now diverges a lot from its sibling PR at #7125 . Did you push changes here unintentionally? this needs to match the other PR.

[litao-buptsse]

@srowen #7125 is for master branch and passed all the test case. You can consider to merge it to master.

But for branch-1.4, not only contains the bugs, but also need some refactor to cache up with the master. My first commit litao-buptsse@cbae84e is just intend to fix the bugs. The following commits of the #7055 PR is intend to refactor the branch-1.4 to catch up with master branch.

#7125 for master is ready to merge to master.
But I don't know if I should refactor the branch-1.4? Or it it spark commiter to cherry-pick some commits from master to branch-1.4?

If no need to do the refactor job, litao-buptsse@cbae84e can fix the bugs for branch-1.4 and passed all tests.
If I can continue do the branch-1.4 refactor job, I will follow @tgravescs 's advice and review my code again.

[srowen]

@litao-buptsse one JIRA = one logical issue, so I'd generally expect PRs for one JIRA to be quite similar when back-ported. However this has a lot more change; are you sure you mean for all of this to be in this PR?

In any event, at this point, your change to master ended up very simple and easy to cherry-pick, so you can close this. When it's merged it can be back-ported easily. That's standard procedure.

[litao-buptsse]

@srowen OK I will close this PR and remain only #7125 for this issue