[SPARK-32178][TESTS] Disable test-dependencies.sh from Jenkins jobs

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to disable dependency tests(test-dependencies.sh) from Jenkins.

Why are the changes needed?

• First of all, GitHub Action provides the same test capability already and stabler.
• Second, currently, test-dependencies.sh fails very frequently in AmpLab Jenkins environment. For example, in the following irrelevant PR, it fails 5 times during 6 hours.
  • [SPARK-32100][CORE][TESTS][FOLLOWUP] Reduce the required test resources #29001

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the Jenkins without test-dependencies.sh invocation.

[dongjoon-hyun]

Could you review this please, @HyukjinKwon and @maropu ?

[HyukjinKwon]

@dongjoon-hyun, could we maybe test this it in amp-jenkins-worker-04? I was doubting if other builds pass fine given that it has sort of broken .m2 cache. For some reasons #28957 consistently uses amp-jenkins-worker-04. Let me try to port it there and see if tests pass.

[dongjoon-hyun]

How can we specify the worker? Maybe, by triggering multiple times until we reach worker-04?

could we maybe test this it in amp-jenkins-worker-04?

[HyukjinKwon]

I don't know .. maybe that's the only way ..

[dongjoon-hyun]

BTW, another purpose of this PR is off-loading AmpLab Jenkins jobs. We will save more resources there and reduce the test time by taking advantage the existing GitHub test coverage.

[dongjoon-hyun]

Retest this please.

[dongjoon-hyun]

Retest this please

[dongjoon-hyun]

Retest this please

[dongjoon-hyun]

Retest this please

[dongjoon-hyun]

Retest this please

[dongjoon-hyun]

Retest this please

[dongjoon-hyun]

Retest this please

[dongjoon-hyun]

Retest this please

[HyukjinKwon]

#29004 (comment) got started in worker 4!

[dongjoon-hyun]

Wow. We are lucky! :)

[HyukjinKwon]

Okay, let me just merge this one. At least the build got started properly, and Github Actions will test the dependencies properly.

[HyukjinKwon]

Merged to master.

[dongjoon-hyun]

Thank you for review and merge. worker-04 also looks good until now.

[maropu]

Nice update, thanks, @dongjoon-hyun !

[dongjoon-hyun]

Thank you, @maropu !

[SparkQA]

Test build #125005 has finished for PR 29004 at commit e8c916c.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #125001 has finished for PR 29004 at commit e8c916c.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[HeartSaVioR]

Would we want to offload gendoc as well? Similar situation occurs for gendoc in worker-05 and Github Action passed.

#28975
#28975 (comment)

[dongjoon-hyun]

Oh, let me try. Thank you for reporting, @HeartSaVioR .

[dongjoon-hyun]

Hmm. It seems that we need another GitHub Action jobs for that because the current jekyll build triggers the following which is different from Jenkins.

    puts "Running 'build/sbt -Pkinesis-asl clean compile unidoc' from " + pwd + "; this may take a few minutes..."
    system("build/sbt -Pkinesis-asl clean compile unidoc") || raise("Unidoc generation failed")

In @HeartSaVioR 's PR, it was the following. I guess it depends on the PRBuilder configuration at each time.

Building Unidoc API Documentation
========================================================================
[info] Building Spark unidoc using SBT with these arguments:  -Phadoop-3.2 -Phive-2.3 -Pspark-ganglia-lgpl -Pkubernetes -Pmesos -Phadoop-cloud -Phive -Phive-thriftserver -Pkinesis-asl -Pyarn unidoc

[dongjoon-hyun]

Let me summarize this issue in a WIP PR~ We can offload it if that is not used in the release process.

[dongjoon-hyun]

Or, we can add a test coverage for this.

[HeartSaVioR]

I guess it depends on the PRBuilder configuration at each time.

Ah OK that might be tricky. It'd be great if we can imitate the same behavior.

[dongjoon-hyun]

Ya. Let's continue the discuss on #29017