[SPARK-33914][SQL][DOCS] Describe the structure of unified DS v1 and v2 tests

[MaxGekk]

What changes were proposed in this pull request?

Add comments for the unified datasource tests, describe what kind of tests they contain, and put refs to other test suits.

Why are the changes needed?

To improve code maintenance.

Does this PR introduce any user-facing change?

No

How was this patch tested?

By running ./dev/scalastyle.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37975/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37975/

[SparkQA]

Test build #133385 has finished for PR 30929 at commit 0a8a0b8.

• This patch fails SparkR unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #133394 has finished for PR 30929 at commit 0b557b2.

• This patch fails SparkR unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[MaxGekk]

Not clear how my changes are related to the error:

[error] /home/runner/work/spark/spark/mllib/target/java/org/apache/spark/mllib/util/MLlibTestSparkContext.java:10:1:  error: illegal combination of modifiers: public and protected
[error]   protected  class testImplicits {
[error]              ^

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37988/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37989/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37989/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37988/

[SparkQA]

Test build #133397 has finished for PR 30929 at commit af4d508.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #133398 has finished for PR 30929 at commit ac74991.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

org.apache.spark.execution.command.AlterTableDropPartitionSuiteBase ?

[MaxGekk]

No, I think the ref is correct, see

spark/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/AlterTableDropPartitionSuite.scala

Line 23 in 29cca68

trait AlterTableDropPartitionSuiteBase extends command.AlterTableDropPartitionSuiteBase {

[dongjoon-hyun]

My bad~

[dongjoon-hyun]

org.apache.spark.sql.execution.command.AlterTableAddPartitionSuiteBase?

[MaxGekk]

You pointed out the current trait which contains tests for v1 and v2. org.apache.spark.sql.execution.command.v1.AlterTableAddPartitionSuiteBase is the base trait for v1 In-Memory and Hive catalogs.

[HyukjinKwon]

+1 I like this comments!

[cloud-fan]

thanks, merging to master!

[dongjoon-hyun]

+1, late LGTM. Thank you.