[SPARK-38569][BUILD] Rename external top level dir to connector

[alkis]

What changes were proposed in this pull request?

Rename the external top level directory to connector.

Why are the changes needed?

external is a hardwired special name for bazel and this causes all sorts of issues with both native bazel or extensions like bazel-compile-commands-extractor. Spark forks using bazel to build Spark have to go through hoops to make things work if at all.

Amongst other things, making the source code browsable through top level external symlink is impossible with a top level external directory in the repository.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Ran ./build/mvn -DskipTests clean package and ./dev/run-tests.

[cloud-fan]

It makes sense to avoid using reserved name for some build systems. Can we create a JIRA ticket for it?

[alkis]

It makes sense to avoid using reserved name for some build systems. Can we create a JIRA ticket for it?

Sure thing. Done.

[alkis]

FYI for more details on how this causes major issues in practice hedronvision/bazel-compile-commands-extractor#30

[cloud-fan]

LGTM, can you fix the merge conflcts?

[dongjoon-hyun]

Since this is big, shall we discuss a little more in the dev mailing list before merging, please?

[dongjoon-hyun]

Although Apache Spark is irrelevant to bazel, I fully understand the requirement from downstream projects here.
What we need is simply an officially discussion with a broader community audience.

[alkis]

Although Apache Spark is irrelevant to bazel, I fully understand the requirement from downstream projects here. What we need is simply an officially discussion with a broader community audience.

Thank you Dongjoon. This is my first contribution to Apache Spark so bear with my perhaps silly questions. Could you advice what topics we want to discuss - the move itself, the final directory name or something else? While the PR looks big, at its core this is a single directory rename (maybe there are risks I am unaware of?).

[AmplabJenkins]

Can one of the admins verify this patch?

[dongjoon-hyun]

@alkis .

You are asking a broader upstream breaking restructuring for your specific build system's limitation. Your suggestion might cause another conflicts and headache on another downstream forks.

So, please send an email about your intention and share this PR link to dev@spark mailing list first and build a consensus among the other community members. If there is any objections, we need a discussion, don't we?

If we agree to rename, the next step is the consensus on new name. From my side, I don't think vendor is a correct usage here. Why Apache Spark's Avro and Kafka module should be under vendor? Does it make sense?

BTW, although bezel is not supported by Apache Spark and the community recommend to use Maven and SBT so far, I'm not against your use case of bazel.

[alkis]

Thank you Dongjoon. Makes sense. I started a conversation at dev@spark mailing list.

Stale.

[dongjoon-hyun]

Could you revise the PR title and description according to the PR content?

[alkis]

Could you revise the PR title and description according to the PR content?

Done.

[cloud-fan]

thanks, merging to master!

[alkis]

thanks, merging to master!

Woohoo, thank you!