Skip to content

[SPARK-26048][build] Enable flume profile when creating 2.x releases. #23931

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

[SPARK-26048][build] Enable flume profile when creating 2.x releases. #23931

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion dev/create-release/release-build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,14 +116,17 @@ SCALA_2_11_PROFILES=
if [[ $SPARK_VERSION > "2.3" ]]; then
BASE_PROFILES="$BASE_PROFILES -Pkubernetes"
if [[ $SPARK_VERSION < "3.0." ]]; then
SCALA_2_11_PROFILES="-Pkafka-0-8"
SCALA_2_11_PROFILES="-Pkafka-0-8 -Pflume"
fi
Copy link
Member

@dongjoon-hyun dongjoon-hyun Mar 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, @vanzin , @dbtsai , @srowen , @cloud-fan .

For branch-2.4 release, I thought we are supposed to use dev/create-release/release-build.sh in branch-2.4.

branch-2.4 already has -Pflume here. Is this PR aiming to use master branch code to do release 2.4.x?

if [[ $SPARK_VERSION > "2.3" ]]; then
  BASE_PROFILES="$BASE_PROFILES -Pkubernetes -Pflume"
  SCALA_2_11_PROFILES="-Pkafka-0-8"
else

Copy link
Contributor Author

@vanzin vanzin Mar 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the past I've avoided using the release scripts from the branches for a bunch of reasons. But mainly:

  • if you find a problem with the release script in the middle of building an RC, it's awkward to change the script in the branch without creating a new RC.
  • as master moves forward, it becomes tricky to backport fixes from the master scripts to other branches

So I've always used the release scripts from master, and I think it's much simpler for everybody if they also do that.

(Note that even though the 2.4 script has the flume profile enabled, 2.4.0 was shipped without a flume artifact published to maven.)

Copy link
Member

@srowen srowen Mar 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had thought the same, but I think the practice is to let the main script create releases for prior branches too. I'd defer to release managers for policy on this. We ought to be able to drop support in the script for EOL releases. I did that a while ago in order to keep it at all sane after adding scala 2.12 support.

Copy link
Member

@dongjoon-hyun dongjoon-hyun Mar 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @vanzin and @srowen . It would be great if we explicitly guide to use the master branch's release script always at https://spark.apache.org/release-process.html .

else
PUBLISH_SCALA_2_10=1
fi

PUBLISH_SCALA_2_12=0
SCALA_2_12_PROFILES="-Pscala-2.12"
if [[ $SPARK_VERSION < "3.0." ]]; then
SCALA_2_12_PROFILES="-Pflume"
fi
if [[ $SPARK_VERSION > "2.4" ]]; then
PUBLISH_SCALA_2_12=1
fi
Expand Down