[SPARK-26048][build] Enable flume profile when creating 2.x releases. #23931
Conversation
|
@dbtsai this doesn't need to be in the 2.4 branch but it would be nice to build the next RC with this change. |
| if [[ $SPARK_VERSION < "3.0." ]]; then | ||
| SCALA_2_11_PROFILES="-Pkafka-0-8" | ||
| SCALA_2_11_PROFILES="-Pkafka-0-8 -Pflume" | ||
| fi |
There was a problem hiding this comment.
Hi, @vanzin , @dbtsai , @srowen , @cloud-fan .
For branch-2.4 release, I thought we are supposed to use dev/create-release/release-build.sh in branch-2.4.
branch-2.4 already has -Pflume here. Is this PR aiming to use master branch code to do release 2.4.x?
if [[ $SPARK_VERSION > "2.3" ]]; then
BASE_PROFILES="$BASE_PROFILES -Pkubernetes -Pflume"
SCALA_2_11_PROFILES="-Pkafka-0-8"
elseThere was a problem hiding this comment.
In the past I've avoided using the release scripts from the branches for a bunch of reasons. But mainly:
- if you find a problem with the release script in the middle of building an RC, it's awkward to change the script in the branch without creating a new RC.
- as master moves forward, it becomes tricky to backport fixes from the master scripts to other branches
So I've always used the release scripts from master, and I think it's much simpler for everybody if they also do that.
(Note that even though the 2.4 script has the flume profile enabled, 2.4.0 was shipped without a flume artifact published to maven.)
There was a problem hiding this comment.
I had thought the same, but I think the practice is to let the main script create releases for prior branches too. I'd defer to release managers for policy on this. We ought to be able to drop support in the script for EOL releases. I did that a while ago in order to keep it at all sane after adding scala 2.12 support.
There was a problem hiding this comment.
Thanks, @vanzin and @srowen . It would be great if we explicitly guide to use the master branch's release script always at https://spark.apache.org/release-process.html .
|
@vanzin Okay. I enabled -Pflume in my build for rc6. Thanks! |
|
Test build #102922 has finished for PR 23931 at commit
|
|
Hey, isn't flume support deprecated since 2.3? https://spark.apache.org/docs/2.3.0/streaming-flume-integration.html |
|
ah got it. It's a followup of a001814#diff-01ca42240614718522afde4d4885b40d Although we removed flume in 3.0, the release script should still include flume when releasing 2.x LGTM |
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
+1, LGTM. Merged to master.
|
It's weird that with flume enabled, I got an error on signature validation on flume test jar when I published the jars to staging. Any idea?
|
|
Locally, or in the bundle that was pushed to Sonatype, do you see the hash / signature files for this module, or others? I'm not sure how it would be different, but I'd start there. |
|
Interestingly, I download the files and run , and it works without any problem. I guess it's a transient issue in sonatype. Do you know if I can ask sonatype to recheck the signature without building it again? |
|
Hm, I suppose verification of the keys depends on it having your key. I wonder how it knows to download the KEYS file from Apache to verify? See https://central.sonatype.org/pages/working-with-pgp-signatures.html ; I wonder if you need to publish your signing key to one of these 'central' servers? What other log detail can you get about the error? |
6 participants
No description provided.