Skip to content

[2.4][SPARK-26048][SPARK-24530] Cherrypick all the missing commits to 2.4 release script #24503

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from
Closed

[2.4][SPARK-26048][SPARK-24530] Cherrypick all the missing commits to 2.4 release script #24503

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
78a0d36
Resolved SPARK-26048. https://github.com/apache/spark/pull/23931
gatorsmile May 1, 2019
c74dfde
[MINOR][BUILD] ensure call to translate_component has correct number …
Jan 17, 2019
88e0791
[SPARK-24530][FOLLOWUP] run Sphinx with python 3 in docker
cloud-fan Oct 2, 2018
3ed2e8c
[BUILD][MINOR] release script should not interrupt by svn
cloud-fan Oct 30, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions dev/create-release/do-release-docker.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,9 @@ if [ -n "$JAVA" ]; then
JAVA_VOL="--volume $JAVA:/opt/spark-java"
fi

# SPARK-24530: Sphinx must work with python 3 to generate doc correctly.
echo "SPHINXPYTHON=/opt/p35/bin/python" >> $ENVFILE

echo "Building $RELEASE_TAG; output will be at $WORKDIR/output"
docker run -ti \
--env-file "$ENVFILE" \
Expand Down
7 changes: 5 additions & 2 deletions dev/create-release/release-build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,9 @@ fi

PUBLISH_SCALA_2_12=0
SCALA_2_12_PROFILES="-Pscala-2.12"
if [[ $SPARK_VERSION < "3.0." ]]; then
SCALA_2_12_PROFILES="-Pscala-2.12 -Pflume"
fi
if [[ $SPARK_VERSION > "2.4" ]]; then
PUBLISH_SCALA_2_12=1
fi
Expand Down Expand Up @@ -327,7 +330,7 @@ if [[ "$1" == "package" ]]; then
svn add "svn-spark/${DEST_DIR_NAME}-bin"

cd svn-spark
svn ci --username $ASF_USERNAME --password "$ASF_PASSWORD" -m"Apache Spark $SPARK_PACKAGE_VERSION"
svn ci --username $ASF_USERNAME --password "$ASF_PASSWORD" -m"Apache Spark $SPARK_PACKAGE_VERSION" --no-auth-cache
cd ..
rm -rf svn-spark
fi
Expand Down Expand Up @@ -355,7 +358,7 @@ if [[ "$1" == "docs" ]]; then
svn add "svn-spark/${DEST_DIR_NAME}-docs"

cd svn-spark
svn ci --username $ASF_USERNAME --password "$ASF_PASSWORD" -m"Apache Spark $SPARK_PACKAGE_VERSION docs"
svn ci --username $ASF_USERNAME --password "$ASF_PASSWORD" -m"Apache Spark $SPARK_PACKAGE_VERSION docs" --no-auth-cache
cd ..
rm -rf svn-spark
fi
Expand Down
2 changes: 1 addition & 1 deletion dev/create-release/releaseutils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -236,7 +236,7 @@ def translate_component(component, commit_hash, warnings):
# The returned components are already filtered and translated
def find_components(commit, commit_hash):
components = re.findall(r"\[\w*\]", commit.lower())
components = [translate_component(c, commit_hash)
components = [translate_component(c, commit_hash, [])
for c in components if c in known_components]
return components

Expand Down
4 changes: 2 additions & 2 deletions dev/create-release/spark-rm/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,8 @@ RUN echo 'deb http://cran.cnr.Berkeley.edu/bin/linux/ubuntu xenial/' >> /etc/apt
pip install $BASE_PIP_PKGS && \
pip install $PIP_PKGS && \
cd && \
virtualenv -p python3 p35 && \
. p35/bin/activate && \
virtualenv -p python3 /opt/p35 && \
. /opt/p35/bin/activate && \
pip install $BASE_PIP_PKGS && \
pip install $PIP_PKGS && \
# Install R packages and dependencies used when building.
Expand Down