Add license and notice to spark client jar and push jar to maven #1830

gh-yzou · 2025-06-07T01:10:30Z

This PR does following:

add default license and notice to spark client jar
update the spark client jar name to align with polaris published jar
enable publish spark client jar to maven

flyrain · 2025-06-07T04:48:13Z

site/content/in-dev/unreleased/polaris-spark-client.md

 --conf spark.sql.catalog.<spark-catalog-name>.token-refresh-enabled=true
 ```
-Assume the released Polaris Spark client you want to use is `org.apache.polaris:polaris-iceberg-1.8.1-spark-runtime-3.5_2.12:1.0.0`,
+Assume the released Polaris Spark client you want to use is `org.apache.polaris:polaris-spark-3.5_2.12:1.0.0`,


Are we sure spark can download the bundle jar with the following config? Can we test it out? I'm not sure how do we test it against local mvn repo though, one way is to merge this into nightly maven repo first, then test it out.

--packages org.apache.polaris:polaris-spark-3.5_2.12:1.0.0

I haven't tested it out yet. once it is into nightly maven repo, i can test it out and update the doc if needed.

flyrain · 2025-06-07T04:52:15Z

plugins/spark/v3.5/spark/build.gradle.kts

-  archiveClassifier = null
-  archiveBaseName =
-    "polaris-iceberg-${icebergVersion}-spark-runtime-${sparkMajorVersion}_${scalaVersion}"
+  archiveClassifier = "bundle"


Think a bit more, I think it'd be nice to have the iceberg version in the package name. Otherwise, debugging would be tricky. Users couldn't figure out the Iceberg lib version easily.

Wondering if we could apply the Iceberg version as we did for Scala version.

The name of the jar now seems <project_name>-<polaris_version>-.jar, and the scala_version is embedded in the project_name, such as spark-3.5_2.12. I don't think it is a good idea to have iceberg version as project name, since the iceberg version might gets updated more frequently.
i don't think we can touch polaris_version also, then classifier is the only place . however, it seems wired that the iceberg version is after the polaris version. I think what we can do is publish on the webpage about the iceberg client compatibility. Given in long term, we may not need iceberg version in the package name also.

Users can open up the jar file to figure out the embedded Iceberg version. But my concern is that a lot of users don't know that or couldn't do that easily. I think it's still valuable to provide the iceberg versions as it's embedded. Unless we figure out a way to not embed it.

I don't think it is a good idea to have iceberg version as project name, since the iceberg version might gets updated more frequently.

We need to release a new version for this plugin every time we update the embedded Iceberg version.

it seems wired that the iceberg version is after the polaris version.

True. The only option is to play with the project name.

It's not a blocker to me for this PR. We could figure it out later.

We need to release a new version for this plugin every time we update the embedded Iceberg version.

I don't think we need to release a new version for this plugin every time update the embedded Iceberg version. i only need to release a new version every time we do a release, whatever iceberg version shipped during the release, it will be the version, right? However, if we we have it in the project name, then the project name has to be updated every time the iceberg version is updated, which I don't think it is a good idea, which can be very frequently.

I tried couple of ways to retain the original jar name, so far didn't find a good way yet, we can continue investigating. However, i hope in long term, we don't have to ship iceberg runtime in the bundle, and I think we should definitely have a doc/webpage somewhere to doc the compatible iceberg version regardless whether it is in the name or not.

only need to release a new version every time we do a release

+1

if we we have it in the project name, then the project name has to be updated every time the iceberg version is updated

I don't think we need to do that.

i hope in long term, we don't have to ship iceberg runtime in the bundle, and I think we should definitely have a doc/webpage somewhere to doc the compatible iceberg version regardless whether it is in the name or not.

Thanks for the investigation! Let's doc it first.

I don't think we need to do that.

Sorry, i meant every time we update the iceberg dependency version in Polaris, for example, when polaris is updated from 1.7 -> 1.8 -> 1.9, i don't mean every time when there is new iceberg version available.

Actually, after some thought, i don't think we should include the iceberg version in the package name. The polaris client is a client we provided for spark to communicate with polaris, the spark scala version is used to tell the spark scala compatibility when choosing which library to use. For the iceberg, even though we ship an iceberg runtime along with our package today, we don't really guarantee any compatibility with the iceberg client, in other words, user can not use an iceberg runtime jar with the polaris jar we provided. The iceberg client is more like a dependency we ship with, which i think makes more sense to be doc correctly, instead of add it in the package name, which actually could be confusing to users in other aspects also.

jbonofre

LICENSE and NOTICE are not correct. I can help on that.

I propose:

We merge this PR as it is and I will fix LICENSE and NOTICE in another PR
You give me access to you branch and I directly push a commit fixing LICENSE and NOTICE

flyrain · 2025-06-07T16:19:24Z

LICENSE and NOTICE are not correct. I can help on that.

I propose:

We merge this PR as it is and I will fix LICENSE and NOTICE in another PR

You give me access to you branch and I directly push a commit fixing LICENSE and NOTICE

+1 on option 1. Let's get this in first.

gh-yzou · 2025-06-07T23:58:02Z

@jbonofre @flyrain thanks a lot for the help on this! we can definitely continue look into it to see if there is a good way to retain the original jar name in the published maven jar

* main: Update dependency io.netty:netty-codec-http2 to v4.2.2.Final (apache#1819) * main: Update dependency com.google.cloud:google-cloud-storage-bom to v2.53.0 (apache#1811) * main: Pin actions/setup-java action to c5195ef (apache#1809) * main: Update dependency boto3 to v1.38.31 (apache#1803) * Downgrade open api generator to 7.11 (apache#1823) * main: Update dependency io.smallrye.config:smallrye-config-core to v3.13.1 (apache#1826) * JDBC: Use PreparedStatement (apache#1802) * Remove "unused" files in client/python (apache#1678) * main: Update dependency org.junit:junit-bom to v5.13.1 (apache#1832) * main: Update dependency boto3 to v1.38.32 (apache#1828) * main: Update dependency io.opentelemetry:opentelemetry-bom to v1.51.0 (apache#1829) * Spark: Add license and notice to spark client jar and push jar to maven (apache#1830) * Update versions in LICENSE and NOTICE (apache#1833) * Core: Fix maxCode generation for ConnectionTypeEnum (apache#1827) * main: Update dependency com.github.ben-manes.caffeine:caffeine to v3.2.1 (apache#1836) * main: Update dependency software.amazon.awssdk:bom to v2.31.59 (apache#1835) * Fix the issue where the Polaris Server exposes backend metadata in the error response body when the database is not bootstrapped. (apache#1837) * Fix LICENSE and NOTICE in Spark plugin (apache#1834) * main: Update dependency io.micrometer:micrometer-bom to v1.15.1 (apache#1842) * main: Update dependency boto3 to v1.38.33 (apache#1841) * NoSQL: bump dependencies * Last merged commit 9934fc4 --------- Co-authored-by: Mend Renovate <[email protected]> Co-authored-by: Honah (Jonas) J. <[email protected]> Co-authored-by: Prashant Singh <[email protected]> Co-authored-by: JB Onofré <[email protected]> Co-authored-by: gh-yzou <[email protected]> Co-authored-by: Pooja Nilangekar <[email protected]> Co-authored-by: Bo Wang <[email protected]>

gh-yzou added 4 commits June 6, 2025 16:51

add test

eb9b01a

add change

8489880

update changes

01dd63e

add test

a888809

gh-yzou requested review from adutra, ashvina, dennishuo, dimas-b and eric-maynard as code owners June 7, 2025 01:10

github-project-automation bot added this to Basic Kanban Board Jun 7, 2025

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Jun 7, 2025

gh-yzou requested review from HonahX, MonkeyCanCode, RussellSpitzer, ajantha-bhat, collado-mike, ebyhr, flyrain, jackye1995, jbonofre, pingtimeout, singhpk234, snazy, takidau and vvcephei as code owners June 7, 2025 01:10

add change

8a742e2

flyrain reviewed Jun 7, 2025

View reviewed changes

jbonofre reviewed Jun 7, 2025

View reviewed changes

flyrain enabled auto-merge (squash) June 7, 2025 16:19

flyrain approved these changes Jun 7, 2025

View reviewed changes

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Jun 7, 2025

jbonofre approved these changes Jun 7, 2025

View reviewed changes

flyrain merged commit be3fee8 into apache:main Jun 7, 2025
9 checks passed

github-project-automation bot moved this from Ready to merge to Done in Basic Kanban Board Jun 7, 2025

flyrain mentioned this pull request Jun 18, 2025

Prepare and distribute spark plugin runtime artifact #1616

Closed

Add license and notice to spark client jar and push jar to maven #1830

Add license and notice to spark client jar and push jar to maven #1830

Uh oh!

Conversation

gh-yzou commented Jun 7, 2025

Uh oh!

flyrain Jun 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

flyrain Jun 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gh-yzou Jun 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jbonofre left a comment

Choose a reason for hiding this comment

Uh oh!

flyrain commented Jun 7, 2025

Uh oh!

Uh oh!

gh-yzou commented Jun 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

flyrain Jun 7, 2025 •

edited

Loading

flyrain Jun 7, 2025 •

edited

Loading

gh-yzou Jun 7, 2025 •

edited

Loading