Use official spark image #2899
Merged
Use official spark image #2899
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
singhpk234
reviewed
Oct 25, 2025
Contributor
singhpk234
left a comment
singhpk234
left a comment
There was a problem hiding this comment.
LGTM, can you please double confirm you successfully ran the getting started post updating these ?
Contributor
Author
|
Yes
|
dimas-b
approved these changes
Oct 28, 2025
github-project-automation
bot
moved this from PRs In Progress
to Ready to merge
in Basic Kanban Board
Contributor
Author
@singhpk234 let me know if there are additional concerns before I merge this one. |
singhpk234
approved these changes
Oct 28, 2025
Contributor
singhpk234
left a comment
singhpk234
left a comment
There was a problem hiding this comment.
LGTM, thanks @MonkeyCanCode !
github-project-automation
bot
moved this from Ready to merge
to Done
in Basic Kanban Board
Contributor
Author
|
Thanks for the review.
|
snazy
added a commit
to snazy/polaris
that referenced
this pull request
Nov 20, 2025
* Fix Jandex Maven coordinates (apache#2888) The entry `jandex = { module = "io.smallrye.jandex:jandex", version ="3.5.0" }` is wrong (coordinates are `io.smallrye:jandex`), and Jandex is defined elsewhere as `smallrye-jandex`. Interestingly, these (broken) coordinates seem to cause the consistent re-creation of the Quarkus 3.29.0 PR (the cause is a mystery). * Update plugin com.gradle.develocity to v4.2.2 (apache#2597) * Site: Hugo docs relative links (apache#2892) * Update dependency software.amazon.awssdk:bom to v2.36.2 (apache#2901) * Update GitHub Artifact Actions (apache#2895) * Formatting: apply Spotless to :polaris-distribution (apache#2900) * Build: Capture jcstress output in a log file (apache#2890) The jcstress output is pretty verbose and prints a lot to the console. This change captures the output in a log file. In case of a test failure, the output is logged to the console, but only in case of a failure. * Prep: Site for 1.2 release (apache#2877) * Adding 1.2.0 as one of active releases (apache#2916) Co-authored-by: Yufei Gu <yufei.apache.org> * Use official spark image (apache#2899) * Update dependency ipykernel to v7.1.0 (apache#2918) * Added missing features doc (apache#2898) * Added missing features doc * Added missing features doc * Site: Add a blog for StarRocks and Apache Polaris Integration (apache#2851) * NoSQL: Node IDs - API, SPI + general implementation (apache#2728) * NoSQL: Node IDs - API, SPI + general implementation This PR provides a mechanism to assign a Polaris-cluster-wide unique node-ID to each Polaris instance, which is then used when generating Polaris-cluster-wide unique Snowflake-IDs. The change is fundamental for the NoSQL work, but also demanded for the existing relational JDBC persistence. Does not include any persistence specific implementation. * NoSQL: Fail node-management-impl init after timeout Also move the expensive part to a `@PostConstruct` to not block CDI entirely from initializing. * Update dependency io.prometheus:prometheus-metrics-exporter-servlet-jakarta to v1.4.2 (apache#2929) * Build-logic: `GitInfo` refactor (apache#2908) Allows use of `GitInfo` for other use cases than just Jar manifest attributes. SBOM generation will be another use case for Git information. * Memoize ASF project information (apache#2909) Information included in Polaris publications pulls some information about the project from ASF project metadata sources (Whimsey). This information is currently only used when generating Maven poms, but will also be needed in SBOMs. This change adds a new, memoized `AsfProject` information object, which holds the project infromation from Whimsey. * Build: Simplify signing + fix execution in polaris-distribution (apache#2906) This change simplifies generation of non-publication artifacts by adding a function taking the task which outputs shall be signed. That function takes care of setting up the correct task dependencies and task execution. Also fixes an issue that signing does not always happen when running `./gradlew :polaris-distribution:assemble`, because the task dependency graph for the archive tasks and the corresponding signing tasks isn't properly set up. * Proposed Test Fix (apache#2936) Co-authored-by: Travis Michael Bowen <[email protected]> * Update docker.io/prom/prometheus Docker tag to v3.7.3 (apache#2944) * Update Quarkus Platform and Group to v3.29.0 (apache#2934) * Update Gradle to v9.2.0 (apache#2938) Co-authored-by: Robert Stupp <[email protected]> * Update dependency openapi-generator-cli to v7.17.0 (apache#2940) * Implement OpaPolarisAuthorizer (apache#2680) * Update dependency com.github.ben-manes.caffeine:caffeine to v3.2.3 (apache#2923) * Prefer PolarisPrincipal.getRoles in Resolver (apache#2925) it should be sufficient to rely on `SecurityContext.getUserPrincipal` alone, we dont need to call `isUserInRole` explicitly. note due to the `ResolverTest` testing with non-existent roles we have to add null-filtering to the `Resolver`. * Move `nodeids` to `nosql` package parent (apache#2931) Following up on apache#2728 this change moves "nodeids" code to the `org.apache.polaris.persistence.nosql.nodeids` package. * Update actions/stale digest to 39bea7d (apache#2950) * Update dependency org.junit:junit-bom to v5.14.1 (apache#2951) * docs(2843): Add documentation around Polaris-Tools (apache#2946) * Add documentation around Polaris-Tools * Related to apache#2843 * Add getting started with Apache Ozone (apache#2853) * Add getting started with Apache Ozone Use Apache Ozone as an example S3 impl. that does not have STS. * fix typo in MinIO readme * Update dependency com.azure:azure-sdk-bom to v1.3.0 (apache#2754) * docs: add feature configuration section to Hive federation guide (apache#2952) Add documentation for required feature flags when enabling Hive Metastore federation. Users must configure three properties in `application.properties` before Hive federation will work: - `SUPPORTED_CATALOG_CONNECTION_TYPES` - `SUPPORTED_EXTERNAL_CATALOG_AUTHENTICATION_TYPES` - `ENABLE_CATALOG_FEDERATION` Inspired from [this](https://apache-polaris.slack.com/archives/C084XDM50CB/p1761851426511259) Slack thread. Co-authored-by: Prathyush Shankar <[email protected]> * Change getting-start docker file to use official spark image from outdated jupyter image (apache#2943) * Use official spark image * Use official spark image * Use official spark image * Use official spark image * Use official spark image * Use Iterable for realms in BootstrapCommand (apache#2956) * Simplify digest generation (apache#2907) Similarly to the change to simplify artifact signing, this change simplifies digest generation by introducing a function to digest the output files of any task. That function takes care of setting up the correct task dependencies and task execution. Also removes an unnecessary double buffering during digest generation. * Build: `GitInfo` function to build a raw github content URL (apache#2910) * NoSQL: nodeids renames * NoSQL: Update test for Caffeine 3.2.3 The read of `Eviction` properties is "just" a volatile read since Caffeine 3.2.3 and trigger cleanups asynchronously. Before 3.2.3, cleanups happened synchronously. This change breaks the initially present assertions of this test, but not the functionality of the production code. See ben-manes/caffeine#1897 * Last merged commit cec41c4 --------- Co-authored-by: Mend Renovate <[email protected]> Co-authored-by: olsoloviov <[email protected]> Co-authored-by: Prashant Singh <[email protected]> Co-authored-by: Yufei Gu <[email protected]> Co-authored-by: Yong Zheng <[email protected]> Co-authored-by: Youngwb <[email protected]> Co-authored-by: Travis Bowen <[email protected]> Co-authored-by: Travis Michael Bowen <[email protected]> Co-authored-by: Sung Yun <[email protected]> Co-authored-by: Christopher Lambert <[email protected]> Co-authored-by: Dmitri Bourlatchkov <[email protected]> Co-authored-by: Adam Christian <105929021+adam-christian-software@users.noreply.github.com> Co-authored-by: carc-prathyush-shankar <[email protected]> Co-authored-by: Prathyush Shankar <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently we are using
jupyter/all-spark-notebook:spark-3.5.0which is really old and no sign of support newer version from juputer. With this old image, we are then downloading newer spark binary which is a bit problematic if we want to support spark 4 (as there is no base image for that and downloading spark4 on top of this old base image may not work). To have more control over this, I suggest we should use official spark image instead.