Update to Iceberg 1.2.0 #72
Conversation
nastra
force-pushed
the
update-to-iceberg-1.2.0
branch
from
ebf2f01 to
0f4400e
Compare
nastra
force-pushed
the
update-to-iceberg-1.2.0
branch
from
0f4400e to
0d1f390
Compare
nastra
force-pushed
the
update-to-iceberg-1.2.0
branch
from
0d1f390 to
90caeb7
Compare
nastra
force-pushed
the
update-to-iceberg-1.2.0
branch
from
90caeb7 to
b78ff5d
Compare
| environment: | ||
| - MINIO_ROOT_USER=admin | ||
| - MINIO_ROOT_PASSWORD=password | ||
| - MINIO_DOMAIN=minio |
There was a problem hiding this comment.
this is required for Minio to enable virtual-host-style access to the bucket (which is the default in the new AWS SDK version
There was a problem hiding this comment.
all of the networking changes are basically only to make virtual-host-style access to the bucket properly work
There was a problem hiding this comment.
the alternative would be to configure path-style access (which is deprecated) similar to #70
| networks: | ||
| iceberg_net: | ||
| aliases: | ||
| - warehouse.minio |
There was a problem hiding this comment.
this alias is required so that clients can access the bucket
|
|
||
| # Download URL connection client required for S3FileIO | ||
| RUN curl https://repo1.maven.org/maven2/software/amazon/awssdk/url-connection-client/2.17.257/url-connection-client-2.17.257.jar -Lo /opt/spark/jars/url-connection-client-2.17.257.jar | ||
| RUN curl https://repo1.maven.org/maven2/software/amazon/awssdk/url-connection-client/${AWSSDK_VERSION}/url-connection-client-${AWSSDK_VERSION}.jar -Lo /opt/spark/jars/url-connection-client-${AWSSDK_VERSION}.jar |
There was a problem hiding this comment.
Did the change to switch over to the apache client make it into 1.2? Just wondering if we still need this.
There was a problem hiding this comment.
apache/iceberg@053028c didn't make it into 1.2 unfortunately, so we still need this
This also requires the changes from databricks/iceberg-rest-image#7