Integration test broken

[kevinjqliu]

Apache Iceberg version

main (development)

Please describe the bug 🐞

As of current commit 3/2/2024

Error log:

>           assert spark_partition_for_justification == expected_partition_record
E           assert Record[timestamp_field_hour=464611] == Record[timestamp_field_hour=464603]
E             Full diff:
E             - Record[timestamp_field_hour=464603]
E             ?                                 ^^
E             + Record[timestamp_field_hour=464611]
E             ?                                 ^^

tests/integration/test_partitioning_key.py:771: AssertionError

Reproduce:

git checkout main
git pull
make test-integration

Possibly related to #453

[HonahX]

I also hit this today. I think this is because the spark session uses our computer's local timezone when inserting timestamp values into that table. We may config the Spark Session to always use UTC timezone in conftest.py:

spark = (
        SparkSession.builder.appName("PyIceberg integration test")
        .config("spark.sql.session.timeZone", "UTC")
        .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
        .config("spark.sql.catalog.integration", "org.apache.iceberg.spark.SparkCatalog")
        .config("spark.sql.catalog.integration.catalog-impl", "org.apache.iceberg.rest.RESTCatalog")
...

The github CI should pass without this change because the Github Action VM uses UTC timezone by default.

[kevinjqliu]

makes sense, I'm in PT (UTC-8) which matches the difference between 464603 and 464611.

Just ran make test-integration with the

        .config("spark.sql.session.timeZone", "UTC")

and everything passed

[kevinjqliu]

Do you already have a PR out for the change? Or should I push one up?

[HonahX]

Could you please open one up? I can quickly merge that 😄 . Thanks!

[kevinjqliu]

@HonahX opened #494