-
Notifications
You must be signed in to change notification settings - Fork 28.9k
[SPARK-36346][SQL][FOLLOWUP] Rename withAllOrcReaders to withAllNativeOrcReaders
#34733
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
withAllOrcReaders to withAllNativeOrcReaders
|
Kubernetes integration test starting |
MaxGekk
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we test Hive's implementation under withAllOrcReaders as well? If yes, can we just extend withAllOrcReaders in that case? Also I see that spark.sql.orc.enableVectorizedReader is set in Hive's tests, so, withAllNativeOrcReaders can be used in such tests which is confusing slightly.
|
Kubernetes integration test status failure |
That will be a big change and we need to move many ORC tests to sql/hive. Another idea is to remove (or at least deprecate) the hive ORC reader as the native orc reader has been used by default since Spark 2.4. |
|
Test build #145698 has finished for PR 34733 at commit
|
|
To @MaxGekk , as @cloud-fan mentioned, it's a big change and not a good approach because To @cloud-fan , yes, the deprecation is possible, but Hive ORC reader has still been used by some users. Do we officially resolve all old issues with native readers? I'm still not sure about that. |
|
Thank you all! Merged to master. |
What changes were proposed in this pull request?
This PR aims to rename
withAllOrcReaderstowithAllNativeOrcReaders.Why are the changes needed?
Apache Spark have non-native ORC reader too. In
SQLmodule, we only test native readers.Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass the CIs.