[SPARK-37492][SQL] Optimize Orc test code with withAllNativeOrcReaders #34723
Conversation
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
|
Test build #145669 has finished for PR 34723 at commit
|
There was a problem hiding this comment.
Sorry, but hold on please, @beliefer .
Although this PR itself is correct, the previous commit seems to misleading.
withAllOrcReaders is not correct in Apache Spark context because this only tests native OrcReaders. We have hive ORC reader too.
|
Let's proceed this after #34733 . |
I see. |
|
BTW, is this PR include all instances like this, @beliefer ? Could you check other places too? |
beliefer
changed the title
beliefer
force-pushed
the
optimize-orc
branch
from
8c464a0 to
96bb4f5
Compare
Yes. I checked every places. |
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
| SQLConf.ORC_VECTORIZED_READER_ENABLED.key -> vectorized) { | ||
| withSQLConf( | ||
| SQLConf.ORC_IMPLEMENTATION.key -> orcImpl) { | ||
| withAllNativeOrcReaders { |
There was a problem hiding this comment.
Ur, @beliefer . This is technically correct, but logically misleading again. When orcImpl=hive, we are not testing nativeOrcReaders. The original code is more natural.
There was a problem hiding this comment.
I'll merge this PR without this part.
There was a problem hiding this comment.
@dongjoon-hyun Thank you for the reminder.
|
Test build #145738 has finished for PR 34723 at commit
|
4 participants
What changes were proposed in this pull request?
This PR used to optimize some Orc test code in OrcSourceSuite.scala
Why are the changes needed?
Improve code.
Does this PR introduce any user-facing change?
'No'. Just improve code for UT.
How was this patch tested?
Jenkins test.