Skip to content

[SPARK-22383][SQL] Generate code to directly get value of primitive type array from ColumnVector for table cache #19601

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 12 commits into from
Closed

[SPARK-22383][SQL] Generate code to directly get value of primitive type array from ColumnVector for table cache #19601

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
2db2bb1
initial commit
kiszk Oct 29, 2017
642110e
add UnsafeColumnVector to support array for table cache
kiszk Oct 30, 2017
ef1fb3d
fix faiulres in CacheTableSuite, HiveCompatibilitySuite, and HiveQuer…
kiszk Oct 30, 2017
c5882ce
remove wrong assert to fix failures
kiszk Oct 30, 2017
f6b963b
avoid to override ColumnVector.getArray()
kiszk Nov 5, 2017
c81528c
fix test failures
kiszk Nov 5, 2017
e8fcd34
Remove ColumnVector.putUnsafeData()
kiszk Nov 10, 2017
886833e
rebase with master
kiszk Nov 15, 2017
71df28b
address review comment
kiszk Nov 16, 2017
b8cef29
fix compilation error
kiszk Nov 16, 2017
274b777
fix failures of rebase
kiszk Nov 21, 2017
20d2ba2
rebase with master
kiszk Nov 28, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 0 additions & 22 deletions sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVectorUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,28 +93,6 @@ public static void populate(WritableColumnVector col, InternalRow row, int field
}
}

/**
* Returns the array data as the java primitive array.
* For example, an array of IntegerType will return an int[].
* Throws exceptions for unhandled schemas.
*/
public static Object toPrimitiveJavaArray(ColumnarArray array) {
Copy link
Member

@viirya viirya Nov 16, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious that why we have this method? Looks it is just used in test.

Copy link
Member Author

@kiszk kiszk Nov 16, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right. This method was used only for test. I removed this by replacing this with another method.

DataType dt = array.data.dataType();
if (dt instanceof IntegerType) {
int[] result = new int[array.length];
ColumnVector data = array.data;
for (int i = 0; i < result.length; i++) {
if (data.isNullAt(array.offset + i)) {
throw new RuntimeException("Cannot handle NULL values.");
}
result[i] = data.getInt(array.offset + i);
}
return result;
} else {
throw new UnsupportedOperationException();
}
}

private static void appendValue(WritableColumnVector dst, DataType t, Object o) {
if (o == null) {
if (t instanceof CalendarIntervalType) {
Expand Down
Loading