Skip to content

[SPARK-28151][SQL] Fix MsSqlServerDialect Byte/Short/Float type mappings ( DRAFT) #24969

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from
Closed

[SPARK-28151][SQL] Fix MsSqlServerDialect Byte/Short/Float type mappings ( DRAFT) #24969

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
39d22d5
Fix for ByteType support while writing and reading with JDBC based da…
shivsood Jun 21, 2019
3edc863
Fix for ShortType and FloatType when appending to SQLServerTables
shivsood Jun 25, 2019
b874527
Addressed Review comments on PR. 1. Fixed test cases related to TINYI…
shivsood Jul 3, 2019
8a7d960
fixed styling issues from dev/stylecheck run
shivsood Jul 8, 2019
6bf6e3d
Fixing issues in orignal commit. Change was not in the right place. S…
shivsood Jul 8, 2019
f2700eb
fixing styling issues
shivsood Jul 9, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 10 additions & 8 deletions ...egration-tests/src/test/scala/org/apache/spark/sql/jdbc/MsSqlServerIntegrationSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,26 +118,28 @@ class MsSqlServerIntegrationSuite extends DockerJDBCIntegrationSuite {
val row = rows(0)
val types = row.toSeq.map(x => x.getClass.toString)
assert(types.length == 12)

assert(types(0).equals("class java.lang.Boolean"))
assert(types(1).equals("class java.lang.Integer"))
assert(types(2).equals("class java.lang.Integer"))
assert(types(1).equals("class java.lang.Byte"))
assert(types(2).equals("class java.lang.Short"))
assert(types(3).equals("class java.lang.Integer"))
assert(types(4).equals("class java.lang.Long"))
assert(types(5).equals("class java.lang.Double"))
assert(types(6).equals("class java.lang.Double"))
assert(types(7).equals("class java.lang.Double"))
assert(types(6).equals("class java.lang.Float"))
assert(types(7).equals("class java.lang.Float"))
assert(types(8).equals("class java.math.BigDecimal"))
assert(types(9).equals("class java.math.BigDecimal"))
assert(types(10).equals("class java.math.BigDecimal"))
assert(types(11).equals("class java.math.BigDecimal"))
assert(row.getBoolean(0) == false)
assert(row.getInt(1) == 255)
assert(row.getInt(2) == 32767)

assert(row.getByte(1) == 255.toByte)
assert(row.getShort(2) == 32767)
assert(row.getInt(3) == 2147483647)
assert(row.getLong(4) == 9223372036854775807L)
assert(row.getDouble(5) == 1.2345678901234512E14) // float = float(53) has 15-digits precision
assert(row.getDouble(6) == 1.23456788103168E14) // float(24) has 7-digits precision
assert(row.getDouble(7) == 1.23456788103168E14) // real = float(24)
assert(row.getFloat(6) == 1.23456788103168E14) // float(24) has 7-digits precision
assert(row.getFloat(7) == 1.23456788103168E14) // real = float(24)
assert(row.getAs[BigDecimal](8).equals(new BigDecimal("123.00")))
assert(row.getAs[BigDecimal](9).equals(new BigDecimal("12345.12000")))
assert(row.getAs[BigDecimal](10).equals(new BigDecimal("922337203685477.5800")))
Expand Down
2 changes: 1 addition & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -550,7 +550,7 @@ object JdbcUtils extends Logging {

case ByteType =>
(stmt: PreparedStatement, row: Row, pos: Int) =>
stmt.setInt(pos + 1, row.getByte(pos))
stmt.setByte(pos + 1, row.getByte(pos))
Copy link
Member

@dongjoon-hyun dongjoon-hyun Jul 9, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is orthogonal to MySqlServerDialect. Can we proceed this PR without this line?

Copy link
Member

@dongjoon-hyun dongjoon-hyun Jul 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or, we can proceed this first as another PR with a proper regression test.

Copy link
Contributor Author

@shivsood shivsood Jul 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dongjoon-hyun JDBCUtils.scala fix is required for completeness of the ByteType Fix. Without this fix Integrations test in "external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/MsSqlServerIntegrationSuite.scala" will fail.

I dont think the "MsSqlServerIntegrationSuite.scala" integration tests are running in CI/CD. I was running these locally following a review suggestion. Is that's true, i can still remove this fix and submit this as part of another PR.

Copy link
Member

@dongjoon-hyun dongjoon-hyun Jul 9, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then, you need to follow the second option I gave. Please make another PR. We need to review that first before this PR.

Or, we can proceed this first as another PR with a proper regression test.

Copy link
Contributor Author

@shivsood shivsood Jul 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dongjoon-hyun Can u please elaborate on proper regression test. I have test cases added/fixed for ByteType in JDBCSuite.scala and MsSqlServerIntegrationSuite.scala. Are you expecting some other test? Can u please point me to type of test.

Another way to cleanly split this PR would be as follows.

  1. Remove ByteType fix from this PR and jdbcutils.scala fix also get removed. This PR will only have fix for FloatType and Short.
  2. 2nd PR with complete ByteType fix with more elaborate integration test.

I prefer this for better seperation of concern in each of the PRs , while still retaining the completeness. Let me know what you think.

Thanks for your patience and very helpful guidance.

Copy link
Member

@dongjoon-hyun dongjoon-hyun Jul 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I meant is like your 2nd option.

2nd PR with complete ByteType fix with more elaborate integration test.

Here, the proper regression test means a test causing failure for the other Dialects. (e.g. PostgreSQL/MySQL/...) because JdbcUtils is shared across Dialects.

Copy link
Contributor Author

@shivsood shivsood Jul 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dongjoon-hyun Would create 2 PRs then. PR1 ( this PR) i will remove ByteType and JDBCUtilis fix. Will create PR2 for ByteType and JDBCUtilis fix with full integration test. PR2 will take time as i have to investigate how to test for other dialects.

Copy link
Member

@dongjoon-hyun dongjoon-hyun Jul 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep. Please try as you want. Then, ping me later when the PR is ready.


case BooleanType =>
(stmt: PreparedStatement, row: Row, pos: Int) =>
Expand Down
9 changes: 8 additions & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/jdbc/MsSqlServerDialect.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,12 @@ private object MsSqlServerDialect extends JdbcDialect {
// String is recommend by Microsoft SQL Server for datetimeoffset types in non-MS clients
Option(StringType)
} else {
None
sqlType match {
case java.sql.Types.TINYINT => Some(ByteType)
case java.sql.Types.SMALLINT => Some(ShortType)
case java.sql.Types.REAL => Some(FloatType)
case _ => None
}
}
}

Expand All @@ -39,6 +44,8 @@ private object MsSqlServerDialect extends JdbcDialect {
case StringType => Some(JdbcType("NVARCHAR(MAX)", java.sql.Types.NVARCHAR))
case BooleanType => Some(JdbcType("BIT", java.sql.Types.BIT))
case BinaryType => Some(JdbcType("VARBINARY(MAX)", java.sql.Types.VARBINARY))
case ByteType => Some(JdbcType("TINYINT", java.sql.Types.TINYINT))
case ShortType => Some(JdbcType("SMALLINT", java.sql.Types.SMALLINT))
case _ => None
}

Expand Down
15 changes: 15 additions & 0 deletions sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -895,6 +895,21 @@ class JDBCSuite extends QueryTest
"BIT")
assert(msSqlServerDialect.getJDBCType(BinaryType).map(_.databaseTypeDefinition).get ==
"VARBINARY(MAX)")
assert(msSqlServerDialect.getJDBCType(ByteType).map(_.databaseTypeDefinition).get ==
"TINYINT")
assert(msSqlServerDialect.getJDBCType(ShortType).map(_.databaseTypeDefinition).get ==
"SMALLINT")
}

test("SPARK-28151 MsSqlServerDialect catalyst type mapping") {
val msSqlServerDialect = JdbcDialects.get("jdbc:sqlserver")
val metadata = new MetadataBuilder().putLong("scale", 1)
assert(msSqlServerDialect.getCatalystType(java.sql.Types.TINYINT, "TINYINT", 1,
metadata).get == ByteType)
assert(msSqlServerDialect.getCatalystType(java.sql.Types.SMALLINT, "SMALLINT", 1,
metadata).get == ShortType)
assert(msSqlServerDialect.getCatalystType(java.sql.Types.REAL, "REAL", 1,
metadata).get == FloatType)
}

test("table exists query by jdbc dialect") {
Expand Down