column nullable setting not retained as part of read

[darrenpricedcww]

When reading in a column set as not null this is not retained as part of the spark.read.
All columns are showing as nullable = true

Is there a way to change this behaviour to retain the null setting from the source?

[darrenpricedcww]

tableName = "dbo.MyTable"

df = spark.read
.format("com.microsoft.sqlserver.jdbc.spark")
.option("url", myJdbcUrl)
.option("accessToken", accessToken)
.option("dbTable", tableName)
.load()

df.printSchema()

[luxu1-ms]

@darrenpricedcww Can you try jdbc default connector if it has the same nullable results? replace .format("com.microsoft.sqlserver.jdbc.spark") with .format("jdbc").

[darrenpricedcww]

Hi luxu1-ms,

I can confirm I get the same results using just JDBC as advised.
For info the Databricks Synapse Driver (com.databricks.spark.sqldw) is retaining the column null setting as I would expect.

Regards,
Darren

[shivsood]

Note : connector delegates read to JDBC driver.

[darrenpricedcww]

Hi All,

Do I need to do anytyhing to request this enhancment to the JDBC driver or has that been done?

Thanks,
Darren

[shivsood]

@darrenpricedcww default spark JDBC driver enhancements can be requested on the Apache Spark forum.

[darrenpricedcww]

Issue logged on Apache Spark website, please vote if anyone else is interested in getting this fixed.
https://issues.apache.org/jira/browse/SPARK-36758

[luxu1-ms]

@darrenpricedcww please refer to this PR, column nullable is true. apache/spark#18445

[luxu1-ms]

inactive issue.