Skip to content

Commit c8c4441

Browse files
rdbluecloud-fan
authored andcommitted
[SPARK-23418][SQL] Fail DataSourceV2 reads when user schema is passed, but not supported.
## What changes were proposed in this pull request? DataSourceV2 initially allowed user-supplied schemas when a source doesn't implement `ReadSupportWithSchema`, as long as the schema was identical to the source's schema. This is confusing behavior because changes to an underlying table can cause a previously working job to fail with an exception that user-supplied schemas are not allowed. This reverts commit adcb25a0624, which was added to #20387 so that it could be removed in a separate JIRA issue and PR. ## How was this patch tested? Existing tests. Author: Ryan Blue <[email protected]> Closes #20603 from rdblue/SPARK-23418-revert-adcb25a0624.
1 parent 95e25ed commit c8c4441

File tree

1 file changed

+1
-12
lines changed

1 file changed

+1
-12
lines changed

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala

Lines changed: 1 addition & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -174,13 +174,6 @@ object DataSourceV2Relation {
174174
v2Options: DataSourceOptions,
175175
userSchema: Option[StructType]): StructType = {
176176
val reader = userSchema match {
177-
// TODO: remove this case because it is confusing for users
178-
case Some(s) if !source.isInstanceOf[ReadSupportWithSchema] =>
179-
val reader = source.asReadSupport.createReader(v2Options)
180-
if (reader.readSchema() != s) {
181-
throw new AnalysisException(s"${source.name} does not allow user-specified schemas.")
182-
}
183-
reader
184177
case Some(s) =>
185178
source.asReadSupportWithSchema.createReader(s, v2Options)
186179
case _ =>
@@ -195,11 +188,7 @@ object DataSourceV2Relation {
195188
filters: Option[Seq[Expression]] = None,
196189
userSpecifiedSchema: Option[StructType] = None): DataSourceV2Relation = {
197190
val projection = schema(source, makeV2Options(options), userSpecifiedSchema).toAttributes
198-
DataSourceV2Relation(source, options, projection, filters,
199-
// if the source does not implement ReadSupportWithSchema, then the userSpecifiedSchema must
200-
// be equal to the reader's schema. the schema method enforces this. because the user schema
201-
// and the reader's schema are identical, drop the user schema.
202-
if (source.isInstanceOf[ReadSupportWithSchema]) userSpecifiedSchema else None)
191+
DataSourceV2Relation(source, options, projection, filters, userSpecifiedSchema)
203192
}
204193

205194
private def pushRequiredColumns(reader: DataSourceReader, struct: StructType): Unit = {

0 commit comments

Comments
 (0)