[SPARK-19611][SQL] Preserve metastore field order when merging inferred schema #17249
Conversation
…ed schema The ```HiveMetastoreCatalog.mergeWithMetastoreSchema()``` method added in apache#16944 may not preserve the same field order as the metastore schema in some cases, which can cause queries to fail. This change ensures that the metastore field order is preserved. A test for ensuring that metastore order is preserved was added to ```HiveSchemaInferenceSuite.``` The particular failure usecase from apache#16944 was tested manually as well.
|
Pinging @cloud-fan and @dongjoon-hyun. Apologies for not catching this the first time around. |
|
@dongjoon-hyun can you verify if it fixes your problem? |
|
I've verified that it does using the same procedure but I'll let @dongjoon-hyun confirm as well. |
|
Sure, I'll test locally, too. |
| StructType(metastoreFields.map { case(name, field) => | ||
| field.copy(name = inferredFields(name).name) | ||
| }.toSeq) | ||
| StructType(metastoreSchema.map(f => f.copy(name = inferredFields(f.name).name))) |
There was a problem hiding this comment.
I see. Now metastoreSchema is used instead of metastoreFields.
There was a problem hiding this comment.
This should ensure the proper ordering is used. Iterating over the metastoreFields map isn't guaranteed to maintain the original ordering.
There was a problem hiding this comment.
Yep. I'm still building this PR locally, but it looks good logically. :) I'll post my result soon.
|
Great. I also verified the patch. scala> sql("SELECT a, b FROM t1").show
+---+---+
| a| b|
+---+---+
|100|200|
+---+---+
scala> sql("SELECT * FROM t1").show
+---+---+-----+---+----+
| a| b|dummy|day|hour|
+---+---+-----+---+----+
|100|200| null| 1| 01|
+---+---+-----+---+----+Thank you so much, @budde and @cloud-fan . |
|
Test build #74337 has finished for PR 17249 at commit
|
|
thanks, merging to master! |
4 participants
What changes were proposed in this pull request?
The
HiveMetastoreCatalog.mergeWithMetastoreSchema()method added in #16944 maynot preserve the same field order as the metastore schema in some cases, which can cause
queries to fail. This change ensures that the metastore field order is preserved.
How was this patch tested?
A test for ensuring that metastore order is preserved was added to
HiveSchemaInferenceSuite.The particular failure usecase from #16944 was tested manually as well.