-
Notifications
You must be signed in to change notification settings - Fork 0
Vision
This is a very mature library already, and not much evolvement is needed.
Right now the library is build against Spark 2.4.x. The next major version of Spark 3.x has been out for some time already. Version 4.x is on the horizon. It would be great to have the library updated to be sure to support Spark 3.x perhaps 4.x as well.
The ErrorHandling capability within spark-commons is rather specific, isolated and generic enough to be extracted into a separate library. That would make it easier to integrate into other projects and evolve independently.
The idea is to create an extension library for spark-functions that would contain additional functions that are not part of the core library. These functions would be more specialized, perhaps targeting specific use cases or data types. Some existing functions from spark-commons, namely from the za.co.absa.spark.commons.sql.functions and za.co.absa.spark.commons.implicits.ColumnImplicits, could be moved into this new library.