Skip to content

Commit d461552

Browse files
committed
Prevent failure on corrupt JSON records
Return failed record when a record cannot be parsed. Allows parsing of files containing corrupt records of any form.
1 parent 2db4662 commit d461552

File tree

1 file changed

+1
-5
lines changed

1 file changed

+1
-5
lines changed

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JacksonParser.scala

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -266,11 +266,7 @@ object JacksonParser {
266266
} else {
267267
array.toArray[InternalRow](schema)
268268
}
269-
case _ =>
270-
sys.error(
271-
s"Failed to parse record $record. Please make sure that each line of " +
272-
"the file (or each string in the RDD) is a valid JSON object or " +
273-
"an array of JSON objects.")
269+
case _ => failedRecord(record)
274270
}
275271
}
276272
} catch {

0 commit comments

Comments
 (0)