Error reading nested tags in Open StreetMap XML-Files

[martin-weber]

Hello,

I try to parse Open Streetmap XML-Files using the latest spark-xml and pyspark (scala 2.11.8., spark 2.0.1 with jupyter notebooks as well with databricks community).

The xml data can be fetched using:
wget -O winterthur_map.osm "http://overpass-api.de/api/map?bbox=8.6432,47.4579,8.8210,47.5442"

Extracting the way-tags and realtion-tags works fine, but extracting the nodes throws the following error:

---------------------------------------------------------------------------
Py4JJavaError                             Traceback (most recent call last)
<ipython-input-15-fcee35e58e1c> in <module>()
      6 
      7 df = sqlContext.read.format('com.databricks.spark.xml')                 .options(rowTag='node', mode='FAILFAST')                 .schema(my_schema)                 .load(MAP_DATA_SET_WINTERTHUR)
----> 8 df.show(30)

/usr/local/Cellar/apache-spark/2.0.1/libexec/python/pyspark/sql/dataframe.py in show(self, n, truncate)
    285         +---+-----+
    286         """
--> 287         print(self._jdf.showString(n, truncate))
    288 
    289     def __repr__(self):

/usr/local/Cellar/apache-spark/2.0.1/libexec/python/lib/py4j-0.10.3-src.zip/py4j/java_gateway.py in __call__(self, *args)
   1131         answer = self.gateway_client.send_command(command)
   1132         return_value = get_return_value(
-> 1133             answer, self.gateway_client, self.target_id, self.name)
   1134 
   1135         for temp_arg in temp_args:

/usr/local/Cellar/apache-spark/2.0.1/libexec/python/pyspark/sql/utils.py in deco(*a, **kw)
     61     def deco(*a, **kw):
     62         try:
---> 63             return f(*a, **kw)
     64         except py4j.protocol.Py4JJavaError as e:
     65             s = e.java_exception.toString()

/usr/local/Cellar/apache-spark/2.0.1/libexec/python/lib/py4j-0.10.3-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    317                 raise Py4JJavaError(
    318                     "An error occurred while calling {0}{1}{2}.\n".
--> 319                     format(target_id, ".", name), value)
    320             else:
    321                 raise Py4JError(

Py4JJavaError: An error occurred while calling o175.showString.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 13.0 failed 1 times, most recent failure: Lost task 0.0 in stage 13.0 (TID 23, localhost): java.lang.RuntimeException: Malformed line in FAILFAST mode: 

<node id="15998875" lat="47.5296917" lon="8.8199895" version="10" timestamp="2014-05-17T02:31:10Z" changeset="22381881" uid="178186" user="mdk"/>  <node id="15998878" lat="47.5302420" lon="8.8158363" version="13" timestamp="2014-06-10T19:13:01Z" changeset="22856880" uid="266672" user="tf66"/>  <node id="15998881" lat="47.5308430" lon="8.8112575" version="15" timestamp="2014-06-10T19:13:01Z" changeset="22856880" uid="266672" user="tf66"/>  <node id="15998885" lat="47.5291517" lon="8.8246617" version="10" timestamp="2008-11-10T22:47:33Z" changeset="730754" uid="44456" user="twalter"/>  <node id="15998916" lat="47.5315221" lon="8.8060980" version="5" timestamp="2014-06-10T19:13:01Z" changeset="22856880" uid="266672" user="tf66"/>  <node id="15998949" lat="47.5288157" lon="8.8318368" version="10" timestamp="2014-05-17T02:31:10Z" changeset="22381881" uid="178186" user="mdk"/>  <node id="15998977" lat="47.4577472" lon="8.7085158" version="8" timestamp="2014-09-23T23:31:01Z" changeset="25635449" uid="7412" user="mjulius"/>  <node id="15998991" lat="47.5288778" lon="8.8282872" version="11" timestamp="2013-05-12T17:13:27Z" changeset="16099021" uid="390170" user="chewiebug"/>  <node id="15999625" lat="47.5291589" lon="8.8375096" version="11" timestamp="2014-05-17T02:31:10Z" changeset="22381881" uid="178186" user="mdk"/>  <node id="15999629" lat="47.5265460" lon="8.8810894" version="4" timestamp="2014-04-29T16:17:32Z" changeset="22026107" uid="1881716" user="Leonardo Hauschild"/>  <node id="15999636" lat="47.5250933" lon="8.8860317" version="3" timestamp="2008-09-03T11:53:55Z" changeset="512858" uid="7260" user="studerap">    <tag k="created_by" v="JOSM"/>  </node>

    at com.databricks.spark.xml.parsers.StaxXmlParser$.com$databricks$spark$xml$parsers$StaxXmlParser$$failedRecord$1(StaxXmlParser.scala:47)
    at com.databricks.spark.xml.parsers.StaxXmlParser$$anonfun$parse$1$$anonfun$apply$4.apply(StaxXmlParser.scala:92)
    at com.databricks.spark.xml.parsers.StaxXmlParser$$anonfun$parse$1$$anonfun$apply$4.apply(StaxXmlParser.scala:73)
    at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
    at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:370)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$4.apply(SparkPlan.scala:246)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$4.apply(SparkPlan.scala:240)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:803)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:803)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
    at org.apache.spark.scheduler.Task.run(Task.scala:86)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1454)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1442)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1441)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1441)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
    at scala.Option.foreach(Option.scala:257)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:811)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1667)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1622)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1611)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:632)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1890)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1903)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1916)
    at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:347)
    at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:39)
    at org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1.apply(Dataset.scala:2193)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:57)
    at org.apache.spark.sql.Dataset.withNewExecutionId(Dataset.scala:2546)
    at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$execute$1(Dataset.scala:2192)
    at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collect(Dataset.scala:2199)
    at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:1935)
    at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:1934)
    at org.apache.spark.sql.Dataset.withTypedCallback(Dataset.scala:2576)
    at org.apache.spark.sql.Dataset.head(Dataset.scala:1934)
    at org.apache.spark.sql.Dataset.take(Dataset.scala:2149)
    at org.apache.spark.sql.Dataset.showString(Dataset.scala:239)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:280)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:214)
    at java.lang.Thread.run(Thread.java:745)

This is the first part of the OSM-XML-File mentioned in the error message:

<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="Overpass API">
<note>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</note>
<meta osm_base="2016-10-16T09:14:02Z"/>

  <bounds minlat="47.4579" minlon="8.6432" maxlat="47.5442" maxlon="8.821"/>

  <node id="15998875" lat="47.5296917" lon="8.8199895" version="10" timestamp="2014-05-17T02:31:10Z" changeset="22381881" uid="178186" user="mdk"/>
  <node id="15998878" lat="47.5302420" lon="8.8158363" version="13" timestamp="2014-06-10T19:13:01Z" changeset="22856880" uid="266672" user="tf66"/>
  <node id="15998881" lat="47.5308430" lon="8.8112575" version="15" timestamp="2014-06-10T19:13:01Z" changeset="22856880" uid="266672" user="tf66"/>
  <node id="15998885" lat="47.5291517" lon="8.8246617" version="10" timestamp="2008-11-10T22:47:33Z" changeset="730754" uid="44456" user="twalter"/>
  <node id="15998916" lat="47.5315221" lon="8.8060980" version="5" timestamp="2014-06-10T19:13:01Z" changeset="22856880" uid="266672" user="tf66"/>
  <node id="15998949" lat="47.5288157" lon="8.8318368" version="10" timestamp="2014-05-17T02:31:10Z" changeset="22381881" uid="178186" user="mdk"/>
  <node id="15998977" lat="47.4577472" lon="8.7085158" version="8" timestamp="2014-09-23T23:31:01Z" changeset="25635449" uid="7412" user="mjulius"/>
  <node id="15998991" lat="47.5288778" lon="8.8282872" version="11" timestamp="2013-05-12T17:13:27Z" changeset="16099021" uid="390170" user="chewiebug"/>
  <node id="15999625" lat="47.5291589" lon="8.8375096" version="11" timestamp="2014-05-17T02:31:10Z" changeset="22381881" uid="178186" user="mdk"/>
  <node id="15999629" lat="47.5265460" lon="8.8810894" version="4" timestamp="2014-04-29T16:17:32Z" changeset="22026107" uid="1881716" user="Leonardo Hauschild"/>
  <node id="15999636" lat="47.5250933" lon="8.8860317" version="3" timestamp="2008-09-03T11:53:55Z" changeset="512858" uid="7260" user="studerap">
    <tag k="created_by" v="JOSM"/>
  </node>

The following lines are then following, just in case this might help

  <node id="15999644" lat="47.5278966" lon="8.8775565" version="4" timestamp="2014-04-29T16:17:32Z" changeset="22026107" uid="1881716" user="Leonardo Hauschild"/>
  <node id="15999647" lat="47.5302711" lon="8.8651121" version="4" timestamp="2014-06-09T08:39:48Z" changeset="22825155" uid="266672" user="tf66"/>
  <node id="15999651" lat="47.5296471" lon="8.8414284" version="10" timestamp="2014-06-09T08:39:48Z" changeset="22825155" uid="266672" user="tf66"/>

I used the following Python code and this produces the stack-trace above:

from pyspark.sql.types import *
from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)
MAP_DATA_SET_WINTERTHUR = 'winterthur_map.osm'

tag_type = StructType([ \
        StructField("_VALUE", StringType(), True), \
        StructField("_k", StringType(), True), \
        StructField("_v", StringType(), True)])

my_schema = StructType([ \
        StructField("_changeset", LongType() ,True), \
        StructField("_corrupt_record",StringType() ,True), \
        StructField("_id", LongType(), True), \
        StructField("_lat", DoubleType(), True), \
        StructField("_lon", DoubleType(), True), \
        StructField("_timestamp", StringType(), True), \
        StructField("_uid", LongType(), True), \
        StructField("_user", StringType(), True), \
        StructField("_version", LongType(), True), \
        StructField("_VALUE", StringType(), True), \
        StructField("tag", ArrayType(tag_type, True), True)])

df = sqlContext.read.format('com.databricks.spark.xml') \
                .options(rowTag='node', mode='FAILFAST') \
                .schema(my_schema) \
                .load(MAP_DATA_SET_WINTERTHUR)
df.show(30)

Just printing the schema works good.

df_nodes = sqlContext.read.format('com.databricks.spark.xml') \
                .options(rowTag='node', mode='FAILFAST') \
                .load(MAP_DATA_SET_WINTERTHUR)
df_nodes.printSchema()

This produces the following output:

root
 |-- _changeset: long (nullable = true)
 |-- _id: long (nullable = true)
 |-- _lat: double (nullable = true)
 |-- _lon: double (nullable = true)
 |-- _timestamp: string (nullable = true)
 |-- _uid: long (nullable = true)
 |-- _user: string (nullable = true)
 |-- _version: long (nullable = true)
 |-- tag: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- _VALUE: string (nullable = true)
 |    |    |-- _k: string (nullable = true)
 |    |    |-- _v: string (nullable = true)

But using this derived schema still produces the same error as above:

df_nodes_2 = sqlContext.read.format('com.databricks.spark.xml') \
                .options(rowTag='node', mode='FAILFAST') \
                .load(MAP_DATA_SET_WINTERTHUR, schema = df_nodes.schema)
df_nodes_2.show()

Is this a bug or do I miss something. After starring at the file and even reformatting the node and tag elements to one line, I still can not find the reason for this error.

Any help would be great!

[HyukjinKwon]

I guess it is about schema not matching. Will try to reproduce this within this weekend and be back.

[martin-weber]

Thanks. The second last script in the question returns the following inferred schema, which is what I used as starting point and as it failed added the _VALUE to the main "node" StructType, hoping that it will solve the problem... which is not the case.

The Script:

print(df_nodes.schema)

The Output:

StructType(List(
StructField(_changeset,LongType,true),
StructField(_id,LongType,true),
StructField(_lat,DoubleType,true),
StructField(_lon,DoubleType,true),
StructField(_timestamp,StringType,true),
StructField(_uid,LongType,true),
StructField(_user,StringType,true),
StructField(_version,LongType,true),
StructField(tag,ArrayType(
StructType(
List(StructField(_VALUE,StringType,true),
StructField(_k,StringType,true),
StructField(_v,StringType,true))),
true),
true)))

[martin-weber]

I think the problem is the XmlInputFormat which does not correctly detect elements which exist as empty element AND elements with subelements:

Empty Elements:

<node id="123" />

With Subelements:

<node>
  <flag />
</node>

I tried this with

xml = sc.newAPIHadoopFile(
    MAP_DATA_SET_WINTERTHUR,
    'com.databricks.spark.xml.XmlInputFormat',
    'org.apache.hadoop.io.LongWritable',
    'org.apache.hadoop.io.Text',
    conf={
        'xmlinput.start':'<node>',
        'xmlinput.end':'</node>',
        'xmlinput.encoding': 'utf-8'})

from pyspark.sql import SparkSession

spark = SparkSession \
    .builder \
    .appName("GPS Analyse") \
    .getOrCreate()

parts = xml.collect()
parts

which produces (please scroll right to see the full lines):

[(1882,
  '<node id="15998875" version="10" timestamp="2014-05-17T02:31:10Z" uid="178186" user="mdk" changeset="22381881" lat="47.5296917" lon="8.8199895"/>\n  <node id="15998878" version="13" timestamp="2014-06-10T19:13:01Z" uid="266672" user="tf66" changeset="22856880" lat="47.530242" lon="8.8158363"/>\n  <node id="15998881" version="15" timestamp="2014-06-10T19:13:01Z" uid="266672" user="tf66" changeset="22856880" lat="47.530843" lon="8.8112575"/>\n  <node id="15998885" version="10" timestamp="2008-11-10T22:47:33Z" uid="44456" user="twalter" changeset="730754" lat="47.5291517" lon="8.8246617"/>\n  <node id="15998916" version="5" timestamp="2014-06-10T19:13:01Z" uid="266672" user="tf66" changeset="22856880" lat="47.5315221" lon="8.806098"/>\n  <node id="15998949" version="10" timestamp="2014-05-17T02:31:10Z" uid="178186" user="mdk" changeset="22381881" lat="47.5288157" lon="8.8318368"/>\n  <node id="15998977" version="8" timestamp="2014-09-23T23:31:01Z" uid="7412" user="mjulius" changeset="25635449" lat="47.4577472" lon="8.7085158"/>\n  <node id="15998991" version="11" timestamp="2013-05-12T17:13:27Z" uid="390170" user="chewiebug" changeset="16099021" lat="47.5288778" lon="8.8282872"/>\n  <node id="15999625" version="11" timestamp="2014-05-17T02:31:10Z" uid="178186" user="mdk" changeset="22381881" lat="47.5291589" lon="8.8375096"/>\n  <node id="15999629" version="4" timestamp="2014-04-29T16:17:32Z" uid="1881716" user="Leonardo Hauschild" changeset="22026107" lat="47.526546" lon="8.8810894"/>\n  <node id="15999636" version="3" timestamp="2008-09-03T11:53:55Z" uid="7260" user="studerap" changeset="512858" lat="47.5250933" lon="8.8860317">\n    <tag k="created_by" v="JOSM"/>\n  </node>'),
 (9278,
  '<node id="15999644" version="4" timestamp="2014-04-29T16:17:32Z" uid="1881716" user="Leonardo Hauschild" changeset="22026107" lat="47.5278966" lon="8.8775565"/>\n  <node id="15999647" version="4" timestamp="2014-06-09T08:39:48Z" uid="266672" user="tf66" changeset="22825155" lat="47.5302711" lon="8.8651121"/>\n  <node id="15999651" version="10" timestamp="2014-06-09T08:39:48Z" uid="266672" user="tf66" changeset="22825155" lat="47.5296471" lon="8.8414284"/>\n  <node id="15999652" version="5" timestamp="2014-06-09T08:39:48Z" uid="266672" user="tf66" changeset="22825155" lat="47.5230248" lon="8.8924783"/>\n  <node id="15999660" version="11" timestamp="2014-06-09T08:39:48Z" uid="266672" user="tf66" changeset="22825155" lat="47.5301817" lon="8.8462336"/>\n  <node id="15999663" version="11" timestamp="2014-06-09T08:39:48Z" uid="266672" user="tf66" changeset="22825155" lat="47.5303233" lon="8.8492799"/>\n  <node id="15999672" version="4" timestamp="2014-06-09T08:39:48Z" uid="266672" user="tf66" changeset="22825155" lat="47.5300882" lon="8.8692992"/>\n  <node id="15999682" version="5" timestamp="2014-06-09T08:39:48Z" uid="266672" user="tf66" changeset="22825155" lat="47.5181329" lon="8.8985811"/>\n  <node id="15999699" version="5" timestamp="2014-06-09T08:39:48Z" uid="266672" user="tf66" changeset="22825155" lat="47.5205209" lon="8.8959793"/>\n  <node id="15999727" version="10" timestamp="2014-06-09T08:39:48Z" uid="266672" user="tf66" changeset="22825155" lat="47.5301456" lon="8.8572301"/>\n  <node id="15999757" version="6" timestamp="2014-06-09T08:39:49Z" uid="266672" user="tf66" changeset="22825155" lat="47.516078" lon="8.9016568"/>\n  <node id="17284583" version="7" timestamp="2014-09-23T23:31:01Z" uid="7412" user="mjulius" changeset="25635449" lat="47.4582709" lon="8.7085695"/>\n  <node id="17291983" version="4" timestamp="2014-06-10T19:13:01Z" uid="266672" user="tf66" changeset="22856880" lat="47.5316154" lon="8.806115"/>\n  <node id="17291985" version="11" timestamp="2014-06-10T19:13:01Z" uid="266672" user="tf66" changeset="22856880" lat="47.5309641" lon="8.8110597"/>\n  <node id="17291986" version="11" timestamp="2014-06-10T19:13:01Z" uid="266672" user="tf66" changeset="22856880" lat="47.5303308" lon="8.8158613"/>\n  <node id="17291987" version="8" timestamp="2014-05-17T02:31:11Z" uid="178186" user="mdk" changeset="22381881" lat="47.5297827" lon="8.8199923"/>\n  <node id="17291988" version="8" timestamp="2014-05-17T02:31:11Z" uid="178186" user="mdk" changeset="22381881" lat="47.5292412" lon="8.8246716"/>\n  <node id="17291989" version="8" timestamp="2014-05-17T02:31:11Z" uid="178186" user="mdk" changeset="22381881" lat="47.5289329" lon="8.8293619"/>\n  <node id="17291990" version="8" timestamp="2014-05-17T02:31:11Z" uid="178186" user="mdk" changeset="22381881" lat="47.5289057" lon="8.8318414"/>\n  <node id="17291992" version="9" timestamp="2014-05-17T02:31:11Z" uid="178186" user="mdk" changeset="22381881" lat="47.5292508" lon="8.8374973"/>\n  <node id="17291995" version="8" timestamp="2014-06-09T08:39:49Z" uid="266672" user="tf66" changeset="22825155" lat="47.5297382" lon="8.8413978"/>\n  <node id="17291996" version="9" timestamp="2014-06-09T08:39:49Z" uid="266672" user="tf66" changeset="22825155" lat="47.5302746" lon="8.8462347"/>\n  <node id="17292000" version="9" timestamp="2014-06-09T08:39:49Z" uid="266672" user="tf66" changeset="22825155" lat="47.5302433" lon="8.8571037"/>\n  <node id="17292002" version="5" timestamp="2014-06-09T08:39:49Z" uid="266672" user="tf66" changeset="22825155" lat="47.5303581" lon="8.8650907"/>\n  <node id="17292004" version="5" timestamp="2014-06-09T08:39:49Z" uid="266672" user="tf66" changeset="22825155" lat="47.5301721" lon="8.8693658"/>\n  <node id="17292006" version="2" timestamp="2014-06-09T08:39:49Z" uid="266672" user="tf66" changeset="22825155" lat="47.5292947" lon="8.8734748"/>\n  <node id="17292007" version="6" timestamp="2014-06-09T08:39:49Z" uid="266672" user="tf66" changeset="22825155" lat="47.529381" lon="8.8735047"/>\n  <node id="17292008" version="6" timestamp="2014-04-29T16:17:32Z" uid="1881716" user="Leonardo Hauschild" changeset="22026107" lat="47.5279776" lon="8.8776073"/>\n  <node id="17292009" version="6" timestamp="2014-04-29T16:17:32Z" uid="1881716" user="Leonardo Hauschild" changeset="22026107" lat="47.5266105" lon="8.8811809"/>\n  <node id="17292010" version="5" timestamp="2014-04-29T16:17:33Z" uid="1881716" user="Leonardo Hauschild" changeset="22026107" lat="47.5252154" lon="8.8859877"/>\n  <node id="17292013" version="2" timestamp="2014-06-09T08:39:49Z" uid="266672" user="tf66" changeset="22825155" lat="47.5239798" lon="8.8901459"/>\n  <node id="17292014" version="7" timestamp="2014-06-09T08:39:49Z" uid="266672" user="tf66" changeset="22825155" lat="47.524045" lon="8.8902596"/>\n  <node id="17292015" version="6" timestamp="2014-06-09T08:39:49Z" uid="266672" user="tf66" changeset="22825155" lat="47.5231156" lon="8.892528"/>\n  <node id="17292017" version="5" timestamp="2014-06-09T08:39:49Z" uid="266672" user="tf66" changeset="22825155" lat="47.520539" lon="8.8961268"/>\n  <node id="17292018" version="5" timestamp="2014-06-09T08:39:49Z" uid="266672" user="tf66" changeset="22825155" lat="47.5222222" lon="8.8940475"/>\n  <node id="17292020" version="3" timestamp="2014-06-09T08:39:49Z" uid="266672" user="tf66" changeset="22825155" lat="47.5221516" lon="8.8939671"/>\n  <node id="17292022" version="5" timestamp="2014-06-09T08:39:49Z" uid="266672" user="tf66" changeset="22825155" lat="47.5181828" lon="8.8986937"/>\n  <node id="17298003" version="6" timestamp="2014-06-09T08:39:49Z" uid="266672" user="tf66" changeset="22825155" lat="47.5161649" lon="8.901721"/>\n  <node id="19486041" version="6" timestamp="2013-10-01T14:15:02Z" uid="481380" user="Nzara" changeset="18127967" lat="47.5033348" lon="8.8181471"/>\n  <node id="19486044" version="5" timestamp="2011-07-06T22:08:58Z" uid="334389" user="ueliw0" changeset="8652872" lat="47.5010407" lon="8.8149097"/>\n  <node id="19486045" version="7" timestamp="2011-07-06T22:08:59Z" uid="334389" user="ueliw0" changeset="8652872" lat="47.5003745" lon="8.8128874"/>\n  <node id="19486047" version="4" timestamp="2011-07-06T22:08:59Z" uid="334389" user="ueliw0" changeset="8652872" lat="47.5002666" lon="8.8098208"/>\n  <node id="19486050" version="5" timestamp="2015-02-17T21:37:26Z" uid="178186" user="mdk" changeset="28919946" lat="47.5005555" lon="8.8060688"/>\n  <node id="19486053" version="7" timestamp="2015-02-17T21:37:26Z" uid="178186" user="mdk" changeset="28919946" lat="47.499729" lon="8.8030104"/>\n  <node id="21311157" version="5" timestamp="2015-02-17T21:37:26Z" uid="178186" user="mdk" changeset="28919946" lat="47.5002745" lon="8.8044231"/>\n  <node id="21311158" version="5" timestamp="2015-02-17T21:37:26Z" uid="178186" user="mdk" changeset="28919946" lat="47.5005606" lon="8.8073485"/>\n  <node id="21311159" version="6" timestamp="2011-07-06T22:08:57Z" uid="334389" user="ueliw0" changeset="8652872" lat="47.500225" lon="8.8102452">\n    <tag k="ele" v="486"/>\n    <tag k="name" v="Schottikon"/>\n    <tag k="network" v="ZVV"/>\n    <tag k="operator" v="SBB"/>\n    <tag k="public_transport" v="stop_position"/>\n    <tag k="railway" v="stop"/>\n    <tag k="source" v="Bundesamt für Verkehr (BAV), DIDOK 2009"/>\n    <tag k="train" v="yes"/>\n    <tag k="uic_name" v="Schottikon"/>\n    <tag k="uic_ref" v="8506035"/>\n  </node>'),
 (14092,
  '<node id="21311160" version="7" timestamp="2011-07-06T22:09:00Z" uid="334389" user="ueliw0" changeset="8652872" lat="47.5006754" lon="8.8140081"/>\n  <node id="21311161" version="6" timestamp="2011-07-06T22:09:00Z" uid="334389" user="ueliw0" changeset="8652872" lat="47.5015754" lon="8.8158235"/>\n  <node id="21311162" version="6" timestamp="2013-10-01T14:15:04Z" uid="481380" user="Nzara" changeset="18127967" lat="47.5038762" lon="8.8192518"/>\n  <node id="21539900" version="5" timestamp="2014-06-09T08:39:51Z" uid="266672" user="tf66" changeset="22825155" lat="47.5158063" lon="8.9022255"/>\n  <node id="21539901" version="7" timestamp="2014-06-09T08:39:51Z" uid="266672" user="tf66" changeset="22825155" lat="47.5159095" lon="8.9022361"/>\n  <node id="21539902" version="2" timestamp="2014-06-09T08:39:51Z" uid="266672" user="tf66" changeset="22825155" lat="47.5163812" lon="8.9011015"/>\n  <node id="21539903" version="5" timestamp="2014-06-09T08:39:51Z" uid="266672" user="tf66" changeset="22825155" lat="47.5164657" lon="8.9011604"/>\n  <node id="21539904" version="7" timestamp="2014-06-09T08:39:51Z" uid="266672" user="tf66" changeset="22825155" lat="47.5234174" lon="8.891887"/>\n  <node id="21539905" version="4" timestamp="2014-12-13T01:15:14Z" uid="178186" user="mdk" changeset="27433683" lat="47.52366" lon="8.8910223"/>\n  <node id="21539906" version="6" timestamp="2014-06-09T08:39:51Z" uid="266672" user="tf66" changeset="22825155" lat="47.5246747" lon="8.8881277"/>\n  <node id="21539907" version="2" timestamp="2014-06-09T08:39:51Z" uid="266672" user="tf66" changeset="22825155" lat="47.52457" lon="8.888162"/>\n  <node id="21539908" version="4" timestamp="2014-06-09T08:39:51Z" uid="266672" user="tf66" changeset="22825155" lat="47.5256986" lon="8.8837248"/>\n  <node id="21539909" version="6" timestamp="2014-04-29T16:17:33Z" uid="1881716" user="Leonardo Hauschild" changeset="22026107" lat="47.5257672" lon="8.8838534"/>\n  <node id="21539910" version="6" timestamp="2014-06-09T08:39:51Z" uid="266672" user="tf66" changeset="22825155" lat="47.5287106" lon="8.8756846"/>\n  <node id="21539911" version="3" timestamp="2014-04-29T16:17:32Z" uid="1881716" user="Leonardo Hauschild" changeset="22026107" lat="47.5286296" lon="8.8756214"/>\n  <node id="21539912" version="5" timestamp="2014-06-09T08:39:51Z" uid="266672" user="tf66" changeset="22825155" lat="47.5298964" lon="8.8712866"/>\n  <node id="21539913" version="2" timestamp="2014-06-09T08:39:51Z" uid="266672" user="tf66" changeset="22825155" lat="47.5298015" lon="8.8712404"/>\n  <node id="21539914" version="5" timestamp="2014-06-09T08:39:51Z" uid="266672" user="tf66" changeset="22825155" lat="47.5303516" lon="8.8669942"/>\n  <node id="21539915" version="3" timestamp="2014-06-09T08:39:51Z" uid="266672" user="tf66" changeset="22825155" lat="47.5302537" lon="8.8670017"/>\n  <node id="21539916" version="3" timestamp="2014-06-09T08:39:51Z" uid="266672" user="tf66" changeset="22825155" lat="47.5301206" lon="8.8599666"/>\n  <node id="21539917" version="7" timestamp="2014-06-09T08:39:51Z" uid="266672" user="tf66" changeset="22825155" lat="47.5302173" lon="8.8600277"/>\n  <node id="21539918" version="12" timestamp="2014-06-09T08:39:51Z" uid="266672" user="tf66" changeset="22825155" lat="47.5303298" lon="8.8515299"/>\n  <node id="21539919" version="10" timestamp="2014-06-09T08:39:51Z" uid="266672" user="tf66" changeset="22825155" lat="47.5303851" lon="8.8483248"/>\n  <node id="21539920" version="11" timestamp="2013-05-12T17:13:27Z" uid="390170" user="chewiebug" changeset="16099021" lat="47.5290502" lon="8.8363795"/>\n  <node id="21539921" version="8" timestamp="2014-05-17T02:31:13Z" uid="178186" user="mdk" changeset="22381881" lat="47.5290866" lon="8.8357822"/>\n  <node id="21539922" version="11" timestamp="2013-05-12T17:13:27Z" uid="390170" user="chewiebug" changeset="16099021" lat="47.528962" lon="8.826872"/>\n  <node id="21539923" version="8" timestamp="2014-05-17T02:31:13Z" uid="178186" user="mdk" changeset="22381881" lat="47.5290574" lon="8.8269098"/>\n  <node id="21546330" version="5" timestamp="2011-10-10T23:20:06Z" uid="7412" user="mjulius" changeset="9527023" lat="47.4588033" lon="8.7089406"/>\n  <node id="21546331" version="6" timestamp="2011-10-10T23:20:06Z" uid="7412" user="mjulius" changeset="9527023" lat="47.4595504" lon="8.7089153"/>\n  <node id="25426773" version="6" timestamp="2012-12-16T18:31:42Z" uid="178186" user="mdk" changeset="14297157" lat="47.5041366" lon="8.7246636"/>\n  <node id="25426776" version="5" timestamp="2013-08-26T22:27:16Z" uid="178186" user="mdk" changeset="17526834" lat="47.4903745" lon="8.7239761"/>\n  <node id="25426777" version="4" timestamp="2012-06-23T16:28:23Z" uid="178186" user="mdk" changeset="11990845" lat="47.5051886" lon="8.7384868">\n    <tag k="highway" v="crossing"/>\n  </node>'),
 (15659,
  '<node id="25426787" version="7" timestamp="2013-08-27T00:16:28Z" uid="178186" user="mdk" changeset="17527633" lat="47.4986795" lon="8.7554761"/>\n  <node id="25426790" version="7" timestamp="2014-06-10T19:13:01Z" uid="266672" user="tf66" changeset="22856880" lat="47.5267296" lon="8.735563"/>\n  <node id="25426791" version="12" timestamp="2012-01-15T20:24:26Z" uid="178186" user="mdk" changeset="10402834" lat="47.5113937" lon="8.7215678"/>\n  <node id="25426792" version="7" timestamp="2013-08-27T00:16:28Z" uid="178186" user="mdk" changeset="17527633" lat="47.5049067" lon="8.760263"/>\n  <node id="25426795" version="6" timestamp="2013-08-27T00:16:28Z" uid="178186" user="mdk" changeset="17527633" lat="47.4837899" lon="8.7022003"/>\n  <node id="25426804" version="7" timestamp="2012-06-09T21:20:33Z" uid="178186" user="mdk" changeset="11849209" lat="47.4944111" lon="8.7112899"/>\n  <node id="25426806" version="4" timestamp="2012-09-16T16:45:17Z" uid="470089" user="pitg600" changeset="13133024" lat="47.4918548" lon="8.7597633"/>\n  <node id="25426824" version="8" timestamp="2013-08-19T23:05:58Z" uid="178186" user="mdk" changeset="17419062" lat="47.4974493" lon="8.7337237"/>\n  <node id="25426827" version="6" timestamp="2013-08-08T01:32:20Z" uid="178186" user="mdk" changeset="17261334" lat="47.5009987" lon="8.7408259"/>\n  <node id="25426834" version="6" timestamp="2014-11-27T17:29:54Z" uid="577816" user="martin0203" changeset="27072617" lat="47.5147996" lon="8.7575484">\n    <tag k="crossing" v="uncontrolled"/>\n    <tag k="highway" v="crossing"/>\n  </node>'),
 (16905,
  '<node id="25426847" version="6" timestamp="2016-03-09T06:48:39Z" uid="2110714" user="captain_slow" changeset="37701425" lat="47.4914681" lon="8.7145365"/>\n  <node id="25426854" version="5" timestamp="2010-09-21T23:20:20Z" uid="178186" user="mdk" changeset="5841750" lat="47.5044617" lon="8.7196017"/>\n  <node id="25426860" version="2" timestamp="2011-04-10T07:53:49Z" uid="78090" user="Roman Fischer" changeset="7820073" lat="47.5063473" lon="8.7720639"/>\n  <node id="25426872" version="4" timestamp="2016-04-13T11:30:13Z" uid="178186" user="mdk" changeset="38522412" lat="47.5108258" lon="8.7570041"/>\n  <node id="25426873" version="6" timestamp="2012-06-09T21:20:34Z" uid="178186" user="mdk" changeset="11849209" lat="47.4903443" lon="8.704835"/>\n  <node id="25426879" version="5" timestamp="2011-12-31T23:10:25Z" uid="178186" user="mdk" changeset="10258494" lat="47.5083615" lon="8.7243642"/>\n  <node id="25426880" version="10" timestamp="2014-06-10T19:13:01Z" uid="266672" user="tf66" changeset="22856880" lat="47.5257471" lon="8.7296538"/>\n  <node id="25426882" version="7" timestamp="2012-06-09T21:20:34Z" uid="178186" user="mdk" changeset="11849209" lat="47.4939946" lon="8.7106323">\n    <tag k="highway" v="traffic_signals"/>\n  </node>'),
 (18619,
  '<node id="25426884" version="3" timestamp="2012-11-10T00:06:26Z" uid="178186" user="mdk" changeset="13815933" lat="47.485034" lon="8.7679153"/>\n  <node id="25426894" version="6" timestamp="2013-06-01T20:41:15Z" uid="178186" user="mdk" changeset="16383115" lat="47.4944825" lon="8.7567161"/>\n  <node id="25426898" version="10" timestamp="2014-06-10T19:13:01Z" uid="266672" user="tf66" changeset="22856880" lat="47.5320305" lon="8.7892039"/>\n  <node id="25426900" version="5" timestamp="2011-11-10T01:36:35Z" uid="178186" user="mdk" changeset="9786127" lat="47.4880593" lon="8.7528133"/>\n  <node id="25426902" version="8" timestamp="2013-07-06T21:59:07Z" uid="178186" user="mdk" changeset="16854057" lat="47.498229" lon="8.7379017"/>\n  <node id="25426905" version="12" timestamp="2013-12-15T03:42:50Z" uid="178186" user="mdk" changeset="19458523" lat="47.5146361" lon="8.7212415"/>\n  <node id="25426912" version="3" timestamp="2012-01-24T23:21:52Z" uid="178186" user="mdk" changeset="10489546" lat="47.4829964" lon="8.7715284"/>\n  <node id="25426915" version="4" timestamp="2014-02-16T00:15:06Z" uid="178186" user="mdk" changeset="20587312" lat="47.5047985" lon="8.7735619"/>\n  <node id="25426917" version="9" timestamp="2011-10-13T22:30:12Z" uid="7412" user="mjulius" changeset="9551675" lat="47.4871232" lon="8.7023569"/>\n  <node id="25426919" version="6" timestamp="2012-06-09T21:20:34Z" uid="178186" user="mdk" changeset="11849209" lat="47.4972985" lon="8.715829"/>\n  <node id="25426922" version="10" timestamp="2015-01-15T21:40:51Z" uid="178186" user="mdk" changeset="28174225" lat="47.5079332" lon="8.7093797">\n    <tag k="crossing" v="traffic_signals"/>\n    <tag k="highway" v="traffic_signals"/>\n  </node>'),
 (18965,
  '<node id="25426924" version="9" timestamp="2013-08-25T19:12:15Z" uid="178186" user="mdk" changeset="17505523" lat="47.4928507" lon="8.7433431"/>\n  <node id="25426925" version="15" timestamp="2013-08-08T22:30:38Z" uid="178186" user="mdk" changeset="17273135" lat="47.5024939" lon="8.730581">\n    <tag k="highway" v="traffic_signals"/>\n  </node>'),
 (19156,
  '<node id="25426928" version="4" timestamp="2012-06-09T21:20:34Z" uid="178186" user="mdk" changeset="11849209" lat="47.477385" lon="8.7030625">\n    <tag k="highway" v="crossing"/>\n  </node>'),
 (20450,
  '<node id="25426939" version="5" timestamp="2014-11-30T16:18:21Z" uid="178186" user="mdk" changeset="27135035" lat="47.503106" lon="8.724858"/>\n  <node id="25426974" version="3" timestamp="2011-10-28T23:43:59Z" uid="178186" user="mdk" changeset="9681042" lat="47.5225189" lon="8.7216449"/>\n  <node id="25426976" version="6" timestamp="2014-06-10T19:13:01Z" uid="266672" user="tf66" changeset="22856880" lat="47.5297279" lon="8.7551183"/>\n  <node id="25426977" version="7" timestamp="2013-08-27T00:16:28Z" uid="178186" user="mdk" changeset="17527633" lat="47.4922159" lon="8.7586273"/>\n  <node id="25426978" version="3" timestamp="2013-12-11T23:46:09Z" uid="178186" user="mdk" changeset="19404529" lat="47.5098251" lon="8.6891799"/>\n  <node id="25426986" version="6" timestamp="2013-12-16T21:07:00Z" uid="178186" user="mdk" changeset="19488974" lat="47.5208665" lon="8.721843"/>\n  <node id="25426988" version="6" timestamp="2012-01-24T00:33:12Z" uid="178186" user="mdk" changeset="10480641" lat="47.5019357" lon="8.7613508"/>\n  <node id="25426994" version="16" timestamp="2014-06-10T19:12:59Z" uid="266672" user="tf66" changeset="22856880" lat="47.5304258" lon="8.7736311">\n    <tag k="highway" v="motorway_junction"/>\n    <tag k="name" v="Oberwinterthur"/>\n    <tag k="ref" v="72"/>\n  </node>'),

It only detects lines ending with the configured endtag but not empty elements.

[HyukjinKwon]

Oh, thanks for debugging closely! Yes, it seems an issue. Maybe is this a duplicate of #92 ?

[martin-weber]

It is not 100% the same as #92 because a schema can be derived as you can see in my posts. But it might be related, by having to the same root cause.

[Synesso]

I witness the same condition with OSM XML. In my case, I am mapping to case class.

    sqlContext.read
      .format("com.databricks.spark.xml")
      .option("rootTag", "osm")
      .option("rowTag", "node")
      .load("map.osm")
      .rdd.map { r =>
        Node(r.getAs[Long]("_id"), Point(r.getAs[Double]("_lat"), r.getAs[Double]("_lon")))
      }

The result is that any node with 1 or more child tags are parsed. The remainder (self-closing tags) have values defaulting to zero. i.e. Node(0, Point(0.0, 0.0))

If I modify some of the input to be in the format <node ...></node> with no child tags, the process throws this exception:

Caused by: java.util.NoSuchElementException
	at com.sun.xml.internal.stream.EventFilterSupport.nextEvent(EventFilterSupport.java:79)
	at com.databricks.spark.xml.parsers.StaxXmlParserUtils$.checkEndElement(StaxXmlParserUtils.scala:31)
	at com.databricks.spark.xml.util.InferSchema$.com$databricks$spark$xml$util$InferSchema$$inferObject(InferSchema.scala:208)
	at com.databricks.spark.xml.util.InferSchema$$anonfun$3$$anonfun$apply$2.apply(InferSchema.scala:101)
	at com.databricks.spark.xml.util.InferSchema$$anonfun$3$$anonfun$apply$2.apply(InferSchema.scala:89)
	at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
	at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
	at scala.collection.Iterator$class.foreach(Iterator.scala:893)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
	at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
	at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1336)
	at scala.collection.TraversableOnce$class.aggregate(TraversableOnce.scala:214)
	at scala.collection.AbstractIterator.aggregate(Iterator.scala:1336)
	at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$24.apply(RDD.scala:1112)
	at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$24.apply(RDD.scala:1112)
	at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$25.apply(RDD.scala:1113)
	at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$25.apply(RDD.scala:1113)
	at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:785)
	at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:785)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)

[edenfisher]

Any solution suggested to this issue?

[HyukjinKwon]

I had a plan to sweep all issues in this library on this weekend but I was too busy. Will try to handle this within the next week.

[queirozfcom]

Is there a workaround for this? I'm having the same issue when reading rows that have only attributes and no child nodes..

[gaortiz1]

This works for me in java

`
final SQLContext sqlContext = new SQLContext(SPARK_SESSION);

    StructType nd = new StructType(new StructField[] {
            new StructField("_ref", DataTypes.StringType, true, Metadata.empty())
    });

    StructType way = new StructType(new StructField[] {
            new StructField("_id", DataTypes.StringType, true, Metadata.empty()),
            new StructField("_changeset", DataTypes.StringType, true, Metadata.empty()),
            new StructField("_timestamp", DataTypes.StringType, true, Metadata.empty()),
            new StructField("_uid", DataTypes.StringType, true, Metadata.empty()),
            new StructField("_visible", DataTypes.StringType, true, Metadata.empty()),
            new StructField("nd", DataTypes.createArrayType(nd), true, Metadata.empty())
    });

    final Dataset<Row> load = sqlContext.read()
            .format("com.databricks.spark.xml")
            .option("rowTag", "way")
            .schema(way)
            .load("C:\\Users\\bisbi\\Desktop\\map.osm");

    load.show();

`

[BioQwer]

@martin-weber in your xml file you have self closing xml nodes <node />
i fix it in #285
now it should work