Self-closing tags are not supported as top-level rows

[upalkhouski]

Self-closing tags that are supported by xml standard are not supported currently.

Trying to read xml with self-closing tags generates empty schema.

Python Code snippet:

df = sqlContext.load(source="com.databricks.spark.xml", rowTag = 'book', path = 'file:///home/cloudera/pp/1/books.xml')
df.printSchema()

XML:

<?xml version="1.0"?>
<catalog>
   <book id="bk101"/>
   <book id="bk102"/>
   <book id="bk103"/>
   <book id="bk104"/>
</catalog>

Schema output:
root

Spark Version: 1.3

[HyukjinKwon]

@upalkhouski Thanks for opening this issue. I think this might have to be supported at XmlInputFormat.

BTW, could you maybe change the title to the one such as Self-closing tags are not supported as top-level rows(not sure this example title infers what I mean to say though) so that we can differentiate self-closing tags being parsed as each column and as each row?

[upalkhouski]

the title has been updated as requested.

I have realized that if I add value to the node it still does not work:

not working example:
<book id="bk104">TITLE</book>

however if I add additional child then it finally starts working:
<book id="bk104"><title>TITLE</title></book>

Looks like nodes that contain both attributes and value are not supported either as top-level rows

[HyukjinKwon]

@upalkhouski Thank you for looking into this deep. I tested the case too and found it does not work. Let me submit a PR soon.

Just a note to myself.

with the data below:

<bkval>
  <book id="bk116">bk_115</book>
  <book id="bk117">bk_116</book>
</bkval>

with the codes below:

val path = "path-to-file"
sqlContext.read
  .format("xml")
  .option("rowTag", "book")
  .load(path)
  .show()

produces an exception below as well:

java.util.NoSuchElementException
    at com.sun.xml.internal.stream.EventFilterSupport.nextEvent(EventFilterSupport.java:79)
    at com.databricks.spark.xml.parsers.StaxXmlParserUtils$.checkEndElement(StaxXmlParserUtils.scala:31)
    at com.databricks.spark.xml.util.InferSchema$.com$databricks$spark$xml$util$InferSchema$$inferObject(InferSchema.scala:208)
    at com.databricks.spark.xml.util.InferSchema$$anonfun$3$$anonfun$apply$2.apply(InferSchema.scala:101)
    at com.databricks.spark.xml.util.InferSchema$$anonfun$3$$anonfun$apply$2.apply(InferSchema.scala:89)
    at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:396)
    at scala.collection.Iterator$class.foreach(Iterator.scala:742)

I haven't looked into this yet. Let me maybe open another issue if it is not the same one with the above.

[vipulg]

Any idea if this is being worked upon

[HyukjinKwon]

Hi, @vipulg
<book id="bk104">TITLE</book> this case should be handled in StaxXmlParser
<book id="bk101"/> this case should be handled in both StaxXmlParser and XmlInputFormat.

Maybe I can fix the first case soon and then will try to fix the second case, though I think the latter case will take a bit of time to be fixed. Meanwhile, please feel free to submit a PR if anyone has some time!

[D059566]

Hi,

I have a similar issue where I want to read an XML file with multiple attributes per tag. Here's the file structure:

<?xml version="1.0" encoding="UTF-8"?>
<report format="1.0">
   <creationTime millis="1468158875331" readable="2016-07-10 13:54:35 +0000" />
   <project artifactid="fin.ap.balances.display" gitUrl="ssh://git.wdf.sap.corp:2/path/path/path" groupid="com.sap.prod.prod" parentArtifactId="name.name" parentVersion="1.12.2" version="4.0.7-SNAPSHOT">
      <check columnNumber="0" context="4.0.6" errorType="PREVIOUS_PROJECT_VERSION" filePath="/hompath/path/path" lineNumber="0" message="Reporting :: Previous version checked for compatibility&#xA;For details, see: https://githudoc.doc.doc.doc.docm.md" severity="Info" />
      <check columnNumber="0" context="Directories in '/src/main/webapp': [WEB-INF, model, view, util, css, img, i18n]" errorType="PROJECT_OLD_STRUCTURE" filePath="/hpathpathpath/ath/webapp" lineNumber="0" message="Reporting :: Using old project structure&#xA;For details, see: https://github.wdf.sap.coath.oath/pathpath.nmd" severity="Info" />
   </project>
</report>

--> Is there any way I can have com.databricks.spark.xml write all the attributes into one cell as a string and I come up with my own way of splitting and transforming this into a table? Are there any updates on the proposed fix?

thanks much,
best,

jonathan