Make encode-marc21 more resilient if order is not correct

[TobiasNx]

When the leader comes not as first, encode-marc21 breaks: java.lang.IllegalStateException: must not be appending data fields

https://metafacture.org/playground/?flux=%22https%3A//d-nb.info/1106253078/about/marcxml%22%0A%7C+open-http%28accept%3D%22application/xml%22%29%0A%7C+decode-xml%0A%7C+handle-marcxml%0A%7C+fix%28transformationFile%29%0A%7C+encode-marc21%0A%7C+print%0A%3B&transformation=move_field%28%22leader%22%2C%22@leader%22%29%0Acopy_field%28%22@leader%22%2C%22leader.status%22%29%0Asubstring%28%22leader.status%22%2C%225%22%2C%221%22%29%0Acopy_field%28%22@leader%22%2C%22leader.type%22%29%0Asubstring%28%22leader.status%22%2C%226%22%2C%221%22%29%0Acopy_field%28%22@leader%22%2C%22leader.bibliographicLevel%22%29%0Asubstring%28%22leader.status%22%2C%227%22%2C%221%22%29%0Acopy_field%28%22@leader%22%2C%22leader.typeOfControl%22%29%0Asubstring%28%22leader.status%22%2C%228%22%2C%221%22%29%0Acopy_field%28%22@leader%22%2C%22leader.characterCodingScheme%22%29%0Asubstring%28%22leader.status%22%2C%229%22%2C%221%22%29%0Acopy_field%28%22@leader%22%2C%22leader.encodingLevel%22%29%0Asubstring%28%22leader.status%22%2C%2217%22%2C%221%22%29%0Acopy_field%28%22@leader%22%2C%22leader.catalogingForm%22%29%0Asubstring%28%22leader.status%22%2C%2218%22%2C%221%22%29%0Acopy_field%28%22@leader%22%2C%22leader.multipartLevel%22%29%0Asubstring%28%22leader.status%22%2C%2219%22%2C%221%22%29%0Aremove_field%28%22@field%22%29

encode-marc21 should not care if the order of elements is correct when entering it should though output in the correct order.

[dr0i]

The fix is missing. Could you provide it?
(also interested if this is solved in 'https://test.metafacture.org/playground/ as a sideefect of #526 ?)

[TobiasNx]

The fix is not missing, this is the old playground spec. The url still has the fix, but the playground does not interpret it anymore. I will repair the link

[TobiasNx]

Fix the link. See it in the first comment.

[dr0i]

Seems indeed to be fixed as a sideeffect.

[TobiasNx]

The test was identified in context of #454 but it is another problem.

[TobiasNx]

Seems indeed to be fixed as a sideeffect.

nope, you are looking at encode-marcxml the problem is with encode-marc21

https://test.metafacture.org/playground/?flux=%22https%3A//d-nb.info/1106253078/about/marcxml%22%0A%7C+open-http%28accept%3D%22application/xml%22%29%0A%7C+decode-xml%0A%7C+handle-marcxml%0A%7C+fix%28transformationFile%29%0A%7C+encode-marc21%0A%7C+print%0A%3B&transformation=move_field%28%22leader%22%2C%22@leader%22%29%0Acopy_field%28%22@leader%22%2C%22leader.status%22%29%0Asubstring%28%22leader.status%22%2C%225%22%2C%221%22%29%0Acopy_field%28%22@leader%22%2C%22leader.type%22%29%0Asubstring%28%22leader.status%22%2C%226%22%2C%221%22%29%0Acopy_field%28%22@leader%22%2C%22leader.bibliographicLevel%22%29%0Asubstring%28%22leader.status%22%2C%227%22%2C%221%22%29%0Acopy_field%28%22@leader%22%2C%22leader.typeOfControl%22%29%0Asubstring%28%22leader.status%22%2C%228%22%2C%221%22%29%0Acopy_field%28%22@leader%22%2C%22leader.characterCodingScheme%22%29%0Asubstring%28%22leader.status%22%2C%229%22%2C%221%22%29%0Acopy_field%28%22@leader%22%2C%22leader.encodingLevel%22%29%0Asubstring%28%22leader.status%22%2C%2217%22%2C%221%22%29%0Acopy_field%28%22@leader%22%2C%22leader.catalogingForm%22%29%0Asubstring%28%22leader.status%22%2C%2218%22%2C%221%22%29%0Acopy_field%28%22@leader%22%2C%22leader.multipartLevel%22%29%0Asubstring%28%22leader.status%22%2C%2219%22%2C%221%22%29%0Aremove_field%28%22@field%22%29

[dr0i]

I am not 99% sure but I think your fix creates a problematic marc21, if any, because you set the leader double time (use | encode-json(prettyprinting="true") to see that ) - and, more importantly, the leader.subfields must only consist of one value, not the whole leader:

   ....
  "@leader" : "00000naa a2200000uc 4500",
  "leader" : {
    "status" : "n",
    "type" : "00000naa a2200000uc 4500",
    "bibliographicLevel" : "00000naa a2200000uc 4500",
    "typeOfControl" : "00000naa a2200000uc 4500",
    "characterCodingScheme" : "00000naa a2200000uc 4500",
    "encodingLevel" : "00000naa a2200000uc 4500",
    "catalogingForm" : "00000naa a2200000uc 4500",
    "multipartLevel" : "00000naa a2200000uc 4500"
  }

I think the order is irrelevant at least.

[TobiasNx]

Ah I deleted @field instead of @leader. I fixed this, but there is still an error, but another one:
https://test.metafacture.org/playground/?flux=%22https%3A//d-nb.info/1106253078/about/marcxml%22%0A%7C+open-http%28accept%3D%22application/xml%22%29%0A%7C+decode-xml%0A%7C+handle-marcxml%0A%7C+fix%28transformationFile%29%0A%7C+encode-marc21%0A%7C+print%0A%3B&transformation=move_field%28%22leader%22%2C%22@leader%22%29%0Acopy_field%28%22@leader%22%2C%22leader.status%22%29%0Asubstring%28%22leader.status%22%2C%225%22%2C%221%22%29%0Acopy_field%28%22@leader%22%2C%22leader.type%22%29%0Asubstring%28%22leader.status%22%2C%226%22%2C%221%22%29%0Acopy_field%28%22@leader%22%2C%22leader.bibliographicLevel%22%29%0Asubstring%28%22leader.status%22%2C%227%22%2C%221%22%29%0Acopy_field%28%22@leader%22%2C%22leader.typeOfControl%22%29%0Asubstring%28%22leader.status%22%2C%228%22%2C%221%22%29%0Acopy_field%28%22@leader%22%2C%22leader.characterCodingScheme%22%29%0Asubstring%28%22leader.status%22%2C%229%22%2C%221%22%29%0Acopy_field%28%22@leader%22%2C%22leader.encodingLevel%22%29%0Asubstring%28%22leader.status%22%2C%2217%22%2C%221%22%29%0Acopy_field%28%22@leader%22%2C%22leader.catalogingForm%22%29%0Asubstring%28%22leader.status%22%2C%2218%22%2C%221%22%29%0Acopy_field%28%22@leader%22%2C%22leader.multipartLevel%22%29%0Asubstring%28%22leader.status%22%2C%2219%22%2C%221%22%29%0Aremove_field%28%22@leader%22%29%0A

[TobiasNx]

okay the error was still in the fix:

https://test.metafacture.org/playground/?flux=%22https%3A//d-nb.info/1106253078/about/marcxml%22%0A%7C+open-http%28accept%3D%22application/xml%22%29%0A%7C+decode-xml%0A%7C+handle-marcxml%0A%7C+fix%28transformationFile%29%0A%7C+encode-marc21%0A%7C+print%0A%3B&transformation=move_field%28%22leader%22%2C%22@leader%22%29%0Acopy_field%28%22@leader%22%2C%22leader.status%22%29%0Asubstring%28%22leader.status%22%2C%225%22%2C%221%22%29%0Acopy_field%28%22@leader%22%2C%22leader.type%22%29%0Asubstring%28%22leader.type%22%2C%226%22%2C%221%22%29%0Acopy_field%28%22@leader%22%2C%22leader.bibliographicLevel%22%29%0Asubstring%28%22leader.bibliographicLevel%22%2C%227%22%2C%221%22%29%0Acopy_field%28%22@leader%22%2C%22leader.typeOfControl%22%29%0Asubstring%28%22leader.typeOfControl%22%2C%228%22%2C%221%22%29%0Acopy_field%28%22@leader%22%2C%22leader.characterCodingScheme%22%29%0Asubstring%28%22leader.characterCodingScheme%22%2C%229%22%2C%221%22%29%0Acopy_field%28%22@leader%22%2C%22leader.encodingLevel%22%29%0Asubstring%28%22leader.encodingLevel%22%2C%2217%22%2C%221%22%29%0Acopy_field%28%22@leader%22%2C%22leader.catalogingForm%22%29%0Asubstring%28%22leader.catalogingForm%22%2C%2218%22%2C%221%22%29%0Acopy_field%28%22@leader%22%2C%22leader.multipartLevel%22%29%0Asubstring%28%22leader.multipartLevel%22%2C%2219%22%2C%221%22%29%0Aremove_field%28%22@leader%22%29%0A

I did not use the correct paths in the substring function but always copy_field("@leader","leader.status")

My bad. closing