[Java.Interop.Tools.JavaSource] Improve <code> parsing

[pjcollins]

Fixes: #1071

The latest API docs update contained a couple dozen parsing issues due
to a handful of issues parsing <code> elements such as:

<code>null</null> - wrong closing tag
<code>android:label="@string/resolve_title"</code> - @ tag rule seemingly taking precedence
<code>Activity.RESULT_OK<code> - trailing open tag rather than closing tag
<code><pre><p>content</code></pre></p> - additional inline html tags
<code class=prettyprint>content<code> - attributes on the code element

These issues have been fixed by attempting to capture entire <code>
elements (and the known issues mentioned above) using a Regex based
terminal.

[pjcollins]

Testing further with: https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=7913417&view=results
Sample diff for these changes can be found at xamarin/android-api-docs#49.

[jonpryor]

From the original PR message:

Instead, an HTML [post-]processing step has been added which replaces, removes, or decodes well known HTML tags after the javadoc is parsed.

This concerns me, in that it means that there is no way to ensure valid XML. Imagine we have this Javadoc fragment:

/**
 * <em>this is never closed
 */
public static void foo() {
}

then (I think; haven't tested) this would result in FixUpHtml() returning "<i>this is never closed", which could eventually result in:

/// <i>this is never closed

(A more common example would be <li> usage, for which there is almost never a corresponding </li>. See e.g. java/text/Collator.java and search for <LI>.)

The good news is that this doesn't error out, but it would result in a CS1570 warning:

warning CS1570: XML comment has badly formed XML -- 'Expected an end tag for element 'i'.'

This would only be a problem for those building with $(TreatWarningsAsErrors)=True (which is bananas for binding projects)…

…but is it otherwise a problem? I philosophically don't like it, but this might be the better path forward anyway…

[jonpryor]

From the PR message:

Readability of our generated docs should be improved by both of these changes, as there will be fewer encoded character entities in places where they are not necessary.

On the one hand, "yes", but on the other hand, whitespace is (normally) not significant, so if we look at xamarin/android-api-docs#49 and compare to what we have now, consider:

• https://learn.microsoft.com/en-us/dotnet/api/android.app.admin.devicepolicymanager.generatekeypair?view=xamarin-android-sdk-13
• https://github.com/xamarin/android-api-docs/pull/49/files#diff-18564fa5cda8d8249b99da1017644a77442722a7c1c3a6e243f3e3254854b146L4568

The <ul> & co is replaced, removing the HTML. In the diff, this looks nicer. In context?

Before:

This API can be called by the following to generate a new private/public key pair: <ul> <li>Device owner</li> <li>Profile owner</li> <li>Delegated certificate installer</li> <li>Credential management app</li> </ul> If the device supports key generation via secure hardware…

After:

This API can be called by the following to generate a new private/public key pair: Device owner Profile owner Delegated certificate installer Credential management app If the device supports key generation via secure hardware

The "raw html" is absolute garbage, yes, but is the resulting text actually better? IMHO it is not better, as there is no longer a way to even know when the first sentence ends and the next sentence begins. </ul> If the device… is ugly, certainly, but at least you know "If the device" begins a new sentence.

("Best", of course, would be adding Irony logic to look for <ul/> & <li/> and translate into mdoc(5) <list type="bullet"/>, but that's significantly more work. Something we should do eventually, but not in this commit.)

[jonpryor]

Overall, it may be better/simpler to separate the <a/> parsing changes from the <code/> parsing changes.

[pjcollins]

I moved the <a/> element parsing fixes into #1126. I'll need to rethink the rest of these changes. This does produce a more readable doc in many cases, but as you've pointed out this is not going to be better / more desirable for all cases.

[pjcollins]

It would be great to find a way to reduce the amount of encoded html entities in our docs to help with readability, but the attempt to do so as part of this PR was lacking. I've updated this to only include an update for <code> element parsing as suggested.

[jonpryor]

I would like to extend this test to see what happens with the content after the <code>, e.g. from android/telephony/gsm/SmsManager.java:

     *  The result code will be <code>Activity.RESULT_OK<code> for success,
     *  or one of these errors:
     *  <code>RESULT_ERROR_GENERIC_FAILURE</code>

or shorter: <code>should be code<code> but what about this <code>and this?</code>. Yes, <code>should be code<code> results in <c>should be code</c>, but what about the rest? Would we get:

<c>should be code</c><c> but what about this </c><c>and this?</c>

which is entirely defensible and reasonable, but is worth "calling out" in the unit tests.

[pjcollins]

We don't appear to create an additional code element for the intermediate content, presumably because we capture the incorrectly placed open <code> tag while parsing the previous code element. I've added another test for this.

[jonpryor]

WIP commit message:

Fixes: https://github.com/xamarin/java.interop/issues/1071

The latest API docs update contained a couple dozen parsing issues
due to `<code/>` parsing, including:

  * Closing element doesn't match opening element: `<code>null</null>`
  * Content including `@`: `<code>android:label="@string/resolve_title"</code>`
  * Closing element is actually an opening element:
    `<code>Activity.RESULT_OK<code>`
  * Improper element nesting: `<code><pre><p>content</code></pre></p>`
  * Use of attributes: `<code class=prettyprint>content<code>`

Fix this by replacing `CodeElementDeclaration` to use a new
`CodeElementContentTerm` terminal, which is a "greedy regex" which
grabs `<code` until one of:

  * `</code>`
  * `</null>`
  * `<code>`