US 38-098 22.14.6.4p1 [format.string.escaped] Escaping for debugging and logging

[wg21bot]

The subject subclause describes how characters or strings are "escaped" to be formatted more suitably "for debugging or for logging".

The actual suitability for debugging or for logging depends on the needs of the application, and there is a conflict between formatting for human readability of the textual content and formatting for clarity and fidelity of encoding nuances. Indeed, for the latter, there can still be (for stateful encodings) a conflict between formatting for human visual inspection versus formatting for machine consumption of the output sequence as a C++ string/character literal.

The current design introduces extensions to the API and to the format string syntax that assume that there is one specific default that should be chosen "for debugging or for logging". The reasoning behind the chosen default and the extensibility of the current design does not appear to be sufficiently explored.

Note 1:
An example, for Unicode encodings, of a choice between prioritizing between human readability of the textual content and visual clarity of encoding nuances is in the treatment of characters having Unicode property Grapheme_Extend=Yes. The current design favors visual clarity of encoding nuances by outputing such characters as escape sequences.

Note 2:
For stateful encodings, the lack of return to the initial shift state at the end of the sequence cannot be represented using a C++ string/character literal unless if a prior shift sequence from the initial shift state is rendered via escape sequence(s). It is not clear that scanning forward is generally always an option (nor is it clear that doing so is desirable).

Proposed change:

Narrow the purported scope and affirm the design choices of the default behavior:
Modify "logging" to "technical logging" and spell out the priorities in order in the description (this has the benefit of clearly communicating intention and providing guidance for implementation choices).
1. The output is intended to be a C++ string/character literal that reproduces the encoded sequence. (This seems to be taken for granted and not made explicit in the current draft.)
2. Prefer visually distinguishing between different methods of encoding "equivalent" textual content.

Make any adjustments necessary to the API or the format string syntax associated with "escaped" strings to allow for future additions for alternative escaping.

[brycelelbach]

@tahonermann you get first crack at this. Send it back to us when it's ready.

[tahonermann]

SG16 discussed this issue during its 2022-10-19 telecon. The following polls were taken:

• Poll 2.1: [US 38-098] SG16 agrees that the formatted code units in the escaped string are intended
  to be usable as a string literal that reproduces the input.
  • Attendees: 8
  • No objection to unanimous consent.
• Poll 2.2: [US 38-098] SG16 agrees that the escaped string is intended to be readable for its textual
  content in any Unicode script.
  • Attendees: 8
  • No objection to unanimous consent.
• Poll 2.3: [US 38-098] SG16 agrees that separators and non-printable characters
  ([format.string.escaped]p(2.2.1.2))
  shall be escaped in the escaped string.
  • Attendees: 8
  • No objection to unanimous consent.
• Poll 2.4: [US 38-098] SG16 agrees that combining code points shall not be escaped unless there is no
  leading code point or the previous character was escaped.
  • Attendees: 8
  • No objection to unanimous consent.

The first poll confirms the intent that an escaped string be usable as a string literal (e.g., that it can be copied and pasted into a C++ program) such that, when evaluated as a string literal, the input used to produce the escaped string is reproduced. No changes are required to satisfy this poll.

The second poll clarifies that it is intended that the escaped string be readable by humans. The context for this poll was concern about producing visually ambiguous output. The SG16 conclusion is that escaped strings are not intended to produce visually unambiguous results; it is ok for the escaped string to contain unescaped characters that might be confused with other characters (e.g., characters considered "confusables" by Unicode). No changes are required to satisfy this poll.

The third poll clarifies that it is intended that separator and non-printable characters continue to be escaped. No changes are required to satisfy this poll.

The last poll indicates a change in direction relative to the current wording. SG16 desires that combining characters (those with the Unicode property Grapheme_­Extend=Yes) shall be escaped if they are not preceded by a non-escaped lead character (or another combining character that is preceded by a lead character). Satisfying this poll will require normative changes to [format.string.escaped]p2.

Examples (that would be appropriate to add to example 1 in [format.string.escaped] follow:

    string s6 = format("{:?}", "\u0301");        // s6 has value "\u{301}"
                                                 // (the combining character is escaped since it does
                                                 // not follow a non-escaped non-combining character)
    string s7 = format("{:?}", "\\\u0301");      // s7 has value "\\\u{301}"
                                                 // (the combining character is escaped since it follows
                                                 // an escaped character)
    string s8 = format("{:?}", "e\u0301\u0323"); // s8 has value "ẹ́"
                                                 // (U+0065, U+0301, U+0323; the combining characters
                                                 // are not escaped since they follow a non-combining
                                                 // character)

(Note: This comment was edited to remove one of the original examples as it was incorrect. The remaining examples are consistent with those Victor seeks to add via P2713R0.)