Deterministic JSON serialization and deserialization support in Painless (should be deterministic)

[ypid-geberit]

Describe the feature:

The Painless API reference does not contain any function that deals with JSON. Would it be possible to provide basic JSON functionally in Painless? If possible the serialization should be deterministic, see below.

I am including full ES document dumps in emails send by watches. Because the toJson Mustache function is not deterministic I needed to implement my own deterministic serialization function in Painless and realized that this is somewhat painful (e.g. not as it should be ;-) ).

This has been asked before: https://discuss.elastic.co/t/painless-scripting-json-functions/98511

For reference, this is my workaround. My code does not produce valid JSON and probably has bugs (not extensively tested). To work without regex support in Painless, I also put HTML indent support into this function.

/* JSON-like serialization Painless {{{
 * Ref: https://github.com/elastic/elasticsearch/issues/37585
 * License: Public domain (because the code is too simple).
 */
String get_indent_string(boolean html_output, int indent) {
  if (html_output) {
    return String.join("", Collections.nCopies(indent, "  "));
  } else {
    return String.join("", Collections.nCopies(indent, "  "));
  }
}
String serializ_leaf(def object) {
  if (object == null) {
    return "null";
  } else if (object instanceof String) {
    return '"' + object + '"';
  } else {
    return object.toString();
  }
}

ArrayList object_to_human_readable_string_lines(def object) {
  return object_to_human_readable_string_lines(object, false, 0, true);
}

ArrayList object_to_human_readable_string_lines(def object, boolean html_output) {
  return object_to_human_readable_string_lines(object, html_output, 0, true);
}

ArrayList object_to_human_readable_string_lines(def object, boolean html_output, int indent) {
  return object_to_human_readable_string_lines(object, html_output, indent, true);
}

ArrayList object_to_human_readable_string_lines(def object, boolean html_output, int indent, boolean indent_first_line) {
  ArrayList serialized_lines = new ArrayList();

  if (object instanceof Collection) {
    serialized_lines.add("[");
    indent += 1;
    for (def nested_object : object) {
      if (nested_object instanceof Collection || nested_object instanceof Map) {
        serialized_lines.addAll(object_to_human_readable_string_lines(nested_object, html_output, indent));
      } else {
        serialized_lines.add(get_indent_string(html_output, indent) + serializ_leaf(nested_object) + ",");
      }
    }
    indent -= 1;
    serialized_lines.add(get_indent_string(html_output, indent) + "]" + (indent == 0 ? "" : ","));
  } else if (object instanceof Map) {
    serialized_lines.add(get_indent_string(html_output, (indent_first_line ? indent : 0)) + "{");
    indent += 1;
    List keys = object.keySet().stream().sorted().collect(Collectors.toList());
    for (String key : keys) {
      def nested_object = object[key];
      if (nested_object instanceof Collection || nested_object instanceof Map) {
        ArrayList nested_serialized_lines = object_to_human_readable_string_lines(nested_object, html_output, indent, false);
        serialized_lines.add(get_indent_string(html_output, indent) + key + ": " + nested_serialized_lines.remove(0));
        serialized_lines.addAll(nested_serialized_lines);
      } else {
        serialized_lines.add(get_indent_string(html_output, indent) + key + ": " + serializ_leaf(nested_object) + ",");
      }
    }
    indent -= 1;
    serialized_lines.add(get_indent_string(html_output, indent) + "}" + (indent == 0 ? "" : ","));
  } else {
    serialized_lines.add(serializ_leaf(object) + ",");
  }

  return serialized_lines;
}
/* }}} */

[elasticmachine]

Pinging @elastic/es-core-infra

[honzakral]

👍 on json support in painless. My ideal interface would be a pair of functions for (de)serializaing json objects. Example usage (naming lifter from python, should be adapted to painless' naming convention):

Strng json = '{"obj": {"l": [1,2, 3]}}';
HashMap o = loads(json);
o.obj.l.add(4);
// return {"obj":{"l":[1,2,3,4]}}
return dumps(o);

Ideally this would also support dumps(o, true) which would prettify the output, same as dumps(o, indent=4, sort_keys=True) in Python.

Supported types for dumps should also include all Date related types which would be serialized into the ISO format.

[honzakral]

PR sent in #60925

[ypid-geberit]

#63278 has been merged which is based on #60925, thanks! That is a great step forward 👍

It does not yet ensure deterministic serialization so I would like to keep this issue open. See discussion in #60925 (comment)

Cc: @stu-elastic

[stu-elastic]

Agreed this issue should be kept open, there is need for deterministic serialization.

[rjernst]

This has been open for quite a while, and we haven't made much progress on this due to focus in other areas. For now I'm going to close this as something we aren't planning on implementing. We can re-open it later if needed.

[ypid]

@rjernst Can you please set the correct issue status then? "Deterministic" is in the title but is not solved. This could otherwise give then wrong impression when people search for issues.