[GR-37572] Introduce agent experimental options documentation.

PullRequest: graal/11434

Author: Aleksandar Gradinac

docs/reference-manual/native-image/Agent.md

@@ -39,7 +39,7 @@ It is advisable to manually review the generated configuration files. Because th
 
 The generated configuration files can be supplied to the `native-image` tool by placing them in a `META-INF/native-image/` directory on the class path, for example, in a JAR file used in the image build. This directory (or any of its subdirectories) is searched for files with the names `jni-config.json`, `reflect-config.json`, `proxy-config.json` and `resource-config.json`, which are then automatically included in the build. Not all of those files must be present. When multiple files with the same name are found, all of them are included.
 
-## Building Native Image with Java Reflection Example
+## Build a Native Executable with Java Reflection Example
 
 For demonstration purposes, save the following code as _ReflectionExample.java_ file:
 
@@ -155,6 +155,10 @@ Filter files have the following structure:
     {"excludeClasses": "com.oracle.svm.**"},
     {"includeClasses": "com.oracle.svm.tutorial.*"},
     {"excludeClasses": "com.oracle.svm.tutorial.HostedHelper"}
+  ],
+  "regexRules": [
+    {"includeClasses": ".*"},
+    {"excludeClasses": ".*\\$\\$Generated[0-9]+"}
   ]
 }
 ```
@@ -171,6 +175,12 @@ In the next rule however, lookups from those classes that are directly in packag
 Finally, lookups from the `HostedHelper` class is excluded again. Each of these rules partially overrides the previous ones.
 For example, if the rules were in the reverse order, the exclusion of `com.oracle.svm.**` would be the last rule and would override all other rules.
 
+The `regexRules` section also contains a sequence of rules.
+Its structure is the same as that of the `rules` section, but rules are specified as regular expression patterns which are matched against the entire fully qualified class identifier.
+The `regexRules` section is optional.
+If a `regexRules` section is specified, a class will be considered included if (and only if) both `rules` and `regexRules` include the class and neither of them exclude it.
+With no `regexRules` section, only the `rules` section determines whether a class is included or excluded.
+
 For testing purposes, the built-in filter for Java class library lookups can be disabled by adding the `no-builtin-caller-filter` option, but the resulting configuration files are generally unsuitable for a native image build.
 Similarly, the built-in filter for Java VM-internal accesses based on heuristics can be disabled with `no-builtin-heuristic-filter` and will also generally lead to less usable configuration files.
 For example: `-agentlib:native-image-agent=no-builtin-caller-filter,no-builtin-heuristic-filter,config-output-dir=...`
@@ -185,7 +195,7 @@ Using the `access-filter-file` option, a custom filter file that follows the fil
 The option can be specified more than once to add multiple filter files and can be combined with the other filter options.
 For example: `-agentlib:access-filter-file=/path/to/access-filter-file,caller-filter-file=/path/to/caller-filter-file,config-output-dir=...`
 
-### Specifying Configuration Files as Native Image Arguments
+### Specify Configuration Files as Native Image Arguments
 
 A directory containing configuration files that is not part of the class path can be specified to `native-image` via `-H:ConfigurationFileDirectories=/path/to/config-dir/`.
 This directory must directly contain all four files: `jni-config.json`, `reflect-config.json`, `proxy-config.json` and `resource-config.json`.
@@ -243,3 +253,8 @@ native-image-configure generate --input-dir=/path/to/config-dir-0/ --input-dir=/
 
 This command reads one set of configuration files from `/path/to/config-dir-0/` and another from `/path/to/config-dir-1/` and then writes a set of configuration files that contains both of their information to `/path/to/merged-config-dir/`.
 An arbitrary number of `--input-dir` arguments with sets of configuration files can be specified. See `native-image-configure help` for all options.
+
+### Experimental Options
+
+The native-image-agent has options which are currently experimental and might be enabled in future releases, but can also be changed or removed entirely.
+See the [ExperimentalAgentOptions.md](ExperimentalAgentOptions.md) guide.

docs/reference-manual/native-image/ExperimentalAgentOptions.md

@@ -0,0 +1,86 @@
+---
+layout: docs
+toc_group: native-image
+link_title: Experimental Agent Options
+permalink: /reference-manual/native-image/ExperimentalAgentOptions/
+---
+
+# Experimental Agent Options
+
+The `native-image-agent` tool has options which are currently experimental and might be enabled in future releases, but can also be changed or removed entirely.
+These options are described here.
+
+## Support For Predefined Classes
+
+Native-image needs all classes to be known at image build time (a "closed-world assumption").
+However, Java has support for loading new classes at runtime.
+To emulate class loading, the agent can be told to trace dynamically loaded classes and save their bytecode for later use by the image builder.
+This functionality can be enabled by adding `experimental-class-define-support` to the agent option string, e.g.: `-agentlib:native-image-agent=config-output-dir=config,experimental-class-define-support`
+Apart from the standard configuration files, the agent will create an `agent-extracted-predefined-classes` directory in the configuration output directory and write bytecode of newly loaded classes on the go.
+The configuration directory can then be used by image builder without additional tweaks,.
+The classes will be loaded during the image build, but will not be initialized or made available to the application.
+At runtime, if there is an attempt to load a class with the same name and bytecodes as one of the classes encountered during tracing, the predefined class will be supplied to the application.
+
+### Known Limitations
+
+ - Native images support "loading" a predefined class only once per execution, by just a single class loader.
+ - Predefined classes are initialized when they are "loaded" at runtime and cannot be initialized at build time.
+ - The agent collects all classes which are not loaded by one of the Java VM's built-in class loaders (with some exceptions), that is, from the class path or module path. This includes classes loaded by any custom class loaders.
+ - Classes that are generated with varying data in their name or bytecodes, such as sequential or random numbers or timestamps, can generally not be matched to predefined classes at runtime. In these cases, the way such classes are generated needs to be adjusted.
+
+## Printing Configuration With Origins
+
+For debugging, it may be useful to know the origin of certain configuration entries.
+By supplying `experimental-configuration-with-origins` to the agent option string, the agent will output configuration files with configuration entries broken down to the calling context (stack trace) they originate from in tree form.
+This option should be used in conjunction with `config-output-dir=<path>` to tell the agent where to output the configuration files.
+An example agent option string: `-agentlib:native-image-agent=config-output-dir=config-with-origins/,experimental-configuration-with-origins`
+
+## Omitting Configuration From The Agent's Output
+
+The agent can omit traced configuration entries present in existing configuration files.
+There are two ways to specify these existing configuration files:
+ - By using configuration files from the class path or module path. When `experimental-omit-config-from-classpath` is added to the agent option string, the class path and module path of the running application are scanned for `META-INF/native-image/**/*.json` configuration files.
+ - By explicitly pointing the agent to an existing configuration file directory using `config-to-omit=<path>`.
+
+## Generating Conditional Configuration Using The Agent
+
+The agent can, using a heuristic, generate configuration with reachability conditions on user specified classes.
+The agent will track configuration origins and try to deduce the conditions automatically.
+User classes are specified via an agent filter file (for more information on the format, see [Agent.md](Agent.md)).
+Additionally, the resulting configuration can further be filtered using another filter file.
+
+Currently, this feature supports two modes:
+ 1. Generating conditional configuration in a single run with the agent.
+ 2. Generating conditional configuration from multiple runs with the agent and finally merging the collected data.
+
+### Generating Conditional Configuration During An Agent Run
+
+To enable this mode, add `experimental-conditional-config-filter-file=<path>` to the agent's command line, where `<path>` points to an agent filter file.
+Classes that are considered included by this filter will be designated as user code classes.
+To further filter the generated configuration, you can use `conditional-config-class-filter-file=<path>`, where `<path>` is a path to an agent filter file.
+
+### Generating Conditional Configuration From Multiple Agent Runs
+
+Conditional configuration can be generated from multiple agent runs that reach different code paths in the application.
+Each agent run produces configuration with metadata. `native-image-configure` is then used to merge the collected data and produce a conditional configuration.
+To run the agent in this mode, add `experimental-conditional-config-part` to the agent's command line.
+Once all the agent runs have finished, you can generate a conditional configuration by invoking:
+```shell
+native-image-configure generate-conditional --user-code-filter=<path-to-filter-file> --class-name-filter=<path-to-filter-file> --input-dir=<path-to-agent-run-output-1> --input-dir=<path-to-agent-run-ouput-2> ... --output-dir=<path-to-resulting-conditional-config>
+```
+where:
+ - `--user-code-filter=<path-to-filter-file>`: path to an agent filter file that specifies user classes
+ - (optional) `--class-name-filter=<path-to-filter-file>`: path to an agent filter file that further filters the generated config
+
+### The Underlying Heuristics
+
+Conditions are generated using the call tree of the application. The heuristics work as follows:
+ 1. For each unique method, create a list of all nodes in the call tree that correspond to the method
+ 2. For each unique method, if the method has more than one call node in the tree:
+  - Find common configuration across all call nodes of that method
+  - For each call node of the method, propagate configuration that isn't common across these calls to the caller node
+ 3. Repeat 2. until an iteration produced no changes in the call tree.
+ 4. For each node that contains configuration, generate a conditional configuration entry with the method's class as the condition.
+
+The primary goal of this heuristic is to attempt to find where a method creates different configuration entries depending on the caller (for example, a method that wraps `Class.forName` calls.)
+This implies that the heuristic will not work well for code that generates configuration through a different dependency (for example, same method returns calls `Class.forName` with different class parameters depending on a system property).

substratevm/src/com.oracle.svm.configure/resources/Help.txt

@@ -5,6 +5,7 @@ This tool can be used to prepare a configuration of JNI, reflection and
 resources for a native-image build.
 
 Usage: native-image-configure generate [options]
+       native-image-configure generate-conditional [options]
        native-image-configure generate-filters [options]
        native-image-configure command-file <command-file-path>
        native-image-configure help
@@ -83,6 +84,25 @@ generate                  generates configuration file(s) from all inputs.
                           This option disables builtin heuristics that identify
                           further internal JNI, reflection and resource usages.
 
+generate-conditional      generates conditional configuration from data
+                          collected by previous agent runs.
+    --input-dir=<path>
+                          reads configuration and metadata from a directory that
+                          was previously populated by a run with the agent in
+                          the partial configuration mode.
+    --output-dir=<path>
+                          writes a set of conditional configuration files to
+                          the given path.
+    --user-code-filter=<path>
+                          specifies a filter file used to classify classes as
+                          user application classes. Generated conditions will
+                          only reference these classes.
+    --class-name-filter=<path>
+                          specifies a filter file used to exclude classes from
+                          the computed configuration. Both the configuration
+                          and the conditions in the configuration will be
+                          tested against this filter.
+
 generate-filters          builds a class filter according to the parameters.
                           Filter rules are created according to the order of
                           these parameters, and filter rules are applied in