Skip to content

Commit ed6d791

Browse files
author
Aleksandar Gradinac
committed
[GR-34422] Use toolchains for musl cross-compilation and improve docs for building static native-images.
PullRequest: graal/10047
2 parents 0127f23 + 852ceca commit ed6d791

File tree

7 files changed

+112
-114
lines changed

7 files changed

+112
-114
lines changed

docs/reference-manual/embedding/embed-languages.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ permalink: /reference-manual/embed-languages/
1919
* [Embed languages in Guest Languages](#embed-languages-in-guest-languages)
2020
* [Build a Shell for Many Languages](#build-a-shell-for-many-languages)
2121
* [Step Through with Execution Listeners](#step-through-with-execution-listeners)
22-
* [Configure Sandbox Resource Limits](#configure-sandbox-resource-limits)
22+
* [Enterprise Sandbox Resource Limits](#enterprise-sandbox-resource-limits)
2323

2424
The GraalVM Polyglot API lets you embed and run code from guest languages in JVM-based host applications.
2525

@@ -632,5 +632,5 @@ In this code:
632632
- The `context.eval()` call evaluates a specified snippet of guest language code.
633633
- The `listener.close()` closes a listener earlier, however execution listeners are automatically closed with the engine.
634634

635-
<!-- Configure Sandbox Resource Limits -->
635+
<!-- Enterprise Sandbox Resource Limits -->
636636
{% include_relative sandbox-options.md %}

docs/reference-manual/native-image/CertificateManagement.md

Lines changed: 24 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -6,60 +6,40 @@ permalink: /reference-manual/native-image/CertificateManagement/
66
---
77
# Certificate Management in Native Image
88

9-
Native-image provides multiple ways to specify the certificate file used to
10-
define the default TrustStore. In the following sections we describe the
11-
available buildtime and runtime options. Note the default behavior for
12-
native-image is to capture and use the default TrustStore from the buildtime
13-
host environment.
9+
Native Image provides multiple ways to specify the certificate file used to define the default TrustStore.
10+
In the following sections we describe the available build-time and run-time options.
11+
Note: The default behavior for `native-image` is to capture and use the default TrustStore from the build-time host environment.
1412

15-
## Buildtime Options
13+
## Build-time Options
1614

17-
During the image building process, native-image captures the host environment's
18-
default TrustStore and embeds it into the native image. This TrustStore is
19-
by default created from the root certificate file provided within the JDK, but
20-
can be changed to use a different certificate file by setting the buildtime
21-
system property "javax.net.ssl.trustStore" (see [Properties](Properties.md) for
22-
how to do so).
15+
During the image building process, the `native-image` builder captures the host environment's default TrustStore and embeds it into the native executable.
16+
This TrustStore is by default created from the root certificate file provided within the JDK, but can be changed to use a different certificate file by setting the build-time system property `javax.net.ssl.trustStore` (see [Properties](Properties.md) for how to do it).
2317

24-
Since the contents of the buildtime certificate file is embedded into the image
25-
executable, the file itself does not need to present in the target environment.
18+
Since the contents of the build-time certificate file is embedded into the native executable, the file itself does not need to be present in the target environment.
2619

27-
## Runtime Options
20+
## Run-time Options
2821

29-
The certificate file can also be changed dynamically at runtime via setting
30-
the "javax.net.ssl.trustStore\*" system properties.
22+
The certificate file can also be changed dynamically at run time via setting the `javax.net.ssl.trustStore\*` system properties.
3123

32-
If any of the following system properties are set during image execution,
33-
native-image also requires "javax.net.ssl.trustStore" to be set and for it
34-
to point to an accessible certificate file:
35-
- javax.net.ssl.trustStore
36-
- javax.net.ssl.trustStoreType
37-
- javax.net.ssl.trustStoreProvider
38-
- javax.net.ssl.trustStorePassword
24+
If any of the following system properties are set during the image execution, `native-image` also requires `javax.net.ssl.trustStore` to be set, and for it to point to an accessible certificate file:
25+
- `javax.net.ssl.trustStore`
26+
- `javax.net.ssl.trustStoreType`
27+
- `javax.net.ssl.trustStoreProvider`
28+
- `javax.net.ssl.trustStorePassword`
3929

40-
If any of these properties are set and "javax.net.ssl.trustStore" does not point
41-
to an accessible file, then an UnsupportedFeatureError will be thrown.
30+
If any of these properties are set and `javax.net.ssl.trustStore` does not point to an accessible file, then an `UnsupportedFeatureError` will be thrown.
4231

43-
Note that this behavior is different than OpenJDK. When the
44-
"javax.net.ssl.trustStore" system property is unset/invalid, OpenJDK will
45-
fallback to using a certificate file shipped within the JDK; however, such
46-
files will not be present alongside the image executable and hence cannot be
47-
used as a fallback.
32+
Note that this behavior is different than OpenJDK.
33+
When the `javax.net.ssl.trustStore` system property is unset or invalid, OpenJDK will fallback to using a certificate file shipped within the JDK.
34+
However, such files will not be present alongside the image executable and hence cannot be used as a fallback.
4835

49-
During the execution, it also possible to dynamically change the
50-
"javax.net.ssl.trustStore\*" properties and for the default TrustStore to be
51-
updated accordingly.
36+
During the execution, it also possible to dynamically change the `javax.net.ssl.trustStore\*` properties and for the default TrustStore to be updated accordingly.
5237

53-
Finally, whenever all of the "javax.net.ssl.trustStore\*" system properties
54-
listed above are unset, the default TrustStore will be the one captured during
55-
buildtime, as described in the [prior section](#buildtime-options).
38+
Finally, whenever all of the `javax.net.ssl.trustStore\*` system properties listed above are unset, the default TrustStore will be the one captured during the build time, as described in the [prior section](#build-time-options).
5639

5740
## Untrusted Certificates
5841

59-
During the image building process, a list of untrusted certificates is loaded
60-
from the file <java.home>/lib/security/blacklisted.certs. This file is used
61-
when validating certificates at both buildtime and runtime. In other words,
62-
when a new certificate file is specified at runtime via setting the
63-
"javax.net.ssl.trustStore\*" system properties, the new certificates will still
64-
be checked against the <java.home>/lib/security/blacklisted.certs loaded at
65-
image buildtime.
42+
During the image building process, a list of untrusted certificates is loaded from the file `<java.home>/lib/security/blacklisted.certs`.
43+
This file is used when validating certificates at both build time and run time.
44+
In other words, when a new certificate file is specified at run time via setting the `javax.net.ssl.trustStore\*` system properties, the new certificates will still be checked against the `<java.home>/lib/security/blacklisted.certs` loaded at
45+
image build time.

docs/reference-manual/native-image/Reflection.md

Lines changed: 18 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,12 @@ toc_group: native-image
44
link_title: Reflection on Native Image
55
permalink: /reference-manual/native-image/Reflection/
66
---
7+
8+
* [Automatic Detection](#automatic-detection)
9+
* [Manual Configuration](#manual-configuration)
10+
* [Conditional Configuration](#conditional-configuration)
11+
* [Configuration with Features](#configuration-with-features)
12+
713
# Reflection Use in Native Images
814

915
Java reflection support (the `java.lang.reflect.*` API) enables Java code to examine its own classes, methods, fields and their properties at run time.
@@ -128,7 +134,9 @@ Also, `-H:ReflectionConfigurationResources` can be specified to load one or seve
128134

129135
### Conditional Configuration
130136

131-
With conditional configuraiton, a class configuration entry is applied only if a provided `condition` is satisfied. The only currently supported condition is `typeReachable`, which enables the configuration entry if the specified type is reachable through other code. For example, to support reflective access to `sun.misc.Unsafe.theUnsafe` only when `io.netty.util.internal.PlatformDependent0` is reachable, the configuration should look like:
137+
With conditional configuration, a class configuration entry is applied only if a provided `condition` is satisfied.
138+
The only currently supported condition is `typeReachable`, which enables the configuration entry if the specified type is reachable through other code.
139+
For example, to support reflective access to `sun.misc.Unsafe.theUnsafe` only when `io.netty.util.internal.PlatformDependent0` is reachable, the configuration should look like:
132140

133141
```json
134142
{
@@ -140,11 +148,14 @@ With conditional configuraiton, a class configuration entry is applied only if a
140148
}
141149
```
142150

143-
Conditional configuration is the *preferred* way to specify reflection configuration: if code doing a reflective access is not reachable, it is unnecessary to include its corresponding reflection entry. The consistent usage of `condition` results in *smaller binaries* and *better build times* as the image builder can selectively include reflectively accessed code.
151+
Conditional configuration is the **preferred** way to specify reflection configuration: if code doing a reflective access is not reachable, it is unnecessary to include its corresponding reflection entry.
152+
The consistent usage of `condition` results in *smaller binaries* and *better build times* as the image builder can selectively include reflectively accessed code.
144153

145-
If a `condition` is omitted, the element is always included. When the same `condition` is used for two distinct elements in two configuration entries, both elements will be included when the condition is satisfied. When a configuration entry should be enabled if one of several types are reachable, it is necessary to add two configuration entries: one entry for each condition.
154+
If a `condition` is omitted, the element is always included.
155+
When the same `condition` is used for two distinct elements in two configuration entries, both elements will be included when the condition is satisfied.
156+
When a configuration entry should be enabled if one of several types are reachable, it is necessary to add two configuration entries: one entry for each condition.
146157

147-
When used with [assisted configuration](BuildConfiguration.md#assisted-configuration-of-native-image-builds), conditional entries of existing configuration will not be fused with agent-collected entries as agent-collected entries.
158+
When used with [assisted configuration](BuildConfiguration.md#assisted-configuration-of-native-image-builds), conditional entries of existing configuration will not be fused with agent-collected entries.
148159

149160
### Configuration with Features
150161

@@ -173,4 +184,6 @@ Reflection can be used without restrictions during a native image generation, fo
173184
At this point, code can collect information about methods and fields and store them in their own data structures, which are then reflection-free at run time.
174185

175186
### Unsafe Accesses
176-
The `Unsafe` class, although its use is discouraged, provides direct access to the memory of Java objects. The `Unsafe.objectFieldOffset()` method provides the offset of a field within a Java object. Note that the offsets that are queried during native image generation can be different from the offsets at run time.
187+
The `Unsafe` class, although its use is discouraged, provides direct access to the memory of Java objects.
188+
The `Unsafe.objectFieldOffset()` method provides the offset of a field within a Java object.
189+
Note that the offsets that are queried during native image generation can be different from the offsets at run time.

docs/reference-manual/native-image/StaticImages.md

Lines changed: 60 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -4,58 +4,76 @@ toc_group: native-image
44
link_title: Static Native Images
55
permalink: /reference-manual/native-image/StaticImages/
66
---
7-
# Static Native Images
7+
# Static and Mostly Static Images
88

9-
Static native images are statically linked binaries which can be used without any additional library dependencies.
10-
This makes them suitable for use in a Docker container.
9+
With GraalVM Native Image you can create static or mostly static images, depending on the purposes.
10+
11+
**Static native images** are statically linked binaries which can be used without any additional library dependencies.
12+
This makes them easier to distribute and to deploy on slim or distroless container images.
13+
They are created by statically linking against [musl-libc](https://musl.libc.org/), a lightweight, fast and simple `libc` implementation.
14+
15+
**Mostly static native images** statically link against all libraries except `libc`.
16+
This approach is ideal for deploying such native images on distroless container images.
17+
Note that it currently only works when linking against `glibc`.
1118

1219
## Prerequisites
13-
- Right now, this only works on Linux AMD64 on Java 11.
14-
- You will need `gcc`, `make`, and `configure`.
15-
- Create a directory that will hold the libraries you build. You will refer to this directory as `${RESULT_DIR}`.
16-
- Download the latest `musl` release [here](https://musl.libc.org/). This document will use `musl-1.2.0`.
17-
- Download the latest `zlib` release [here](https://zlib.net/). This document will use `zlib-1.2.11`.
1820

19-
## Build a Static Native Image
21+
- Linux AMD64 operating system
22+
- GraalVM distribution for Java 11 with [Native Image support](README.md#install-native-image)
23+
- A 64-bit `musl` toolchain, `make`, and `configure`
24+
- The latest `zlib` library
2025

21-
If you have `musl-gcc` on the path, you can build a native image statically linked against `muslc` with the following options: `--static --libc=musl`.
22-
To verify that `musl-gcc` is on the path, run `musl-gcc -v`.
26+
## Preparation
2327

24-
To build a static native image, use:
25-
```shell
26-
native-image --static --libc=musl [other arguments] Class
27-
```
28+
You should get the `musl` toolchain first, and then compile and install `zlib` into the toolchain.
29+
30+
1. Download the `musl` toolchain from [musl.cc](musl.cc). [This one](http://musl.cc/x86_64-linux-musl-native.tgz) is recommended. Extract the toolchain to a directory of your choice. This directory will be referred as `$TOOLCHAIN_DIR`.
31+
2. Download the latest `zlib` library sources from [here](https://zlib.net/) and extract them. This guide uses `zlib-1.2.11`.
32+
3. Set the following environment variable:
33+
```bash
34+
CC=$TOOLCHAIN_DIR/bin/gcc
35+
```
36+
4. Change into the `zlib` directory, and then run the following commands to compile and install `zlib` into the toolchain:
37+
```bash
38+
./configure --prefix=$TOOLCHAIN_DIR --static
39+
make
40+
make install
41+
```
42+
43+
## Build Static Native Image
2844

29-
## Build a Mostly Static Native Image
45+
1. First, ensure `$TOOLCHAIN_DIR/bin` is present on your `PATH` variable.
46+
To verify this, run:
47+
```bash
48+
x86_64-linux-musl-gcc
49+
```
50+
You should get a similar output printed:
51+
```bash
52+
x86_64-linux-musl-gcc: fatal error: no input files
53+
compilation terminated.
54+
```
3055

31-
As of GraalVM version 20.2, you can build a “mostly static” native image which link statically everything except `libc`. Native images built this way are convenient to run in Docker containers, for example, based on
32-
[distroless minimal Linux, glibc-based systems](https://github.com/GoogleContainerTools/distroless/blob/master/base/README.md).
56+
2. Build a static native image by using this command:
57+
```shell
58+
native-image --static --libc=musl [other arguments] Class
59+
```
3360

34-
To build a mostly-static native image native image, use:
61+
## Build Mostly Static Native Image
62+
63+
As of GraalVM version 20.2, you can build a “mostly static” native image which statically links everything except `libc`.
64+
Statically linking all your libraries except `glibc` ensures your application has all the libraries it needs to run on any Linux `glibc`-based distribution.
65+
66+
To build a mostly-static native image native image, use this command:
3567
```shell
3668
native-image -H:+StaticExecutableWithDynamicLibC [other arguments] Class
3769
```
3870

39-
#### Building musl
40-
- Extract the musl release `tarball` and `cd` into the extracted directory.
41-
- Run `./configure --disable-shared --prefix=${RESULT_DIR}`.
42-
- Run `make`.
43-
- Run `make install`.
44-
Other than building `musl` libraries, the build also creates a `gcc` wrapper called `musl-gcc` in the `${RESULT_DIR}/bin` directory.
45-
You should now put this wrapper on your `PATH` by running `export PATH=$PATH:${RESULT_DIR}/bin`.
46-
47-
#### Building zlib
48-
- Extract the zlib release `tarball` and `cd` into the extracted directory.
49-
- You need to compile zlib and link it against musl so set `CC` to `musl-gcc`: `export CC=musl-gcc`.
50-
- Run `./configure --static --prefix=${RESULT_DIR}`.
51-
- Run `make`.
52-
- Run `make install`.
53-
54-
#### Getting libstdc++
55-
`libstdc++` is obtained by building gcc. There are multiple approaches to obtaining it:
56-
1. Build gcc with `musl-gcc`.
57-
2. Use `libstdc++.a` from your distribution. If you choose this path, check the [FAQs](https://www.musl-libc.org/faq.html) page, "How do I use the musl-gcc wrapper?":
58-
> The existing libstdc++ is actually compatible with musl in most cases and could be used by copying it into the musl library path, but the C++ header files are usually not compatible.
59-
> Since you do not need C++ header files, this approach should work. If you run into issues, make sure they are not caused by your ditribution's `libstdc++.a`.
60-
3. Take `libstdc++.a` from Alpine.
61-
In each case, `libstdc++.a` must be placed in `${RESULT_DIR}/lib`.
71+
> Note: This currently only works for `glibc`.
72+
73+
## Frequently Asked Questions
74+
75+
### What is the recommended base Docker image for deploying a static or mostly static native image?
76+
77+
A fully static native image gives you the most flexibility to choose a base image - it can run on anything including a `FROM scratch` image.
78+
A mostly static native image requires a container image that provides `glibc`, but has no additional requirements.
79+
In both cases, choosing the base image mostly depends on what your particular native image needs without having to worry about run-time library dependencies.

substratevm/ci_includes/gate.hocon

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -28,17 +28,19 @@ builds += [
2828
${labsjdk-ce-11} ${svm-common-linux-gate} ${linux-deploy} {
2929
name: "gate-svm-build-ce-11"
3030
downloads: {
31-
"MUSL_LIBS": {
32-
"name": "musl-libs",
31+
"MUSL_TOOLCHAIN": {
32+
"name": "musl-toolchain",
3333
"version": "1.0",
3434
"platformspecific": true
3535
}
3636
}
3737
environment : {
38-
PATH : "$MUSL_LIBS/bin:$PATH"
38+
# Note that we must add the toolchain to the end of the PATH so that the system gcc still remains the first choice
39+
# for building the rest of GraalVM. The musl toolchain also provides a gcc executable that would shadow the system one
40+
# if it were added at the start of the PATH.
41+
PATH : "$PATH:$MUSL_TOOLCHAIN/bin"
3942
}
4043
run: [
41-
["$MUSL_LIBS/fix_paths.sh"]
4244
${svm-cmd-gate} ["build,helloworld,test,nativeimagehelp,muslcbuild"]
4345
]
4446
}

substratevm/mx.substratevm/mx_substratevm.py

Lines changed: 1 addition & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -116,16 +116,6 @@ def svmbuild_dir(suite=None):
116116
return join(suite.dir, 'svmbuild')
117117

118118

119-
def is_musl_gcc_wrapper_on_path():
120-
mx.logv('Probing if musl-gcc exists on path.')
121-
throwaway_capture = mx.LinesOutputCapture()
122-
try:
123-
ret_code = mx.run(['musl-gcc', '-v'], nonZeroIsFatal=False, out=throwaway_capture, err=throwaway_capture)
124-
return ret_code == 0
125-
except OSError as _:
126-
return False
127-
128-
129119
def is_musl_supported():
130120
jdk = mx.get_jdk(tag='default')
131121
if mx.is_linux() and mx.get_arch() == "amd64" and mx.get_jdk(tag='default').javaCompliance == '11':
@@ -243,10 +233,7 @@ def vm_executable_path(executable, config=None):
243233

244234
def run_musl_basic_tests():
245235
if is_musl_supported():
246-
if is_musl_gcc_wrapper_on_path():
247-
helloworld(['--output-path', svmbuild_dir(), '--static', '--libc=musl'])
248-
else:
249-
mx.abort('Attempted to run musl tests without a musl-gcc wrapper.')
236+
helloworld(['--output-path', svmbuild_dir(), '--static', '--libc=musl'])
250237

251238

252239
@contextmanager
@@ -1578,7 +1565,5 @@ def javac_image(args):
15781565
doc_string = "Runs a musl based Hello World static native-image with custom build arguments."
15791566
@mx.command(suite.name, command_name='muslhelloworld', usage_msg='[options]', doc_function=lambda: doc_string)
15801567
def musl_helloworld(args, config=None):
1581-
if not is_musl_gcc_wrapper_on_path():
1582-
mx.abort('musl-gcc wrapper not detected on path. Cannot run musl helloworld. Please consult substratevm/StaticImages.md')
15831568
final_args = ['--static', '--libc=musl'] + args
15841569
run_helloworld_command(final_args, config, 'muslhelloworld')

substratevm/src/com.oracle.svm.core.posix/src/com/oracle/svm/core/posix/linux/libc/MuslLibC.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ public List<String> getAdditionalQueryCodeCompilerOptions() {
4949

5050
@Override
5151
public String getTargetCompiler() {
52-
return "musl-gcc";
52+
return "x86_64-linux-musl-gcc";
5353
}
5454

5555
@Override

0 commit comments

Comments
 (0)