8282567: Improve source-date handling in build system

Reviewed-by: erikj, sgehwolf

Author: magicus

doc/building.html

@@ -890,30 +890,28 @@ <h3 id="getting-help">Getting Help</h3>
 <p>If you need general help or advice about developing for the JDK, you can also contact the Adoption Group. See the section on <a href="#contributing-to-openjdk">Contributing to OpenJDK</a> for more information.</p>
 <h2 id="reproducible-builds">Reproducible Builds</h2>
 <p>Build reproducibility is the property of getting exactly the same bits out when building, every time, independent on who builds the product, or where. This is for many reasons a harder goal than it initially appears, but it is an important goal, for security reasons and others. Please see <a href="https://reproducible-builds.org">Reproducible Builds</a> for more information about the background and reasons for reproducible builds.</p>
-<p>Currently, it is not possible to build OpenJDK fully reproducibly, but getting there is an ongoing effort. There are some things you can do to minimize non-determinism and make a larger part of the build reproducible:</p>
+<p>Currently, it is not possible to build OpenJDK fully reproducibly, but getting there is an ongoing effort.</p>
+<p>An absolute prerequisite for building reproducible is to speficy a fixed build time, since time stamps are embedded in many file formats. This is done by setting the <code>SOURCE_DATE_EPOCH</code> environment variable, which is an <a href="https://reproducible-builds.org/docs/source-date-epoch/">industry standard</a>, that many tools, such as gcc, recognize, and use in place of the current time when generating output.</p>
+<p>To generate reproducible builds, you must set <code>SOURCE_DATE_EPOCH</code> before running <code>configure</code>. The value in <code>SOURCE_DATE_EPOCH</code> will be stored in the configuration, and used by <code>make</code>. Setting <code>SOURCE_DATE_EPOCH</code> before running <code>make</code> will have no effect on the build.</p>
+<p>You must also make sure your build does not rely on <code>configure</code>'s default adhoc version strings. Default adhoc version strings <code>OPT</code> segment include user name and source directory. You can either override just the <code>OPT</code> segment using <code>--with-version-opt=&lt;any fixed string&gt;</code>, or you can specify the entire version string using <code>--with-version-string=&lt;your version&gt;</code>.</p>
+<p>This is a typical example of how to build the JDK in a reproducible way:</p>
+<pre><code>export SOURCE_DATE_EPOCH=946684800
+bash configure --with-version-opt=adhoc
+make</code></pre>
+<p>Note that regardless if you specify a source date for <code>configure</code> or not, the JDK build system will set <code>SOURCE_DATE_EPOCH</code> for all build tools when building. If <code>--with-source-date</code> has the value <code>updated</code> (which is the default unless <code>SOURCE_DATE_EPOCH</code> is found by in the environment by <code>configure</code>), the source date value will be determined at build time.</p>
+<p>There are several aspects of reproducible builds that can be individually adjusted by <code>configure</code> arguments. If any of these are given, they will override the value derived from <code>SOURCE_DATE_EPOCH</code>. These arguments are:</p>
 <ul>
-<li>Turn on build system support for reproducible builds</li>
+<li><p><code>--with-source-date</code></p>
+<p>This option controls how the JDK build sets <code>SOURCE_DATE_EPOCH</code> when building. It can be set to a value describing a date, either an epoch based timestamp as an integer, or a valid ISO-8601 date.</p>
+<p>It can also be set to one of the special values <code>current</code>, <code>updated</code> or <code>version</code>. <code>current</code> means that the time of running <code>configure</code> will be used. <code>version</code> will use the nominal release date for the current JDK version. <code>updated</code>, which means that <code>SOURCE_DATE_EPOCH</code> will be set to the current time each time you are running <code>make</code>. All choices, except for <code>updated</code>, will set a fixed value for the source date timestamp.</p>
+<p>When <code>SOURCE_DATE_EPOCH</code> is set, the default value for <code>--with-source-date</code> will be the value given by <code>SOURCE_DATE_EPOCH</code>. Otherwise, the default value is <code>updated</code>.</p></li>
+<li><p><code>--with-hotspot-build-time</code></p>
+<p>This option controls the build time string that will be included in the hotspot library (<code>libjvm.so</code> or <code>jvm.dll</code>). When the source date is fixed (e.g. by setting <code>SOURCE_DATE_EPOCH</code>), the default value for <code>--with-hotspot-build-time</code> will be an ISO 8601 representation of that time stamp. Otherwise the default value will be the current time when building hotspot.</p></li>
+<li><p><code>--with-copyright-year</code></p>
+<p>This option controls the copyright year in some generated text files. When the source date is fixed (e.g. by setting <code>SOURCE_DATE_EPOCH</code>), the default value for <code>--with-copyright-year</code> will be the year of that time stamp. Otherwise the default is the current year at the time of running configure. This can be overridden by <code>--with-copyright-year=&lt;year&gt;</code>.</p></li>
+<li><p><code>--enable-reproducible-build</code></p>
+<p>This option controls some additional behavior needed to make the build reproducible. When the source date is fixed (e.g. by setting <code>SOURCE_DATE_EPOCH</code>), this flag will be turned on by default. Otherwise, the value is determined by heuristics. If it is explicitly turned off, the build might not be reproducible.</p></li>
 </ul>
-<p>Add the flag <code>--enable-reproducible-build</code> to your <code>configure</code> command line. This will turn on support for reproducible builds where it could otherwise be lacking.</p>
-<ul>
-<li>Do not rely on <code>configure</code>'s default adhoc version strings</li>
-</ul>
-<p>Default adhoc version strings OPT segment include user name, source directory and timestamp. You can either override just the OPT segment using <code>--with-version-opt=&lt;any fixed string&gt;</code>, or you can specify the entire version string using <code>--with-version-string=&lt;your version&gt;</code>.</p>
-<ul>
-<li>Specify how the build sets <code>SOURCE_DATE_EPOCH</code></li>
-</ul>
-<p>The JDK build system will set the <code>SOURCE_DATE_EPOCH</code> environment variable during building, depending on the value of the <code>--with-source-date</code> option for <code>configure</code>. The default value is <code>updated</code>, which means that <code>SOURCE_DATE_EPOCH</code> will be set to the current time each time you are running <code>make</code>.</p>
-<p>The <a href="https://reproducible-builds.org/docs/source-date-epoch/"><code>SOURCE_DATE_EPOCH</code> environment variable</a> is an industry standard, that many tools, such as gcc, recognize, and use in place of the current time when generating output.</p>
-<p>For reproducible builds, you need to set this to a fixed value. You can use the special value <code>version</code> which will use the nominal release date for the current JDK version, or a value describing a date, either an epoch based timestamp as an integer, or a valid ISO-8601 date.</p>
-<p><strong>Hint:</strong> If your build environment already sets <code>SOURCE_DATE_EPOCH</code>, you can propagate this using <code>--with-source-date=$SOURCE_DATE_EPOCH</code>.</p>
-<ul>
-<li>Specify a hotspot build time</li>
-</ul>
-<p>Set a fixed hotspot build time. This will be included in the hotspot library (<code>libjvm.so</code> or <code>jvm.dll</code>) and defaults to the current time when building hotspot. Use <code>--with-hotspot-build-time=&lt;any fixed string&gt;</code> for reproducible builds. It's a string so you don't need to format it specifically, so e.g. <code>n/a</code> will do. Another solution is to use the <code>SOURCE_DATE_EPOCH</code> variable, e.g. <code>--with-hotspot-build-time=$(date --date=@$SOURCE_DATE_EPOCH)</code>.</p>
-<ul>
-<li>Copyright year</li>
-</ul>
-<p>The copyright year in some generated text files are normally set to the current year. This can be overridden by <code>--with-copyright-year=&lt;year&gt;</code>. For fully reproducible builds, this needs to be set to a fixed value.</p>
 <h2 id="hints-and-suggestions-for-advanced-users">Hints and Suggestions for Advanced Users</h2>
 <h3 id="bash-completion">Bash Completion</h3>
 <p>The <code>configure</code> and <code>make</code> commands tries to play nice with bash command-line completion (using <code>&lt;tab&gt;</code> or <code>&lt;tab&gt;&lt;tab&gt;</code>). To use this functionality, make sure you enable completion in your <code>~/.bashrc</code> (see instructions for bash in your operating system).</p>

doc/building.md

@@ -1527,57 +1527,85 @@ https://reproducible-builds.org) for more information about the background and
 reasons for reproducible builds.
 
 Currently, it is not possible to build OpenJDK fully reproducibly, but getting
-there is an ongoing effort. There are some things you can do to minimize
-non-determinism and make a larger part of the build reproducible:
+there is an ongoing effort.
+
+An absolute prerequisite for building reproducible is to speficy a fixed build
+time, since time stamps are embedded in many file formats. This is done by
+setting the `SOURCE_DATE_EPOCH` environment variable, which is an [industry
+standard]( https://reproducible-builds.org/docs/source-date-epoch/), that many
+tools, such as gcc, recognize, and use in place of the current time when
+generating output.
+
+To generate reproducible builds, you must set `SOURCE_DATE_EPOCH` before running
+`configure`. The value in `SOURCE_DATE_EPOCH` will be stored in the
+configuration, and used by `make`. Setting `SOURCE_DATE_EPOCH` before running
+`make` will have no effect on the build.
+
+You must also make sure your build does not rely on `configure`'s default adhoc
+version strings. Default adhoc version strings `OPT` segment include user name
+and source directory. You can either override just the `OPT` segment using
+`--with-version-opt=<any fixed string>`, or you can specify the entire version
+string using `--with-version-string=<your version>`.
 
-  * Turn on build system support for reproducible builds
+This is a typical example of how to build the JDK in a reproducible way:
+
+```
+export SOURCE_DATE_EPOCH=946684800
+bash configure --with-version-opt=adhoc
+make
+```
 
-Add the flag `--enable-reproducible-build` to your `configure` command line.
-This will turn on support for reproducible builds where it could otherwise be
-lacking.
+Note that regardless if you specify a source date for `configure` or not, the
+JDK build system will set `SOURCE_DATE_EPOCH` for all build tools when building.
+If `--with-source-date` has the value `updated` (which is the default unless
+`SOURCE_DATE_EPOCH` is found by in the environment by `configure`), the source
+date value will be determined at build time.
 
-  * Do not rely on `configure`'s default adhoc version strings
+There are several aspects of reproducible builds that can be individually
+adjusted by `configure` arguments. If any of these are given, they will override
+the value derived from `SOURCE_DATE_EPOCH`. These arguments are:
 
-Default adhoc version strings OPT segment include user name, source directory
-and timestamp. You can either override just the OPT segment using
-`--with-version-opt=<any fixed string>`, or you can specify the entire version
-string using `--with-version-string=<your version>`.
+ * `--with-source-date`
 
-  * Specify how the build sets `SOURCE_DATE_EPOCH`
+    This option controls how the JDK build sets `SOURCE_DATE_EPOCH` when
+    building. It can be set to a value describing a date, either an epoch based
+    timestamp as an integer, or a valid ISO-8601 date.
 
-The JDK build system will set the `SOURCE_DATE_EPOCH` environment variable
-during building, depending on the value of the `--with-source-date` option for
-`configure`. The default value is `updated`, which means that
-`SOURCE_DATE_EPOCH` will be set to the current time each time you are running
-`make`.
+    It can also be set to one of the special values `current`, `updated` or
+    `version`. `current` means that the time of running `configure` will be
+    used. `version` will use the nominal release date for the current JDK
+    version. `updated`, which means that `SOURCE_DATE_EPOCH` will be set to the
+    current time each time you are running `make`. All choices, except for
+    `updated`, will set a fixed value for the source date timestamp.
 
-The [`SOURCE_DATE_EPOCH` environment variable](
-https://reproducible-builds.org/docs/source-date-epoch/) is an industry
-standard, that many tools, such as gcc, recognize, and use in place of the
-current time when generating output.
+    When `SOURCE_DATE_EPOCH` is set, the default value for `--with-source-date`
+    will be the value given by `SOURCE_DATE_EPOCH`. Otherwise, the default value
+    is `updated`.
 
-For reproducible builds, you need to set this to a fixed value. You can use the
-special value `version` which will use the nominal release date for the current
-JDK version, or a value describing a date, either an epoch based timestamp as an
-integer, or a valid ISO-8601 date.
+ * `--with-hotspot-build-time`
 
-**Hint:** If your build environment already sets `SOURCE_DATE_EPOCH`, you can
-propagate this using `--with-source-date=$SOURCE_DATE_EPOCH`.
+    This option controls the build time string that will be included in the
+    hotspot library (`libjvm.so` or `jvm.dll`). When the source date is fixed
+    (e.g. by setting `SOURCE_DATE_EPOCH`), the default value for
+    `--with-hotspot-build-time` will be an ISO 8601 representation of that time
+    stamp. Otherwise the default value will be the current time when building
+    hotspot.
 
-  * Specify a hotspot build time
+ * `--with-copyright-year`
 
-Set a fixed hotspot build time. This will be included in the hotspot library
-(`libjvm.so` or `jvm.dll`) and defaults to the current time when building
-hotspot. Use `--with-hotspot-build-time=<any fixed string>` for reproducible
-builds. It's a string so you don't need to format it specifically, so e.g. `n/a`
-will do. Another solution is to use the `SOURCE_DATE_EPOCH` variable, e.g.
-`--with-hotspot-build-time=$(date --date=@$SOURCE_DATE_EPOCH)`.
+    This option controls the copyright year in some generated text files. When
+    the source date is fixed (e.g. by setting `SOURCE_DATE_EPOCH`), the default
+    value for `--with-copyright-year` will be the year of that time stamp.
+    Otherwise the default is the current year at the time of running configure.
+    This can be overridden by `--with-copyright-year=<year>`.
 
-  * Copyright year
+ * `--enable-reproducible-build`
 
-The copyright year in some generated text files are normally set to the current
-year. This can be overridden by `--with-copyright-year=<year>`. For fully
-reproducible builds, this needs to be set to a fixed value.
+    This option controls some additional behavior needed to make the build
+    reproducible. When the source date is fixed (e.g. by setting
+    `SOURCE_DATE_EPOCH`), this flag will be turned on by default. Otherwise, the
+    value is determined by heuristics. If it is explicitly turned off, the build
+    might not be reproducible.
 
 ## Hints and Suggestions for Advanced Users
 

make/InitSupport.gmk

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2011, 2021, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2011, 2022, Oracle and/or its affiliates. All rights reserved.
 # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
 #
 # This code is free software; you can redistribute it and/or modify it
@@ -310,17 +310,16 @@ else # $(HAS_SPEC)=true
   # level of reproducible builds
   define SetupReproducibleBuild
     ifeq ($$(SOURCE_DATE), updated)
-      SOURCE_DATE := $$(shell $$(DATE) +"%s")
-    endif
-    export SOURCE_DATE_EPOCH := $$(SOURCE_DATE)
-    ifeq ($$(IS_GNU_DATE), yes)
-      export SOURCE_DATE_ISO_8601 := $$(shell $$(DATE) --utc \
-                                                       --date="@$$(SOURCE_DATE_EPOCH)" \
-                                                       +"%Y-%m-%dT%H:%M:%SZ" 2> /dev/null)
-    else
-      export SOURCE_DATE_ISO_8601 := $$(shell $$(DATE) -u \
-                                                       -j -f "%s" "$$(SOURCE_DATE_EPOCH)" \
-                                                       +"%Y-%m-%dT%H:%M:%SZ" 2> /dev/null)
+      # For static values of SOURCE_DATE (not "updated"), these are set in spec.gmk
+      export SOURCE_DATE_EPOCH := $$(shell $$(DATE) +"%s")
+      ifeq ($$(IS_GNU_DATE), yes)
+        export SOURCE_DATE_ISO_8601 := $$(shell $$(DATE) --utc \
+            --date="@$$(SOURCE_DATE_EPOCH)" +"$$(ISO_8601_FORMAT_STRING)" \
+            2> /dev/null)
+      else
+        export SOURCE_DATE_ISO_8601 := $$(shell $$(DATE) -u -j -f "%s" \
+            "$$(SOURCE_DATE_EPOCH)" +"$$(ISO_8601_FORMAT_STRING)" 2> /dev/null)
+      endif
     endif
   endef
 

make/autoconf/configure.ac

@@ -152,6 +152,7 @@ BOOTJDK_SETUP_DOCS_REFERENCE_JDK
 #
 ###############################################################################
 
+JDKOPT_SETUP_REPRODUCIBLE_BUILD
 JDKOPT_SETUP_JDK_OPTIONS
 
 ###############################################################################
@@ -207,7 +208,6 @@ PLATFORM_SETUP_OPENJDK_TARGET_BITS
 PLATFORM_SETUP_OPENJDK_TARGET_ENDIANNESS
 
 # Configure flags for the tools. Need to know if we should build reproducible.
-JDKOPT_SETUP_REPRODUCIBLE_BUILD
 FLAGS_SETUP_FLAGS
 
 # Setup debug symbols (need objcopy from the toolchain for that)

make/autoconf/help.m4

@@ -296,6 +296,13 @@ AC_DEFUN_ONCE([HELP_PRINT_SUMMARY_AND_WARNINGS],
   printf "* OpenJDK target: OS: $OPENJDK_TARGET_OS, CPU architecture: $OPENJDK_TARGET_CPU_ARCH, address length: $OPENJDK_TARGET_CPU_BITS\n"
   printf "* Version string: $VERSION_STRING ($VERSION_SHORT)\n"
 
+  if test "x$SOURCE_DATE" != xupdated; then
+    source_date_info="$SOURCE_DATE ($SOURCE_DATE_ISO_8601)"
+  else
+    source_date_info="Determined at build time"
+  fi
+  printf "* Source date:    $source_date_info\n"
+
   printf "\n"
   printf "Tools summary:\n"
   if test "x$OPENJDK_BUILD_OS" = "xwindows"; then

make/autoconf/hotspot.m4

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2011, 2020, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2011, 2022, Oracle and/or its affiliates. All rights reserved.
 # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
 #
 # This code is free software; you can redistribute it and/or modify it
@@ -114,12 +114,26 @@ AC_DEFUN_ONCE([HOTSPOT_SETUP_MISC],
     HOTSPOT_TARGET_CPU_ARCH=zero
   fi
 
+
   AC_ARG_WITH([hotspot-build-time], [AS_HELP_STRING([--with-hotspot-build-time],
-  [timestamp to use in hotspot version string, empty for on-the-fly @<:@empty@:>@])])
+  [timestamp to use in hotspot version string, empty means determined at build time @<:@source-date/empty@:>@])])
+
+  AC_MSG_CHECKING([what hotspot build time to use])
 
   if test "x$with_hotspot_build_time" != x; then
     HOTSPOT_BUILD_TIME="$with_hotspot_build_time"
+    AC_MSG_RESULT([$HOTSPOT_BUILD_TIME (from --with-hotspot-build-time)])
+  else
+    if test "x$SOURCE_DATE" = xupdated; then
+      HOTSPOT_BUILD_TIME=""
+      AC_MSG_RESULT([determined at build time (default)])
+    else
+      # If we have a fixed value for SOURCE_DATE, use it as default
+      HOTSPOT_BUILD_TIME="$SOURCE_DATE_ISO_8601"
+      AC_MSG_RESULT([$HOTSPOT_BUILD_TIME (from --with-source-date)])
+    fi
   fi
+
   AC_SUBST(HOTSPOT_BUILD_TIME)