Skip to content

Conversation

@Domest0s
Copy link
Contributor

@Domest0s Domest0s commented Sep 18, 2025

jcmd provides great diagnostics but many commands lack a timestamp in their output.
Adding a timestamp to the output of some would add value for those debugging JVM data.

Some diagnostic commands already provide timestamps. For example Thread.print already prints one of "yyyy-MM-dd HH:mm:ss" format.

Adding timestamps to all diagnostic jcmd commands with a non-throw-away STDOUT.

The exceptions are:

  • VM.uptime - command run with -date argument will also print a timestamp;
  • VM.system_properties - already lists timestamp
  • Thread.dump_to_file - the content dumped to file already has a timestamp;
  • VM.version

Progress

  • Change must be properly reviewed (1 review required, with at least 1 Reviewer)
  • Change must not contain extraneous whitespace
  • Commit message must refer to an issue

Issue

  • JDK-8357828: Add a timestamp to jcmd diagnostic commands (Enhancement - P4)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/27368/head:pull/27368
$ git checkout pull/27368

Update a local copy of the PR:
$ git checkout pull/27368
$ git pull https://git.openjdk.org/jdk.git pull/27368/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 27368

View PR using the GUI difftool:
$ git pr show -t 27368

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/27368.diff

Using Webrev

Link to Webrev Comment

@bridgekeeper
Copy link

bridgekeeper bot commented Sep 18, 2025

👋 Welcome back Domest0s! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

@openjdk
Copy link

openjdk bot commented Sep 18, 2025

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

@openjdk
Copy link

openjdk bot commented Sep 18, 2025

@Domest0s The following labels will be automatically applied to this pull request:

  • hotspot-runtime
  • serviceability

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

}

void ThreadDumpToFileDCmd::execute(DCmdSource source, TRAPS) {
print_local_time(output());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume you didn't mean to print a time stamp here as the output already has a timestamp. Also for -format=json and sending the output to stdout, then it will be unparsable if a timestamp appears before the JSON object.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Alan. That's right.
My mistake. I declared the intention to not have timestamps for commands with a throwaway stdout. This also includes "Thread.dump_to_file" command but then mistakenly added print_local_time() in here.

@Domest0s Domest0s marked this pull request as ready for review September 19, 2025 14:03
@openjdk openjdk bot added the rfr Pull request is ready for review label Sep 19, 2025
@mlbridge
Copy link

mlbridge bot commented Sep 19, 2025

Webrevs

@Domest0s
Copy link
Contributor Author

Internal testing tier 1-5 did not show any issues.

@dholmes-ora
Copy link
Member

This was raised and discussed somewhere else not that long ago, but I don't recall where and it fizzled out as I recall. But I will re-raise the issues - what form should this timestamp take? Shouldn't it be configurable to allow matching with what may be presented by Unified Logging e.g. VM uptime rather than wall-clock time? The value of a timestamp on dmcd output is to allow it to be placed in chronological order relative to other output - so you need to be able to match the "timestamp" of that other output.

And rather than add timestamp printing code to each dcmd it would make more sense to me to have a global dcmd flag that says "print a timestamp" (with a given format). That way users opt-in and we don't have to remember to add this for new Dcmds.

@coffeys
Copy link
Contributor

coffeys commented Sep 22, 2025

The Thread.print dcmd already uses the yyyy-MM-dd HH:mm:ss format but yes, having all timestamps in a consistent format and time zone is useful for log comparisons.

I'd second the global dcmd flag suggestion which would allow the option of the timestamp(on or off). Some commands should have timestamp on by default though. Those where output is sensitive to the time that the command was issued. e.g.
Thread dumps
...and as per JBS suggestions, other examples include: (but not limited to)
GC.finalizer_info
GC.heap_info

Looking at such data output in isolation with the absence of timestamps makes it far less valuable IMO, especially when one wants to compare it to application logs, GC logs etc.

@kevinjwalls
Copy link
Contributor

I think it should be clearer what the benefit is, who the users are.

In a remote support situation where the the user is gathering info, then timestamps are good.
Running "date; jcmd ... " is an alternative.

An optional global timestamp setting could be in the Jcmd launcher, which would mean it doesn't have to be implemented in each native diagnostic command impl.
i.e. a jcmd option before the PID.

jcmd -T PID command... (or whatever option)
jcmd -T -f commands.txt ... could that print a timestamp before every inidvidual command?

That might be straightforward, but then you can get double timestamps if the jcmd implementation already prints one (so do you remove the timestamp from Thread.print if the jcmd launcher starts printing it?)

Also complicated for some to commands enable a timestamp by default, the jcmd launcher does not know anything about command implementations.
So maybe it needs some common code on the native side.

Do we print a timestamp if the command is not recognised?

Lots of questions. 8-)

@Domest0s
Copy link
Contributor Author

Speaking of having "print a timestamp" as a global flag. Placing be the flag early for the launcher, before pid | main class gives some benefits:

  • we can document the flag in the general documentation "jcmd -h"
  • we could place the timestamp on the same line with PID printed by launcher.
$ jcmd -T ${PID} VM.info
12345: 2025-01-01 23:59:59
. . .

Is there any issues to that?

On the other hand, I see dcmd implements a special rule to handle "-h", "-help", and "--help" flags placed at the end of jcmd arguments list. We could allow the new "timestamp" flag to be passed past the jcmd PID command, Process the flag in DCmdFactory and pass the boolean down to a concrete DCmd instance.

@kevinjwalls
Copy link
Contributor

Hi, that sounds promising.
I do think we should state the benefit of having this (we don't expect much latency as this is a local attach).

Passing something on to the native impl is interesting - then, if the jcmd launcher prints pid:, then timestamp on a newline, can Thread.print find out a timestamp is already printed, and not duplicate it?

3975026:
2025-09-23 10:31:54
Full thread dump ...

Or, should Thread.print stop printing a timestamp, and rely on the launcher to do it. That would avoid that communication, but would mean we have to default to timestamps ON, or Thread.print output format will change (which is not completely illegal, but will no doubt surprise some people/scripts). Or, have the two sides communicate and Thread.print prints the timestamp, if it has not been shown already. Nothing there is as simple as we'd like. 8-)

@AlanBateman
Copy link
Contributor

AlanBateman commented Sep 23, 2025

It might be useful for @larry-cable to comment on this too. I think he's exploring some of these command serving up JSON rather than plain text. Any "infrastructure" that print a plain text timestamp could mess that up.

@tstuefe
Copy link
Member

tstuefe commented Sep 23, 2025

Piling on...

I think this feature would be useful. I wished for this more often than not.

I think the printing should be done on the server side, not on the client side (be it a date before the invocation of jcmd, or jcmd itself printing the date), since the timestamp on the server side may be different than that of the client jcmd.

I would prefer a timestamp in the UL "time" decoration format: Full ISO-8601, including timezone offset. See https://openjdk.org/jeps/158 . That would also keep in line with gc logs etc.

I am not sure we need a specific option to disable or enable this. Who would want to disable this feature?

@mlbridge
Copy link

mlbridge bot commented Sep 23, 2025

Mailing list message from Laurence Cable on serviceability-dev:

On 9/23/25 2:59 AM, Thomas Stuefe wrote:

On Thu, 18 Sep 2025 14:57:44 GMT, Ivan Bereziuk <duke at openjdk.org> wrote:

`jcmd` provides great diagnostics but many commands lack a timestamp in their output.
Adding a timestamp to the output of some would add value for those debugging JVM data.

Some diagnostic commands already provide timestamps. For example `Thread.print` already prints one of "yyyy-MM-dd HH:mm:ss" format.

Adding timestamps to all diagnostic `jcmd` commands with a non-throw-away STDOUT.

The exceptions are:
* `VM.uptime` - command run with `-date` argument will also print a timestamp;
* `VM.system_properties` - already lists timestamp
* `Thread.dump_to_file` - the content dumped to file already has a timestamp;
* `VM.version`
Piling on...

I think this feature would be useful. I wished for this more often than not.

I think the printing should be done on the server side, not on the client side (be it a `date` before the invocation of jcmd, or jcmd itself printing the date), since the timestamp on the server side may be different than that of the client jcmd.

I would prefer a timestamp in the UL "time" decoration format: Full ISO-8601, including timezone offset. See https://openjdk.org/jeps/158 . That would also keep in line with gc logs etc.

I am not sure we need a specific option to disable or enable this. Who would want to disable this feature?

anyone who is parsing the output of jcmd's programmatically ... since
this change can potentially break that code

@Domest0s
Copy link
Contributor Author

anyone who is parsing the output of jcmd's programmatically ... since this change can potentially break that code

@tstuefe Do you think we would need a CSR for this kind of change?

@tstuefe
Copy link
Member

tstuefe commented Sep 24, 2025

@larry-cable

anyone who is parsing the output of jcmd's programmatically ... since
this change can potentially break that code

yeah, true

@Domest0s
Hmm, rather than breaking compatibility and requiring a CSR, I now also opt for an option. We are usually not so strict with changing individual commands, but I agree with Larry that changing the format of all commands may be a problem.

Thinking this a bit further. We may have the same discussion in the future whenever we do fundamental changes to jcmd output.

As an arbitrary example, let's say someone wants to report the jcmd runtime back at the end of every command (I recently needed that, but abandoned the PR). E.g. a trailing "321ms". Or some other statistics or process specs. Do we want a new diagnostic switch every time? That could get confusing quickly, and these switches need testing.

How about a single "enable jcmd legacy output format" setting. By default on. Behind this setting we guard all future changes that severely change the output of commands. Anyone wanting a stable output would define that setting.

And how about making this switch a standard jcmd option, not a JVM setting? Typically, this decision is made by the analyst running jcmd. That is the person wanting to post-process the output. A JVM switch means the analyst has to restart the JVM with new command-line parameters. That is uncomfortable, and the analyst may even be able to do that (often can't restart client production servers; maybe not even change VM parameters).

Opinions?

@mlbridge
Copy link

mlbridge bot commented Sep 24, 2025

Mailing list message from Laurence Cable on serviceability-dev:

see inline...

On 9/24/25 2:37 AM, Thomas Stuefe wrote:

On Thu, 18 Sep 2025 14:57:44 GMT, Ivan Bereziuk <duke at openjdk.org> wrote:

`jcmd` provides great diagnostics but many commands lack a timestamp in their output.
Adding a timestamp to the output of some would add value for those debugging JVM data.

Some diagnostic commands already provide timestamps. For example `Thread.print` already prints one of "yyyy-MM-dd HH:mm:ss" format.

Adding timestamps to all diagnostic `jcmd` commands with a non-throw-away STDOUT.

The exceptions are:
* `VM.uptime` - command run with `-date` argument will also print a timestamp;
* `VM.system_properties` - already lists timestamp
* `Thread.dump_to_file` - the content dumped to file already has a timestamp;
* `VM.version`
@larry-cable
anyone who is parsing the output of jcmd's programmatically ... since
this change can potentially break that code

yeah, true

thanks!

@Domest0s
Hmm, rather than breaking compatibility and requiring a CSR, I now also opt for an option. We are usually not so strict with changing individual commands, but I agree with Larry that changing the format of all commands may be a problem.

cheers!

Thinking this a bit further. We may have the same discussion in the future whenever we do fundamental changes to jcmd output.

agreed!

As an arbitrary example, let's say someone wants to report the jcmd runtime back at the end of every command (I recently needed that, but abandoned the PR). E.g. a trailing "321ms". Or some other statistics or process specs. Do we want a new diagnostic switch every time? That could get confusing quickly, and these switches need testing.

agreed

How about a single "enable jcmd legacy output format" setting. By default on. Behind this setting we guard all future changes that severely change the output of commands. Anyone wanting a stable output would define that setting.

agreed

And how about making this switch a standard *jcmd* option, not a JVM setting? Typically, this decision is made by the analyst running jcmd. That is the person wanting to post-process the output. A JVM switch means the analyst has to restart the JVM with new command-line parameters. That is uncomfortable, and the analyst may even be able to do that (often can't restart client production servers; maybe not even change VM parameters).

Opinions?

makes sense, I have a (much) larger dream for jcmd etc that I want to
work on:

1) add Unix Domain Socket support to the (sun) enbedded httpserver
2) expose a UDS endpoint for Jcmds'
3)? add http+json as a transport for commands and responses
4) add mechanism for DCmd to render in json

with json as a payload format such additions would be much simpler...(in
theory at least)

@tstuefe
Copy link
Member

tstuefe commented Sep 24, 2025

@Domest0s Let's see if everyone agrees on this.

My proposal would entail

  • adding a new standard option to jcmd
  • somehow funneling that option to the JVM
  • the option should be optional, so that:
    • old jcmd still works with new JVMs (and produces legacy jcmd output)
    • new jcmd still works with old JVMs (and produces legacy jcmd output)

How complex would this be? There are two approaches to this. You can either create a new attach listener protocol (we already have ATTACH_API_V2 vs ATTACH_API_V1, so the precedence is there). Or, you can expand the jcmd parsing (see jcmd(AttachOperation* op, attachStream* out) in attachListener.cpp).

Not sure whether this would be complex. I understand if this is expanding the scope of your PR too much, in which case I think you should add a new JVM command line option that preserves legacy jcmd output format.

I think you need a CSR in both cases. Not exactly because compatibility would be broken (the jcmd output should stay the same unless this option is explicitly enabled), but because it makes sense to think a bit about the format of this new jcmd option or this new command line flag, and to make the new command line flag product.

Just my 5 cents, let's see what others think first

@tstuefe
Copy link
Member

tstuefe commented Sep 24, 2025

makes sense, I have a (much) larger dream for jcmd etc that I want to work on:

  1. add Unix Domain Socket support to the (sun) enbedded httpserver 2) expose a UDS endpoint for Jcmds' 3)? add http+json as a transport for commands and responses 4) add mechanism for DCmd to render in json

with json as a payload format such additions would be much simpler...(in theory at least)

That sounds interesting, and useful.

@alexmenkov
Copy link

My proposal would entail
* adding a new standard option to jcmd
* somehow funneling that option to the JVM
* the option should be optional, so that:
* old jcmd still works with new JVMs (and produces legacy jcmd output)
* new jcmd still works with old JVMs (and produces legacy jcmd output)
How complex would this be? There are two approaches to this. You can either create a new attach listener protocol (we already have ATTACH_API_V2 vs ATTACH_API_V1, so the precedence is there). Or, you can expand the jcmd parsing (see jcmd(AttachOperation* op, attachStream* out) in attachListener.cpp).

ATTACH_API_V2 supports "options".
Client (jcmd) detects options supported by the target VM ("getversion options" command) and can set option values in attach command request.
Currently the only supported option is "streaming" (it allows turn off streaming output).
The option is needed only for tests and needs to work for all attach tools (jcmd, jstack, etc.), so it can be set by specifying java property launching attach tool (like jcmd -J-Djdk.attach.allowStreamingOutput=false PID command)

@larry-cable
Copy link
Contributor

larry-cable commented Sep 24, 2025 via email

@AlanBateman
Copy link
Contributor

AlanBateman commented Sep 25, 2025

Do you think we would need a CSR for this kind of change?

The output from several commands have changed over many releases/years, the output from Thread.print in particular. AFAIK, none of these changes has a CSR, there were only CSRs when new commands were added. That said, if there is some evidence that the output is being parsed then there is a compatibility concern so creating a CSR and RN would prudent.

@mlbridge
Copy link

mlbridge bot commented Sep 25, 2025

Mailing list message from Laurence Cable on serviceability-dev:

On 9/25/25 12:00 AM, Alan Bateman wrote:

On Wed, 24 Sep 2025 09:09:22 GMT, Ivan Bereziuk <duke at openjdk.org> wrote:

Do you think we would need a CSR for this kind of change?
The output from several commands have changed over many releases/years, the output from Thread.dump in particular. AFAIK, none of these changes has a CSR, there were only CSRs when new commands were added. That said, if there is some evidence that the output is being parsed then there is a compatibility concern so creating a CSR and RN would prudent.

I am personally aware of at least one cloud svc that parses jcmd o/p
that may be affected by such a change

@Domest0s
Copy link
Contributor Author

Domest0s commented Oct 2, 2025

 My proposal would entail
 * adding a new standard option to jcmd
 * somehow funneling that option to the JVM
 * the option should be optional, so that:
 * old jcmd still works with new JVMs (and produces legacy jcmd output)
 * new jcmd still works with old JVMs (and produces legacy jcmd output)
 How complex would this be? There are two approaches to this. You
 can either create a new attach listener protocol (we already have
 ATTACH_API_V2 vs ATTACH_API_V1, so the precedence is there). Or,
 you can expand the jcmd parsing (see |jcmd(AttachOperation* op,
 attachStream* out)| in attachListener.cpp).

ATTACH_API_V2 supports "options".
Client (jcmd) detects options supported by the target VM ("getversion
options" command) and can set option values in attach command request.
Currently the only supported option is "streaming" (it allows turn off
streaming output).
The option is needed only for tests and needs to work for all attach
tools (jcmd, jstack, etc.), so it can be set by specifying java
property launching attach tool (like |jcmd
-J-Djdk.attach.allowStreamingOutput=false PID command|)

Agreed. I will be working on the proposal. Thank you.

@mlbridge
Copy link

mlbridge bot commented Oct 7, 2025

Mailing list message from Laurence Cable on serviceability-dev:

at some point (soon) we should also define a (common) JSON envelope
(schema) for those jcmds that implement a JSON content variant...

IMO it should potentially include:

- jcmd "name"
- JVM version string
- JVM pid
- timestamp
- host "identity"
- content version #
- ...

all jcmds supporting JSON o/p should wrap their content in this std
envelope ...

the motivation for this is that since emitting JSON is primarily
intended to enable programmatic parsing of such content, having the content
be self describing is highly desirable, particularly in a cloud
environment ...

- Larry

On 10/7/25 1:14 PM, Ivan Bereziuk wrote:

@Domest0s Domest0s marked this pull request as draft October 30, 2025 12:23
@openjdk openjdk bot removed the rfr Pull request is ready for review label Oct 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

8 participants