The source or way to affect the time captured by p_timestamp

[a-zb]

I've seen another user ask this before, but it seems that it's not a "good enough" answer in my view.

One, I understand p_timestamp is internal.
Having said that, it isn't internal when it is used in the user interface and confusing the users.

Example:
I have a property in the event that provides a 'created_at' timestamp.
Event is sent from the same computer where Parseable is running via fluentbit from a script.

My timedatectl output is as following:
Local time: Sun 2024-10-20 14:28:22 EDT
Universal time: Sun 2024-10-20 18:28:22 UTC
RTC time: Sun 2024-10-20 18:28:22
Time zone: America/Toronto (EDT, -0400)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no

created_at looks as following when it is captured, correct: 2024-10-19T18:11:05.702800Z
The p_timestamp that gets captured is, oddly in UTC time: 2024-10-19T22:11:05.717

When user clicks on the stream, the p_timestamp is shown, that's a source of confusion #1.
When user clicks on the event , the side bar with details shows the following:
Timestamp

19/10/2024 (10:11:05 PM) EDT

It's obviously wrong and source of confusion #2.

What drives the format of the p_timestamp capture so that it gets captured correctly or at least includes the -0400 timezone part ?

[a-zb]

A follow up observation after much testing.
I have tried variety of timestamp ingestion inside my event 'created_at'
RFC3339 extended with nanosecond format is most useful in my case.

If I leave stream timestamp partition field as default, using p_timestamp, then my event 'created_at' is left as is.

I send : 2024-10-20T23:28:49.212-04:00
and parseable records: 2024-10-20T23:28:49.212-04:00

If on the other hand, I define stream partitioning to use the 'created_at' event property, then parseable converts the data:

I send: 2024-10-20T23:28:49.212-04:00
and parseable records: 2024-10-21T03:28:52.220

So, in the end , it looks like internally it has to be UTC time and it's quite fine. So be it. It probably relies on Parquet convention.

What I want to know, is how to change the display format in user interface since users who are sitting in front of the browser see the UTC time, which is not correct in my view.

The time-range widget in top right has the right idea, which is to show in "local browser time", the 'created_at' is now off by 4 hours and like I've mentioned above, clicking on the event shows the details sidebar with:

21/10/2024 (03:28:49 AM) EDT

Since the time range widget is working correctly, there must be account for the local to utc conversion somewhere, just not shown.

Am I missing some setting here ?

[nitisht]

Thanks for reporting this in details @a-zb , let us get back on this

[balaji-jr]

@a-zb
The incorrect time range in the popup view has been fixed in parseablehq/console#342. This fix will be included in the upcoming release.

[balaji-jr]

Additionally, we have introduced minimal timezone support, planned for the very next release. You will be able to see your local timezone wherever a time is displayed. However, the datetime columns in the log-table will still be displayed in UTC for now. We will also include a setting to control timestamp parsing, allowing you to choose the date format and preferred timezone in the log table.

[a-zb]

The incorrect time range in the popup view has been fixed in parseablehq/console#342. This fix will be included in the upcoming release.

Kudos. Impressive that you have it in the pipeline.

@balaji-jr by "wherever a time is displayed .. but not in the log table for now' - So for the interim, the ingested timestamp will be shown in UTC, but users will be informed about which timezone it is in ? Then eventually, a change will be released that formats the timestamps to a TZ? Am I close to understanding that ?

[nitisht]

Then eventually, a change will be released that formats the timestamps to a TZ? Am I close to understanding that ?

Hey @a-zb , yes we've fixed the issue here in console PR parseablehq/console#346

[nitisht]

New changes will be available in next release. Closing the issue for now, feel free to reopen / comment if something is not working as expected.