Skip to content

Enforce default date formats for tsdb data streams #83517

@martijnvg

Description

@martijnvg

Supporting custom date formats when resolving a tsdb data stream to a backing index makes the logic on the coordinating node complicated, because the target backing index is unknown. It is only known after parsing the @timestamp field, but in order to parse the timestamp field the date format should be known and this information is stored in the mapping of a backing index.

In order to avoid unnecessary complexity when coordinating a bulk request we need to make the following changes:

  • Only support strict_date_optional_time and strict_date_optional_time_nanos date formats for the @timestamp field in tsdb data streams. For now we don't expect that we need to support other date formats. In order to do this we also need to prohibit field mapping for the @timestamp field that allow any other date format for backing indices of tsdb data streams.
  • When resolving the backing index for tsdb data streams on the coordinating node, truncate millisecond and nanosecond precision from the the timestamp only for determining to what backing index the document should be indexed into. Also the DataStreamIndexSettingsProvider should be adjusted to truncate ms/ns precision when setting the index.time_series.start_time and index.time_series.end_time index settings. Note that, on the data node when documents get indexed the timestamp do get parsed into a date value that confirms with the field mapping of the @timestamp field.

Note that currently only strict_date_optional_time and ms since epoch provided as long is supported for tsdb data streams (see this TODO).

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions