-
Notifications
You must be signed in to change notification settings - Fork 25.6k
Closed
Labels
:Data Management/Data streamsData streams and their lifecyclesData streams and their lifecyclesTeam:Data ManagementMeta label for data/management teamMeta label for data/management team
Description
Supporting custom date formats when resolving a tsdb data stream to a backing index makes the logic on the coordinating node complicated, because the target backing index is unknown. It is only known after parsing the @timestamp field, but in order to parse the timestamp field the date format should be known and this information is stored in the mapping of a backing index.
In order to avoid unnecessary complexity when coordinating a bulk request we need to make the following changes:
- Only support
strict_date_optional_timeandstrict_date_optional_time_nanosdate formats for the@timestampfield in tsdb data streams. For now we don't expect that we need to support other date formats. In order to do this we also need to prohibit field mapping for the@timestampfield that allow any other date format for backing indices of tsdb data streams. - When resolving the backing index for tsdb data streams on the coordinating node, truncate millisecond and nanosecond precision from the the timestamp only for determining to what backing index the document should be indexed into. Also the
DataStreamIndexSettingsProvidershould be adjusted to truncate ms/ns precision when setting theindex.time_series.start_timeandindex.time_series.end_timeindex settings. Note that, on the data node when documents get indexed the timestamp do get parsed into a date value that confirms with the field mapping of the@timestampfield.
Note that currently only strict_date_optional_time and ms since epoch provided as long is supported for tsdb data streams (see this TODO).
Metadata
Metadata
Assignees
Labels
:Data Management/Data streamsData streams and their lifecyclesData streams and their lifecyclesTeam:Data ManagementMeta label for data/management teamMeta label for data/management team