Skip to content

Rollup search ignores the "format" option in date_histogram #34391

@kiju98

Description

@kiju98

Elasticsearch version (bin/elasticsearch --version): Version: 6.4.0, Build: default/tar/595516e/2018-08-17T23:18:47.308994Z, JVM: 1.8.0_162

Plugins installed: []

JVM version (java -version): 1.8.0_162, Java HotSpot(TM) 64-Bit Server VM, 25.162-b12, Oracle Corporation

OS version (uname -a if on a Unix-like system): Linux oss-elastic02 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

I tried a rollup search but "format" option in date_histogram was ignored.

Steps to reproduce:

  1. Prepare an index.
PUT stat
{
  "mappings": {
    "doc": {
      "properties": {
        "agent": {
          "properties": {
            "@timestamp": {
              "type": "date"
            }
          }
        },
        "cpu": {
          "properties": {
            "idle": {
              "properties": {
                "pct": {
                  "type": "float"
                }
              }
            },
            "iowait": {
              "properties": {
                "pct": {
                  "type": "float"
                }
              }
            },
            "sys": {
              "properties": {
                "pct": {
                  "type": "float"
                }
              }
            },
            "total": {
              "properties": {
                "pct": {
                  "type": "scaled_float",
                  "store": true,
                  "scaling_factor": 100
                }
              }
            },
            "user": {
              "properties": {
                "pct": {
                  "type": "float"
                }
              }
            }
          }
        }
      }
    }
  }
}

PUT stat/doc/1
{
  "agent": {
    "@timestamp": "2018-09-01T09:19:00.000Z"
  },
  "cpu": {
    "total": {
      "pct": 11
    },
    "iowait": {
      "pct": 0
    },
    "sys": {
      "pct": 1
    },
    "idle": {
      "pct": 88.6
    },
    "user": {
      "pct": 10.5
    }
  }
}
  1. Create and start a rollup job.
PUT _xpack/rollup/job/rollup-stat
{
  "index_pattern": "stat",
  "rollup_index": "rollup-stat",
  "cron": "* * * * * ?",
  "page_size": 2000,
  "groups": {
    "date_histogram": {
      "interval": "1h",
      "field": "agent.@timestamp"
    }
  },
  "metrics": [
    {
      "field": "cpu.total.pct",
      "metrics": [
        "min",
        "max",
        "avg"
      ]
    }
  ]
}

POST _xpack/rollup/job/rollup-stat/_start
  1. Run a rollup query
GET rollup-stat/_rollup_search
{
  "size": 0,
  "aggs": {
    "1day_val": {
      "date_histogram": {
        "field": "agent.@timestamp",
        "interval": "1d",
        "format": "yyyy-MM-dd",
        "time_zone": "UTC"
      },
      "aggs": {
        "cpu_day_max": {
          "max": {
            "field": "cpu.total.pct"
          }
        }
      }
    }
  }
}

and see the format option in the date_histogram aggregation was ignored as below:

{
  "took": 2,
  "timed_out": false,
  "terminated_early": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 0,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "1day_val": {
      "meta": {},
      "buckets": [
        {
          "key_as_string": "2018-09-01T00:00:00.000Z",
          "key": 1535760000000,
          "doc_count": 1,
          "cpu_day_max": {
            "value": 11
          }
        }
      ]
    }
  }
}

"key_as_string" should be "2018-09-01".

Provide logs (if relevant):

Metadata

Metadata

Assignees

No one assigned

    Labels

    :StorageEngine/RollupTurn fine-grained time-based data into coarser-grained datav6.4.0

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions