[ML] Error - Failed to calculate splitting significance

[sophiec20]

Found in

$./autodetect --version
Model State Version 34
Quantile State Version 3
autodetect (64 bit): Version 7.0.0-alpha1-SNAPSHOT (Build 06a0f99f4383b4) Copyright (c) 2018 Elasticsearch BV

QA smoke test job xf_not_set_height_by_category_20180704-1059_700-alpha1_2446

• metric(height) by category
• bucket_span = 2m
• exclude_frequent NOT set

Error in logfile found on Windows and Linux for MLStack-build 2446.

[CAdaptiveBucketing.cc@676] Failed to calculate splitting significance: Error in function boost::math::binomial_distribution::binomial_distribution: Success fraction argument is -0.10081309000651041, but must be >= 0 and <= 1 !

No commits were added to ml-cpp since last successful execution.

Unable to replicate locally on Mac.

Possible sporadic issue - will continue to watch out for this.

2018-07-04T10:59:45.086 : [127][INFO][o.e.x.m.j.p.a.AutodetectProcessManager] [Ni0JZ10] Opening job [xf_not_set_height_by_category_20180704-1059_700-alpha1_2446]
2018-07-04T10:59:45.088 : [128][INFO][o.e.x.m.j.p.a.AutodetectProcessManager] [Ni0JZ10] [xf_not_set_height_by_category_20180704-1059_700-alpha1_2446] Loading model snapshot [N/A], job latest_record_timestamp [N/A]
2018-07-04T10:59:45.460 : [129][INFO][o.e.x.m.j.p.l.CppLogMessageHandler] [xf_not_set_height_by_category_20180704-1059_700-alpha1_2446] [autodetect/18693] [CResourceMonitor.cc@66] Setting model memory limit to 1024 MB
2018-07-04T10:59:45.477 : [130][INFO][o.e.x.m.j.p.a.AutodetectProcessManager] [Ni0JZ10] Successfully set job state to [opened] for job [xf_not_set_height_by_category_20180704-1059_700-alpha1_2446]
2018-07-04T10:59:45.514 : [131][INFO][o.e.x.m.d.DatafeedJob] [xf_not_set_height_by_category_20180704-1059_700-alpha1_2446] Datafeed started (from: 1970-01-01T00:00:00.000Z to: 2018-07-04T10:59:45.488Z) with frequency [60000ms]
2018-07-04T10:59:48.747 : [132][INFO][o.e.x.m.j.p.DataCountsReporter] [Ni0JZ10] [xf_not_set_height_by_category_20180704-1059_700-alpha1_2446] 10000 records written to autodetect; missingFieldCount=0, invalidDateCount=0, outOfOrderCount=0
2018-07-04T10:59:51.501 : [133][INFO][o.e.x.m.j.p.DataCountsReporter] [Ni0JZ10] [xf_not_set_height_by_category_20180704-1059_700-alpha1_2446] 20000 records written to autodetect; missingFieldCount=0, invalidDateCount=0, outOfOrderCount=0
2018-07-04T10:59:54.171 : [134][INFO][o.e.x.m.j.p.DataCountsReporter] [Ni0JZ10] [xf_not_set_height_by_category_20180704-1059_700-alpha1_2446] 30000 records written to autodetect; missingFieldCount=0, invalidDateCount=0, outOfOrderCount=0
2018-07-04T10:59:56.431 : [135][INFO][o.e.x.m.j.p.DataCountsReporter] [Ni0JZ10] [xf_not_set_height_by_category_20180704-1059_700-alpha1_2446] 40000 records written to autodetect; missingFieldCount=0, invalidDateCount=0, outOfOrderCount=0
2018-07-04T10:59:58.936 : [136][INFO][o.e.x.m.j.p.DataCountsReporter] [Ni0JZ10] [xf_not_set_height_by_category_20180704-1059_700-alpha1_2446] 50000 records written to autodetect; missingFieldCount=0, invalidDateCount=0, outOfOrderCount=0
2018-07-04T11:00:01.063 : [137][ERROR][o.e.x.m.j.p.l.CppLogMessageHandler] [xf_not_set_height_by_category_20180704-1059_700-alpha1_2446] [autodetect/18693] [CAdaptiveBucketing.cc@676] Failed to calculate splitting significance: Error in function boost::math::binomial_distribution::binomial_distribution: Success fraction argument is -0.10081309000651041, but must be >= 0 and <= 1 !
2018-07-04T11:00:01.084 : [138][INFO][o.e.x.m.j.p.DataCountsReporter] [Ni0JZ10] [xf_not_set_height_by_category_20180704-1059_700-alpha1_2446] 60000 records written to autodetect; missingFieldCount=0, invalidDateCount=0, outOfOrderCount=0
2018-07-04T11:00:03.428 : [139][INFO][o.e.x.m.j.p.DataCountsReporter] [Ni0JZ10] [xf_not_set_height_by_category_20180704-1059_700-alpha1_2446] 70000 records written to autodetect; missingFieldCount=0, invalidDateCount=0, outOfOrderCount=0
2018-07-04T11:00:05.833 : [140][INFO][o.e.x.m.j.p.DataCountsReporter] [Ni0JZ10] [xf_not_set_height_by_category_20180704-1059_700-alpha1_2446] 80000 records written to autodetect; missingFieldCount=0, invalidDateCount=0, outOfOrderCount=0
2018-07-04T11:00:09.228 : [141][INFO][o.e.x.m.j.p.DataCountsReporter] [Ni0JZ10] [xf_not_set_height_by_category_20180704-1059_700-alpha1_2446] 90000 records written to autodetect; missingFieldCount=0, invalidDateCount=0, outOfOrderCount=0
2018-07-04T11:00:11.693 : [142][INFO][o.e.x.m.j.p.DataCountsReporter] [Ni0JZ10] [xf_not_set_height_by_category_20180704-1059_700-alpha1_2446] 100000 records written to autodetect; missingFieldCount=0, invalidDateCount=0, outOfOrderCount=0
2018-07-04T11:00:12.656 : [143][INFO][o.e.x.m.d.DatafeedJob] [xf_not_set_height_by_category_20180704-1059_700-alpha1_2446] Lookback has finished
2018-07-04T11:00:12.657 : [144][INFO][o.e.x.m.d.DatafeedManager] [no_realtime] attempt to stop datafeed [datafeed-xf_not_set_height_by_category_20180704-1059_700-alpha1_2446] for job [xf_not_set_height_by_category_20180704-1059_700-alpha1_2446]
2018-07-04T11:00:12.657 : [145][INFO][o.e.x.m.d.DatafeedManager] [no_realtime] try lock [20s] to stop datafeed [datafeed-xf_not_set_height_by_category_20180704-1059_700-alpha1_2446] for job [xf_not_set_height_by_category_20180704-1059_700-alpha1_2446]...
2018-07-04T11:00:12.657 : [146][INFO][o.e.x.m.d.DatafeedManager] [no_realtime] stopping datafeed [datafeed-xf_not_set_height_by_category_20180704-1059_700-alpha1_2446] for job [xf_not_set_height_by_category_20180704-1059_700-alpha1_2446], acquired [true]...
2018-07-04T11:00:12.657 : [147][INFO][o.e.x.m.d.DatafeedManager] [no_realtime] datafeed [datafeed-xf_not_set_height_by_category_20180704-1059_700-alpha1_2446] for job [xf_not_set_height_by_category_20180704-1059_700-alpha1_2446] has been stopped
2018-07-04T11:00:12.684 : [148][INFO][o.e.x.m.j.p.a.AutodetectProcessManager] [Ni0JZ10] Closing job [xf_not_set_height_by_category_20180704-1059_700-alpha1_2446], because [close job (api)]
2018-07-04T11:00:12.685 : [149][INFO][o.e.x.m.j.p.l.CppLogMessageHandler] [xf_not_set_height_by_category_20180704-1059_700-alpha1_2446] [autodetect/18693] [CCmdSkeleton.cc@45] Handled 101002 records
2018-07-04T11:00:12.685 : [150][INFO][o.e.x.m.j.p.l.CppLogMessageHandler] [xf_not_set_height_by_category_20180704-1059_700-alpha1_2446] [autodetect/18693] [CAnomalyJob.cc@1441] Pruning all models
2018-07-04T11:00:12.718 : [151][INFO][o.e.x.m.j.p.a.NativeAutodetectProcess] [xf_not_set_height_by_category_20180704-1059_700-alpha1_2446] State output finished
2018-07-04T11:00:12.764 : [152][INFO][o.e.x.m.j.p.a.o.AutoDetectResultProcessor] [xf_not_set_height_by_category_20180704-1059_700-alpha1_2446] 25000 buckets parsed from autodetect output
2018-07-04T11:00:13.814 : [153][INFO][o.e.x.m.j.p.a.AutodetectCommunicator] [xf_not_set_height_by_category_20180704-1059_700-alpha1_2446] job closed

[tveasey]

This indicates the m_Endpoints is not ordered, which is worrying and shouldn't happen. Also it is surprising that this behaviour is platform dependent. We should probably add in a check that m_Endpoints is sorted at the end of CAdaptiveBucketing::refine and ensure it is as a safeguard. But it would be good to understand how this is happening.

[hendrikmuhs]

This does not reproduce for me on my linux box. I will create a PR to improve the log message, hopefully that will help us finding the root cause.

[davidkyle]

I saw the same error in 7.6 with a real time job that came up automatically on node restart.

[2020-03-16T16:28:11,000][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [DaveKs-MacBook-Pro-2533.local] [farequote] Loading model snapshot [1584031569] with latest_record_timestamp [2014-06-27T23:59:56.000Z], job latest_record_timestamp [2014-06-27T23:59:56.000Z]
[2020-03-16T16:28:11,049][INFO ][o.e.x.m.j.p.a.NativeAutodetectProcessFactory] [DaveKs-MacBook-Pro-2533.local] Restoring quantiles for job 'farequote'
[2020-03-16T16:28:11,417][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [DaveKs-MacBook-Pro-2533.local] [farequote] [autodetect/43251] [CResourceMonitor.cc@71] Setting model memory limit to 14 MB
[2020-03-16T16:28:11,471][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [DaveKs-MacBook-Pro-2533.local] [farequote] [autodetect/43251] [CAnomalyJob.cc@839] Processing is already complete to time 1403911800
...
[2020-03-16T16:28:12,168][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [DaveKs-MacBook-Pro-2533.local] Successfully set job state to [opened] for job [farequote]
[2020-03-16T16:28:12,652][INFO ][o.e.x.m.d.DatafeedJob    ] [DaveKs-MacBook-Pro-2533.local] [farequote] Datafeed started (from: 2014-06-27T23:59:56.001Z to: real-time) with frequency [600000ms]
...
[2020-03-16T16:34:04,412][ERROR][o.e.x.m.p.l.CppLogMessageHandler] [DaveKs-MacBook-Pro-2533.local] [farequote] [autodetect/43251] [CAdaptiveBucketing.cc@681] Failed to calculate splitting significance: Error in function boost::math::binomial_distribution<double>::binomial_distribution: Success fraction argument is -0.046742892795138886, but must be >= 0 and <= 1 ! interval = -4038.59 period = 86400 type = Seasonal[0.002401,1800.000000]
[2020-03-16T16:34:15,208][ERROR][o.e.x.m.p.l.CppLogMessageHandler] [DaveKs-MacBook-Pro-2533.local] [farequote] [autodetect/43251] [CAdaptiveBucketing.cc@681] Failed to calculate splitting significance: Error in function boost::math::binomial_distribution<double>::binomial_distribution: Success fraction argument is -0.046742892795138886, but must be >= 0 and <= 1 ! interval = -4038.59 period = 86400 type = Seasonal[0.002401,1800.000000]
[2020-03-16T16:34:15,209][ERROR][o.e.x.m.p.l.CppLogMessageHandler] [DaveKs-MacBook-Pro-2533.local] [farequote] [autodetect/43251] [CAdaptiveBucketing.cc@681] Failed to calculate splitting significance: Error in function boost::math::binomial_distribution<double>::binomial_distribution: Success fraction argument is -0.021616753472222222, but must be >= 0 and <= 1 ! interval = -1867.69 period = 86400 type = Seasonal[0.002400,1800.000000]
[2020-03-16T16:36:29,875][ERROR][o.e.x.m.p.l.CppLogMessageHandler] [DaveKs-MacBook-Pro-2533.local] [farequote] [autodetect/43251] [CAdaptiveBucketing.cc@681] Failed to calculate splitting significance: Error in function boost::math::binomial_distribution<double>::binomial_distribution: Success fraction argument is -0.025200918692129631, but must be >= 0 and <= 1 ! interval = -2177.36 period = 86400 type = Seasonal[0.002400,1800.000000] | repeated [3]
[2020-03-16T16:36:29,882][ERROR][o.e.x.m.p.l.CppLogMessageHandler] [DaveKs-MacBook-Pro-2533.local] [farequote] [autodetect/43251] [CAdaptiveBucketing.cc@681] Failed to calculate splitting significance: Error in function boost::math::binomial_distribution<double>::binomial_distribution: Success fraction argument is -0.064357458043981478, but must be >= 0 and <= 1 ! interval = -5560.48 period = 86400 type = Seasonal[0.002400,1800.000000]
[2020-03-16T16:36:46,891][ERROR][o.e.x.m.p.l.CppLogMessageHandler] [DaveKs-MacBook-Pro-2533.local] [farequote] [autodetect/43251] [CAdaptiveBucketing.cc@681] Failed to calculate splitting significance: Error in function boost::math::binomial_distribution<double>::binomial_distribution: Success fraction argument is -0.091362892433449069, but must be >= 0 and <= 1 ! interval = -7893.75 period = 86400 type = Seasonal[0.002400,1800.000000]
[2020-03-16T16:36:46,898][ERROR][o.e.x.m.p.l.CppLogMessageHandler] [DaveKs-MacBook-Pro-2533.local] [farequote] [autodetect/43251] [CAdaptiveBucketing.cc@681] Failed to calculate splitting significance: Error in function boost::math::binomial_distribution<double>::binomial_distribution: Success fraction argument is -0.021287254050925925, but must be >= 0 and <= 1 ! interval = -1839.22 period = 86400 type = Seasonal[0.002400,1800.000000]
[2020-03-16T16:38:43,980][ERROR][o.e.x.m.p.l.CppLogMessageHandler] [DaveKs-MacBook-Pro-2533.local] [farequote] [autodetect/43251] [CAdaptiveBucketing.cc@681] Failed to calculate splitting significance: Error in function boost::math::binomial_distribution<double>::binomial_distribution: Success fraction argument is -0.021287254050925925, but must be >= 0 and <= 1 ! interval = -1839.22 period = 86400 type = Seasonal[0.002400,1800.000000]
[2020-03-16T16:38:43,980][ERROR][o.e.x.m.p.l.CppLogMessageHandler] [DaveKs-MacBook-Pro-2533.local] [farequote] [autodetect/43251] [CAdaptiveBucketing.cc@681] Failed to calculate splitting significance: Error in function boost::math::binomial_distribution<double>::binomial_distribution: Success fraction argument is -0.021275273075810185, but must be >= 0 and <= 1 ! interval = -1838.18 period = 86400 type = Seasonal[0.002400,1800.000000]
[2020-03-16T16:42:15,801][WARN ][r.suppressed             ] [DaveKs-MacBook-Pro-2533.local] path: /_snapshot/ml-data-snapshots-7-share, params: {pretty=true, repository=ml-data-snapshots-7-share}

After which there were no more errors.

[davidkyle]

Reproduction on v7.6.0

You will need the farequote dataset from 2014, I indexed the data with the following mappings

{
  "farequote" : {
    "mappings" : {
      "properties" : {
        "Country" : {
          "type" : "keyword"
        },
        "airline" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword"
            }
          }
        },
        "responsetime" : {
          "type" : "float"
        },
        "sourcetype" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "time" : {
          "type" : "date"
        }
      }
    }
  }
}

1. Create a job with the config

{
  "analysis_config": {
    "bucket_span": "30m",
    "detectors": [
      {
        "detector_description": "count partitionfield=\"airline.keyword\"",
        "function": "count",
        "partition_field_name": "airline.keyword",
        "detector_index": 0
      },
      {
        "detector_description": "distinct_count(\"airline.keyword\") partitionfield=\"airline.keyword\"",
        "function": "distinct_count",
        "field_name": "airline.keyword",
        "partition_field_name": "airline.keyword",
        "detector_index": 1
      }
    ],
    "influencers": [
      "airline.keyword"
    ]
  },

2. Run the lookback job and when it is done click 'Start job running in real time'
3. Wait
4. The job is now processing almost 6 years of empty buckets
5. Eventually you will see the error

[ERROR][o.e.x.m.p.l.CppLogMessageHandler] [DaveKs-MacBook-Pro-2533.local] [fq-no-crash] [autodetect/44120] [CAdaptiveBucketing.cc@681] Failed to calculate splitting significance: Error in function boost::math::binomial_distribution<double>::binomial_distribution: Success fraction argument is -0.046742892795138886, but must be >= 0 and <= 1 ! interval = -4038.59 period = 86400 type = Seasonal[0.002401,1800.000000]

On my laptop it took ~ 5minutes before the first error occured.