IndexOutOfBoundsException[index (0) must be less than size (0)];

[clintongormley]

Hiya

I'm getting these IndexOutOfBoundsException[index (0) must be less than size (0)]; errors, which are kinda meaningless, on queries that work 99% of the time.

There are some errors in the logs, but they don't correspond to the times of these query errors:

Query errors:

05:38

ReduceSearchPhaseException[Failed to execute phase [fetch], [reduce] ]; nested: IndexOutOfBoundsException[index (0) must be less than size (0)]; 
With vars:
$VAR1 = {
        'request' => {
            'cmd' =>
                '/iannounce_object/anniversary,bestwish,birthday,memorial,specialday,wedding/_search?search_type=dfs_query_then_fetch',
            'data' => {
                'sort'   => [ { '_score' => 'desc' } ],
                'fields' => [],
                'from'   => 0,
                'query'  => {
                    'filtered' => {
                        'filter' => {
                            'bool' => {
                                'must' => [
                                    { 'term' => { 'source' => 'tmmi_wvle' } },
                                    { 'term' => { 'status' => 'active' } },
                                    { 'term' => { 'region' => 'tmmi' } },
                                    { 'term' => { 'parent_id' => '441895' } }
                                ]
                            }
                        },
                        'query' => {
                            'dis_max' => {
                                'queries' => [ {
                                        'query_string' => {
                                            'fields' => [ 'name' ],
                                            'boost'  => '1',
                                            'query' =>
                                                '"HUGH ANDREW DAVID KNAPP"~9'
                                        }
                                    },
                                    {   'filtered' => {
                                            'filter' => {
                                                'term' => { 'has_name' => 0 }
                                            },
                                            'query' => {
                                                'query_string' => {
                                                    'fields' =>
                                                        [ 'short_text' ],
                                                    'boost' => '1.3',
                                                    'query' =>
                                                        '"HUGH ANDREW DAVID KNAPP"~9'
                                                }
                                            }
                                        }
                                    },
                                    {   'query_string' => {
                                            'boost' => 1,
                                            'query' =>
                                                'HUGH ANDREW DAVID KNAPP'
                                        }
                                    }
                                ],
                                'tie_breaker' => '0.7'
                            }
                        }
                    }
                },
                'size' => 300
            },
            'method' => 'GET'
        },
        'status_code' => 500,
        'response'    => {
            'error' =>
                'ReduceSearchPhaseException[Failed to execute phase [fetch], [reduce] ]; nested: IndexOutOfBoundsException[index (0) must be less than size (0)]; '
        },
        'server'     => 'http://192.168.10.51:9200',
        'status_msg' => 'Internal Server Error'
    };

07:05

ReduceSearchPhaseException[Failed to execute phase [fetch], [reduce] ]; nested: IndexOutOfBoundsException[index (0) must be less than size (0)]; 
With vars:
$VAR1 = {
    'request' => {
        'cmd' => '/ia_object/notice/_search?search_type=dfs_query_then_fetch',
        'data' => {
            'sort'   => [ { '_score' => 'desc' } ],
            'fields' => [],
            'from'   => 0,
            'query'  => {
                'filtered' => {
                    'filter' => {
                        'bool' => {
                            'must' => [
                                { 'term' => { 'sub_type'    => 'obit' } },
                                { 'term' => { 'status'      => 'active' } },
                                { 'term' => { 'location_id' => '25' } }
                            ]
                        }
                    },
                    'query' => {
                        'dis_max' => {
                            'queries' => [ {
                                    'query_string' => {
                                        'fields' => [ 'name' ],
                                        'boost'  => '1',
                                        'query'  => '"Dorothy DAYLE"~7'
                                    }
                                },
                                {   'filtered' => {
                                        'filter' =>
                                            { 'term' => { 'has_name' => 0 } },
                                        'query' => {
                                            'query_string' => {
                                                'fields' => [ 'text' ],
                                                'boost'  => '1.3',
                                                'query' => '"Dorothy DAYLE"~7'
                                            }
                                        }
                                    }
                                },
                                {   'query_string' => {
                                        'boost' => 1,
                                        'query' => 'Dorothy DAYLE'
                                    }
                                }
                            ],
                            'tie_breaker' => '0.7'
                        }
                    }
                }
            },
            'size' => 300
        },
        'method' => 'GET'
    },
    'status_code' => 500,
    'response'    => {
        'error' =>
            'ReduceSearchPhaseException[Failed to execute phase [fetch], [reduce] ]; nested: IndexOutOfBoundsException[index (0) must be less than size (0)]; '
    },
    'server'     => 'http://192.168.10.51:9200',
    'status_msg' => 'Internal Server Error'
};

Errors in non-master:

[18:30:50,391][WARN ][monitor.jvm              ] [Uncle Ben Parker] Long GC collection occurred, took [12.5s], breached threshold [10s]
[23:07:16,048][WARN ][monitor.jvm              ] [Uncle Ben Parker] Long GC collection occurred, took [10.3s], breached threshold [10s]

Errors in master:

[05:14:32,291][DEBUG][action.search.type       ] [Rock] [4845938] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [Uncle Ben Parker][inet[/192.168.10.50:9300]][search/phase/fetch/id]
Caused by: org.elasticsearch.search.SearchContextMissingException: No search context found for id [4845938], timed out
        at org.elasticsearch.search.SearchService.findContext(SearchService.java:279)
        at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:262)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:444)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:435)
        at org.elasticsearch.transport.netty.MessageChannelHandler$3.run(MessageChannelHandler.java:195)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
[05:14:32,293][DEBUG][action.search.type       ] [Rock] [4845936] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [Uncle Ben Parker][inet[/192.168.10.50:9300]][search/phase/fetch/id]
Caused by: org.elasticsearch.search.SearchContextMissingException: No search context found for id [4845936], timed out
        at org.elasticsearch.search.SearchService.findContext(SearchService.java:279)
        at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:262)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:444)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:435)
        at org.elasticsearch.transport.netty.MessageChannelHandler$3.run(MessageChannelHandler.java:195)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)



[05:29:08,976][DEBUG][action.search.type       ] [Rock] [2444873] Failed to execute query phase
org.elasticsearch.search.query.QueryPhaseExecutionException: [ia_object_1283933184][4]: query[filtered(+_all:margaret +_all:jordan)->FilterCacheFilterWrapper(BooleanFilter( +FilterCacheFilterWrapper(publish_date:[1284422400000 TO *]) +FilterCacheFilterWrapper(org.elasticsearch.common.lucene.search.TermFilter@3a1bb115) +FilterCacheFilterWrapper(location_id:[23 TO 23])))],from[0],size[300],sort[<score>]: Query Failed [Failed to execute main query]
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:130)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:198)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:159)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeQuery(TransportSearchDfsQueryThenFetchAction.java:141)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.access$200(TransportSearchDfsQueryThenFetchAction.java:64)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$1.run(TransportSearchDfsQueryThenFetchAction.java:114)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: df for term margaret not available
        at org.elasticsearch.search.dfs.CachedDfSource.docFreq(CachedDfSource.java:51)
        at org.apache.lucene.search.Similarity.idfExplain(Similarity.java:769)
        at org.apache.lucene.search.spans.SpanWeight.<init>(SpanWeight.java:51)
        at org.elasticsearch.common.lucene.all.AllTermQuery$AllTermWeight.<init>(AllTermQuery.java:61)
        at org.elasticsearch.common.lucene.all.AllTermQuery.createWeight(AllTermQuery.java:55)
        at org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:188)
        at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:362)
        at org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:63)
        at org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:63)
        at org.apache.lucene.search.Query.weight(Query.java:101)
        at org.elasticsearch.search.internal.ContextIndexSearcher.createWeight(ContextIndexSearcher.java:100)
        at org.apache.lucene.search.Searcher.search(Searcher.java:98)
        at org.apache.lucene.search.Searcher.search(Searcher.java:108)
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:126)
        ... 8 more
[05:29:08,976][DEBUG][action.search.type       ] [Rock] [2444873] Failed to execute fetch phase
org.elasticsearch.search.SearchContextMissingException: No search context found for id [2444873], timed out
        at org.elasticsearch.search.SearchService.findContext(SearchService.java:279)
        at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:262)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:309)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeFetch(TransportSearchDfsQueryThenFetchAction.java:226)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.access$700(TransportSearchDfsQueryThenFetchAction.java:64)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$4.run(TransportSearchDfsQueryThenFetchAction.java:199)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
[05:29:08,983][DEBUG][action.search.type       ] [Rock] [2444874] Failed to execute query phase
org.elasticsearch.search.SearchContextMissingException: No search context found for id [2444874], timed out
        at org.elasticsearch.search.SearchService.findContext(SearchService.java:279)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:190)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:159)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeQuery(TransportSearchDfsQueryThenFetchAction.java:141)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.access$200(TransportSearchDfsQueryThenFetchAction.java:64)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$1.run(TransportSearchDfsQueryThenFetchAction.java:114)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
[05:29:31,785][DEBUG][action.search.type       ] [Rock] [2445049] Failed to execute query phase
org.elasticsearch.search.query.QueryPhaseExecutionException: [ia_object_1283933184][3]: query[filtered((name:"roy light"~7 | filtered(text:"roy light"~7^1.3)->FilterCacheFilterWrapper(org.elasticsearch.common.lucene.search.TermFilter@3e3f4f7) | (_all:roy _all:light))~0.7)->FilterCacheFilterWrapper(BooleanFilter( +FilterCacheFilterWrapper(org.elasticsearch.common.lucene.search.TermFilter@fdb5bf46) +FilterCacheFilterWrapper(org.elasticsearch.common.lucene.search.TermFilter@3a1bb115) +FilterCacheFilterWrapper(location_id:[30 TO 30])))],from[0],size[300],sort[<score>]: Query Failed [Failed to execute main query]
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:130)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:198)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:159)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeQuery(TransportSearchDfsQueryThenFetchAction.java:141)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.access$200(TransportSearchDfsQueryThenFetchAction.java:64)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$1.run(TransportSearchDfsQueryThenFetchAction.java:114)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: df for term roy not available
        at org.elasticsearch.search.dfs.CachedDfSource.docFreq(CachedDfSource.java:51)
        at org.apache.lucene.search.Similarity.idfExplain(Similarity.java:769)
        at org.apache.lucene.search.PhraseQuery$PhraseWeight.<init>(PhraseQuery.java:122)
        at org.apache.lucene.search.PhraseQuery.createWeight(PhraseQuery.java:264)
        at org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.<init>(DisjunctionMaxQuery.java:107)
        at org.apache.lucene.search.DisjunctionMaxQuery.createWeight(DisjunctionMaxQuery.java:184)
        at org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:63)
        at org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:63)
        at org.apache.lucene.search.Query.weight(Query.java:101)
        at org.elasticsearch.search.internal.ContextIndexSearcher.createWeight(ContextIndexSearcher.java:100)
        at org.apache.lucene.search.Searcher.search(Searcher.java:98)
        at org.apache.lucene.search.Searcher.search(Searcher.java:108)
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:126)
        ... 8 more
[05:29:31,786][DEBUG][action.search.type       ] [Rock] [2445050] Failed to execute query phase
org.elasticsearch.search.query.QueryPhaseExecutionException: [ia_object_1283933184][0]: query[filtered((name:"roy light"~7 | filtered(text:"roy light"~7^1.3)->FilterCacheFilterWrapper(org.elasticsearch.common.lucene.search.TermFilter@3e3f4f7) | (_all:roy _all:light))~0.7)->FilterCacheFilterWrapper(BooleanFilter( +FilterCacheFilterWrapper(org.elasticsearch.common.lucene.search.TermFilter@fdb5bf46) +FilterCacheFilterWrapper(org.elasticsearch.common.lucene.search.TermFilter@3a1bb115) +FilterCacheFilterWrapper(location_id:[30 TO 30])))],from[0],size[300],sort[<score>]: Query Failed [Failed to execute main query]
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:130)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:198)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:159)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeQuery(TransportSearchDfsQueryThenFetchAction.java:141)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.access$200(TransportSearchDfsQueryThenFetchAction.java:64)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$1.run(TransportSearchDfsQueryThenFetchAction.java:114)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: df for term roy not available
        at org.elasticsearch.search.dfs.CachedDfSource.docFreq(CachedDfSource.java:51)
        at org.apache.lucene.search.Similarity.idfExplain(Similarity.java:769)
        at org.apache.lucene.search.PhraseQuery$PhraseWeight.<init>(PhraseQuery.java:122)
        at org.apache.lucene.search.PhraseQuery.createWeight(PhraseQuery.java:264)
        at org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.<init>(DisjunctionMaxQuery.java:107)
        at org.apache.lucene.search.DisjunctionMaxQuery.createWeight(DisjunctionMaxQuery.java:184)
        at org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:63)
        at org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:63)
        at org.apache.lucene.search.Query.weight(Query.java:101)
        at org.elasticsearch.search.internal.ContextIndexSearcher.createWeight(ContextIndexSearcher.java:100)
        at org.apache.lucene.search.Searcher.search(Searcher.java:98)
        at org.apache.lucene.search.Searcher.search(Searcher.java:108)
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:126)
        ... 8 more
[05:29:31,804][DEBUG][action.search.type       ] [Rock] [2445050] Failed to execute query phase
org.elasticsearch.search.SearchContextMissingException: No search context found for id [2445050], timed out
        at org.elasticsearch.search.SearchService.findContext(SearchService.java:279)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:190)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:159)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeQuery(TransportSearchDfsQueryThenFetchAction.java:141)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.access$200(TransportSearchDfsQueryThenFetchAction.java:64)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$1.run(TransportSearchDfsQueryThenFetchAction.java:114)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
[05:29:31,880][DEBUG][action.search.type       ] [Rock] [2445049] Failed to execute fetch phase
org.elasticsearch.search.SearchContextMissingException: No search context found for id [2445049], timed out
        at org.elasticsearch.search.SearchService.findContext(SearchService.java:279)
        at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:262)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:309)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeFetch(TransportSearchDfsQueryThenFetchAction.java:226)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.access$700(TransportSearchDfsQueryThenFetchAction.java:64)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$4.run(TransportSearchDfsQueryThenFetchAction.java:199)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)






[06:51:44,352][DEBUG][action.search.type       ] [Rock] [2488219] Failed to execute query phase
org.elasticsearch.search.query.QueryPhaseExecutionException: [iannounce_object_1283872186][0]: query[filtered(+_all:amy +_all:hotson)->FilterCacheFilterWrapper(BooleanFilter( +FilterCacheFilterWrapper(org.elasti
csearch.common.lucene.search.TermFilter@3d4c17e4) +FilterCacheFilterWrapper(org.elasticsearch.common.lucene.search.TermFilter@3d3f2f85) +FilterCacheFilterWrapper(org.elasticsearch.common.lucene.search.TermFilter
@3a1bb115) +FilterCacheFilterWrapper(org.elasticsearch.common.lucene.search.TermFilter@40ea8da2) +FilterCacheFilterWrapper(parent_id:[1 TO 1])))],from[0],size[300],sort[<score>]: Query Failed [Failed to execute 
main query]
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:130)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:198)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:159)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeQuery(TransportSearchDfsQueryThenFetchAction.java:141)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.access$200(TransportSearchDfsQueryThenFetchAction.java:64)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$1.run(TransportSearchDfsQueryThenFetchAction.java:114)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: df for term amy not available
        at org.elasticsearch.search.dfs.CachedDfSource.docFreq(CachedDfSource.java:51)
        at org.apache.lucene.search.Similarity.idfExplain(Similarity.java:769)
        at org.apache.lucene.search.spans.SpanWeight.<init>(SpanWeight.java:51)
        at org.elasticsearch.common.lucene.all.AllTermQuery$AllTermWeight.<init>(AllTermQuery.java:61)
        at org.elasticsearch.common.lucene.all.AllTermQuery.createWeight(AllTermQuery.java:55)
        at org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:188)
        at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:362)
        at org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:63)
        at org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:63)
        at org.apache.lucene.search.Query.weight(Query.java:101)
        at org.elasticsearch.search.internal.ContextIndexSearcher.createWeight(ContextIndexSearcher.java:100)
        at org.apache.lucene.search.Searcher.search(Searcher.java:98)
        at org.apache.lucene.search.Searcher.search(Searcher.java:108)
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:126)
        ... 8 more
[06:51:44,353][DEBUG][action.search.type       ] [Rock] [2488220] Failed to execute query phase
org.elasticsearch.search.query.QueryPhaseExecutionException: [iannounce_object_1283872186][2]: query[filtered(+_all:amy +_all:hotson)->FilterCacheFilterWrapper(BooleanFilter( +FilterCacheFilterWrapper(org.elasti
csearch.common.lucene.search.TermFilter@3d4c17e4) +FilterCacheFilterWrapper(org.elasticsearch.common.lucene.search.TermFilter@3d3f2f85) +FilterCacheFilterWrapper(org.elasticsearch.common.lucene.search.TermFilter
@3a1bb115) +FilterCacheFilterWrapper(org.elasticsearch.common.lucene.search.TermFilter@40ea8da2) +FilterCacheFilterWrapper(parent_id:[1 TO 1])))],from[0],size[300],sort[<score>]: Query Failed [Failed to execute 
main query]
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:130)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:198)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:159)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeQuery(TransportSearchDfsQueryThenFetchAction.java:141)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.access$200(TransportSearchDfsQueryThenFetchAction.java:64)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$1.run(TransportSearchDfsQueryThenFetchAction.java:114)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: df for term amy not available
        at org.elasticsearch.search.dfs.CachedDfSource.docFreq(CachedDfSource.java:51)
        at org.apache.lucene.search.Similarity.idfExplain(Similarity.java:769)
        at org.apache.lucene.search.spans.SpanWeight.<init>(SpanWeight.java:51)
        at org.elasticsearch.common.lucene.all.AllTermQuery$AllTermWeight.<init>(AllTermQuery.java:61)
        at org.elasticsearch.common.lucene.all.AllTermQuery.createWeight(AllTermQuery.java:55)
        at org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:188)
        at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:362)
        at org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:63)
        at org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:63)
        at org.apache.lucene.search.Query.weight(Query.java:101)
        at org.elasticsearch.search.internal.ContextIndexSearcher.createWeight(ContextIndexSearcher.java:100)
        at org.apache.lucene.search.Searcher.search(Searcher.java:98)
        at org.apache.lucene.search.Searcher.search(Searcher.java:108)
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:126)
        ... 8 more
[06:51:44,353][DEBUG][action.search.type       ] [Rock] [2488219] Failed to execute query phase
org.elasticsearch.search.SearchContextMissingException: No search context found for id [2488219], timed out
        at org.elasticsearch.search.SearchService.findContext(SearchService.java:279)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:190)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:159)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeQuery(TransportSearchDfsQueryThenFetchAction.java:141)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.access$200(TransportSearchDfsQueryThenFetchAction.java:64)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$1.run(TransportSearchDfsQueryThenFetchAction.java:114)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
[06:51:44,354][DEBUG][action.search.type       ] [Rock] [2488220] Failed to execute query phase
org.elasticsearch.search.SearchContextMissingException: No search context found for id [2488220], timed out
        at org.elasticsearch.search.SearchService.findContext(SearchService.java:279)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:190)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:159)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeQuery(TransportSearchDfsQueryThenFetchAction.java:141)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.access$200(TransportSearchDfsQueryThenFetchAction.java:64)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$1.run(TransportSearchDfsQueryThenFetchAction.java:114)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)

[clintongormley]

More random errors:

[16:45:46,071][DEBUG][action.search.type       ] [Rock] [5833008] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [Uncle Ben Parker][inet[/192.168.10.50:9300]][search/phase/fetch/id]
Caused by: java.lang.NullPointerException
        at org.elasticsearch.search.internal.InternalSearchHits.writeTo(InternalSearchHits.java:156)
        at org.elasticsearch.search.fetch.FetchSearchResult.writeTo(FetchSearchResult.java:100)
        at org.elasticsearch.transport.support.TransportStreams.buildResponse(TransportStreams.java:141)
        at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:72)
        at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:65)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:445)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:435)
        at org.elasticsearch.transport.netty.MessageChannelHandler$3.run(MessageChannelHandler.java:195)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
[16:45:46,073][DEBUG][action.search.type       ] [Rock] [5833009] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [Uncle Ben Parker][inet[/192.168.10.50:9300]][search/phase/fetch/id]
Caused by: java.lang.NullPointerException
        at org.elasticsearch.search.internal.InternalSearchHits.writeTo(InternalSearchHits.java:156)
        at org.elasticsearch.search.fetch.FetchSearchResult.writeTo(FetchSearchResult.java:100)
        at org.elasticsearch.transport.support.TransportStreams.buildResponse(TransportStreams.java:141)
        at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:72)
        at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:65)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:445)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:435)
        at org.elasticsearch.transport.netty.MessageChannelHandler$3.run(MessageChannelHandler.java:195)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
[16:45:46,075][DEBUG][action.search.type       ] [Rock] [5833008] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [Uncle Ben Parker][inet[/192.168.10.50:9300]][search/phase/fetch/id]
Caused by: java.lang.NullPointerException
        at org.elasticsearch.search.internal.InternalSearchHits.writeTo(InternalSearchHits.java:156)
        at org.elasticsearch.search.fetch.FetchSearchResult.writeTo(FetchSearchResult.java:100)
        at org.elasticsearch.transport.support.TransportStreams.buildResponse(TransportStreams.java:141)
        at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:72)
        at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:65)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:445)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:435)
        at org.elasticsearch.transport.netty.MessageChannelHandler$3.run(MessageChannelHandler.java:195)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
[16:45:46,077][DEBUG][action.search.type       ] [Rock] [5833009] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [Uncle Ben Parker][inet[/192.168.10.50:9300]][search/phase/fetch/id]
Caused by: java.lang.NullPointerException
        at org.elasticsearch.search.internal.InternalSearchHits.writeTo(InternalSearchHits.java:156)
        at org.elasticsearch.search.fetch.FetchSearchResult.writeTo(FetchSearchResult.java:100)
        at org.elasticsearch.transport.support.TransportStreams.buildResponse(TransportStreams.java:141)
        at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:72)
        at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:65)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:445)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:435)
        at org.elasticsearch.transport.netty.MessageChannelHandler$3.run(MessageChannelHandler.java:195)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
[16:45:46,079][DEBUG][action.search.type       ] [Rock] [5833010] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [Uncle Ben Parker][inet[/192.168.10.50:9300]][search/phase/fetch/id]
Caused by: java.lang.NullPointerException
        at org.elasticsearch.search.internal.InternalSearchHits.writeTo(InternalSearchHits.java:156)
        at org.elasticsearch.search.fetch.FetchSearchResult.writeTo(FetchSearchResult.java:100)
        at org.elasticsearch.transport.support.TransportStreams.buildResponse(TransportStreams.java:141)
        at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:72)
        at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:65)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:445)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:435)
        at org.elasticsearch.transport.netty.MessageChannelHandler$3.run(MessageChannelHandler.java:195)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
[16:45:46,082][DEBUG][action.search.type       ] [Rock] [5833010] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [Uncle Ben Parker][inet[/192.168.10.50:9300]][search/phase/fetch/id]
Caused by: java.lang.NullPointerException
        at org.elasticsearch.search.internal.InternalSearchHits.writeTo(InternalSearchHits.java:156)
        at org.elasticsearch.search.fetch.FetchSearchResult.writeTo(FetchSearchResult.java:100)
        at org.elasticsearch.transport.support.TransportStreams.buildResponse(TransportStreams.java:141)
        at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:72)
        at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:65)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:445)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchFetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:435)
        at org.elasticsearch.transport.netty.MessageChannelHandler$3.run(MessageChannelHandler.java:195)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)










[16:50:12,372][DEBUG][action.search.type       ] [Rock] [2934788] Failed to execute query phase
org.elasticsearch.search.query.QueryPhaseExecutionException: [iannounce_object_1283872186][4]: query[filtered(+_all:smithurst +_all:elsie +_all:may)->FilterCacheFilterWrapper(BooleanFilter( +FilterCacheFilterWr
apper(org.elasticsearch.common.lucene.search.TermFilter@66fa1702) +FilterCacheFilterWrapper(org.elasticsearch.common.lucene.search.TermFilter@3a1bb115) +FilterCacheFilterWrapper(parent_id:[4323320 TO 4323320]))
)],from[0],size[300],sort[<score>]: Query Failed [Failed to execute main query]
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:130)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:198)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:159)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeQuery(TransportSearchDfsQueryThenFetchAction.java:141)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.access$200(TransportSearchDfsQueryThenFetchAction.java:64)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$1.run(TransportSearchDfsQueryThenFetchAction.java:114)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: df for term smithurst not available
        at org.elasticsearch.search.dfs.CachedDfSource.docFreq(CachedDfSource.java:51)
        at org.apache.lucene.search.Similarity.idfExplain(Similarity.java:769)
        at org.apache.lucene.search.spans.SpanWeight.<init>(SpanWeight.java:51)
        at org.elasticsearch.common.lucene.all.AllTermQuery$AllTermWeight.<init>(AllTermQuery.java:61)
        at org.elasticsearch.common.lucene.all.AllTermQuery.createWeight(AllTermQuery.java:55)
        at org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:188)
        at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:362)
        at org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:63)
        at org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:63)
        at org.apache.lucene.search.Query.weight(Query.java:101)
        at org.elasticsearch.search.internal.ContextIndexSearcher.createWeight(ContextIndexSearcher.java:100)
        at org.apache.lucene.search.Searcher.search(Searcher.java:98)
        at org.apache.lucene.search.Searcher.search(Searcher.java:108)
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:126)
        ... 8 more









[17:15:55,179][DEBUG][action.search.type       ] [Rock] [2953639] Failed to execute fetch phase
org.elasticsearch.search.SearchContextMissingException: No search context found for id [2953639], timed out
        at org.elasticsearch.search.SearchService.findContext(SearchService.java:279)
        at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:262)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:309)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeFetch(TransportSearchDfsQueryThenFetchAction.java:226)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.access$700(TransportSearchDfsQueryThenFetchAction.java:64)
        at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$4.run(TransportSearchDfsQueryThenFetchAction.java:199)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
[17:15:55,185][DEBUG][action.search.type       ] [Rock] [5877973] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [Uncle Ben Parker][inet[/192.168.10.50:9300]][search/phase/fetch/id]
Caused by: java.lang.NullPointerException
[17:15:55,192][DEBUG][action.search.type       ] [Rock] [5877973] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [Uncle Ben Parker][inet[/192.168.10.50:9300]][search/phase/fetch/id]
Caused by: java.lang.NullPointerException
[17:15:55,223][DEBUG][action.search.type       ] [Rock] [5877972] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [Uncle Ben Parker][inet[/192.168.10.50:9300]][search/phase/fetch/id]
Caused by: java.lang.NullPointerException



[18:06:13,863][DEBUG][action.search.type       ] [Rock] [ia_object_1283933184][1], node[7500316d-3f57-4677-a169-2788cb532db7], [R], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@5
9149939] while moving to second phase
java.lang.NullPointerException
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.moveToSecondPhase(TransportSearchQueryThenFetchAction.java:101)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:218)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.access$100(TransportSearchTypeAction.java:80)
        at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onResult(TransportSearchTypeAction.java:196)
        at org.elasticsearch.search.action.SearchServiceTransportAction$2.handleResponse(SearchServiceTransportAction.java:142)
        at org.elasticsearch.search.action.SearchServiceTransportAction$2.handleResponse(SearchServiceTransportAction.java:135)
        at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:142)
        at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:102)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:754)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:302)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:214)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:274)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:261)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)

[kimchy]

Hi,

There are three types of exceptions here that are important to investigate:

• java.lang.IllegalArgumentException: df for term roy not available

This happens when using the dfs_... type query.

• NullPointException in InternalSearchHits.writeTo(InternalSearchHits.java:156)

Happens when trying to serialize the result of a fetch request back to the calling node.

• failed ... moving to second phase, NullPointerException at TransportSearchQueryThenFetchAction.java:101

Fails when trying to move to the fetch phase on the query_then_fetch execution.

To be honest, I don't really understand why any of these exceptions should happen, very strange!. I know its probably hard, but is there a chance for a recreation? Does it happen when you execute similar searches on the same dataset consistently?

Regarding the context not found exception, I have improved the timeout logic in upcoming 0.11, so hopefully this will help solve the problem. It might also relate to the other problems.

[clintongormley]

Re chance of recreation - no. I only have a few types of queries running against the cluster, and almost none of them fail. So these are aberrations. They typically work, until they fail :)

The one thing I have noted though is that they only happen when there is more than one node running. When I was running on a single node, I didn't get these errors.

[clintongormley]

This has been fixed