[bootstrap] 6.7.0 RPM installation expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip]

[inqueue]

Elasticsearch version (bin/elasticsearch --version): 6.7.0 (Upgraded from 6.6.1)

[2019-02-28T07:44:54,138][INFO ][o.e.n.Node] [json-sut-c1] 
version[6.7.0], pid[16514], build[default/rpm/0002203/2019-02-27T11:21:09.100270Z], 
OS[Linux/3.10.0-957.5.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server 
VM/1.8.0_191/25.191-b12]

Description of the problem including expected versus actual behavior:
Elasticsearch 6.7.0 will not start using the RPM installation on CentOS Linux release 7.6.1810 (Core). The log complains GeoLite2-ASN.mmdb is missing, but it is there:

[root@json-sut-c1 elasticsearch]# pwd && tree ingest-geoip
/etc/elasticsearch
ingest-geoip
├── GeoLite2-ASN.mmdb
├── GeoLite2-City.mmdb
├── GeoLite2-City.mmdb.gz
├── GeoLite2-Country.mmdb
└── GeoLite2-Country.mmdb.gz

0 directories, 5 files

Steps to reproduce:

sudo yum install elasticserach
systemctl start elasticsearch

Provide logs:

[2019-02-28T07:45:05,349][ERROR][o.e.b.Bootstrap          ] [json-sut-c1] Exception
java.lang.RuntimeException: java.io.IOException: expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.getProcessors(IngestGeoIpPlugin.java:78) ~[?:?]
	at org.elasticsearch.ingest.IngestService.processorFactories(IngestService.java:105) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.ingest.IngestService.<init>(IngestService.java:88) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.node.Node.<init>(Node.java:387) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.7.0.jar:6.7.0]
	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.7.0.jar:6.7.0]
Caused by: java.io.IOException: expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.assertDatabaseExistence(IngestGeoIpPlugin.java:150) ~[?:?]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.loadDatabaseReaders(IngestGeoIpPlugin.java:102) ~[?:?]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.getProcessors(IngestGeoIpPlugin.java:76) ~[?:?]
	... 14 more
[2019-02-28T07:45:05,355][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [json-sut-c1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: java.io.IOException: expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.7.0.jar:6.7.0]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.7.0.jar:6.7.0]
Caused by: java.lang.RuntimeException: java.io.IOException: expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.getProcessors(IngestGeoIpPlugin.java:78) ~[?:?]
	at org.elasticsearch.ingest.IngestService.processorFactories(IngestService.java:105) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.ingest.IngestService.<init>(IngestService.java:88) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.node.Node.<init>(Node.java:387) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.7.0.jar:6.7.0]
	... 6 more
Caused by: java.io.IOException: expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.assertDatabaseExistence(IngestGeoIpPlugin.java:150) ~[?:?]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.loadDatabaseReaders(IngestGeoIpPlugin.java:102) ~[?:?]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.getProcessors(IngestGeoIpPlugin.java:76) ~[?:?]
	at org.elasticsearch.ingest.IngestService.processorFactories(IngestService.java:105) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.ingest.IngestService.<init>(IngestService.java:88) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.node.Node.<init>(Node.java:387) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.7.0.jar:6.7.0]
	... 6 more

Please let me know if you need access to the repro host.

[danielmitterdorfer]

I am not sure this is a bug. The error message says that it expects the file to NOT exist at this location:

java.lang.RuntimeException: java.io.IOException: expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip]

This is caused by a leftover from an old installation and can either be resolved via uninstalling the plugin in 6.6.1 before attempting the upgrade with the --purge option, i.e.:

./bin/elasticsearch-plugin remove --purge ingest-geoip

or manually removing the files in the config directory after the upgrade.

[elasticmachine]

Pinging @elastic/es-core-infra

[inqueue]

The error message says that it expects the file to NOT exist at this location:

Sorry it was late and I missed that. That is the issue @danielmitterdorfer; the ingest-geoip plugin was removed without the purge option. I removed blocker and added the docs label. Those were files leftover from the ingest-geoip plugin that was manually installed in 6.6.1. rm -rf ingest-geoip fixed it.

I do not exactly see why we have to fail the node for files not in use. This will happen to anyone that hasn't purged those files, so it at least warrants a clear note in the documentation.

[danielmitterdorfer]

I do not exactly see why we have to fail the node for files not in use.

I did not implement that check but I'd reason that we want to fail fast here because those files were previously used by the plugin and we want to avoid that people think that they are still used by the module and wonder why they are not picked up.