Performance issue GeoIP2 vs GeoIP Legacy

[evengard88]

I'm testing GeoIP2 vs GeoIP and getting a very strange result, it seems the new api is way slower.
maxMind1 - Legacy API
maxMind2 - new API, FileMode.MEMORY_MAPPED
maxMind2v2 - new API, FileMode.MEMORY
Caches are on;

Here are my benchmark results:

Benchmark                      Mode     Cnt       Score      Error  Units
MindMaxBenchmark.maxMind1     thrpt      50  176225.276 ± 1603.768  ops/s
MindMaxBenchmark.maxMind2     thrpt      50     825.766 ±   12.428  ops/s
MindMaxBenchmark.maxMind2v2   thrpt      50     767.863 ±   35.938  ops/s
MindMaxBenchmark.maxMind1      avgt      50      ≈ 10⁻⁵              s/op
MindMaxBenchmark.maxMind2      avgt      50       0.001 ±    0.001   s/op
MindMaxBenchmark.maxMind2v2    avgt      50       0.001 ±    0.001   s/op
MindMaxBenchmark.maxMind1    sample  131466      ≈ 10⁻⁵              s/op
MindMaxBenchmark.maxMind2    sample   38187       0.001 ±    0.001   s/op
MindMaxBenchmark.maxMind2v2  sample   34064       0.001 ±    0.001   s/op
MindMaxBenchmark.maxMind1        ss      50      ≈ 10⁻⁴              s/op
MindMaxBenchmark.maxMind2        ss      50       0.028 ±    0.005   s/op
MindMaxBenchmark.maxMind2v2      ss      50       0.026 ±    0.004   s/op

Services are initialized at the test start like that:

final DatabaseReader mindMaxDB = new DatabaseReader.Builder(...)
                    .withCache(new CHMCache())
                    .fileMode(Reader.FileMode.MEMORY)
                    .build();

and the test code is like (some null checks are ommited):

        public LocationData resolve(InetAddress inetAddress) {

                CityResponse cityResponse = mindMaxDB.city(inetAddress);
                Location location = cityResponse.getLocation();
                return new LocationData(
                        inetAddress, cityResponse.getCountry().getIsoCode(), cityResponse.getCity().getName()
                );
            }

The Legacy API:
Service created like that

LookupService lookupService = new LookupService(file, LookupService.GEOIP_MEMORY_CACHE);

and test code is

 com.maxmind.geoip.Location location = lookupService.getLocation(inetAddress);
                            return new LocationData(
                                    inetAddress, location.countryCode,
                                    location.city);

I'd like to know if I'm missing something?

[oschwald]

I am not quite sure how to read or interpret your benchmark results based on what you posted. This library is slower than the legacy API for two reasons: (1) the search tree is much deeper as it includes IPv6 address, (2) we include considerably more data for each IP. That said, a lookup should be on the order of 20μs on modern hardware once the caches are warm.

[rudgirello]

We are seeing similar results in our system (RTB domain). GeoIP Legacy appears to be 500 times faster. Pretty staggering. Our average query time for the Legacy City DB is ~8 microseconds, for GeoIP2 - ~5 milliseconds. I believe this is due to the massive size of the new V2 DB and due to the various encodings causing substantial overhead. Perhaps, it's a good idea to provide a way to turn off secondary encodings.

[oschwald]

If your queries are taking 5 ms, there is almost certainly something wrong. Are you opening the database for every request?

[rudgirello]

No, our usage pattern was quite optimal, as far as I could see. To be fair, we did have to alleviate a couple of synchronized blocks in the V1 client code; it's a custom build that is. And, btw, by encodings I meant locales.

Is this the only issue tracking the performance of the V2 client? Given the great spec for the MMDB format, we are considering writing our own fast client based on it. Of course, we'd rather not do it, given the wheel has already been invented...

[rudgirello]

@oschwald, provided you are surprised to see such high query time figures, what are the expected figures? What hardware did you use? We are running spinning disks (commodity 10K RPM), 16GB RAM (12GB heap), 24 logical cores, Debian 7.

[oschwald]

20 μs would be in the normal range with modern commodity hardware. If you think spinning disks are an issue (my test machines have SSDs), you could try memory mode. You could also try enabling the caching as mentioned in the documentation.

[evengard88]

I have used memory mode in my tests, the database is opened once, caches are on and the average time is still 1 millisecond per request.
It seems to me that something is configured wrongly and my test is a copy-paste from the documentation;

Could you post the code that would give 20 μs per request?

[oschwald]

Sure, here is a benchmark. On my machine, I get about 105,000 lookups per second without caching and 225,000 lookups per second with caching. This is about 9.5 μs and 4 μs per lookup.

[rudgirello]

@oschwald, these are the values that we observed too for a single thread. The slow downs manifest themselves as we add and increase concurrency. At around 20-30 concurrent queries we are at the 1-5ms figures. At this scale any synchronized block potentially kills your performance. In this regard, Locale instances must be treated with utmost care as their retrieval involves "synchronized" like there's no tomorrow.

[oschwald]

Unlike the legacy API, this API does not synchronize internally. If you have a reproducible benchmark that shows an issue, I am happy to look at it.

[rudgirello]

@oschwald, we'll try to set up a benchmark reproducing this behaviour. It might be tricky, as the specificity of user traffic may have a profound effect (e.g., IP address clashes). Will get back to you next week. Thanks for looking into this!

[evengard88]

I've just tried single thread version with my benchmark, it works much faster, so the behavior is reproduced.

My benchmark is written with jmh, i could share it if needed.

[oschwald]

Please do share your benchmark. If there is some sort of locking issue when using a reader from multiple threads, I'd like to fix it.

[evengard88]

here is my benchmark
https://github.com/evengard88/bench

test.ParamsHolder - holds parameters (ip addresses), could be modified to read from external sources.
test.MaxMind2Holder - Service holder, wraps com.maxmind.geoip2.DatabaseReader. Addresses to DB file should be modified.
test.jmh.MindMaxBenchmark - the benchmark. Annotations could be modified to watch different modes (multi-thread vs single thread and etc)

the whole thing started with
mvn clean package && java -Xmx500m -Xms500m -Djmh.ignoreLock=true -jar target/benchmarks.jar