[jwrosewell]

Problem

The performance figures for IP Intelligence come from the console benchmark PerformanceBenchmark.java, run with the Enterprise data file. When it does not ask for every Property, the benchmark on main asked for RegisteredName alone (line 219). #48 made it ask for AsnName as well (lines 245 and 246 on rc), so that it runs with the free Asn data file, which has no RegisteredName. The benchmark still reads one Property, but with the Enterprise data file the extra request lowers the lookups a second it reports, by about 11 per cent on rc against main. The release draft #112 measured that drop in run 43173346463.

Run 43283829748 found the cause by moving one change at a time. It used one ubuntu-22.04-arm runner, the image the Ubuntu_ARM_Java_17 configuration uses, and 32 rounds. Every tree was built and measured with its own ci/run-performance-tests.ps1, with the same data files and the same generated evidence, and each round ran every tree once with the order turning by one place. IP Intelligence lookups a second:

Tree	Median	Lowest to highest	Against main
main f69da52	327,869	266,667 to 361,174
rc 0b9257f	290,910	236,337 to 308,285	11.3 per cent lower, higher in 1 of 32 rounds
rc without the AsnName line	346,696	288,288 to 367,816	5.7 per cent higher, higher in 23 of 32 rounds
main with the AsnName line added	277,056	227,920 to 290,909	15.5 per cent lower, higher in 0 of 32 rounds
main with rc's cxx submodule and its generated ConfigIpiSwig.java	330,579	294,659 to 348,584	0.8 per cent higher, higher in 20 of 32 rounds

So the AsnName line accounts for the whole drop in both directions, and with the same request rc is faster than main. Moving the C library alone makes no difference to main. Device Detection lookups a second were the same in every tree.

What the specification says

ip-intelligence-specification/required-examples.md, lines 85 to 87 says "A performance, reload or update example, whose Properties do not matter to what it shows, uses the Asn data file, which is far smaller to download." Neither that file nor the Performance Console section says which Properties a performance example asks for. This change keeps the Asn data file as the default and asks for the one Property the benchmark reads.

What changed

Examples only. No package changes.

• benchmarkProperties opens the data file with the low memory profile, as DataFileHelper.lacksEnterpriseProperties already does, and returns RegisteredName where the file has it (the Lite and Enterprise data files) and AsnName otherwise (the free Asn data file). The benchmark asks for that one Property when it does not ask for all of them. The run that asks for every Property is unchanged.
• The example's header comment says what it asks for.
• PerformanceBenchmarkPropertiesTest checks the choice with the Enterprise or Lite data file, and with the Asn data file.

The first commit, 52c3564, adds the test and moves the two setProperty calls into benchmarkProperties with the behaviour unchanged, so that the test fails there. The second, 60302c5, is the change.

Tests

• Red, job 131441588677 on 52c3564, Ubuntu with Java 17, the Enterprise, Lite and Asn data files present. Both tests failed, with "expected:<[RegisteredName]> but was:<[RegisteredName, AsnName]>" and "expected:<[AsnName]> but was:<[RegisteredName, AsnName]>".
• Green, job 131441588613 on 60302c5, the same set up. 2 tests, none failed, none skipped.

Measured with this change

Job 131441587980, one ubuntu-22.04-arm runner, 18 rounds of main, rc and this head taking turns, each measured with its own ci/run-performance-tests.ps1, all three logging the Enterprise data file. IP Intelligence lookups a second:

Tree	Median	Lowest to highest
main f69da52	332,986	321,285 to 347,826
rc 0b9257f	294,664	272,109 to 317,460
this head 60302c5	356,749	299,065 to 399,002

This head was higher than rc in 18 of 18 rounds, by 21.1 per cent at the median, and higher than main in 13 of 18, by 7.1 per cent.

Build

Full build 43292170844 on head 60302c5 finished green. Every job's Setting 'PullRequestSha' line gives 60302c5, and Configure printed "Left out 6 macOS entries". Ubuntu_ARM_Java_17 reported 350,109 IP Intelligence lookups a second, and its log shows PerformanceBenchmarkPropertiesTest ran 2 tests with none failed or skipped. Compare Performance was skipped in this build.

macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.

rc moved to 36c153d (#116) after the build started. #116 changes no file this draft changes and does not move the cxx submodule, and the draft merges cleanly, so the head stays at 60302c5.

What remains

• The .NET, Go and C library performance examples ask for both Properties, on main as on rc (.NET, Go, C), so their figures were measured that way throughout. With this change Java asks for fewer Properties than they do. Whether every performance example should ask only for the Property it reads, or all should ask for both, is not decided here. If both is chosen, this draft closes and the lower Java figure becomes the one its history compares with.
• This change does not touch the usage sharing overhead, which the same runs measured higher on rc than on main.