Skip to content

Move cxx to the 51degrees-cxx release 4.5.105 and read by index on an engine built with no names - #82

Merged
jwrosewell merged 12 commits into
rcfrom
feature/take-cxx-rc-865cee97
Nov 11, 2026
Merged

jwrosewell merged 12 commits into
rcfrom
feature/take-cxx-rc-865cee97

Conversation

@jwrosewell

@jwrosewell jwrosewell commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Moves the cxx submodule to 7c4ac436, the second 51degrees-cxx release of 11 November 2026, which 51degrees-cxx#150 merged into main at 15:05 UTC, and with it reads by index on an engine built with no property names, which #101 held back until this release. The publish workflow has tagged it 4.5.105, and git/matching-refs/tags/4.5.105 gives 7c4ac436e47cd15beed304cad2b8151d50a1efd1, so the submodule is at the tagged commit.

Head e24cbdf8, which is rc at 528e0df3 merged into 905e7484. The merge brings in only #104's JSON Builder change, #49's README change and #108's CI change, none of which touches the on-premise engines, so the links below are at 905e748.

cxx taken

rc builds against c6d4e4d9, the 10 November release, which the automated Update submodules pull request #103 moved it to. 7c4ac436 is one commit after c6d4e4d9 and brings in three 51degrees-cxx pull requests.

  • #148 keeps the first data file's property names, sorted by name, across every reload of an engine created with no names, so a property's index names the same property after a reload, and the C++ reads by index work on such an engine. A property that only a later data file has reads as unavailable, by index or by name, until the engine is created again, and its metadata reports it as not available.
  • #147 makes the C and C++ IP Intelligence examples read _51DEGREES_IPI_PATH in place of 51DEGREES_IPI_PATH.
  • #149 merges main back into rc with no change to the tree.

What the specification says

Finding Properties at 927ac677, merged from specifications#137, says the following.

An Engine created with no names takes as its names those of every Property of the first data file it loads, sorted by name, which is the order of that data set's list of available Properties. It keeps them across every data refresh, so its positions are fixed too. The native code does this. An implementation configured with no names MUST create the Engine with no names, and MUST NOT pass the data set's names itself, because an Engine created with names walks graphs during process, as reading graphs describes, where one created with no names walks a graph only when a Property of it is first read.

A Property that only a later data file has gets no position until the Engine is created again, so a read of it, by position or by name, gives no value, with the invalid Property reason.

Every read MUST use the position. A typed accessor, such as IsMobile, resolves its position once, when the Engine is created or when its data object is first made, and reads by position from then on. A read by name looks the name up in a table from the names the Engine was created with to their positions, and then reads by position. That table is built once, when the Engine is created, from the names the native code gives, for example through the Engine's getRequiredProperties, and MUST NOT be built again from a later data set.

What changed

On rc, each on-premise engine makes its map from names to indexes from the names it was built with, and an engine built with no names has an empty map, so every read on it goes by name.

  1. DeviceDetectionHashEngine and IPIntelligenceOnPremiseEngine each gain fixPropertyIndexes. When the engine was built with no names, it makes the map once from the native engine's getRequiredProperties(), and marks it made, so no later load makes it again. An engine built with names is marked made in the constructor (line 113), so its map is never replaced.
  2. fixPropertyIndexes runs first in setEngineMetaData (Device Detection line 427, IP Intelligence line 417), which every load calls. The first load runs inside the constructor through addDataFile (line 114), so the map is in place before the engine processes anything.
  3. getRequiredProperties() gives the name at each required index in index order (src/common/EngineBase.cpp lines 119 to 137 at 7c4ac436), so a name's place in that list is its index. After the first load those are the first data file's names, which a reload keeps.
  4. The engine is still built with no names, as the specification requires, so processing walks each component's graph only when a property of it is first read.
  5. A property only a later data file has is not in the map, so it reads by name, and the native code gives no value for it, with the invalid property reason, until the engine is built again, as #148 describes.
  6. The comments on the map, on getPropertyIndex and on indexOf in DeviceDataHashDefault and IPIntelligenceDataHashDefault now say this.

Tests

Test What it checks
PropertyIndex_NoNamesFromFirstFile, Device Detection and IP Intelligence An engine built with no names has an index for every property of its data file, no two share one, and the native results' getPropertyName names that property at that index
PropertyIndex_NoNamesSurvivesReload, Device Detection An engine built with no names on the Lite file, reloaded from the TAC file, whose properties are a superset in another order, keeps every index, gives no index to a property only the TAC file has, and reads every property both files have as an engine built on the TAC file reads it
PropertyIndex_NoNamesSurvivesReload, IP Intelligence An engine built with no names keeps every index through a reload from its data file
PropertyIndex_NoNamesReadByIndexAgainstByName, Device Detection The measurement below, for an engine built with no names

PropertyIndex_NoNamesReadsByName in both test classes, which required no index on such an engine, is replaced by the first row. The Device Detection names leave out the match metrics the engine adds, such as Difference, because they are not in the data file and so have no index.

  • Red, on 4a5c3bf (the tests without the change): run 43056910573. Device Detection 3 of 11 fail and IP Intelligence 2 of 6 fail, each a new test, with "Expected an index for 'BrowserName'." or "Expected an index for 'RegisteredCountry'." or "Expected an index for 'IsMobile'.". The first red run, 43054611798, stopped after the Device Detection step, so the workflow was changed to run the IP Intelligence step as well.
  • Green, on 905e748: run 43054611599, 11 and 6 pass.

The cost

The engine is built with no names before and after, so processing does not change. Building adds one call to getRequiredProperties() after the first load.

A read by name against a read by index on the native results of an engine built with no names, IsMobile read 20,000 times by each in each of 5 rounds after a warm-up, in nanoseconds per read as lowest, median and highest. The engine built with names is the same test as before, for comparison.

From full build 43072290831, PropertyIndex_NoNamesReadByIndexAgainstByName and PropertyIndex_ReadByIndexAgainstByName in the same job.

Job No names, by name No names, by index Names given, by name Names given, by index
Ubuntu_ARM_Java_17 283/293/317 174/179/180 276/276/409 176/176/177
Ubuntu_Java_8 190/193/390 119/122/234 189/192/193 118/120/121
Ubuntu_Java_11 267/278/382 145/166/187 168/172/180 95/96/97
Ubuntu_Java_17 341/360/722 211/228/402 326/333/337 210/211/217
Ubuntu_Java_21 309/317/680 193/204/404 340/342/344 247/248/249
Windows_Java_8 381/386/756 213/218/242 359/362/364 212/215/222
Windows_Java_11 217/221/395 122/123/222 228/370/414 140/184/251
Windows_Java_17 246/466/511 155/296/315 234/236/246 158/159/167
Windows_Java_21 366/414/718 221/226/473 348/351/709 224/231/468

The read by index is faster than the read by name in every job, for the engine built with no names as for the one built with names, so the index stands. The green run gave 303 ns by name against 187 ns by index with no names.

The typed read through the pipeline, before and after the change, in one job (run 43058951094). The job built 4a5c3bf (before) and 905e748 (after), then timed getIsMobile() and get("IsMobile") 50,000 times each in each of 7 rounds after a warm-up, three times for each commit in turn. Medians in nanoseconds per read.

Round Engine getIsMobile() before after get("IsMobile") before after
1 no names 603 435 713 475
2 no names 588 435 672 567
3 no names 604 504 669 585
1 names given 421 413 462 433
2 names given 415 397 440 435
3 names given 407 400 434 456

The engine with no names reads faster after the change in every round. The engine built with names, whose code does not change, moves by a few nanoseconds either way.

SWIG wrappers

No regeneration is needed. Both hash_java.i files include only cxx/src/common/JavaTypes.i and cxx/src/dd/hash/hash.i or cxx/src/ipi/ipi.i. The cxx .i files declare every class SWIG wraps, and the headers are only copied into the wrappers from %{ %} blocks. git diff cfad8978 7c4ac436 -- '*.i' and git diff c6d4e4d9 7c4ac436 -- '*.i' in 51degrees-cxx both give one change, two lines of a // comment above copyValueAsUTF8String in src/ipi/ResultsIpi.i lines 115 and 116, which SWIG does not copy into a wrapper.

The committed wrappers are the ones this draft generated on Ubuntu runners with SWIG 4.5.1, built from the swig/swig v4.5.1 tag, using the README commands. With cxx at cfad8978 (run 42796095651), Java_Ipi_Engine.cpp and every generated Java class of both engines came out byte for byte as committed. With cxx at 865cee97 (run 42672898614), only ConfigIpiSwig.java changed, because cxx#122 scopes %newobject to MetaData, so getComponents, getProperties, getValues and getProfiles return a collection the configuration owns.

ConfigIpiSwig.java differs from rc because the wrapper on rc was last generated on 2 November (fe9d0cbd), before cxx#122, and #103 moved cxx on without generating it again.

Ownership workaround removed

#69 added borrowed() to BuilderTests and OnPremisePerformanceProfileTests, which set swigCMemOwn to false by reflection because four getters marked their collection as owned. With the wrapper generated again, all eleven getters give a borrowed collection, so the workaround is deleted. BuilderTests.Config_CollectionsAreBorrowed reads the flag on each of the eleven collections, deletes each proxy and reads every setting again.

  • Red, with the workaround removed and the old ConfigIpiSwig.java: run 42673597556, at the ownership assertion, BuilderTests.java:123.
  • Green, with the wrapper generated again: run 42674143386.

Full build

Run 43072290831 on tmp/pr-matrix-82b, started by tools/pr-matrix.py without macOS, passed. All 10 Setting 'PullRequestSha' lines, in Configure and the nine build jobs, give this head, e24cbdf, and Configure printed "Left out 6 macOS entries". Every job passed, being Ubuntu Java 8, 11, 17 and 21, Ubuntu ARM Java 17, and Windows Java 8, 11, 17 and 21.

The earlier full build on 905e748, run 43054665777, failed only at "Pipeline developer examples" in Ubuntu Java 17, where SimpleCloudEngine_Test found no resource key, and the other jobs were cancelled. #108 fixed that on rc, and this head takes it.

macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.

Remaining

  • This draft does not test the missing property reason Java gives for a property that only a later data file has. The specification gives PropertyExcludedFromEngineConfiguration for it.
  • The temporary branches tmp/index-java-nonames-red, tmp/index-java-nonames-green, tmp/index-java-timing, tmp/pr-matrix-82 and tmp/pr-matrix-82b are still to be deleted.

The configuration's collections are borrowed. Test that each one the
configuration hands out is borrowed, and stop forcing the flag by
reflection in the two tests that read them.
SWIG 4.5.1 now marks the configuration's collection getters as borrowed,
as the corrected MetaData.i asks. The Device Detection wrapper is unchanged
apart from comments recording the include path, so it is left as it was.
Takes the two cxx commits after b4f769ad, from
51Degrees/51degrees-cxx#130 and 51Degrees/51degrees-cxx#146. This is
the cxx commit for the release.
An engine built with no names should take its names from the native
engine after the first data file loads, give every property of that file
an index the native results agree with, and keep those indexes through a
reload from a file whose properties are a superset in another order. The
read by name against the read by index is measured for such an engine as
it is for one built with names.
51degrees-cxx 7c4ac436 keeps the first data file's property names, in
their order, across every reload of an engine created with no names, and
its C++ reads by index work on such an engine. The engine therefore makes
its map of names to indexes once, from the native engine's required
properties after the first data file loads, and never again. A property
only a later data file has is not in the map, so it reads by name and
gives no value until the engine is built again.
@jwrosewell jwrosewell changed the title Take the 51degrees-cxx rc commit and generate the SWIG wrapper again Move cxx to the 51degrees-cxx release 4.5.105 and read by index on an engine built with no names Nov 11, 2026
@jwrosewell
jwrosewell marked this pull request as ready for review November 11, 2026 19:33
@jwrosewell
jwrosewell merged commit c34b30d into rc Nov 11, 2026
2 checks passed
@jwrosewell
jwrosewell deleted the feature/take-cxx-rc-865cee97 branch November 11, 2026 19:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant