Repository navigation
Use LowMemory in the IP Intelligence examples built for the Enterprise file - #47
Merged
Merged
Conversation
On the free Lite data file, which a fresh clone downloads, the getting started example showed only the properties Lite has, and the web page showed HardwareVendor, HardwareName and DeviceType as "Unknown (the property 'hardwarevendor' is not in the results, so the current resource key does not include it)", a message written for the cloud service. Getting started, offline processing and both web pages now show each property only the Enterprise data file has as needing the Enterprise data file, with a link to ask for a copy. Metadata, match metrics and performance end with what the Enterprise data file adds and the same link when they run on the Lite file. Without a licence key the data file update example threw "No license key available" with a stack trace. It now shows the Lite data file it would start from, says that an update needs a licence key, gives the link and finishes. The automatic update examples link to the contact page for a licence key, in place of pricing. The shared helpers gain a check of whether the data file has a property, read from the data file's own list rather than the engine's, the text and notes the examples write, and a check of whether the data file is Lite. The data file age warning added 30 to the published date as text rather than adding 30 days, so it showed for a data file published four days earlier, and it now links to the contact page. The free-file warning in the data file log is removed, as each example now says what the Enterprise data file adds in its own output. The file search recursed through every folder up to the root of the disk and overflowed the stack when the file was not there. It now keeps a list of the folders still to search and skips any it cannot read. Downloading a missing data file uses PowerShell 7 when it is on the path, and otherwise Windows PowerShell without the module path it would inherit from PowerShell 7. Inside GitHub Actions the examples no longer send their download source, so the data repository's script names the build and test runs are not counted as people using the examples.
On the free Lite data file, which a fresh clone downloads, getting started and offline processing printed "Unknown (Property iprangestart not found in ip)" for every property only the Enterprise data file has, and the web pages did the same. They now show each one as needing the Enterprise data file and give a link to ask for a copy. The web pages also show their contact message, which they showed only for the tier 'Lite', a tier no IP Intelligence data file has. Metadata and performance say what the Enterprise data file adds. Compare, metrics and suspicious looked for the Enterprise data file, and the mixed example for a fuller Device Detection data file, before the Lite ones, and each overflowed the stack searching the disk for a file that was not there. They now use the Lite data files by default. Compare, metrics, suspicious and translation need properties only the Enterprise data file has, so with a free data file each says which, gives the link and stops, where suspicious printed false and 0 as if they were answers and translation listed every country alphabetically under most probable first. The translation test returns early when the example stops. The performance example downloads the free ASN data file when it finds no data file, rather than the Lite file, and reads AsnName when the data file has no RegisteredName. Without a licence key the data file update example threw "No license key available". It now shows the ASN data file, says that an update needs a licence key, gives the link and finishes. Its check that stops an Enterprise data file being downloaded over a free one compared the tier with 'Lite', and now checks for the Enterprise properties. The automatic update examples link to the contact page for a licence key. The shared helpers change as the Device Detection ones do, and gain a check of whether the data file is free and a stop for an example that needs properties only the Enterprise data file has.
CI puts the Enterprise IP Intelligence data file under the Lite file's name, and a fuller Device Detection data file under the Lite name in the tests folder, so nothing tested the free files a fresh clone uses. Each package gains a test that runs every on-premise console example in its own process, from its own folder, with the data file and licence key settings removed, and checks that it finishes, shows what needs the Enterprise data file and gives the link to ask for a copy. The web examples are checked in the test process on the free files. The Device Detection examples find the Lite data file themselves, in cxx/data/device-detection, where CI links the Lite asset. The IP Intelligence examples are given the free Lite data file, which CI now fetches as the 51Degrees-LiteIpiV41.ipi asset and links into cxx/data/ip-intelligence, and the performance and update examples find the ASN data file themselves. Without an IP Intelligence key the Lite asset is also linked under the Lite name, so it is not downloaded twice. The tests run with the integration tests, which run every test under examples.
The comment on NEEDS_ENTERPRISE said every property the examples ask for is in the Enterprise data file, which nothing in the examples checks. The web page test gave the wrong reason for removing usage sharing, which is removed so that the test sends nothing to 51Degrees.
…e file The Enterprise IP Intelligence data file is over 6 GB, and the MaxPerformance profile holds all of it in memory, so an example that picks the Enterprise file needed that much free memory before its first lookup. The examples built for the Enterprise file, or that pick it when they can find it, now use LowMemory, which reads the file as it goes. These are the suspicious and update data file examples, the three automatic update examples, which download the Enterprise file with a licence key, and the IP Intelligence engine of the mixed console and API examples, whose device detection engine is unchanged. The metrics and compare examples default to LowMemory and take a fourth command line argument, and a matching parameter of run, that names another profile, such as MaxPerformance for speed where the machine has the memory. Examples that load every property from the Lite file, the readme quick starts, the performance example and the device detection examples keep their profiles.
The examples branch moves the IP Intelligence examples onto the free data files, and this branch makes the examples built for the Enterprise data file use LowMemory. Both change the same eight files, so they are resolved here once, keeping the examples branch's changes in each. After that branch, the mixed console example uses the Lite data files for both engines, so it keeps the examples branch's profiles. The suspicious, metrics and compare examples still need the Enterprise data file, so their comments now say they need it rather than that they pick it first. The update data file example's new pipeline for when there is no licence key also uses LowMemory, because a file given to that example is the Enterprise data file.
jwrosewell
changed the base branch from
rc
to
feature/examples-free-data-files
November 8, 2026 15:04
…e test The country translation example needs the Enterprise data file, so it moves from Balanced to LowMemory like the other examples built for that file. The compare test names MaxPerformance, because CI gives it the Enterprise file and holds all of it in memory, where LowMemory looks up about a third as many records a second. The compare header now says the fourth command line argument names the profile, as the metrics header does, and the metrics header says "command line argument" where it said "parameter".
jwrosewell
changed the base branch from
feature/examples-free-data-files
to
rc
November 9, 2026 00:13
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #32. Retarget to rc before #32's branch is deleted, or GitHub closes this pull request.
The rule
An IP Intelligence example that is built for the Enterprise data file, or that picks the Enterprise file when it can find one, now uses the LowMemory performance profile. The Enterprise IP Intelligence file is over 6 GB (6,488,178,052 bytes, or 6.04 GiB, for the file published on 7 November 2026, and it grows a little with each release), and the MaxPerformance profile holds the whole file in memory. Someone running one of these examples with that file needed that much free memory before the first lookup.
Examples that load every property from the free Lite file keep the profile they have. The metrics and compare examples, which process many addresses and where speed matters, start on LowMemory and take a command line argument that chooses MaxPerformance where the machine has the memory for it. Each changed comment says why.
The same change for Java is 51degrees-java#73.
How this sits on #32
#32 moves the examples onto the free data files and changes eight of the files this pull request changes. This branch merges #32's branch and resolves those files once, keeping #32's change in each and applying the rule on top. #32 changes which examples pick the Enterprise file, so the rule lands differently on two of them.
What changed, example by example
All paths are under
ip-intelligence/fiftyone.ipintelligence.onpremise/examples/onpremise/.suspicious-console/suspicious.jsupdatedatafile-console/updatedatafile.jsautomaticupdates/dataFileSystemWatcher.js,updateOnStartUp.js,updatePollingInterval.jsmixed/gettingstarted-api/gettingStarted.jsmetrics-console/metrics.jsruntakes the profile as a sixth parameter with LowMemory as its defaultcompare-console/compare.jsruntakes the profile as a fifth parameter with LowMemory as its defaultcountry-translation-console/gettingStarted.jssrc/ipi/ipi.clines 237 to 249)The compare example's test names MaxPerformance, because CI gives that test the Enterprise file and has the memory to hold all of it. With LowMemory the test would take about three times as long, as measured below. The example's own default stays LowMemory.
The new arguments follow the positional style each example already uses, and the Java change takes its profile the same way. A profile name the engine does not know is refused by the engine with its existing message.
Left as they are, as decided:
mixed/gettingstarted-consoleandmixed/gettingstarted-web, which use the Lite files for both engines.metrics.jsholds raw NUL characters in three string literals, so git does not convert its line endings. Its lines were edited byte for byte, and its diff here is only the profile lines.The trade-off in the metrics and compare comments
The comments say that LowMemory reads the file as it goes and uses little memory but runs much more slowly, and that MaxPerformance holds all of the file in memory and runs much faster. The .NET version of the metrics example measured reading profile values at 166 a second with LowMemory against 2,013 a second with MaxPerformance on the same file (
Metrics-Console/Program.cslines 482 to 495). That example walks profiles, where the Node one looks up sampled addresses, so the figures show the size of the gap rather than what the Node example will do. On the Enterprise file published on 7 November 2026, on Windows x64 with Node.js 24, the compare example looked up about 191 records a second with MaxPerformance and about 68 a second with LowMemory (12,145 records in its first 180 s, when I stopped it). Its test, which looks up 24,597 records, passed with MaxPerformance in 136.7 s, so with LowMemory it would take about six minutes.On the free Lite file, which is 456 MB, the compare example as it was before #32 (when it still ran on Lite rather than stopping) took 38.0 s and ended at 102 MB resident with LowMemory, against 6.0 s and 557 MB with MaxPerformance, one process per profile.
How it was checked, on the merged result
I built both native modules with node-gyp 12 and
-j 1on Node.js 24, Windows x64, and used the free data files.node --checkpasses for each.run(suspicious, update data file, metrics and compare) were required withJEST_WORKER_IDset so their command line code did not start, and the mixed API example exportedserverandsetPipeline. The three automatic update examples, run withoutLICENSE_KEY, printed Run every example on the free data files, and show what needs the Enterprise data file #32's licence key message and exited 0.examplesWithoutSettings.test.js, which runs each example from the command line as a fresh clone does, through the IP Intelligence Jest project. The Lite device detection file was copied for the run intoip-intelligence/fiftyone.ipintelligence.onpremise/tests/(an ignored location), because the mixed tests look for it only in the IP Intelligence data folders and are skipped otherwise.The skipped test is the metrics test, which is
test.skipbecause a full run needs the Enterprise file. #32's test covers the metrics, compare, suspicious, update data file and automatic update examples from the command line, and each says what it needs and stops on the free files as #32 intends.How the review fixes were checked, on head 37705fa
The IP Intelligence native module was built the same way, and the commands were run from the repository root. For the Enterprise runs the file published on 7 November 2026 was copied to
ip-intelligence/fiftyone.ipintelligence.onpremise/tests/51Degrees-EnterpriseIpiV41.ipi(an ignored location), where the test helper looks first.node --checkpasses for each.test.skip.run, ordered all 250 countries with GB, IE and IM first for the sample address, in 1.5 s with 89 MB resident.metrics.jsstill has CRLF line endings and its three NUL characters, and its diff in this commit is only the two header lines.What remains
The full build and test matrix has not run on head 37705fa. It needs to pass before this merges, and its run belongs here.
No version number is changed.
Full build
The full build and test matrix passed on head c9c21bc in run 42571651344 on 9 November 2026, 10 jobs succeeded, run as a dry run so nothing merged.