Repository navigation
Release candidate - #112
Draft
jwrosewell wants to merge 103 commits into
Draft
jwrosewell wants to merge 103 commits into
jwrosewell wants to merge 103 commits into
Conversation
The README now opens with what the code does in the words developers search for, then the install command, try-it links and, where there are any, the repositories it was created from. It ends with a Find out more section, and the logo is a file in the repository. Everything else the README said is kept.
) pipeline-java, device-detection-java, ip-intelligence-java, location-java, device-detection-java-examples and ip-intelligence-java-examples are archived, and the note at the top of each README names the folder of this repository that now holds its code. Four READMEs and 33 example source files still linked those repositories, so each of the 39 links now points at the same file, folder or README heading on main here. Only the URLs change. The History section of the root README, two Java string constants, the creator context demo page and its test, and the four parent poms that the open package text pull request (#27) changes keep their links.
…33) The examples moved to examples/, which the device detection and IP Intelligence Doxyfiles already read, so the old folders go. Point the location IMAGE_PATH at the shared documentation images, and name the native model example console/NativeModelCloud.java in place of the doubled example command.
#32) When the native library fails to load, both on-premise engines suggest rebuilding the binary from the source code and instructions in device-detection-java or ip-intelligence-java, which are archived. The instructions for building the native binary are now in the README of device-detection/ and ip-intelligence/ here, beside the engine source, so the message links those folders. The creator context demo page linked pipeline-java as the source of the demo and of the code it runs on, and now links pipeline/ here. The test that checks the page ends with somewhere to go next asserts the new address.
Co-authored-by: Automation51D <51DCI@51Degrees.com>
Co-authored-by: Automation51D <51DCI@51Degrees.com>
…build when one is left open (#44)
Co-authored-by: Automation51D <51DCI@51Degrees.com>
Co-authored-by: Automation51D <51DCI@51Degrees.com>
…erprise data file (#48)
* Test a check that starts before registration stores its future registerDataFile schedules the first check and only then stores the future it gets back, so a check with a short delay can start in between. If the check reads the future in that gap it finds none, clears nothing, and its own re-registration then finds the stored future and schedules no next check. DataUpdateService_UpdateFromFile_FileUpdated failed this way once in a full build on macOS x64 with Java 11. The two new tests force that order on every run, one for a check that applies an update and one for a check that applies none, which is the path the retry after a 429 response takes. Registration is held after scheduling until the check has read the future or is blocked on a lock that registration holds, and the check's first read waits until registration has stored the future. Every wait is bounded, so a design that deadlocks fails rather than hangs. Each failure message lists the order in which the two threads read and wrote the future. For #70 * Store a data file's future before its check can read it Every read and write of a data file's future now happens while holding one lock in the data update service. registerDataFile holds it from scheduling the check to storing the future, and the check takes it before it cancels, clears or replaces the future, so a check that starts at once waits until registration has stored its future. Unregistering takes it for the same reason. The lock is the service's own rather than the data file's update lock, because the file system watcher holds the update lock while it refreshes the engine and while it waits for the new file to open. Taking that lock for the future would make unregistering wait for a refresh. Only scheduling, cancelling and storing happen while holding the new lock, so it is never held while waiting for the configuration lock or an update lock. For #70 * Word the new comments to match what the lock and the tests cover The lock's comment now says it covers the service's own reads and writes of the future, the test data file's comment says it records the check's reads, and the second test's comment says which checks share its path. No code changes. For #70
The cloud tests under examples/ skipped when no resource key was set, so a run with no key reported success having called nothing. Each now asks its KeyHelper for the key, which reads _51DEGREES_RESOURCE_KEY, then _51DEGREES_RESOURCE_KEY_BESPOKE, then the names the example reads today, each as an environment variable and then as a system property, and fails the test naming every name tried, the product the key has to carry, how to see what a key carries and where to create one. KeyHelper.getOrSetResourceKey passed a null key to System.setProperty, which throws a NullPointerException, whenever no key was found and it was asked not to throw. It now keeps only a valid key. The Device Detection web cloud test sends a known User-Agent and checks that the page shows the browser and platform the cloud engine detected, rather than accepting any 200 response.
* Default IP Intelligence to the Balanced performance profile The IP Intelligence on-premise specification gives Balanced as the default performance profile. The on-premise pipeline builder defaulted to MaxPerformance, and the engine builder applied no profile unless the caller set one, so it ran with whatever configuration the native ConfigIpi constructor starts from, which is the in-memory one in the C library this repository pins. Its setPerformanceProfile documentation and @DefaultValue already said Balanced. The engine builder now applies Balanced to its native configuration in a field initializer, before the constructor sets the concurrency, so a builder whose caller sets no profile is Balanced whatever the native library starts from. The pipeline builder holds no profile until the caller sets one and passes it on only then, because applying Balanced again would replace the concurrency the engine builder set. BuilderTests and the new OnPremisePerformanceProfileTests compare the collection settings each builder gives the engine with the Balanced and in-memory settings the native library defines. * Say which profile the metadata example uses, and the Java way to get the engine The comment above the engine in MetadataOnPrem said the max performance profile while the code sets LowMemory, and gave the C# call for getting the engine from a pipeline. It now names the low memory profile and gives pipeline.getElement(IPIntelligenceOnPremiseEngine.class). * Give up ownership of the collection settings the profile tests read The generated ConfigIpiSwig wrappers mark getComponents, getProperties, getValues and getProfiles as owning the CollectionConfigSwig they return, although each getter returns a reference to a member of the native configuration. When the garbage collector finalized one of those proxies, the native library freed memory inside the configuration, which aborted the test JVM. The tests now give up ownership of every collection they read, so finalizing those proxies frees nothing. The tests that name MaxPerformance and LowMemory are renamed to say that they prove the named profile replaced Balanced, because the two profiles give every collection the same settings.
* Test a check that starts before registration stores its future registerDataFile schedules the first check and only then stores the future it gets back, so a check with a short delay can start in between. If the check reads the future in that gap it finds none, clears nothing, and its own re-registration then finds the stored future and schedules no next check. DataUpdateService_UpdateFromFile_FileUpdated failed this way once in a full build on macOS x64 with Java 11. The two new tests force that order on every run, one for a check that applies an update and one for a check that applies none, which is the path the retry after a 429 response takes. Registration is held after scheduling until the check has read the future or is blocked on a lock that registration holds, and the check's first read waits until registration has stored the future. Every wait is bounded, so a design that deadlocks fails rather than hangs. Each failure message lists the order in which the two threads read and wrote the future. For #70 * Store a data file's future before its check can read it Every read and write of a data file's future now happens while holding one lock in the data update service. registerDataFile holds it from scheduling the check to storing the future, and the check takes it before it cancels, clears or replaces the future, so a check that starts at once waits until registration has stored its future. Unregistering takes it for the same reason. The lock is the service's own rather than the data file's update lock, because the file system watcher holds the update lock while it refreshes the engine and while it waits for the new file to open. Taking that lock for the future would make unregistering wait for a refresh. Only scheduling, cancelling and storing happen while holding the new lock, so it is never held while waiting for the configuration lock or an update lock. For #70 * Word the new comments to match what the lock and the tests cover The lock's comment now says it covers the service's own reads and writes of the future, the test data file's comment says it records the check's reads, and the second test's comment says which checks share its path. No code changes. For #70 * Test the random delay before an available update is in seconds * Add the random delay before an available update as seconds
… assets' Co-authored-by: Automation51D <51DCI@51Degrees.com>
…ets (#65) * Refuse a 51Did text with white space inside it, or both base64 alphabets The 51Did specification (identifier-layout.md, specifications#88 and #90) says a reader ignores white space before and after the text, being any character with the Unicode White_Space property, refuses any other character outside the alphabet, white space inside the text included, and refuses a text holding a character from each alphabet, checking that before it maps - and _ onto + and /. FodId.toStandardBase64 trimmed with String.trim, which removes every character up to U+0020, control characters included, and nothing above it. It then mapped - and _ onto + and / without checking for a mix, and the OWID decoder skips spaces, tabs and line breaks inside the text. So a mixed text always read, and a text with spaces, tabs or line breaks inside it read whenever the padding worked out. toStandardBase64 now trims exactly the 25 White_Space characters, gives null for a text with white space left inside it or with a character from each alphabet, and only then maps one alphabet onto the other. tryFromBase64 reports INVALID_BASE64 for such a text, and fromBase64 throws the OwidException naming INVALID_BASE64. White space alone still reports MISSING_INPUT. This matches 51degrees-dotnet#85. * State why each character in the near-miss test is not white space * Say that the 51Did reader refuses white space inside and mixed alphabets itself The README's status table and the tryFromBase64 documentation said an envelope fault carries the OWID library's status as the envelope reader found it. White space inside the text and characters from both base64 alphabets are now refused by FodId before the OWID library sees the text, under the same INVALID_BASE64 status, so both now say so. The test helper that signs envelopes until one holds both + and / usually needs a few tries, not one or two as its comment said.
…polling carries on (#98) * Test that polling carries on after a refresh from a watched file A new data file found by the file watcher refreshes the engine, and then polling for updates must carry on, as the newFileAvailable steps of the data update specification say. The test runs the watcher's task itself and records each polling timer the service schedules. After the refresh there must be a second timer, held by the data file. * Register the data file again after a refresh from a watched file updatedFileAvailable cancels the polling timer before it refreshes the engine. checkForUpdate registers the data file again afterwards, which sets a new timer, but dataFileUpdated, which the file watcher calls, did not, so HTTP polling stopped after the first refresh from a watched file. The newFileAvailable steps of the data update specification say to resume polling after loading the file and before the callbacks. dataFileUpdated now registers a registered data file again after the refresh and before the callbacks. Registering a file that is already registered only sets the timer, because the watcher and the start-up check are each set up once. The comment on updatedFileAvailable said it registered the file itself, which it does not, and now says the caller does. * Leave a data file Windows still holds to be deleted at exit On Windows the full build failed in the watched file test's clean up, with "The process cannot access the file because it is being used by another process". The data update service reads the file through BinaryReader, which maps it, and Windows keeps a mapped file until the buffer is collected. The test's assertions had passed. The clean up now deletes what it can and leaves the rest to be deleted when the test run ends, as the existing watcher test does by ignoring a failed delete. * Test that a watched file that cannot be opened is given up on The data update specification's Errors table says a watched file that cannot be opened for reading within 10 minutes ends the update with AUTO_UPDATE_TIMEOUT and logs nothing, as .NET does. Java tries for ever with no pause, so the new test, which shortens the wait to one second, never sees the update end. This commit adds the AUTO_UPDATE_TIMEOUT status, the ten minute wait and the half second pause as constants, and the field the test shortens, so the test compiles and fails on behaviour. The loop does not use them yet. The setup the two watched file tests share moves into helpers. * Give up on a watched file that cannot be opened after ten minutes The file watcher tried to open a new data file for ever, with no pause and a warning on every try, while holding the data file's update lock. It now tries every half second for up to ten minutes, logs nothing, and then ends the update with AUTO_UPDATE_TIMEOUT, leaving the engine and the polling timer as they were. This is what .NET does and what the Errors table of the data update specification says. The file is registered again only when the refresh ran, because only the refresh cancels the polling timer. * Test that an interrupted wait for a watched file leaves the update unfinished An interrupt, as unregistering the data file gives the watcher's task, ends the wait for a file that cannot be opened, and the update reports AUTO_UPDATE_TIMEOUT although the file was not tried for the whole wait. The new test requires AUTO_UPDATE_IN_PROGRESS, the status for an update that did not finish, so it fails on the timeout. The two tests of a file that cannot be opened now share one fixture. The stand-in file pauses without sleeping, so an interrupt reaches the service rather than the stand-in. * Leave an interrupted wait for a watched file unfinished, not timed out An interrupt ended the wait and reported AUTO_UPDATE_TIMEOUT, although the file had not been tried for the whole wait. The update now keeps the status it started with, AUTO_UPDATE_IN_PROGRESS, which is what it reports for any watcher event that does not finish an update. The engine and the polling timer are left as they were, as for a timeout. The IOException the loop catches is no longer named e, as nothing reads it.
…e Sequence Element (#80) * Test the web integration switches, the Sequence Element data and usage sharing characters The tests cover the init parameters that set the JavaScript and JSON paths and switch the Sequence Element and the Set Headers Element, the Element Data the Sequence Element writes, how the JSON Builder Element shows that data, and the characters XML 1.0 allows in a shared Evidence value. * Add switches and paths to the web integration, and Element Data to the Sequence Element The filter takes init parameters for the JavaScript path, the JSON path, and whether the Sequence Element and the Set Headers Element are added. The defaults keep today's paths and today's behaviour. The Sequence Element writes session-id and sequence as Element Data as well as Evidence. Usage sharing keeps every character XML 1.0 allows instead of replacing everything above U+00FF. * Test the MVC integration's paths and element switches * Give the MVC integration the filter's paths and element switches, and say what the Lite file lacks The MVC configuration takes the JavaScript path, the JSON path and the two element switches the filter takes, with defaults that keep today's behaviour. The example comments about the Lite data file say it has a small number of Properties, as the specification does. * Test the Sequence Element data under the key sequence * Key the Sequence Element data as sequence, as the other languages do * Do not call an overridable method from the constructor Java 21 warns about it, and the build treats warnings as errors. * Test that the JSON is served from /51dpipeline/json and from the old path * Serve the JSON from /51dpipeline/json by default, and from the old path as well The specification's JSON path becomes the default in the filter and the MVC integration, and the JavaScript they serve calls it. The path 51Degrees.core.json is still answered, and json-path still sets the path. * Test the Sequence Element with session ids and sequences of every kind * Treat the session id and the sequence independently, as the specification says A session id that is not a non-empty string, and a sequence that is not a whole number or text of decimal digits that fits in an int with room to add one, count as absent. The Evidence and the Element Data are written from the same values. * Test that the usage sharing filter never includes the web request * Keep the web request out of the usage sharing filter, as .NET does * Expect a new session id when the Sequence Element is given an empty one * Test that a build with the JSON Builder Element and no Sequence Element is refused The json-path test uses a path that is not the default. * Refuse a build with the JSON Builder Element and no Sequence Element The JSON Builder Element fails on every request without the sequence number, so the build says which option to change. * Use a configuration without the JSON Builder Element when client-side properties are off * Test the exact path match and the rule that a sequence is not negative - FiftyOneJSServiceCoreTests checks that a configured path, with or without its leading slash, answers only that path inside the application, so /anything/json, /json/more and /notjson are not answered, and that the context path is not part of the path. - SequenceElementTests checks that a negative sequence of any whole number type, a sign and white space each count as no sequence, and that 0 gives 1. * Match the JSON and JavaScript paths exactly, and count a negative sequence as absent FiftyOneJSServiceCore.Default answered any request whose whole URL ended with the configured path, so a configured path of json also answered /anything/json. The path is now compared with the whole path of the request inside the application, after the context path, ignoring case and with or without a leading slash, and a request with no URI is not answered. The old 51Degrees.core.json path is matched the same way. The Sequence Element took a negative whole number as a sequence. The Sequence Element specification counts a sequence only from 0 up to 2,147,483,646, so a negative one now counts as absent. * Give a caller with no JSON path the default path
…#99) * Test that the cloud request carries every Evidence value but the reserved names The Cloud Request Engine must send every Evidence value apart from the reserved names resource, license, licence and values, under any prefix and in any letter case, whether or not the cloud service lists the key in its evidencekeys answer. The test that only accepted Evidence is sent is replaced by one that the request carries Evidence the service does not list, and still leaves out reserved names under prefixes it does not list. * Send every Evidence value but the reserved names in the cloud request The Cloud Request Engine sent only the Evidence whose keys the cloud service lists in its evidencekeys answer. It now sends every Evidence value apart from the reserved names resource, license, licence and values, under any prefix and in any letter case, and apart from the web request, as .NET does. What the service uses changes over time, and knowing about the other Evidence helps it improve. The accepted Evidence still serves as the engine's Evidence key filter, for the Pipeline and the web Vary header. * Test that Evidence with no prefix is sent, and that prefixes ignore case Keys only usually carry a prefix, as the Overview of the Evidence specification says, and .NET sends a key with no prefix under its whole key unless it is a reserved name. Java drops it. .NET also compares prefixes without regard to case, where Java treats Query.User-Agent as Evidence with some other prefix, so header Evidence of the same name takes precedence over it with a warning. * Send Evidence with no prefix under its whole key, and compare prefixes ignoring case A key with no separator now has the whole key as both its prefix and the name it is sent under, as .NET's AddQueryData takes it, so it is sent unless it is a reserved name. Prefixes are compared without regard to case when Evidence is grouped and when a conflict is reported, as .NET's KeyHasPrefix and AddQueryData compare them. * Test where Evidence with no prefix ranks in the cloud request Evidence with no prefix ranks with Evidence of any prefix other than query, header and cookie, so header Evidence of the same name takes precedence over it, and a bare key that is itself a prefix, such as query, ranks with that prefix's Evidence, as in .NET. The Evidence specification's precedence order covers prefixes only, so the test records the choice. * Say where Evidence with no prefix ranks, and what hasKeyPrefix compares The getFormData comment now gives the precedence order and where Evidence with no prefix ranks in it, and the hasKeyPrefix comment says a key with no separator is its own prefix. * Test that a repeated cloud request still hits the cache * Make the cache test's second request differ in Evidence the service does not list Two identical requests share a cache entry whatever the key is built from, so the test passed without showing that the key is built from the advertised Evidence. The second request now differs only in header.sec-ch-ua-platform, which the cloud service does not list, and one request must still be posted, carrying the first request's value. The test fails if the cache key takes every Evidence value.
…t name (#88) * Test that the object name refuses the Window properties the specification lists * Refuse the Window property names the specification lists as the object name
…#101) * Read property values by index for an engine built with names * Test the property index rule on both on-premise engines * Test that a property name given twice has the index of its first place C gives a property the place of its name among the names given, the first where the name is given more than once, and compares names without regard to case. The builder drops only an exact repeat, so IsMobile and ismobile both reach C. The engines' map keeps the last index for such a name, so the new tests fail on it. The test comments no longer say which test goes red or describe the change. * Take the property indexes from the names the builder gives, keeping the first Each engine now makes its map from the list of names its builder passes to the native engine, in the constructor, before any data is loaded. That is the list C takes the required property indexes from, one place for each entry in the order given, so the map holds the same indexes without running a detection on empty Evidence to get a results object, which for an engine built with names walked every component's graph at build. A name given more than once keeps its first place, as C keeps it. The map is final and never rebuilt, and the native engine's required properties configuration is no longer read back. * Time a read of IsMobile by name against a read by index PropertyIndex_ReadByIndexAgainstByName builds the native engine with the same names, reads IsMobile 20,000 times by name and 20,000 times by index from the same native results, and 20,000 times through the typed accessor, in five rounds after a warm-up. It requires every read to give the same value and prints the lowest, median and highest time of one read, so each build reports the figures for its platform. * Time the read by name through the overload the reads use A read by name with no index goes through getValueAsBool(Swig.asBytes( name)), which is also how every read by name went on rc, so the timing test now reads by name that way rather than through the String overload. * Test a reload from a data file that lacks a property the engine was named with The engine is built on the TAC file with a name the Lite file lacks, then reloaded from the Lite file. The name keeps its index, a read of it by index gives no value rather than an error, as a read by name does, and IsMobile still reads its own value. C maps a named index to the data set before it reads, and a name the data set lacks maps to -1, so the read reports no value. The test pins that, so it has no red.
) * TEST: Create the 51Did for another address as a licensed caller anIdentifierForAnotherAddress_ReportsEveryFactor asked for a 51Did for a stated address with a resource key alone, and expected a mismatch when the same connection then presented it. The cloud service takes the address of such a caller from the connection, whatever address the caller states, so the verdict is verified and the test fails wherever a licence key is set. The test now creates the identifier as a licensed caller, the caller whose stated address the service uses, with both keys in the form body. The verdict is then a mismatch that names all nine factors and reports the browser address as one that differs. * TEST: Make every cloud test call the live service and fail when it cannot Tests named as cloud tests skipped themselves when no resource key was set, and the example test turned any failure into a skip, so a job that never reached the cloud service still reported success. The resource key the workflows pass never reached Maven either, because ci/run-unit-tests.ps1 took no Keys parameter. - DidClientLiveTests fails with a message naming every variable tried and the fodid product the key must carry, instead of four assume calls and an assumeNoException that swallowed a missing creator context. - The cloud engine example test no longer catches and skips, and the example now pairs a 51Degrees resource key with the 51Degrees cloud endpoint, so a key found in the environment is sent to the service that issued it. - New CloudRequestEngineLiveTests, because every other test in that module answers the engine from a mock and nothing in this repository called the live service through CloudRequestEngine. - ci/run-unit-tests.ps1 takes Keys and passes the key to Maven, and the workflows now supply the key entitled to the bespoke products. - The 51Did workflow runs against the key carrying the fodid product and fails when no key is available. Carried into 51degrees-java from pipeline-java, where it was opened as a draft. The nightly, publish and pull request workflows here already pass the bespoke key, and ci/run-unit-tests.ps1 already hands it to Maven as TestResourceKey, which the tests read, so those parts are not needed. The 51Did workflow keeps its loop over the keys with a paired licence key, drops the basic key, which does not carry fodid, and fails when no key is available. --------- Co-authored-by: James Rosewell <james@51degrees.com>
* Test that a repeat is shared at most once in each interval The usage sharing tracker stores the time evidence was first shared and never moves it on. A repeat is held back for the interval after the first share, and after that it is shared on every request until the cache drops it. The specification describes the interval as a sliding window, which Python and Node follow by storing a new time whenever they share a repeat. The test shares evidence, waits past the interval, shares the repeat, and expects a repeat straight after that not to be shared. It fails until the tracker stores the time a repeat is shared. * Share a repeated evidence set at most once in each interval The tracker now stores the time it shares a repeat in the cached Date, so the interval starts again from that share. A repeat is shared at most once in each interval, which is what the specification's sliding window describes and what Python and Node already do. The check and the update are made under the Date's lock, so two requests cannot both share the same repeat. * Give the tracker tests that do not test expiry an interval no runner reaches The tests shared a 50 millisecond interval, so on a slow runner two requests that should count as one repeat could be more than 50 milliseconds apart and both be tracked. ShareUsageTracker_Session_DoNotTrack failed that way on MacOS_X64_Java_11, with the test taking 2.4 seconds. Only the test that sleeps past the interval to test expiry keeps the short one. ShareUsageTracker_Session_Track no longer sleeps, so the session cookie alone keeps its two requests apart. * Wait past the interval with a margin in the repeat evidence test
* Test that usage sharing pauses sending after failed sends Usage sharing in Java has no back-off. Every batch is sent while the service is failing, and is lost. These tests expect the back-off .NET is being given, at its default settings of ten failed sends within 100 seconds and a first pause of two seconds: - after ten failed sends nothing is sent during the pause, and the evidence processed during it is all sent once the pause ends and the service answers, with no more requests processed; - closing while paused returns at once and sends nothing. They fail until the back-off is added. * Pause usage sharing sends after repeated failures Java now has the same back-off as .NET, with the same five settings and defaults. - After setFailuresToEnterRecovery sends fail within setFailuresWindowSeconds, sending pauses for setInitialRecoveryDelaySeconds. A send fails when it throws or the answer is not 200. - When a pause ends one send is tried. If it fails the delay is multiplied by setRecoveryMultiplier, up to setMaxRecoveryDelaySeconds. If it succeeds the back-off ends. - Processing is not affected. Evidence is queued during a pause and new evidence is dropped only when the queue is full. - The consumer waits out the pause itself, on a latch that closing releases, and then sends until fewer than the minimum entries per message are queued, so the queue drains when a pause ends without more requests being processed. - Closing ends a pause at once and sends nothing while paused, logging how many entries were not sent. ShareUsageBackOff holds the rules and takes the time from the caller, so its tests set the time. The consumer calls the new protected waitWhilePaused before each message and recordSendResult after it. New constructors take the five settings, and the existing ones use the defaults. * Check the whole range of failures to enter recovery in the back-off The builder setter rejects a value outside 1 to 100, and the back-off now does too, so a class derived from ShareUsageBase gets the same check through its constructor. * Keep history out of the comments in the changed send code The comments in the functions this change touches described what the code used to do. They now say only what the code does and why. * Test the usage sharing options, the case of keys and the tracker's cache * Add the usage sharing options to share all Evidence and all query string parameters, read keys without regard to case, and keep one tracker cache * Test the timeouts on the legacy send * Test the back-off time comes from the monotonic clock * Time pauses on a monotonic clock and time out the legacy send * Test that an upload to a service that never answers times out * Time out an upload that is never answered * Test that discarding entries from a full queue is logged without a message per entry * Log entries discarded from a full queue without a message per entry * Test that a failed answer reports the status and the content * Test that a failed upload gives the content of the answer * Report the status and the content of a failed answer * Guard the failure deque against unboxing an empty first entry * Let the upload tests keep restricted headers allowed for the run * Remove a file added by mistake
… and fix the README links (#49) * Show the root logo in every README, link the main contributing guide, and fix the README links Six READMEs showed an older 51Degrees wordmark from 51degrees.com rather than the logo the root README shows. Each now shows the root logo as a Markdown image from raw.githubusercontent.com, using .github/images/logo-160.png, a 160 pixel copy of .github/images/logo.png, the size the root README shows it at. The root CONTRIBUTING.md linked the guide in common-ci. It now links the 51Degrees contributing guide, which leads on to the full guide in common-ci. The Developer Documentation link at the top of six READMEs went to the main page of the product's generated API reference, which has no content. They now go to the product's documentation. The root README's table links both that and the class list of the Java API reference, and the pipeline README's sentence about the API reference links the class list. The dependencies page that the Device Detection and IP Intelligence READMEs linked is gone, and the next sentence in each already links the tested versions page, so the sentence is removed. The location README's Maven link went to search.maven.org, which answers 403, and now goes to the artifact on central.sonatype.com. * Link each product's quick start from the READMEs The documentation links went to section pages of the 51Degrees documentation, such as _device_detection__index.html, which only list the pages in the section. They now go to the page in each section that is most helpful to a developer and the most recently kept up to date. Those are the Device Detection, IP Intelligence and Reverse Geocoding quick starts, which show how to choose between the cloud and on-premise and how to start in each language, and Pipeline Basics for the Pipeline API, which has no quick start. The campaign tags of each link are unchanged.
…ber, and test the behaviour (#104) * Take the sequence number check out, to show the new tests failing The JSON Builder specification says the element MUST fail with an error and MUST NOT set its JSON value when no Sequence Element has produced a sequence number. JsonBuilderElement already does that, and nothing tested it, so the regression audit for this release read it as missing. This commit adds the tests and takes the check out, so the build shows them failing. The next commit puts the check back, unchanged, and the tests pass. What reaches rc is the tests alone. * Put the sequence number check back The check is restored exactly as it is on rc, so what this draft adds to rc is the tests alone. The previous commit took it out only to show the build failing without it. * Test that the cause says a Sequence Element has to come first The JSON Builder specification says the error SHOULD say that a Sequence Element needs to be placed before the JSON Builder Element in the Pipeline. The exception the pipeline records says so, and the cause it carries does not, reading "Sequence number not present in evidence. this is mandatory." This test pins the cause's wording, and fails until the message changes. * Say in the cause that a Sequence Element has to come before the JSON builder The cause read "Sequence number not present in evidence. this is mandatory.", which neither says what to do nor reads as one sentence. It now carries the wording 51degrees-go and 51degrees-rust use, so the same misconfiguration reads the same whichever language a deployment uses. The exception the pipeline records is unchanged. * Pin the error message the JSON Builder specification gives specifications#139 turns the SHOULD into a MUST and gives the exact text, so that the message reads the same in every language. It also says that where an implementation wraps the error in another, the one the caller reads carries the message, which in Java is the PipelineConfigurationException rather than its cause. The constant holds the text and the tests assert it word for word on both the recorded error and the cause. They fail until the element uses it. * Give the specification's message when there is no sequence number The PipelineConfigurationException the caller reads said "Make sure there is a SequenceElement placed before this JsonBuilderElement in the pipeline" and its cause said something else again. Both now carry the text the specification gives, which every language carries. One more test writes that text out beside the constant, so that changing the constant fails rather than moving the other assertions with it. * Pin the JSON Builder links at the specification commit that merged specifications#139 merged at 13:19 as d72eb074, with one more commit saying that the message is the whole text of the error and that a prefix a language's error type writes is not part of it. The links moved from the draft head and from 67ead698, which predates the message. No code changes. * Say in the helper what it now asserts The helper checked that the message named a sequence element and now checks the whole message, so its note says so.
The integration step runs the pipeline developer examples' ExampleTests again without the key the unit test run passes, and the cloud engine example's test now fails when it has no key rather than skipping, so every full build failed there. The step now passes -DTestResourceKey as the example build below it does.
… engine built with no names (#82) * Move cxx to the 51degrees-cxx rc commit 865cee97 * Remove the ownership workaround from the IP Intelligence tests The configuration's collections are borrowed. Test that each one the configuration hands out is borrowed, and stop forcing the flag by reflection in the two tests that read them. * Generate the IP Intelligence SWIG wrapper again for cxx 865cee97 SWIG 4.5.1 now marks the configuration's collection getters as borrowed, as the corrected MetaData.i asks. The Device Detection wrapper is unchanged apart from comments recording the include path, so it is left as it was. * Move cxx to the 51degrees-cxx rc commit b4f769ad Takes the eight cxx commits after 865cee97, from 51Degrees/51degrees-cxx#118, 51Degrees/51degrees-cxx#139, 51Degrees/51degrees-cxx#143, 51Degrees/51degrees-cxx#136, 51Degrees/51degrees-cxx#131, 51Degrees/51degrees-cxx#141, 51Degrees/51degrees-cxx#144 and 51Degrees/51degrees-cxx#145. * Move cxx to the 51degrees-cxx rc commit cfad8978 Takes the two cxx commits after b4f769ad, from 51Degrees/51degrees-cxx#130 and 51Degrees/51degrees-cxx#146. This is the cxx commit for the release. * Move cxx to the 51degrees-cxx release commit 7c4ac436 * Test that an engine built with no names reads by index An engine built with no names should take its names from the native engine after the first data file loads, give every property of that file an index the native results agree with, and keep those indexes through a reload from a file whose properties are a superset in another order. The read by name against the read by index is measured for such an engine as it is for one built with names. * Read by index on an engine built with no names 51degrees-cxx 7c4ac436 keeps the first data file's property names, in their order, across every reload of an engine created with no names, and its C++ reads by index work on such an engine. The engine therefore makes its map of names to indexes once, from the native engine's required properties after the first data file loads, and never again. A property only a later data file has is not in the map, so it reads by name and gives no value until the engine is built again.
* Test the data update logging, the If-Modified-Since time and the licence key check The tests read the messages and levels from the Logging section of the data update specification, check that update on start-up works with auto update off and that the file system watcher follows the auto update switch, and that an engine whose update URL is the Distributor's is refused without a licence key. Tests that built an engine with the defaults now switch auto update off or give a key, as the specification's defaults table says. * Make the data update follow the specification Send the data file's last modified time as If-Modified-Since when the file has no publication date. Register the data file with the update service when update on start-up or auto update is on, and let the auto update switch govern the file system watcher. Log the actions and errors with the messages and levels in the Logging section of the data update specification. Refuse to build with update on start-up or auto update on, the Distributor's URL and no licence key, naming the option to turn off. * Test that HTTP status failures are warnings * Log a 429, a 403 and any other failing HTTP status as a warning The Logging section of the data update specification makes a failed download a warning, with the error detail from the server. * Test that the update on start-up log names the engine type * Name the engine in the update on start-up log before the engine exists The builder gives the data file its engine's type name, taken from the builder's own name, and the update service logs it until the engine is set. * Test the masked license key, the write error, the programmatic file comparison and the license key check across data files - DataUpdateServiceLoggingTests gives the data file a license key in its URL and checks that every message, at every level, and every exception logged with one, shows only the key's last 4 characters, for a 304, a 200, a failed connection, a failed download, a 403 and no response. - A failure writing the temporary file logs the Errors table's text for a write error. - With automatic update off and the file system watcher flag at its default, a programmatic update still compares the file on disk. - A builder with two data files, the second without a license key, refuses the build before it registers either. * Show the license keys only masked in every data update message The data update URL carries the license keys in its query. The messages for checking, downloading, a 304 reply, a failed connection and a failed download now name the URL with each key masked to its last 4 characters, and the error detail and the exception logged with a failure are masked too, because an HTTP library's exception often names the URL. The helper that named a data file by its engine for the old messages is no longer used and is removed. * Log a failure writing the temporary file as the write error A failure while writing the downloaded data to the temporary file was logged as the "Error downloading file" warning. The download stream now records whether a read failed, so a failure that is not a read logs the Errors table's "Error writing to data file location or temp file location" text, "An error occurred when writing to <file path>. Error detail: <error>", at error level, and returns AUTO_UPDATE_ERR_READING_STREAM, whose documentation covers a failed output stream. A failed read is still the download warning. * Compare the file on disk on a programmatic update, and when no watcher exists checkForUpdate skipped the comparison of the data file with the temporary file whenever the file system watcher flag was on. The watcher is created only when automatic update is on, so with automatic update off and the flag at its default, a programmatic update never looked at the file on disk. The data update specification's Programmatic update section says the file at the configured location is used "irrespective of whether Automatic Update from File is enabled". A programmatic update now always compares the files, and the timer compares them whenever no watcher exists, rather than whenever the flag is off. * Check every data file's license key before registering any data file preCreateEngine checked each data file's update service and license key inside the loop that registers the data files, so a builder with several data files could register the first, and run its update on start-up, before refusing the build for a later one. Every data file is now checked first, and a refused build registers none. * Expect the one refresh error in the test that rc added for a failed refresh DataUpdateService_CheckForUpdate_FileRefreshFails_TimerSetAgain, which rc added with the change that refreshes the engine once for an in-memory update, allowed for a warning on each of the 10 refresh tries. This draft logs each try at debug and one error after the last, the Errors table's "An error occurred while applying a data file update to <Engine type name>" row, so the test now allows that one error. The full build of the merge failed on it. * Correct the grammar of the missing data update service message * Test that a failed refresh logs the Errors table's message data-updates.md gives two rows for AUTO_UPDATE_REFRESH_FAILED, being "An error occurred when applying a data update to engine '<Engine type name>' after <tries> tries." for the file, which is tried 10 times, and the same without the tries for data held in memory. * Log the Errors table's message when a refresh fails A refresh from the file, tried 10 times, logs "An error occurred when applying a data update to engine '<Engine type name>' after 10 tries.", and a refresh from data held in memory logs the same without the tries, as data-updates.md's two AUTO_UPDATE_REFRESH_FAILED rows give them, and as .NET and Go word them. The exception is still passed to the logger.
* Add the environment variable name check, red before the rename
Reads every tracked file and fails on a name that starts with the digit,
which a POSIX shell cannot set, and on an old name that is still read
where each variable has one name. Pester tests give the check one line at
a time and a workflow runs both on every pull request.
This commit is the check alone, so it fails over the repository as it
stands and names every place the next commit changes.
* Give every environment variable one _51DEGREES_ name
A POSIX shell cannot set a name that starts with a digit, so the three
names that did move to the leading underscore, and the names read behind
a _51DEGREES_ one are no longer read at all.
51DEGREES_IPI_PATH, 51DEGREES_DD_PATH and 51DEGREES_RESOURCE_KEY become
_51DEGREES_IPI_PATH, _51DEGREES_DD_PATH and _51DEGREES_RESOURCE_KEY.
FOD_CLOUD_API_URL becomes _51DEGREES_CLOUD_ENDPOINT, which is the name
the Rust library already reads, and the constant holding it is renamed
with it. The RESOURCE_KEY and LICENSE_KEY reads behind the _51DEGREES_
names in the 51Did and cloud engine examples and the live 51Did tests
are removed.
* See an old name declared as a key
The old-name half needed a quote, a dollar sign, a brace or a getenv call
in front of the name, so it could not see one declared as a key in a
workflow env: block or listed at the start of a line in a readme. A name
at the start of a line, past any spaces, tabs and a list dash, now counts.
The tests state it, with a workflow line whose key is an old name, a
readme list entry and a shell assignment on the failing side, and a
workflow line whose key is not an old name, and a longer name that ends
in one of these words, on the passing side.
* See an old name braced as a PowerShell or lookup variable
The check misses an old name written as ${env:NAME}, as $ENV:NAME, as
Env:\NAME or as ${{ env.NAME }}, because its prefix set wants the dollar
and the env: adjacent, and its brace branch wants the name straight after
the brace. Both the CI scripts and the pipeline config lookup write the
braced form, so an old name can come back through the file types the check
watches.
These cases fail until the prefix set takes the env token on its own.
* Match the env token on its own, so a braced old name is seen
The prefix set wanted the dollar and the env token adjacent, and its brace
branch wanted the name straight after the brace, so ${env:NAME}, $ENV:NAME,
Env:\NAME and a workflow ${{ env.NAME }} all went past. Matching the token
on its own, in any case and with a following backslash allowed, closes all
four, which is what the Go check already does.
* Wrap the endpoint comment on a whole phrase
…107) * Test that one Evidence name in two letter cases is sent once by precedence Evidence keys ignore letter case, so header.User-Agent and query.user-agent name one field, which is sent once with the name and value of the key that wins by the precedence in evidence.md: query, header, cookie, server, fiftyone and location, then any other prefix alphabetically. * Send one Evidence name once, whatever its letter case, by precedence Evidence keys ignore letter case, so the form fields are now keyed without regard to case. One name under two prefixes, such as header.User-Agent and query.user-agent, is sent once, with the name and value of the key that wins, where before both were sent. The other prefixes now follow evidence.md's order, being server, fiftyone and location, then any other prefix alphabetically, where before server, fiftyone and location were ranked alphabetically among the rest. * Test that two other prefixes are compared as prefixes ab.value ranks before ab-c.Value, because the prefix ab sorts before ab-c, where comparing the whole key put ab-c first, as a hyphen sorts before a dot. _x ranks before a, as the prefixes compare in lower case. * Compare two other prefixes as prefixes, in lower case Two keys under other prefixes were ordered by the whole key, so ab-c.n ranked before ab.n because a hyphen sorts before a dot. evidence.md ranks prefixes, so the prefixes are compared, in lower case as Go, Node, PHP and Python compare them, and the whole key only breaks a tie.
…the properties named (#97) * Test the credentials a Cloud Request Engine accepts and sends Cover the rows of the specification's table of credentials, the data request for a license key alone, a resource key alone, both keys, a resource key with requested properties and all three, the accessible properties request for each, the report of requested properties the answer left out, and the license key staying out of every URL, log entry and error message when a request fails. * Authenticate a Cloud Request Engine on a license key alone A license key without a resource key is accepted when the properties wanted are named. The properties set with setProperty or setProperties, which reached newEngine and were never used, now go to the cloud service as 'values' whenever the list is not empty, beside a resource key, a license key or both, so the service adds them to its answer. The builder refuses at build time only the two combinations the cloud service refuses on every request, being no credential at all and a license key with no requested properties, each with a message that says why the combination cannot work. When a license key is set, the accessible properties request is a POST that carries the license key, and the resource key if there is one, in its body, so discovery finds what the license key grants and the key never appears in a URL. A resource key alone is sent in the query string as before. The engine warns once about requested properties the answer left out. A blank resource key counts as absent, so the data endpoint for a license key alone is the plain json endpoint. * Send only the properties named with setRequestedProperties The previous commit sent the properties set with setProperty and setProperties as 'values'. On rc that list reached newEngine and was never sent, so a configuration that set it, as the .NET cloud web examples do, would change the cloud service's answer. The .NET full build showed that answer losing device.deviceid. The engine builder now keeps its own list, set with setRequestedProperties as a list or as one comma separated string, and only that list is sent. setProperty and setProperties behave exactly as they did on rc, and the refusal for a license key with no requested properties names setRequestedProperties. * Count only the requests a refused build sends in the license key tests * Test that requested property names are sent as given, with two report texts As in .NET, the names sent as values keep the letter case they were given in, and a name given twice in different cases is sent once as first given. With a license key, names the answer leaves out are reported as not covered by the keys, and with a resource key, requested properties and no license key, as not properties, as cloud-request-engine.md says. A null list is refused with the parameter's name. * Send requested property names as given, and say when one is not a property The cloud service reads property names without regard to letter case, so the names sent as values keep the case they were given in, and a repeat in another case is dropped, as .NET now does. A null list of requested properties is refused with an IllegalArgumentException naming the parameter, rather than a NullPointerException from inside the list. With a resource key, requested properties and no license key, a property the resource key does not carry is answered as null with its reason, so a name the answer leaves out is not a property. The warning for that case says so and names no license key, as cloud-request-engine.md asks. * Word the warning for a name that is not a property without the license key The warning for a resource key with requested properties and no license key names no license key, because none was given. * Record the license key tests' fake cloud calls in a synchronized list Discovery on rc fetches the evidence keys on another thread while it fetches the accessible properties, so the fake cloud records calls from two threads. A plain ArrayList could lose a call or hold a null, which failed the resource key discovery and data request tests depending on the order the tests ran in. rc's own discovery tests use a synchronized list for the same reason. * Answer the test cloud's POSTs by URL, as its GETs are rc's Cloud Request Engine tests build engines with a license key, and a license key makes the discovery requests POSTs, which the shared mock answered with the data response. The mock now answers a POST to the accessible properties or evidence keys endpoint as it answers a GET, and the three tests that count the data request count only that request. * Record the license key tests' fake connections in a concurrent map The fake cloud looked each request's URL up in a HashMap that both discovery threads wrote to, so a lost entry left a call with no URL and failed Process_ResourceKeyAndLicenseKey_SendsBoth in one run.
…#102) * Test the web integration switches, the Sequence Element data and usage sharing characters The tests cover the init parameters that set the JavaScript and JSON paths and switch the Sequence Element and the Set Headers Element, the Element Data the Sequence Element writes, how the JSON Builder Element shows that data, and the characters XML 1.0 allows in a shared Evidence value. * Add switches and paths to the web integration, and Element Data to the Sequence Element The filter takes init parameters for the JavaScript path, the JSON path, and whether the Sequence Element and the Set Headers Element are added. The defaults keep today's paths and today's behaviour. The Sequence Element writes session-id and sequence as Element Data as well as Evidence. Usage sharing keeps every character XML 1.0 allows instead of replacing everything above U+00FF. * Test the MVC integration's paths and element switches * Give the MVC integration the filter's paths and element switches, and say what the Lite file lacks The MVC configuration takes the JavaScript path, the JSON path and the two element switches the filter takes, with defaults that keep today's behaviour. The example comments about the Lite data file say it has a small number of Properties, as the specification does. * Test the Sequence Element data under the key sequence * Key the Sequence Element data as sequence, as the other languages do * Do not call an overridable method from the constructor Java 21 warns about it, and the build treats warnings as errors. * Test that the JSON is served from /51dpipeline/json and from the old path * Serve the JSON from /51dpipeline/json by default, and from the old path as well The specification's JSON path becomes the default in the filter and the MVC integration, and the JavaScript they serve calls it. The path 51Degrees.core.json is still answered, and json-path still sets the path. * Test the Sequence Element with session ids and sequences of every kind * Treat the session id and the sequence independently, as the specification says A session id that is not a non-empty string, and a sequence that is not a whole number or text of decimal digits that fits in an int with room to add one, count as absent. The Evidence and the Element Data are written from the same values. * Test that the usage sharing filter never includes the web request * Keep the web request out of the usage sharing filter, as .NET does * Expect a new session id when the Sequence Element is given an empty one * Test that a build with the JSON Builder Element and no Sequence Element is refused The json-path test uses a path that is not the default. * Refuse a build with the JSON Builder Element and no Sequence Element The JSON Builder Element fails on every request without the sequence number, so the build says which option to change. * Use a configuration without the JSON Builder Element when client-side properties are off * Test the exact path match and the rule that a sequence is not negative - FiftyOneJSServiceCoreTests checks that a configured path, with or without its leading slash, answers only that path inside the application, so /anything/json, /json/more and /notjson are not answered, and that the context path is not part of the path. - SequenceElementTests checks that a negative sequence of any whole number type, a sign and white space each count as no sequence, and that 0 gives 1. * Match the JSON and JavaScript paths exactly, and count a negative sequence as absent FiftyOneJSServiceCore.Default answered any request whose whole URL ended with the configured path, so a configured path of json also answered /anything/json. The path is now compared with the whole path of the request inside the application, after the context path, ignoring case and with or without a leading slash, and a request with no URI is not answered. The old 51Degrees.core.json path is matched the same way. The Sequence Element took a negative whole number as a sequence. The Sequence Element specification counts a sequence only from 0 up to 2,147,483,646, so a negative one now counts as absent. * Give a caller with no JSON path the default path * Take the session id and the sequence from the Sequence Element's data * Check the Flow Data's keys before reading the Sequence Element's data * Name the Sequence Element's data key DEFAULT_ELEMENT_DATA_KEY The constant is the key the element uses unless a subclass gives another, so its name says it is the default, as .NET's does. * Give the JavaScript builder tests' mocked Flow Data the Sequence Element's data The builder reads the session id and the sequence from the Sequence Element's data, so the shared mock holds that data as a web integration's pipeline does. JavaScriptBuilderElement_VerifySession failed in the full build without it.
…m configuration-validation (#109) * Use the standard licence text once, with its additional terms in NOTICE The root LICENSE and five copies in device-detection, examples and pipeline held the EUPL v1.2 text with its title in capitals, and location/LICENSE held the 51Degrees header form. The root LICENSE now holds the text byte for byte as 51degrees-cxx has it, NOTICE holds the two additional terms as 51degrees-cxx has them, and the six other copies are removed. Maven Central reads the licence from each pom, so no package needs a file of its own. pipelineOptions.xsd was a copy of the file in configuration-validation. pipeline.core now takes it from that repository as a submodule, as the .NET pipeline does, and adds it to the jar at the same resource path. * Keep the additional terms in the file headers rather than in NOTICE The two additional terms apply where each file's header states them, so the repository has LICENSE alone, without a NOTICE file. The README's licence line goes back to naming LICENSE only. META-INF/NOTICE.txt in pipeline.did is the OWID attribution and stays.
…name (#111) The environment variable names check fails on rc at line 76 of DataUpdateServiceLoggingTests.java, where the constant LICENSE_KEY starts a wrapped line and so matches the check's start of line form for an old name. The constant holds a made up licence key for the test rather than naming an environment variable, so it becomes FULL_KEY, beside MASKED_KEY, and the test reads the same.
DidClientLiveTests declares a licenceKey field, never assigns it, and reads it in identifiersCreatedFor to build the "&license=" part of the form body, so DidClient.encode is handed null and anIdentifierForAnotherAddress_ReportsEveryFactor fails with a NullPointerException. That is the whole of the Cloud 51Did failure on rc. init() now takes the licence key once, keeps it in the field and gives that field to the client, which is the shape the file had before the lookup moved from env() to LiveCloudKeys.find() and was inlined into the constructor call. No fail() is added for a missing licence key, because LiveCloudKeys says the licence key is optional and both tests that need one already skip on client.hasLicenceKey(). One lookup now feeds both that guard and the form body, so the field cannot be null where it is read. Test code only, so nothing a user sees changes.
Each test job downloaded the Asn data file again with the data repository's script, outside the asset cache that the run fills once and every test job restores. They are now fetched as assets with the others, restored from the cache in each test job, and copied or linked to where the tests and examples look for them, so no test job downloads a data file. The notes say CI takes every data file from the assets, and that a new data file needs an entry in common-ci's steps/fetch-assets.ps1.
…#116) * Test that the old cloud endpoint names still work A patch release keeps every public name, and the name of an environment variable a package reads is a public name. These tests ask for FOD_CLOUD_API_URL to be read when _51DEGREES_CLOUD_ENDPOINT is not set, for _51DEGREES_CLOUD_ENDPOINT to win when both are set, for the use of the old name to be logged as deprecated once, and for the two constants that held the old name to hold it again. They cover the cloud request engine builder, the 51Did client, which is run in a Java process of its own so that it reads a real environment, and the cloud engine example. The cases of the environment variable name check now expect the old endpoint name to be allowed. The tests fail until the fix that follows. * Keep reading the old cloud endpoint name and bring back its constants The cloud endpoint variable was renamed from FOD_CLOUD_API_URL to _51DEGREES_CLOUD_ENDPOINT, and the old name was no longer read. A patch release keeps every public name, and that covers the name of an environment variable a package reads. The cloud request engine builder, the 51Did client and the cloud engine example now read _51DEGREES_CLOUD_ENDPOINT first and FOD_CLOUD_API_URL when that is not set or is empty. The first use of the old name in a process is logged as a warning that names its replacement. The engine builder logs through its logger factory, the example through its own, and the 51Did client through java.util.logging, because that module has no logging dependency and this adds none. Constants.FOD_CLOUD_API_URL comes back holding "FOD_CLOUD_API_URL", deprecated and pointing at Constants.CLOUD_ENDPOINT_VARIABLE. DidClient.ENDPOINT_VARIABLE holds "FOD_CLOUD_API_URL" again and is deprecated, and the new DidClient.CLOUD_ENDPOINT_VARIABLE holds "_51DEGREES_CLOUD_ENDPOINT". The environment variable name check no longer fails on FOD_CLOUD_API_URL, because the packages read it again. * Test that every old environment variable name still works Every environment variable name that a package, a test, an example or a CI script reads is a public name, so a patch release keeps reading the old one when the new one is not set. These tests ask for that wherever main read a name that rc had stopped reading. 51DEGREES_IPI_PATH and 51DEGREES_RESOURCE_KEY in the IP Intelligence test helpers and example helpers, 51DEGREES_DD_PATH in the on-premise web example, and RESOURCE_KEY and LICENSE_KEY in the cloud engine example, the 51Did example and the 51Did live test keys. They also ask for the four helper constants that held an old name to hold it again. The cases of the environment variable name check now expect an old name to be allowed only in the files that read it as a deprecated fallback, and refused everywhere else. The tests fail until the fix that follows. * Keep reading every old environment variable name as a deprecated fallback Every environment variable name that a package, a test, an example or a CI script reads is a public name. Each reader that main had and rc changed now reads the new name first and the old name when the new one is not set or is empty, and says once that the old name is deprecated and which name replaces it. 51DEGREES_IPI_PATH is read again by the IP Intelligence test helper FileUtils, the example helper DataFileHelper and the country translation integration test. 51DEGREES_RESOURCE_KEY is read again by the test helper KeyUtils and the example helper KeyHelper. 51DEGREES_DD_PATH is read again by the on-premise web example's configuration. RESOURCE_KEY is read again by the cloud engine example, the 51Did example and the 51Did live test keys, and LICENSE_KEY by the last two. FileUtils.IPI_PATH_ENV_VAR, KeyUtils.RESOURCE_KEY_ENV_VAR, DataFileHelper.IPI_PATH_ENV_VAR and KeyHelper.RESOURCE_KEY_ENV_VAR hold their old values again and are deprecated. IPI_PATH_VARIABLE and RESOURCE_KEY_VARIABLE beside them hold the new names, and the code that named the old constants now names those. The environment variable name check has FOD_CLOUD_API_URL back in its pattern, and allows each old name only in the files that read it as a deprecated fallback or test that read. * Guard two tests against the old names they now meet Two tests run a helper with nothing set, and each checked and cleared only the new name of a variable. The helpers now read the old name when the new one is not set, so each test failed wherever the old name was set. KeyHelperTest.getOrSetTestResourceKey_NoKey_StoresNothing now also checks and clears 51DEGREES_RESOURCE_KEY, and ExamplesWithoutSettingsTest also checks and clears 51DEGREES_IPI_PATH, through the deprecated constant that holds it.
* Test that the IP Intelligence benchmark asks only for the property it reads The properties move into benchmarkProperties, which the new test calls, with the behaviour unchanged, so the test fails here. * Ask the IP Intelligence benchmark only for the property it reads The benchmark asked for RegisteredName and AsnName, so that it ran with the free Asn data file. With the Enterprise data file each lookup then did the work for both properties. It now opens the data file with the low memory profile and asks for RegisteredName where the file has it, and AsnName otherwise.
…nce (#120) * Test that usage sharing finds its values without copying the Evidence * Count how often usage sharing reads all its values in one call Usage sharing finds the values it shares on every call, and most calls are repeats it does not share. A call that is not shared copies every Evidence value twice, once in the element and once in its tracker, and a call that is shared copies them three times. A web request is copied as often. * Find the usage sharing values once a call, without copying the Evidence Without a web request ShareUsageEvidence.find returns the Evidence's own read-only map, which is already keyed without regard to case and holds no null value, rather than a copy of every value. With a web request the element makes the one copy and gives it to its ShareUsageTracker, which makes its key from it once for both the look up and the store of its cache, rather than finding the values again for each. * Count the reads of a call that is not shared The read count tests assert what a call that is not shared costs, which is the call the tracker turns down, as most calls repeat one already shared. A call that is shared reads its values again to share them. * Find the usage sharing values only when they are needed The element no longer finds the values on every call. It finds them for the ignore filter when one is set, and otherwise only once the tracker allows the call to be shared. The tracker finds them for its key as it did, which without a web request copies nothing. Giving the element's copy to the tracker for each call measured slower than the copy it saved on a call with one Evidence value, so ShareUsageTracker is as it is on rc.
* Test that the file system watcher runs on its own setting The watcher is created with automatic updates off and the watcher on, and not with the watcher off. Tests that set automatic updates only to get a watcher no longer do. * Run the file system watcher on its own setting The data update service creates the watcher when the watcher is on, whether or not automatic updates are on. Watching a file downloads nothing, so it needs neither automatic updates nor a license key. Polling, the update on start-up and the license key check are unchanged. * Test that the watcher alone registers a data file and closes its watch service The file system watcher runs on its own setting and needs no license key. An engine whose update source needs a key it does not have is built with automatic updates and the update on start-up off, and its data file should still be registered to be watched. The builder registers a data file only for the two options that download, so the new builder test fails until it also registers one for the watcher. Each watch service runs a thread of its own. Unregistering a watched data file should close its watch service, and a download should resume the watch on the same watch service rather than leave another one running. The watcher's task repeats, and a repeating task that fails is never run again, so it should do nothing while the watch is paused for a download. * Register a data file for the watcher alone and close its watch service The file system watcher runs on its own setting and needs no license key. The builder registered a data file with the data update service only for automatic updates or the update on start-up, so an engine with no license key, which must turn both off, was never watched. It now also registers a data file that is to be watched, when there is a data update service and a data file on disk. The data update service keeps each data file's watch service, as each runs a thread of its own. Unregistering the data file closes it, and the watch resumes on it after a download rather than on a new one, unless the data file was unregistered during the download. Unregistering stops the watcher's task even while the watch is paused, and the task does nothing while the watch is paused rather than fail, as a repeating task that fails is never run again. The notes on setDataFileSystemWatcher no longer say that automatic updates must be on for the watcher to work. * Test that the watcher's task needs a watch key and replaces an earlier one The watcher's task polls the watch key. With no watch key, as when the data file's folder does not exist, no task should be scheduled, as the task would otherwise run on doing nothing now that it no longer fails on a missing key. Registering a data file again that has no watch key should replace the watcher's task rather than leave the earlier one running beside it. The thread tests now follow the watch service threads started during each test, so a thread from an earlier test that ends late makes no difference. * Schedule the watcher's task only with a watch key, replacing an earlier one The watcher's task polls the watch key, so it is scheduled only when the watch key was created. Before, a task was scheduled whatever happened, and failed on its first run when there was no watch key. Now that a run that finds no watch key does nothing, that task would run on for nothing. A data file registered again with no watch key gets a new task, and the earlier task is cancelled rather than left running beside it.
Co-authored-by: Automation51D <51DCI@51Degrees.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Releases what has collected on
rcsince the last release, #45, which merged on 4 November 2026 at 23:31 UTC.This release moves Java toward one behavior across every 51Degrees language. Each element it changes now follows the published 51Degrees specifications, which in turn follow the RFCs they cite, so that the same input gives the same answer in .NET, Java, Node.js, Python, PHP, Go, Rust and C. The native libraries take the C library release 51degrees-cxx 4.5.106, in which a thread waits for a free file handle in place of failing.
Before you upgrade
PipelineConfigurationException, which affects anyone who builds on the free Lite Data File with the default settings. CallsetAutoUpdate(false), or set a key withsetDataUpdateLicenseKey(#79).setAutoUpdate(false)now loads a new file put in place of its Data File, where before the builder registered the file only with automatic updates on and nothing watched it. CallsetDataFileSystemWatcher(false)to keep the old behavior (#119).CloudRequestException, where it failed at first use. The default timeout for each request to the cloud service is 2 seconds, where it was 100, so callsetTimeOutSecondsfor a longer one (#86).MaxPerformancewithsetPerformanceProfileto keep the old behavior (#69).1.2.3.4/24,1.2.3.4, 5.6.7.8and01.2.3.4were read and are now skipped. Pass one address, not a forwarded list (#103).PipelineFilterand the Spring MVC integration answered any path ending51Degrees.core.jsor51Degrees.core.json. They now answer/51Degrees.core.js,/51dpipeline/jsonand the old/51Degrees.core.jsononly, after the context path. A page in a folder must link/51Degrees.core.jsfrom the root, or setjavascript-pathandjson-path(#80).rawweightingandvalue. It wroterawWeighting,valueandweighting. Client code readsrawweighting, and divides it by 65535 for the weighting (#93).AspectPropertyValue.get(name)andasKeyMap()gave the bare value, so checkhasValue()and callgetValue(). A Property the response leaves out throwsPropertyMissingExceptionwhen read, where it gave null (#96).server.http-request(#68).Driftis the sum of every component's drift.getDriftgave the smallest component's drift, so its value rises wherever any component drifted and another did not (#60).httporhttps(#47) and an object name that the script cannot declare, such aswindowordocument(#88).FodId.tryFromBase64reportsINVALID_BASE64,fromBase64throwsOwidException, andDidClientthrowsIllegalArgumentExceptionbefore it calls the cloud service (#65).AUTO_UPDATE_ERR_READING_STREAM, where it returnedAUTO_UPDATE_HTTPS_ERR. Code that switches onAutoUpdateStatusor alerts on error entries is affected (#79, #98)._51DEGREES_CLOUD_ENDPOINTreplacesFOD_CLOUD_API_URL, which still works in this release.CloudRequestEngineBuilderandDidClientread_51DEGREES_CLOUD_ENDPOINT, and readFOD_CLOUD_API_URLonly when the new name is unset or empty. The first use of the old name in a process logs one warning, through the builder's logger factory, and throughjava.util.logginginDidClient.Constants.FOD_CLOUD_API_URLandDidClient.ENDPOINT_VARIABLEhold"FOD_CLOUD_API_URL"again and are@Deprecated, andConstants.CLOUD_ENDPOINT_VARIABLEand the newDidClient.CLOUD_ENDPOINT_VARIABLEhold the new name. A value given in code still comes first. Set_51DEGREES_CLOUD_ENDPOINTto the same value as the old name (#105, #116).What changes
Web integration
PipelineFiltertakesjavascript-path,json-path,sequence-element-enabledandset-headers-element-enabled, andFiftyOneInterceptorConfighas the same four options (#80).server.http-request, a read-only copy that is never shared whole or sent to the cloud service (#68).JavaScript Builder
httporhttps, and the host must be ASCII labels or an IPv6 address in brackets, with no user information, query, fragment or zone identifier, following RFC 3986. A URL from the request that fails is left out of the script and logged (#47, #77).JSON Builder
sequenceentry that holdssession-idandsequence(#80).Cloud
resource,license,licenceandvalues, which it writes itself. One name given under two prefixes, or in two letter cases, is sent once by the order of precedence (#89, #99, #107).setRequestedProperties(#97).*(#85).Usage Sharing
Cookieheader,id.email,id.salt,request-queryandrequest-pathare never shared. Cookies and query string parameters whose names start51D_are shared (#51, #68).escaped="true", where it wasreplaced="true", and holds values of up to 10,000 characters (#81).setShareAllEvidenceandsetShareAllQueryStringParametersare new, and both are off by default (#61, #62).Data updates
setDataUpdateOnStartup(true)works withsetAutoUpdate(false), and a Data File with no published date sends its last modified time asIf-Modified-Since(#79).Device Detection and IP Intelligence
Iterationscomes beforeMethodin the match metric Property metadata (#94).Translation
SourceElementDataKey,FixedLanguageandMissingTranslationBehavior(#75).C library
Examples and documentation
LICENSEfile at its root, the EUPL 1.2 text, and every source file carries the standard 51Degrees header (#109).Environment variables
Each name below has a new name that starts
_51DEGREES_, which every shell can set.CloudRequestEngineBuilderandDidClientread the cloud endpoint, and the examples and tests read the rest. Every old name is still read in this release, but only when its new name is unset or empty, and its first use is reported as deprecated in a logged warning, or on standard error in the 51Did example and the test helpers. A name that starts with a digit is still read as a Java system property as well, as before. Each constant that held an old name holds it again and is marked@Deprecated(#105, #116).FOD_CLOUD_API_URL_51DEGREES_CLOUD_ENDPOINT51DEGREES_DD_PATH_51DEGREES_DD_PATH51DEGREES_IPI_PATH_51DEGREES_IPI_PATH51DEGREES_RESOURCE_KEY,RESOURCE_KEY_51DEGREES_RESOURCE_KEYLICENSE_KEY_51DEGREES_LICENSE_KEYFixed
MaxRandomisationSecondsasks for 600 (#74).JSESSIONIDsession cookie, so session tracking had no effect (#51).query.sequencethat arrived as text failed the Sequence Element with aClassCastException(#68).Issues this closes
Build
Run 43151568806 on
tmp/pr-matrix-112-final, GREEN. Every job succeeded, being Configure, the fifteen Build and Test jobs, Compare Performance and Complete, with Report Compare Performance skipped as it is on a build that passes.MacOS_Java_11,MacOS_Java_17,MacOS_Java_21and the same three on x64. Configure printed no "Left out N macOS entries" line, so nothing was left out. The rest are Ubuntu on Java 8, 11, 17 and 21, Ubuntu ARM on Java 17, and Windows on Java 8, 11, 17 and 21.Setting 'PullRequestSha'lines in the job logs read6644140a928f6a2739626d6db7e826c352333b59, which is this pull request's head, with no other value anywhere in the run.Ubuntu_ARM_Java_17IP Intelligence measured 288,809 lookups a second against a ten result average of 308,101 and a floor of 277,291, which is 1.44 standard deviations below. It is below each of the nine results in the history, which run from 290,381 to 325,203, and an earlier build of this release measured 253,968 and failed the check (run 43130836772). Ubuntu ARM is the configuration whose history is tight enough to read, so that pair of readings is worth more attention than a passing check suggests. OnUbuntu_Java_17Device Detection measured 379,147 against an average of 518,795 and IP Intelligence 194,647 against 267,045, which are 1.09 and 1.30 standard deviations below. OnMacOS_X64_Java_17IP Intelligence measured 30,092 against an average of 100,406, which is 1.23 standard deviations below because that history is very wide. Its floor in the log is 0, because the average less twice the standard deviation is below zero, so that check could not fail whatever was measured, and the same is true of IP Intelligence onMacOS_Java_11.MacOS_Java_11andMacOS_Java_17and both onUbuntu_Java_8.Ubuntu_Java_11,Ubuntu_Java_21,Windows_Java_8,Windows_Java_11,Windows_Java_21,MacOS_Java_21,MacOS_X64_Java_11andMacOS_X64_Java_21each have one, so the check is nine tenths of that one reading and passes by construction. Those figures say nothing about whether anything changed, and should not be read as eight more passes.The full build above ran on
6644140a. 51degrees-java#114, a change to CI scripts and their notes only, merged intorcafterwards, so that test jobs take the free data files from the run's asset cache rather than downloading them, and the head became0b9257f4fac665a7084b830a603ca42109adeee1. A change to CI scripts alone needs no new full build, and the checks were green on that head.51degrees-java#116, which reads the old environment variable names again and brings back their constants, merged into
rcat 21:01 UTC on 12 November 2026, and the head became36c153d5cc824b83f461a556735ecde8793ceaf4. That change is package code, so no full build above covers it.51degrees-java#117, which makes the IP Intelligence performance example ask only for the Property it reads, merged into
rcat 23:19 UTC on 12 November 2026, and the head becameef34a03dc03d16f61a86935b3210e936c148cc3b. It changes examples only. Its body measures the cause of the lower IP Intelligence figure onUbuntu_ARM_Java_17above, which was the request for AsnName that #48 added to that example.51degrees-java#120, which makes Usage Sharing find the values it shares only when it needs them, and without copying the Evidence when there is no web request, merged into
rcat 02:28 UTC on 13 November 2026, and the head became551d916535f37a1935952086290fdd05c8b064c7. It changes package code. Its body measures the cause of the higher usage sharing overhead onrc, which was the copy of every Evidence value that #68 added on every call.51degrees-java#119, which runs the file system watcher on its own setting, merged into
rcat 03:54 UTC on 13 November 2026, and an automated submodule update, #121, at 04:13, so the head is nowd99b7acf. #119 changes package code. #121 moves the C library to 51degrees-cxx 4.5.106, whose C code is that of 4.5.105, and the JavaScript templates gain aLICENSEfile and a test.The macOS release build is queued, run 43364257944 on
tmp/pr-matrix-112rel4, for headd99b7acf60e3f15a3c7f4f7b277a63c04dfb22eb, which is the head ofrc. No release build has finished on this head yet, and this section is written again from that run when it ends.Only the patch number moves.