[jwrosewell]

Closes #10. Follows the License keys section of the logging specification, which 51Degrees/specifications#55 added and #123 reworded, both now on rc.

The problem

A license key is a paid credential, and log entries and error messages travel further than the server, to log services, to support tickets and, through the JSON builder's errors list, to the browser. On rc at 7299688 three places wrote license keys in full.

1. The data update service. FiftyOneUrlFormatter puts the license keys in the query of the update URL. DataUpdateServiceDefault.downloadFile logged that URL at debug level for every download and for a 304, at error level for no response, a 429, a 403 and any other failing status, and logged any exception with its stack trace. The exception an HTTP library raises often names the URL it requested, for example Server returned HTTP response code: 403 for URL: ....
2. The cloud request engine. The cloud answers a refused license key with an error that names the key. CloudRequestEngineDefault.validateResponse put the cloud's error text, or the whole response for a failing status, in the message of the CloudRequestException it raised.
3. The 51Did client. DidClient.readRedeem put the first 200 characters of the body in the message of a redeem error, and the errors list of a 400 answer in the message of the IllegalArgumentException it raised. The redeem request is the one that sends the licence key.

A search of every logger. call and exception message in pipeline, device-detection, ip-intelligence, location and examples for license key values and for URLs or response bodies that can hold one found no other place.

What the specification says

The License keys section of pipeline-specification/features/logging.md, on rc at 5a25065, says the following.

A license key MUST NOT appear in full in a log entry or in an error message, at any log level. Where a message names a license key it MUST show only the key's last 4 characters, with each character before them replaced by one *, so that a reader can tell two keys apart but cannot use either. A key of 4 characters or fewer MUST be replaced entirely, by one * for each character. Every package masks the same way, so ABCDEFGH is written ****EFGH in every language.

Some URLs carry a license key in their query, such as the URL a data update request is sent to. A message that names such a URL MUST name it with the key masked in the same way. The same applies to the message of an error raised by an HTTP library, which often includes the URL it requested.

There MUST be a test that a failed request carrying a license key produces log entries and error messages that hold only the masked form of the key.

#55 also added to the logging section of data-updates.md that the <url> in its messages "is the URL with each license key masked to its last 4 characters".

What changed

1. One helper. fiftyone.pipeline.util.LicenseKeyMask in pipeline.core. Its Javadoc links the specification on rc, where the rule is.
   • mask(key) replaces every character before the last 4 with *, so TESTKEYNOTREALABCDEFGHIJ1234 becomes ************************1234. A key of 4 characters or fewer becomes all *.
   • maskIn(text, keys) masks each key wherever the text holds it, as written and as encoded in a URL with upper or lower case hexadecimal digits, longest key first. An encoded key is replaced by the encoded form of its masked plain form, so it shows the same last 4 characters as the plain key, for example ABCDEFGHIJKLMNOPQRST12%2B%3D becomes ********************12%2B%3D.
   • Every occurrence of a key's text is masked. A key piece of only a few characters, such as one left by a typing mistake in a | list, therefore also masks those characters wherever else they appear. The specification asks for a key of 4 characters or fewer to be masked in full, so the helper keeps doing that, and its Javadoc says so.
   • maskIn(throwable, keys) gives the throwable to log in place of one whose message, causes or suppressed throwables hold a key. Each message in the copy starts with the name of the class it stands for, and the stack traces are the originals. A throwable that holds no key is returned as it is, so logs that never held a key do not change.
2. The data update service. downloadFile names the URL in every message with each key masked, and logs the exception through maskIn. The request itself still uses the unmasked URL. The keys are those licenseKeysToMask gives, being the data update license keys and any key in the URL's LicenseKeys parameter, so a key typed straight into the DataUpdateUrl setting is masked too.
3. The cloud request engine. validateResponse masks the license key in every message before it raises the exception, and so does each key in it when the value holds several separated by | (getLicenseKeys).
4. The 51Did client. readRedeem masks the licence key in both messages, before the body is cut to 200 characters, so a key cannot be left half shown. DidHttpException.getBody() still returns the body as the service sent it, and its Javadoc now says the body is not masked and can hold the license key sent, so a caller that logs it must mask it first. The 51Did package depends on nothing in the Pipeline but org.json, so it cannot use LicenseKeyMask without bringing pipeline.core and its dependencies to every 51Did user. It masks with a private method of eight lines that gives the same result, and its tests pin the same masked value as the helper's tests.

Two things are left as they are.

• Resource keys. The License keys section of logging.md covers license keys only, so resource keys, such as the one in the URL of the accessible properties request, are not masked.
• Response headers. CloudRequestException keeps the response's header fields as the cloud service sent them. The cloud request engine sends the license key only in the form body (getContent), never in a request header or URL, and nothing shows the service writing it into a response header. So there is nothing there to mask.

For the release notes

• A license key no longer appears in full in any log entry or error message. Where a message names one, such as the data update URL in the data update service's log, or the cloud's refusal of a license key, it shows only the key's last 4 characters, with every character before them replaced by *.
• fiftyone.pipeline.util.LicenseKeyMask masks license keys in text and in throwables, for code of your own that logs them.
• A key typed straight into the DataUpdateUrl setting, in its LicenseKeys parameter, is masked in the data update service's log as well.

How to verify

No test calls a live service.

mvn -pl pipeline/pipeline.core,pipeline/pipeline.engines,pipeline/pipeline.cloudrequestengine,pipeline/pipeline.did -am test -Dtest='LicenseKeyMaskTests,DataUpdateServiceLicenseKeyTests,DataUpdateServiceTests,CloudRequestEngineTests,DidClientTests' -Dsurefire.failIfNoSpecifiedTests=false

DidClientTests needs the pipeline/owid-java submodule checked out, as the 51Did package compiles its source.

Class	Test methods on rc	On this branch
LicenseKeyMaskTests (pipeline.core)	0	9
DataUpdateServiceLicenseKeyTests (pipeline.engines)	0	7
CloudRequestEngineTests (pipeline.cloudrequestengine)	18	19
DidClientTests (pipeline.did)	83	86

CloudRequestEngineTests has four parameterised methods, so a run reports more cases than methods.

• d989ba0 adds the helper and LicenseKeyMaskTests, which pass.
• 382fa6c adds the tests of the three places, on their code from rc. Ten are written to fail there: the six in DataUpdateServiceLicenseKeyTests (Forbidden, TooManyRequests, OtherStatus, NotModified, NoResponse and RequestException, each _LogsMaskedKeys), CloudRequestEngineTests.Process_LicenseKeyInCloudError_Masked, and in DidClientTests redeem_RefusedLicenceIsMaskedInTheMessage, redeem_LicenceInErrorsIsMaskedInTheMessage and redeem_ShortLicenceIsMaskedInFull. The data update tests capture everything passed to a mocked logger, at every level, with each throwable's printed stack trace.
• cb0b80e masks the keys in the three places.
• 6318f93 holds the review follow-up tests alone, on cb0b80e's code. LicenseKeyMaskTests.maskIn_EncodedKeyShowsThePlainLast4, LicenseKeyMaskTests.maskIn_MasksAKeyEncodedInLowerCase and DataUpdateServiceLicenseKeyTests.KeyTypedIntoTheUrl_LogsMaskedKey are written to fail there. In run 42727515077 (branch tmp/red-85) all three failed, as expected. maskIn_EncodedKeyShowsThePlainLast4 gave ...********************[****]B%3D where ...12%2B%3D was expected. maskIn_MasksAKeyEncodedInLowerCase left the key unmasked, ...license=ABCDEFGHIJKLMNOPQRST12%2b%3d. KeyTypedIntoTheUrl_LogsMaskedKey found the test key in full in the debug line downloadFile from https://distributor.example/api/v2/download?LicenseKeys=... and in the error line for the 403.
• 4261571 holds the follow-up fixes and the two Javadoc changes.
• e3672e5, this branch's head, puts the Javadoc of downloadFile back above it, as 4261571 added licenseKeysToMask between them.

Other drafts that touch the same lines

#79, at its head adce29b, rewrites the logging of DataUpdateServiceDefault.downloadFile. It names the URL unmasked in its debug line, in the "Checking for update", "Downloaded new data" and "No data newer" info lines, and through connectFailure(url, ...) and downloadFailure(url, ...), and in both catch blocks it copies e.getMessage() into the message text before passing the throwable. Whichever of the two merges second must do four things.

1. Compute licenseKeys = licenseKeysToMask(dataFile, url) and loggedUrl = LicenseKeyMask.maskIn(url, licenseKeys) once at the top of downloadFile, and pass loggedUrl, not url, to every message and helper that names the URL. The real url is used only for new URI(url.trim()).toURL().
2. In both catch blocks, take Throwable masked = LicenseKeyMask.maskIn(e, licenseKeys) and use masked.getMessage() for the error detail text as well as masked as the logged throwable, because an HTTP library's message names the URL.
3. Keep DataUpdateServiceLicenseKeyTests. It captures every logger call at every level, so it fails for any message that is missed.
4. Make the data update follow the specification #79 moves the 429, 403 and other-status lines from error to warn, so those tests' expected level changes on Make the data update follow the specification #79's side. The lines still name the URL through downloadFailure(url, ...), so they need loggedUrl too.

The change that discovers when the Cloud Request Engine is built (#86, at its head 7d4cebd) also conflicts with this branch in CloudRequestEngineDefault.java. So does the change that sends only the Evidence the cloud accepts (#89, at its head 38bccd2), because it is built on #86's branch. Every other file, and the rest of that file, merges without a conflict.

• The conflict is one hunk, just above the method that loads the public properties. This branch adds getLicenseKeys() there. Discover when the Cloud Request Engine is built, and retry on first use only after a failure that can pass #86 adds setTimeouts(HttpURLConnection) there and renames getCloudProperties to fetchPublicProperties.
• Whichever of this branch and Discover when the Cloud Request Engine is built, and retry on first use only after a failure that can pass #86 merges second keeps both helpers, getLicenseKeys() and setTimeouts(HttpURLConnection), and uses Discover when the Cloud Request Engine is built, and retry on first use only after a failure that can pass #86's name for the method, fetchPublicProperties. Send only the Evidence the cloud accepts from the Cloud Request Engine #89 takes the same resolution.

Full build

The full build and test matrix passed every job other than macOS on head e3672e5 in run 42773815977 on 10 November 2026, 11 jobs succeeded, run as a dry run so nothing merged. macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.