[jwrosewell]

This draft follows .NET's data update messages and levels in every place but two, the update on start-up message and the refresh attempt message, where it keeps the specification table's text, as 51degrees-node does.

The problem

Python's data update service wrote its own wording for each event, and logged most failures at Warning where .NET logs them at Error. A customer reading a Python log and a .NET log of the same failure saw two different messages, so 51Degrees support could not match one to the other. Some events .NET reports were not reported at all, and a check that needed no update was often reported as nothing rather than as not needed.

The decision

James Rosewell, 10 November 2026, decision 7: "The data-updates.md Errors table follows .NET's messages and levels", and the other languages align to them now, because "This is all about alignment." Java, Node, Python, Go and Rust use .NET's text and level for each event, in this release. His decision 2 covers the reporting, which is that every check is reported, and a skipped or not-needed check is reported as NotNeeded, as .NET's DataUpdateService.cs does.

What the specification says

data-updates.md at e89679e9 says this at lines 475 to 486.

A row at Error level is an error that carries the message shown. Where the service logs it, the entry is at Error, reads Error during check for data update, and has the error attached. An error of any other kind is logged at Error with the message in the last row, also with the error attached. An error raised by a callback in onUpdateCallbacks is logged in the same way on update polling and the file watcher, as Error during check for data update when it is of the same kind as the Error rows, and with the last row's message otherwise.

and "Either way the callbacks after the one that raised the error are not called". The Errors table that follows, lines 488 to 505, gives each row's text, level and status.

What the other languages did

• 51degrees-node#69, "Write the data update messages and levels the .NET service writes", merged on 11 November as 8b9fc3d1. This draft is its Python half and writes the same text character for character.
• .NET is where the text comes from, in pipeline/Engines/Services/DataUpdateService.cs at b3e90b4, which is the file the specification's table was written from.
• The alignment this draft stands on is #69, whose own siblings are 51degrees-dotnet#115, 51degrees-java#79, 51degrees-node#52 and 51degrees-go#73, all merged.

Python's logger takes a string, so the error is written after the entry text. The text is the one 51degrees-node writes at 74cdc781 (withCause, checkErrorText and unhandledErrorText, lines 58 to 110), character for character, being:

• a row at Error level reads Error during check for data update. Error detail: <row text>, then . Error detail: <cause> when there is a cause (a single space before Error detail: when the row text already ends with a full stop), and nothing more when there is none
• any other error, a callback's included, reads An unhandled error occurred while checking for automatic updates for engine '<Engine type>'. Error detail: <cause>

Event by event

Event	Text and level now
Update on start-up	Updating on start-up, Info, as the table at e89679e9 line 448 and 51degrees-node write it. .NET writes Updating on startup.
Refresh attempt	Attempting to refresh Engine '<name>' with new data, Info, as the table at e89679e9 line 456 and 51degrees-node write it. .NET writes Attempting to refresh engine with new data, naming the engine in a structured field.
Connecting to the update URL failed	Error accessing data update service at '<url>' for engine '<name>', Error, with the cause masked
No response from the update URL	No response from data update service at '<url>' for engine '<name>', Error
429 Too many requests	Too many requests to '<url>' for engine '<name>', Error
403 Forbidden	Access denied to data update service at '<url>' for engine '<name>', Error
Any other failing status	HTTP status code '<code>' from data update service at '<url>' for engine '<name>', Error
No Content-MD5 header	No MD5 hash included in data update response for '<url>'. Unable to verify data integrity, Warning
Integrity check failed	Integrity check failed. MD5 hash in HTTP response '<response hash>' for '<name>'data update does not match calculated hash for the downloaded file '<calculated hash>'., Error, with no space before data update, as .NET (DataUpdateService.cs lines 1461 to 1465 at b3e90b4) and the table write it
No temporary file path	The data file '<identifier>' is checking for updates but does not have a temporary file path configured., Error, status AUTO_UPDATE_TEMP_PATH_NOT_SET
Writing the new data over the data file failed	An error occurred when copying a data file to replace the existing one at '<path>'., Error, with the cause, status AUTO_UPDATE_NEW_FILE_CANT_RENAME, and the update ends, as 51degrees-node writes it at 74cdc781, lines 664 to 676
Writing the temporary copy failed	the last row's message, Error, with the cause, status AUTO_UPDATE_UNKNOWN_ERROR, and the update ends. The table pairs that message with that status at e89679e9 line 505, and .NET pairs them too, its generic catch (Exception) setting AUTO_UPDATE_UNKNOWN_ERROR (DataUpdateService.cs lines 886 to 890 at b3e90b4) where AUTO_UPDATE_NEW_FILE_CANT_RENAME is set only for the copy over the data file at line 1069. 51degrees-node writes the same message with the rename status instead (74cdc781, lines 735 to 742), so this is the one place where following .NET and the table means not following 51degrees-node, and the gap in 51degrees-node is under What remains.
Moving the temporary copy over the data file failed	An error occurred when copying a data file to replace the existing one at '<path>'., Error, with the cause
Refreshing the engine from the data file failed	An error occurred when applying a data update to engine '<name>' after 1 tries., Error, with the cause. Python makes one try and names it, as 51degrees-node does, where .NET makes up to 10 (lines 1190 to 1216 at b3e90b4).
Refreshing the engine from data in memory failed	An error occurred when applying a data update to engine '<name>'., Error, with the cause
A callback raised an error	the last row's message, Error, with the cause, and the callbacks after it are not called
Any other error	the last row's message, Error, with the cause, status AUTO_UPDATE_UNKNOWN_ERROR

Each row at Error level is written under Error during check for data update. Error detail: , as above.

Reporting, under decision 2.

Check	Before	Now
A 304 Not modified	reported not needed	unchanged
A watcher check that found the file changed	reported only on a failed refresh	reported success or the failure
A watcher check that found the file untouched	nothing reported	unchanged, and the reason is below
A watcher check that found a newer file whilst another was being applied	nothing reported	reported in progress

Effect on callers

• Every Error and Warning message the data update service logs changes text, and most failures move from Warning to Error.
• The watcher now reports each change it finds, where before it reported only a failed refresh.
• A download that cannot be written to the data file's path ends the update with AUTO_UPDATE_NEW_FILE_CANT_RENAME, where before this draft it ended with an error nobody caught. The engine keeps the data it has.
• A download whose temporary copy cannot be written ends the update with AUTO_UPDATE_UNKNOWN_ERROR, which is the status the errors table gives the message that failure logs.

What changed

All in pipeline/fiftyone_pipeline_engines/src/fiftyone_pipeline_engines/datafile_update_service.py. This draft was stacked on #69, which merged into rc at 15:06 UTC on 12 November as 8b0b6248. rc is now merged into this draft, so its diff against rc holds this draft's own changes alone, in that file and its two test files. How the merge was resolved is under Merging rc below.

• Each failing HTTP path logs its own message at Error, replacing the one shared download message written at Warning. A response of None, and a data file that asks for a temporary copy but has no temporary directory, are reported.
• _log_check_error writes a row at Error under Error during check for data update. Error detail: with the cause after it, and _log_unhandled writes the last row's message with the cause, as above. The cause of a failed connection is masked, as it can name the URL with its licence key.
• _refresh names the one try (REFRESH_TRIES) for a refresh after a check or a watched change, and ends with a full stop for one from update_from_memory.
• _write_data_file returns a status for every failure and the caller ends the update, because the engine of a data file kept on disk reads that file. Writing straight over the data file carries the errors table's row for the copy that replaces it, with AUTO_UPDATE_NEW_FILE_CANT_RENAME, and a temporary copy that cannot be written carries the last row, with AUTO_UPDATE_UNKNOWN_ERROR. Both callers pass the status back unchanged, the download route through _update_failed and update_from_memory to its caller.
• _with_cause masks the cause, so a licence key cannot reach a log entry through an error that names the update URL, whichever of the two entry texts carries it. That came in with #69, which masked the one place it wrote the cause, and this draft writes both places through one method.
• track_local_data_file_changes reports the changes it finds and says nothing about a file nobody has written to.
• UpdateStatus gains AUTO_UPDATE_TEMP_PATH_NOT_SET and AUTO_UPDATE_UNKNOWN_ERROR, at 12 and 13, which are the numbers .NET gives them in IDataUpdateService.cs at b3e90b4. 11 is .NET's AUTO_UPDATE_NO_CONFIGURATION, which this service has no route to, and AUTO_UPDATE_TIMEOUT is left out for the same reason.
• _retry_text is removed, with the only message that used it gone.

Two places where this service cannot do what the table says

The Warning row is not written, because Python cannot refresh a disk engine from memory, so that row of the table has no implementation here, and it is written out in full below rather than left silent. The row "Writing data passed in from memory to the data file failed" (e89679e9 line 501) is Warning, with the Engine still refreshed. .NET earns that by calling RefreshData(identifier, data) with the data in hand (DataUpdateService.cs lines 394 to 420 at b3e90b4). In Python a data file with download set refreshes through engine.refreshData(), which reads the file on disk (devicedetection_datafile.py lines 107 to 119 and ipintelligence_datafile.py lines 102 to 114), so after a failed write the engine would be left with the old data whilst the callbacks were told the update had succeeded. Both routes therefore end with the write failure. Meeting the row would mean giving the data file's refresh the bytes to use, in the device detection and IP Intelligence packages, which is listed under What remains. 51degrees-node does not meet the row either, its updateFromMemory refusing a data file that has a path at all (74cdc781, lines 768 to 776), where this draft writes the file and then refreshes from it. So the row asks for behaviour that only an Engine able to take the new data directly can give, which is what .NET's RefreshData(identifier, data) does, and no other language it was written from meets it. The table came from specifications#60, merged on 11 November, and whether this row should stand as written needs a change there rather than one here.

A look at an untouched watched file is not reported. e89679e9 lines 345 to 355 say a check by the file watcher MUST be reported once it ends, and count "a downloaded or watched file that is not newer than the current file" as not needed. In .NET and 51degrees-node a watcher check happens only when the operating system says the file was written, so that row covers a second event for one write. Python has no such watcher and looks every 10 seconds (FILE_SYSTEM_WATCHER_INTERVAL_SECONDS), so reporting each look would tell every callback six times a minute, for ever, that no update was needed, which neither of the two languages this draft follows does. A look that finds the file written to is the event here, and it is reported. Overturning this reading costs one line in track_local_data_file_changes and the test beside it.

Merging rc

#69 and #70 merged into rc at 15:06 and 15:08 UTC on 12 November. #69 was squashed into 8b0b6248, so rc holds none of the commits this draft took from #69's head, and the draft no longer merged. a934ee1c merges rc at 435274d8 into the draft with a merge commit, whose first parent is the earlier head cc916dba.

Three files conflicted, being datafile_update_service.py, test_datafile_update_http.py and test_datafile_update_service.py, and each keeps this draft's version. Nothing of rc is lost by that, because rc's version of each of the three is byte for byte the version at #69's head e770a881, which this draft took in at efbfb6d8 and built on. #69's squash is the only commit on rc since the merge base 8ac72361 that touches any of the three, so every line rc changed in them is already here, and the conflicts come only from this draft changing the same lines further. The three files on this head are the same blobs as on cc916dba.

The conflict in the service is in __emit, where rc logs an unhandled error inline and this draft calls _log_unhandled. _log_unhandled and _with_cause build the same sentence with the same masking, "An unhandled error occurred while checking for automatic updates for engine '<type>'. Error detail: " followed by the masked cause.

The guard under "exit if already updating", if data_file.updating: pass, is the same on both sides (lines 122 to 125 here, lines 116 to 119 on rc), so the merge leaves it as it was. #87 is the draft that makes that guard do something, and the two drafts meet there.

Every other file rc changed takes rc's version. That includes the comment #70 changed in the device detection datafileupdate_console.py example, which this branch still had in #69's wording. test_licence_key_masking.py and the IP Intelligence datafileupdate_console.py example were already the same on both sides.

Runs

The runs below are on head a934ee1c338e5d0395a3cdfacb2bf0699bf696ec, which is the earlier head cc916dba with rc at 435274d merged in, as described above. I read every run's log rather than its result, and each temporary branch is one commit on the commit named.

• Red, this draft's two data update test files on plain rc, run 43283805955, failure. The branch tmp/py98rc-red at e4910be5 is rc 435274d plus test_datafile_update_http.py and test_datafile_update_service.py from this head and the temporary workflow, with no source change. 18 failed and 31 passed in the two files. The failures include testAFailedTemporaryCopyIsAnyOtherError ("type object 'UpdateStatus' has no attribute 'AUTO_UPDATE_UNKNOWN_ERROR'"), testTheErrorEntryIsWrittenExactly, testNoTemporaryPathIsReported and each test of a failure's message and level, which rc still writes at Warning in its own words. The 10 licence key masking tests and the 6 other engines tests passed, because Align the data update service to the specification: HTTP date, expected time, update from memory and logging #69 brought the masking into rc, and that is also why fewer fail here than the 38 that failed on the older rc.
• Green, the same tests on the head, run 43283833911, success. The branch tmp/py98rc-green at 94069893 is the head plus the temporary workflow and nothing else. 49 passed in the two data update test files, 10 in the licence key masking tests and 6 in the rest of the engines tests.
• Full build of the head, run 43282138008 on tmp/pr-matrix-98e, success, 12 jobs passed and 2 skipped. I read its log rather than its result, and its 11 Setting 'PullRequestSha' lines all read a934ee1c338e5d0395a3cdfacb2bf0699bf696ec, with Configure printing "Left out 5 macOS entries".

Earlier runs, each on the commit named and all on heads this draft has moved past, are kept so the order the work was done in can be read. The four on cc916dba cover the same three files as this head.

• Red on the earlier head cc916dba without its last commit, run 43198709124, failure, 1 failed and 48 passed. The branch tmp/py98-red4 at faaa90ea held every source change in this draft but the one that pairs a failed temporary copy with its status, and the one failure is testAFailedTemporaryCopyIsAnyOtherError.
• Red on the older rc 8ac7236 with the tests alone, run 43198642071, failure, 38 failed and 11 passed, then 2 failed and 8 passed in the masking tests. Green on cc916dba, run 43198739721, success. Full build of cc916dba, run 43198549171, success, 11 Setting 'PullRequestSha' lines all that head.
• Red for the messages and levels, the tests alone on 9314d95: run 42980461038, 11 of the new tests fail.
• Red for the entry text and the cause, the tests alone on 2f15773: run 42993565106, 14 failed and 32 passed. Green on 8192995, run 42993849053, 45 passed.
• Red for the exact text Node writes, the tests alone on 4c19c24: run 42999458077, 11 failed and 35 passed, among them testTheErrorEntryIsWrittenExactly, which writes two entries out in full, one without a cause and one with. Green on 082c572, run 43000461334, 46 passed.
• Red for the start-up, refresh and integrity texts, the tests alone on 5bf1ec9: run 43000819375, 5 failed and 41 passed. Green on 77ca816, run 43001041931, 46 passed.
• Full build of the earlier head 77ca816: run 43001148365, success.
• Full build of the earlier head 3b5a065: run 43176390642, success, 11 Setting 'PullRequestSha' lines all that head. Its red and green pair on that head are 43177129799 and 43177154341.
• The first green after #69's head was merged in, run 43186051179, is a failure, and it is named here rather than left out. Three callback tests read the unhandled error entry with its cause written twice, because Align the data update service to the specification: HTTP date, expected time, update from memory and logging #69 added a constant that appends the cause to one that already carried it. 3395e52 leaves one constant holding the whole entry, which is what the service writes.
• Full build of the earlier head 3395e52: run 43187305333, success. Its red and green on that head are 43187259196, 43187274349 and 43187243281. That head is superseded by the two commits above, so it is not the build this draft stands on.

macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.

What remains

• Giving a data file's refresh the bytes to use, so that an update from memory can leave the engine on the new data when the file cannot be written, which is the Warning row of the errors table. That is a change in the device detection and IP Intelligence packages as well as this one, so it is not in this draft. Until it is made, that row has no implementation in Python, and 51degrees-node does not meet it either, so an issue in specifications is still to be raised asking whether a row that only .NET can meet should stand as written. The case for it is set out above under Two places where this service cannot do what the table says, with the lines of each language, so whoever raises it has the whole of it.
• 51degrees-node logs the last row's message for a temporary copy that cannot be written and returns AUTO_UPDATE_NEW_FILE_CANT_RENAME (74cdc781, lines 735 to 742), where the table and .NET pair that message with AUTO_UPDATE_UNKNOWN_ERROR. That needs its own draft in 51degrees-node, which this draft does not open.
• Three parts of the specification wait for next week, under decision 2, at data-updates.md e89679e9. A programmatic check, the update on start-up and an update from memory pass an error to their caller rather than logging it (lines 467 to 470). An update from memory reports nothing to the callbacks and gives its result to the caller (lines 362 to 366), where this service and 51degrees-node both report it. A failure with a buffer passed to a programmatic update from memory fails that call with an error, which is not logged or reported (lines 305 to 307).
• The table's '<Engine type name>' in the Errors table and '<Engine type>' in the other tables name the same thing twice, which is another wording for the specification to settle and which no language reads differently.
• Java, Go and Rust carry the same alignment under decision 7 in their own drafts.

The last release into main merged on 2026-11-04.