Repository navigation
11.0.0-beta.1: .NET 11 and EF Core 11, LibRed fixes, dual-table fallback - #308
Merged
Merged
Conversation
…s ACE 2016 LibRedAccess ran only on the 2016 Access Database Engine redistributable, which is out of support and differs from the Microsoft 365 runtime the cross-checks are measured on: it knows neither BIGINT nor DATETIME2, cannot evaluate a backtick-quoted CHECK, and leaves a stored query's LvExtra uninitialised. Those cases skip or are masked there, so CI never ran them. The job now has a second leg, 365, which installs the click-to-run Access Runtime (AccessRuntimeRetail, x64, Current channel) through the Office Deployment Tool, fetched from Microsoft's CDN since it downloads the runtime from there anyway. A failed install fails the step. The 2016 leg is unchanged, and the 365 leg is continue-on-error until it proves stable. push.yml and pull_request.yml change alike; CLAUDE.md and AGENTS.md describe the two legs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since the December 2025 Office builds, ACE refuses every query that reaches
another file ("Operation is not supported for this type of object") unless
AllowQueryRemoteTables is 1. The runtime the 365 leg installs blocks by
default, so on its first run the eight IN externaldatabase append and
make-table cases of StoredActionQueryWriteAccessTests failed when ACE ran the
stored query: the test skips on that message only where ACE creates it, and a
stored append or make-table is still created under the block.
The install step now sets AllowQueryRemoteTables to 1, as on the machine the
cross-checks were measured on, creating the Engines key only where it is
missing. The 2016 redistributable does not block, and its leg is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A contraction was refused only where a lowercase letter met an uppercase one, which holds for every two-letter digraph but not for Hungarian dzs: ACE splits DZs into DZ + s, where LibRed took the whole trigraph. Measured against ACE in every case mix of every contraction order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a connection closes, PageAllocator.ReturnReleasedPages hands back the pages it released and the reservation and groups it holds, rewriting the global usage map on page 1 in a transaction of its own. Every writing statement holds the file's exclusive scope from its first read to its commit; this transaction did not, so a statement on another connection could commit page 1 after the close first wrote it, and the close failed with "Transaction write conflict on page 1". Close and Dispose then threw after the connection's work had already committed, and the pages it should have returned stayed out of the free map. That is MigrationLockConcurrencyTests' intermittent failure on CI: the winning connection releases the lock and closes while seven others keep writing. A stress run of the same pattern (16 connections, no pauses) failed 13 of 16 threads in its first round; it now runs ten rounds clean. The method's body now runs in PageChannel.WriteExclusive, as a statement does, which covers both callers (JetDatabase.Dispose and the encryption path). Its early returns, which read only the connection's own state, stay outside the scope. The bytes written are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ACE has no SQL for an index rename; DAO's Index.Name is the only way, and it was measured: a plain index, a primary key (even one a relationship references), a case-only change and a multi-page definition all rename; the relationship's own index and a name already taken are refused. Jet: ISchemaOperationsProvider gains RenameIndex. DAO sets Index.Name, ADOX refuses and points at DAO, and Precise goes through DAO. JetSchemaOperationsHandling routes "alter table X rename index A to B" to it. LibRed: SchemaEditor.RenameIndex moves the logical index block to its new sorted position and writes the definition twice, as DAO does, so a multi-page definition's continuations land where ACE puts them. The duplicate-name check is split out of InsertIndex, InsertLogicalBlock takes the info bytes, and the relationship guard is split so a rename refuses only the relationship's own index. The SQL statement now runs through JetDatabase.RenameIndex instead of refusing. The parity tests compare the result with DAO's byte for byte, refusals included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A one-to-one (grbit 0x01, DAO's dbRelationUnique) has no SQL form, so RelationshipSpec gains IsOneToOne for direct callers. Measured against DAO, it differs from the same one-to-many in two places only: 0x01 on every MSysRelationships row of the key, and a unique child index (0x81 for 0x80). Being unique, that index shares the block of a unique index over the same columns, though still not the primary key's. AddForeignKey now writes the relationship's MSysObjects object before its index, inside a transaction of its own when the caller has none, as ACE does: a one-to-one refused over duplicate child keys rolls back but leaves MSysObjects' and MSysACEs' counting behind. The object is written by each caller of AddRelationshipRows rather than inside it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An UPDATE and the ALTER COLUMN re-lay build each row in a page-sized
buffer and leave it there for the next one. The row was copied into
that buffer before the record-size check ran, so a row grown past the
page threw an ArgumentException from the copy ("Destination array was
not long enough") where ACE refuses the statement with "Record is too
large". The copy now comes after the check on both paths: UPDATE copies
the encoded row once EnsureRecordFits has passed it, and the re-lay once
RewriteRowRaw has.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LibRed.Core.Fuzz is a coverage-guided harness over LibRed.Core (SharpFuzz + libFuzzer, not in CI; see its README). Its database targets patch a few bytes of a good seed database and walk it as a caller would, read-only or rewriting a row of each table. The walk and the leak contract move out of CorruptFileSweepProbeTests into test/LibRed.Shared/MalformedFile.cs, so the fixed-seed sweep and the fuzzer cannot disagree about what counts as a leak. A write may also refuse with InvalidOperationException, as ACE refuses the same writes; it follows only the tables still listed, since a write that follows a damaged free map can overwrite the catalog, and it tries the first 32 rows of a table rather than every one. What the fuzzer found, each measured against ACE on the same damage: - A catalog row with a null Name or Id is refused; a null Flags reads as no flags. A page read as MSysObjects, MSysQueries or MSysRelationships that lacks one of their columns is not that table, and an MSysObjects with a calculated column, or a table whose calculated column has no Expression property, is refused as ACE refuses it. - A data page's previous row boundary is checked before it is used, a calculated value's width before its status, and a usage map's pages before one below 0 or past int.MaxValue is taken for a bit index. - An index entry a write cannot find is damage (InvalidDataException), as ACE refuses it with "The search key was not found". - On a Text column carrying the AutoNumber flag, an explicit value sets the counter at 0x14 to its first four UTF-16 bytes, as ACE does, rather than failing to convert the text. - A free map that names a page past the 2 GiB limit is refused at the first allocation, as ACE refuses it, instead of growing the file a page at a time towards it. PageChannel.MaxPageCount names the limit WritePage already enforced. - A released-pages map starting below page 0 is left as it is at close and the pages are freed all the same, as ACE does, where sizing it overflowed. The format spec records each measurement, along with what ACE does when the free map offers a live page: it takes it without looking. The last four fixes have tests in LibRed.Core.Tests (IndexTraversalCorruption, ColumnDescriptorFlag, GlobalMapPointer, ReleaseAtClose); the rest are covered by the sweep and the earlier tests that changed with them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Table.Delete asked MSysComplexColumns for every table's complex columns, seeking its ConceptualTableID index with the table's definition page. With that column's type byte damaged from Long to Byte, the seek key of a table past page 255 overflowed the Byte, and every delete from such a table threw an OverflowException. Only a table with a Complex column has values elsewhere to remove, so only it asks the registry now; ACE inserts, updates and deletes rows of other tables on this damage as usual (measured). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…E does On an OLE column whose descriptor carries the AutoNumber flag (a damaged file), an explicit insert passed the binary value to Convert.ToInt32 to update the counter at 0x14, which threw InvalidCastException. ACE sets the counter to the value's first four bytes, as it does a text value's first four UTF-16 bytes (measured: bytes 01 02 03 04 05 06 give 0x04030201), so the binary value now takes the same path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Closing a writable JetDatabase returns the pages its session released, and a release that cannot run on a damaged file was already absorbed: it runs in its own transaction, so it writes nothing and the pages stay released. Only InvalidDataException was caught, though. A released map turned reference form needs a bitmap page, and a free map with a nonzero start page and no free page refuses that allocation with NotSupportedException, which then escaped from Dispose. The close now absorbs that refusal the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…does On a fixed column whose descriptor carries the AutoNumber flag (a damaged file), an explicit insert passed the value to Convert.ToInt32 to update the counter at 0x14; a Date/Time value threw InvalidCastException. Measured against ACE, the column's width decides. The counter is four bytes: a Single takes its own bits (0.25 gives 0x3E800000), while an Integer, Currency or Date/Time column refuses the row, explicit value and all, with "The field is too small to accept the amount of data you attempted to add." after drawing an id, which stays burned. Insert now does the same, refusing before the row is written. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After millions of executions the write target had stopped finding anything new: on a damaged file it only ever rewrote one row of each table. The first input byte still picks the seed by its remainder; its quotient now selects operations from MalformedFile.Operations, run on each user table after the row rewrite: a chained long value written and taken out again, two columns added and dropped, an index built and dropped, rows deleted by a statement that rolls back, a rename and back, and a drop. Each runs as one statement in a transaction of its own, a refusal rolling it back, as QueryEngine runs a writing statement. The patches keep their meaning, so earlier corpora and crashes still replay. database.dict holds the multi-byte values random mutation is slow to find for the patch-list input: the page numbers and offsets of the fields a reader steers by, and the type codes and flag bits they take. The README covers the workload, -use_value_profile and -dict (quoted in PowerShell), and merging old corpora into a fresh one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… type LoadRelationships casts each MSysRelationships value to the type its column should have. With a type byte damaged (szColumn's made 0x0D, a code no type has) the value decodes as bytes, and the cast threw InvalidCastException out of DropIndex's relationship lookup. Measured against ACE on that damage, the foreign-key list fails "Cannot perform this operation" while every table still lists and reads and unrelated DDL runs. LoadRelationships now refuses with InvalidDataException when a column it reads is not its spec's type; the table list is untouched. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CREATE INDEX appends the new index's usage map to the page holding the table's own maps, and AppendEmptyUsageMapRow read that page raw. With its row count damaged past what a page holds, TryAppendRow walked the slot directory off the end of the page and threw ArgumentOutOfRangeException. It now parses the page with DataPage.Read first, as LongValueStore's append does, so the page is refused as InvalidDataException exactly as every read of it already refuses it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LoadRelationships cast every MSysRelationships column straight to its type, so one null (a cleared null-bitmap bit) threw ArgumentNullException or NullReferenceException out of Catalog.Relationships, and with it out of DROP COLUMN's relationship check. ACE, measured with one column of one row made null, leaves the relationship out of its foreign-key list when szObject or szReferencedObject is null and lists it all the same when szRelationship, szColumn, grbit or icolumn is. LibRed now does the same, reading a missing name or column as empty and missing flags or order as zero. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A damaged file can store a column name ACE's DDL refuses, a control character in it. ACE reads such a name as stored, and once the column is renamed away it refuses to give the name back: "is not a valid name" (measured through DAO). LibRed does the same. The workload's rename round trip passed the stored name back in as a caller's argument, so LibRed's refusal of it surfaced as an ArgumentException leak. The round trip now runs only on a name JetName.Validate accepts. A new test pins the measured behaviour on both engines, and the TDEF spec records it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ResolveSlots built a dictionary of every column in the table to look up the index's columns, so a definition whose damage gave two columns one name threw ArgumentException out of CREATE INDEX and ADD FOREIGN KEY, whichever column was asked for. ACE reads such a table, resolves the name to the first column that has it, and creates an index over that name or any other (measured). ResolveSlots now looks each column up with TableDefinition.FindColumn, the first-match lookup every other by-name read already uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since the move to .NET 11 RC 3, the 2016 redistributable takes an access violation in ACECORE on the literal 'lja' < 'lm' in a database stamped Croatian. The fault ends the test process, so every LibRed.Core.AccessTests test after this one went unrun on that leg while the step still reported no failures. The Microsoft 365 runtime runs the query, so the test keeps running there; DATETIME2 support is how the suite tells the two apart. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
EF Core 11's specification tests need xunit.v3 4.0.1, so the xunit.v3 packages move with it, kept on the build EF resolves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Access ODBC driver refuses a string parameter longer than 255 characters, and a binary one longer than 510 bytes, with HY104 "Invalid precision value", whatever the column, so saving or comparing a long value in a memo or long binary column failed (CirrusRedOrg#306). Bound as long text or long binary the same values store and compare, and a value too big for its column gets ACE's "field is too small", as on OLE DB (measured). The string and byte-array mappings already mark such a value unbounded (Size -1); on ODBC they now also set OdbcType.NText or OdbcType.Image, through the set_OdbcType lookup the long and decimal mappings use. A JSON document goes through JetJsonTypeMapping, which had no ConfigureParameter and failed the same way past 255 characters, so it gets one. OLE DB takes long values as they are and is left unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A query with no table of its own (a top-level Any()) reads its FROM from JetDualTable.Name, which defaulted to #Dual. Only databases the provider creates have that table, so the query failed on any database made by Access or another tool. The table is now chosen in this order: JetDualTable.CustomName, then #Dual when the open database has it, then the first table the query itself reads. The migration lock falls back to its own lock table. JetConnection and LibRedConnection report whether #Dual exists; LibRed does so only in compatible SQL mode, the only mode that reads from it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The .NET 11 migration pinned EFCoreVersion and MSLibVersion to an exact build, so a preview or RC could not float underneath the baselines. The GA update kept the exact pin at [11.0.0], so the packages required EF Core, System.Data.Odbc and System.Data.OleDb 11.0.0 and nothing later. Both are [11.0.0,11.0.999] again, as they were for EF 10. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
JetFactory still accepted 9.0.0 of either, though the packages have required 11.0.0 since the move to .NET 11. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This branch moves the providers to the .NET 11 and EF Core 11 release as 11.0.0-beta.1. It fixes two Jet provider failures: a query with no table of its own on a database the provider did not create, and long values over ODBC. It also hardens LibRed against damaged files with a coverage-guided fuzzer. Nearly every LibRed behaviour change was measured against ACE first, on the same statement or the same damage, and CI now cross-checks LibRed against the Microsoft 365 Access Runtime as well as the 2016 redistributable.
Packaging and build
EF Core providers (Jet and shared
EFCore.Jet.Common)#Dual. Jet needs aFROM, so a query such as a top-levelAny()reads one row from a stand-in table. That was always#Dual, which only a database the provider creates has, so on a file made by Access or another tool the query failed with "cannot find the input table or query '#Dual'".JetDualTable.CustomNamewhen set, then#Dualwhen the open database has it, then the first table the query itself reads.MSysRelationships,MSysObjects,MSysQueriesandMSysACEs, andMSysAccessStorageexists only in an.accdbAccess made.ALTER TABLE … RENAME INDEXrenames an index, as DAO'sIndex.Namedoes; ACE has no SQL for it. Jet goes through DAO. LibRed writes the definition as DAO does, so a multi-page definition's continuations land where ACE puts them. A relationship's own index and a name already taken are refused.LibRed file format — damaged files, held to ACE
LibRed.Core.Fuzz(SharpFuzz and libFuzzer; not in CI, see its README). It patches a few bytes of a good database and uses the result as a caller would: reading every table, rewriting a row of each, and a write workload of long values, added and dropped columns and indexes, rolled-back deletes, renames and drops. It shares one definition of a leak with the fixed-seed sweep, and a dictionary gives libFuzzer the offsets, type codes and flag bits a reader steers by.CREATE INDEXvalidates the usage-map page it appends to. An index entry a write cannot find is damage, as ACE's "The search key was not found" is.CREATE INDEXandADD FOREIGN KEYthrew.UPDATEand theALTER COLUMNre-lay threw from an array copy.LibRed file format — writes
DZsisDZ+s, where LibRed took the whole trigraph.Tests and CI
BIGINT/DATETIME2and backtick-quotedCHECKcross-checks actually run. It allows remote-table queries, which that runtime blocks by default. The leg iscontinue-on-erroruntil it proves stable.Docs
src/LibRed/docs/format): each damaged-file behaviour above, the index rename and the one-to-one relationship.CLAUDE.md/AGENTS.md: the version and SDK, the fuzzer and the two LibRedAccess legs.🤖 Generated with Claude Code