Skip to content

11.0.0-beta.1: .NET 11 and EF Core 11, LibRed fixes, dual-table fallback - #308

Merged
ChrisJollyAU merged 28 commits into
CirrusRedOrg:masterfrom
ChrisJollyAU:beta1
Nov 11, 2026
Merged

ChrisJollyAU merged 28 commits into
CirrusRedOrg:masterfrom
ChrisJollyAU:beta1

Conversation

@ChrisJollyAU

@ChrisJollyAU ChrisJollyAU commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

This branch moves the providers to the .NET 11 and EF Core 11 release as 11.0.0-beta.1. It fixes two Jet provider failures: a query with no table of its own on a database the provider did not create, and long values over ODBC. It also hardens LibRed against damaged files with a coverage-guided fuzzer. Nearly every LibRed behaviour change was measured against ACE first, on the same statement or the same damage, and CI now cross-checks LibRed against the Microsoft 365 Access Runtime as well as the 2016 redistributable.

Packaging and build

  • .NET 11 and EF Core 11
  • Version bumped to 11.0.0-beta.1.

EF Core providers (Jet and shared EFCore.Jet.Common)

  • A query with no table of its own no longer needs #Dual. Jet needs a FROM, so a query such as a top-level Any() reads one row from a stand-in table. That was always #Dual, which only a database the provider creates has, so on a file made by Access or another tool the query failed with "cannot find the input table or query '#Dual'".
    • The stand-in is now JetDualTable.CustomName when set, then #Dual when the open database has it, then the first table the query itself reads.
    • No system table can stand in: ACE refuses to read MSysRelationships, MSysObjects, MSysQueries and MSysACEs, and MSysAccessStorage exists only in an .accdb Access made.
    • The migration lock falls back to its own lock table. LibRed does the same in compatible SQL mode; extended mode writes no stand-in.
  • Long strings and binary values save over ODBC. The Access ODBC driver refuses a string parameter over 255 characters, or a binary one over 510 bytes, with HY104 "Invalid precision value", whatever the column, so a long value in a memo or long binary column could not be saved or compared. They now bind as long text and long binary, JSON documents included, and a value too big for its column gets ACE's "field is too small", as on OLE DB. OLE DB is unchanged.
  • ALTER TABLE … RENAME INDEX renames an index, as DAO's Index.Name does; ACE has no SQL for it. Jet goes through DAO. LibRed writes the definition as DAO does, so a multi-page definition's continuations land where ACE puts them. A relationship's own index and a name already taken are refused.

LibRed file format — damaged files, held to ACE

  • A coverage-guided fuzzer, LibRed.Core.Fuzz (SharpFuzz and libFuzzer; not in CI, see its README). It patches a few bytes of a good database and uses the result as a caller would: reading every table, rewriting a row of each, and a write workload of long values, added and dropped columns and indexes, rolled-back deletes, renames and drops. It shares one definition of a leak with the fixed-seed sweep, and a dictionary gives libFuzzer the offsets, type codes and flag bits a reader steers by.
  • What it found, each fixed to ACE's answer on the same damage:
    • A catalog row with a null name or id is refused, and null flags read as none. A page lacking one of a system table's columns is not that table.
    • A null in a relationship row no longer throws: the relationship is left out where an object name is null, and listed otherwise. A relationship column of the wrong type refuses the foreign-key list alone, leaving every table readable.
    • Row boundaries, calculated-value widths and usage-map page ranges are checked before use, and CREATE INDEX validates the usage-map page it appends to. An index entry a write cannot find is damage, as ACE's "The search key was not found" is.
    • A free map naming a page past the 2 GiB limit is refused at the first allocation, instead of growing the file towards it a page at a time.
    • Closing a damaged file no longer throws where the released-pages map cannot grow or starts below page 0; the pages are freed all the same.
    • An index over a name two columns share resolves to the first, where CREATE INDEX and ADD FOREIGN KEY threw.
    • A delete consults the complex-column registry only for a table with a complex column, so a damaged registry no longer stops deletes from every other table.
  • An AutoNumber flag on a column of another type sets the counter as ACE sets it: a text or OLE value's first four bytes, a Single's own bits. An Integer, Currency or Date/Time column refuses the row with "The field is too small…", after drawing an id that stays burned.
  • A row grown past the page is refused with ACE's "Record is too large", where UPDATE and the ALTER COLUMN re-lay threw from an array copy.

LibRed file format — writes

  • A one-to-one relationship can be written through the Core API, as DAO writes it; it has no SQL form. The relationship's catalog object is now written before its index, so a refused one leaves ACE's counters behind.
  • A closing connection returns its pages inside the file's exclusive scope, as a statement does. Another connection's commit could land on the global usage map first, so the close threw "Transaction write conflict on page 1" after its work had committed, and the pages stayed out of the free map. It was an intermittent failure of the concurrent migration-lock test on CI.
  • Digraphs contract only in lowercase, uppercase or title case, as ACE keys them: Hungarian DZs is DZ + s, where LibRed took the whole trigraph.

Tests and CI

  • LibRedAccess has a second leg on the Microsoft 365 Access Runtime, installed through the Office Deployment Tool, where the BIGINT/DATETIME2 and backtick-quoted CHECK cross-checks actually run. It allows remote-table queries, which that runtime blocks by default. The leg is continue-on-error until it proves stable.
  • On the 2016 redistributable, the Croatian-stamped database query is skipped. Since .NET 11 RC 3 it takes an access violation inside ACE that ended the test process, so every later test on that leg went unrun while the step reported no failures.

Docs

  • Format spec (src/LibRed/docs/format): each damaged-file behaviour above, the index rename and the one-to-one relationship.
  • CLAUDE.md/AGENTS.md: the version and SDK, the fuzzer and the two LibRedAccess legs.

🤖 Generated with Claude Code

ChrisJollyAU and others added 26 commits October 23, 2026 18:24
…s ACE 2016

LibRedAccess ran only on the 2016 Access Database Engine redistributable,
which is out of support and differs from the Microsoft 365 runtime the
cross-checks are measured on: it knows neither BIGINT nor DATETIME2, cannot
evaluate a backtick-quoted CHECK, and leaves a stored query's LvExtra
uninitialised. Those cases skip or are masked there, so CI never ran them.

The job now has a second leg, 365, which installs the click-to-run Access
Runtime (AccessRuntimeRetail, x64, Current channel) through the Office
Deployment Tool, fetched from Microsoft's CDN since it downloads the
runtime from there anyway. A failed install fails the step. The 2016 leg is
unchanged, and the 365 leg is continue-on-error until it proves stable.
push.yml and pull_request.yml change alike; CLAUDE.md and AGENTS.md
describe the two legs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since the December 2025 Office builds, ACE refuses every query that reaches
another file ("Operation is not supported for this type of object") unless
AllowQueryRemoteTables is 1. The runtime the 365 leg installs blocks by
default, so on its first run the eight IN externaldatabase append and
make-table cases of StoredActionQueryWriteAccessTests failed when ACE ran the
stored query: the test skips on that message only where ACE creates it, and a
stored append or make-table is still created under the block.

The install step now sets AllowQueryRemoteTables to 1, as on the machine the
cross-checks were measured on, creating the Engines key only where it is
missing. The 2016 redistributable does not block, and its leg is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A contraction was refused only where a lowercase letter met an uppercase one, which holds for every
two-letter digraph but not for Hungarian dzs: ACE splits DZs into DZ + s, where LibRed took the whole
trigraph. Measured against ACE in every case mix of every contraction order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a connection closes, PageAllocator.ReturnReleasedPages hands back the
pages it released and the reservation and groups it holds, rewriting the
global usage map on page 1 in a transaction of its own. Every writing
statement holds the file's exclusive scope from its first read to its
commit; this transaction did not, so a statement on another connection
could commit page 1 after the close first wrote it, and the close failed
with "Transaction write conflict on page 1". Close and Dispose then threw
after the connection's work had already committed, and the pages it should
have returned stayed out of the free map.

That is MigrationLockConcurrencyTests' intermittent failure on CI: the
winning connection releases the lock and closes while seven others keep
writing. A stress run of the same pattern (16 connections, no pauses)
failed 13 of 16 threads in its first round; it now runs ten rounds clean.

The method's body now runs in PageChannel.WriteExclusive, as a statement
does, which covers both callers (JetDatabase.Dispose and the encryption
path). Its early returns, which read only the connection's own state, stay
outside the scope. The bytes written are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ACE has no SQL for an index rename; DAO's Index.Name is the only way, and
it was measured: a plain index, a primary key (even one a relationship
references), a case-only change and a multi-page definition all rename;
the relationship's own index and a name already taken are refused.

Jet: ISchemaOperationsProvider gains RenameIndex. DAO sets Index.Name,
ADOX refuses and points at DAO, and Precise goes through DAO.
JetSchemaOperationsHandling routes "alter table X rename index A to B"
to it.

LibRed: SchemaEditor.RenameIndex moves the logical index block to its
new sorted position and writes the definition twice, as DAO does, so a
multi-page definition's continuations land where ACE puts them. The
duplicate-name check is split out of InsertIndex, InsertLogicalBlock
takes the info bytes, and the relationship guard is split so a rename
refuses only the relationship's own index. The SQL statement now runs
through JetDatabase.RenameIndex instead of refusing. The parity tests
compare the result with DAO's byte for byte, refusals included.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A one-to-one (grbit 0x01, DAO's dbRelationUnique) has no SQL form, so
RelationshipSpec gains IsOneToOne for direct callers. Measured against
DAO, it differs from the same one-to-many in two places only: 0x01 on
every MSysRelationships row of the key, and a unique child index (0x81
for 0x80). Being unique, that index shares the block of a unique index
over the same columns, though still not the primary key's.

AddForeignKey now writes the relationship's MSysObjects object before
its index, inside a transaction of its own when the caller has none, as
ACE does: a one-to-one refused over duplicate child keys rolls back but
leaves MSysObjects' and MSysACEs' counting behind. The object is written
by each caller of AddRelationshipRows rather than inside it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An UPDATE and the ALTER COLUMN re-lay build each row in a page-sized
buffer and leave it there for the next one. The row was copied into
that buffer before the record-size check ran, so a row grown past the
page threw an ArgumentException from the copy ("Destination array was
not long enough") where ACE refuses the statement with "Record is too
large". The copy now comes after the check on both paths: UPDATE copies
the encoded row once EnsureRecordFits has passed it, and the re-lay once
RewriteRowRaw has.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LibRed.Core.Fuzz is a coverage-guided harness over LibRed.Core
(SharpFuzz + libFuzzer, not in CI; see its README). Its database targets
patch a few bytes of a good seed database and walk it as a caller would,
read-only or rewriting a row of each table. The walk and the leak
contract move out of CorruptFileSweepProbeTests into
test/LibRed.Shared/MalformedFile.cs, so the fixed-seed sweep and the
fuzzer cannot disagree about what counts as a leak. A write may also
refuse with InvalidOperationException, as ACE refuses the same writes;
it follows only the tables still listed, since a write that follows a
damaged free map can overwrite the catalog, and it tries the first 32
rows of a table rather than every one.

What the fuzzer found, each measured against ACE on the same damage:

- A catalog row with a null Name or Id is refused; a null Flags reads as
  no flags. A page read as MSysObjects, MSysQueries or MSysRelationships
  that lacks one of their columns is not that table, and an MSysObjects
  with a calculated column, or a table whose calculated column has no
  Expression property, is refused as ACE refuses it.
- A data page's previous row boundary is checked before it is used, a
  calculated value's width before its status, and a usage map's pages
  before one below 0 or past int.MaxValue is taken for a bit index.
- An index entry a write cannot find is damage (InvalidDataException),
  as ACE refuses it with "The search key was not found".
- On a Text column carrying the AutoNumber flag, an explicit value sets
  the counter at 0x14 to its first four UTF-16 bytes, as ACE does,
  rather than failing to convert the text.
- A free map that names a page past the 2 GiB limit is refused at the
  first allocation, as ACE refuses it, instead of growing the file a
  page at a time towards it. PageChannel.MaxPageCount names the limit
  WritePage already enforced.
- A released-pages map starting below page 0 is left as it is at close
  and the pages are freed all the same, as ACE does, where sizing it
  overflowed.

The format spec records each measurement, along with what ACE does when
the free map offers a live page: it takes it without looking. The last
four fixes have tests in LibRed.Core.Tests (IndexTraversalCorruption,
ColumnDescriptorFlag, GlobalMapPointer, ReleaseAtClose); the rest are
covered by the sweep and the earlier tests that changed with them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Table.Delete asked MSysComplexColumns for every table's complex columns,
seeking its ConceptualTableID index with the table's definition page.
With that column's type byte damaged from Long to Byte, the seek key of a
table past page 255 overflowed the Byte, and every delete from such a
table threw an OverflowException. Only a table with a Complex column has
values elsewhere to remove, so only it asks the registry now; ACE
inserts, updates and deletes rows of other tables on this damage as
usual (measured).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…E does

On an OLE column whose descriptor carries the AutoNumber flag (a damaged
file), an explicit insert passed the binary value to Convert.ToInt32 to
update the counter at 0x14, which threw InvalidCastException. ACE sets
the counter to the value's first four bytes, as it does a text value's
first four UTF-16 bytes (measured: bytes 01 02 03 04 05 06 give
0x04030201), so the binary value now takes the same path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Closing a writable JetDatabase returns the pages its session released,
and a release that cannot run on a damaged file was already absorbed:
it runs in its own transaction, so it writes nothing and the pages stay
released. Only InvalidDataException was caught, though. A released map
turned reference form needs a bitmap page, and a free map with a nonzero
start page and no free page refuses that allocation with
NotSupportedException, which then escaped from Dispose. The close now
absorbs that refusal the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…does

On a fixed column whose descriptor carries the AutoNumber flag (a
damaged file), an explicit insert passed the value to Convert.ToInt32 to
update the counter at 0x14; a Date/Time value threw InvalidCastException.
Measured against ACE, the column's width decides. The counter is four
bytes: a Single takes its own bits (0.25 gives 0x3E800000), while an
Integer, Currency or Date/Time column refuses the row, explicit value
and all, with "The field is too small to accept the amount of data you
attempted to add." after drawing an id, which stays burned. Insert now
does the same, refusing before the row is written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After millions of executions the write target had stopped finding
anything new: on a damaged file it only ever rewrote one row of each
table. The first input byte still picks the seed by its remainder; its
quotient now selects operations from MalformedFile.Operations, run on
each user table after the row rewrite: a chained long value written and
taken out again, two columns added and dropped, an index built and
dropped, rows deleted by a statement that rolls back, a rename and back,
and a drop. Each runs as one statement in a transaction of its own, a
refusal rolling it back, as QueryEngine runs a writing statement. The
patches keep their meaning, so earlier corpora and crashes still replay.

database.dict holds the multi-byte values random mutation is slow to
find for the patch-list input: the page numbers and offsets of the
fields a reader steers by, and the type codes and flag bits they take.
The README covers the workload, -use_value_profile and -dict (quoted in
PowerShell), and merging old corpora into a fresh one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… type

LoadRelationships casts each MSysRelationships value to the type its
column should have. With a type byte damaged (szColumn's made 0x0D, a
code no type has) the value decodes as bytes, and the cast threw
InvalidCastException out of DropIndex's relationship lookup. Measured
against ACE on that damage, the foreign-key list fails "Cannot perform
this operation" while every table still lists and reads and unrelated
DDL runs. LoadRelationships now refuses with InvalidDataException when a
column it reads is not its spec's type; the table list is untouched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CREATE INDEX appends the new index's usage map to the page holding the
table's own maps, and AppendEmptyUsageMapRow read that page raw. With
its row count damaged past what a page holds, TryAppendRow walked the
slot directory off the end of the page and threw
ArgumentOutOfRangeException. It now parses the page with DataPage.Read
first, as LongValueStore's append does, so the page is refused as
InvalidDataException exactly as every read of it already refuses it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LoadRelationships cast every MSysRelationships column straight to its
type, so one null (a cleared null-bitmap bit) threw ArgumentNullException
or NullReferenceException out of Catalog.Relationships, and with it out
of DROP COLUMN's relationship check. ACE, measured with one column of one
row made null, leaves the relationship out of its foreign-key list when
szObject or szReferencedObject is null and lists it all the same when
szRelationship, szColumn, grbit or icolumn is. LibRed now does the same,
reading a missing name or column as empty and missing flags or order as
zero.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A damaged file can store a column name ACE's DDL refuses, a control
character in it. ACE reads such a name as stored, and once the column is
renamed away it refuses to give the name back: "is not a valid name"
(measured through DAO). LibRed does the same. The workload's rename round
trip passed the stored name back in as a caller's argument, so LibRed's
refusal of it surfaced as an ArgumentException leak. The round trip now
runs only on a name JetName.Validate accepts. A new test pins the measured
behaviour on both engines, and the TDEF spec records it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ResolveSlots built a dictionary of every column in the table to look up
the index's columns, so a definition whose damage gave two columns one
name threw ArgumentException out of CREATE INDEX and ADD FOREIGN KEY,
whichever column was asked for. ACE reads such a table, resolves the
name to the first column that has it, and creates an index over that
name or any other (measured). ResolveSlots now looks each column up with
TableDefinition.FindColumn, the first-match lookup every other by-name
read already uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since the move to .NET 11 RC 3, the 2016 redistributable takes an access
violation in ACECORE on the literal 'lja' < 'lm' in a database stamped
Croatian. The fault ends the test process, so every LibRed.Core.AccessTests
test after this one went unrun on that leg while the step still reported
no failures. The Microsoft 365 runtime runs the query, so the test keeps
running there; DATETIME2 support is how the suite tells the two apart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
EF Core 11's specification tests need xunit.v3 4.0.1, so the xunit.v3
packages move with it, kept on the build EF resolves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Access ODBC driver refuses a string parameter longer than 255
characters, and a binary one longer than 510 bytes, with HY104 "Invalid
precision value", whatever the column, so saving or comparing a long
value in a memo or long binary column failed (CirrusRedOrg#306). Bound as long text
or long binary the same values store and compare, and a value too big
for its column gets ACE's "field is too small", as on OLE DB (measured).

The string and byte-array mappings already mark such a value unbounded
(Size -1); on ODBC they now also set OdbcType.NText or OdbcType.Image,
through the set_OdbcType lookup the long and decimal mappings use. A JSON
document goes through JetJsonTypeMapping, which had no ConfigureParameter
and failed the same way past 255 characters, so it gets one. OLE DB takes
long values as they are and is left unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A query with no table of its own (a top-level Any()) reads its FROM from
JetDualTable.Name, which defaulted to #Dual. Only databases the provider
creates have that table, so the query failed on any database made by Access
or another tool.

The table is now chosen in this order: JetDualTable.CustomName, then #Dual
when the open database has it, then the first table the query itself reads.
The migration lock falls back to its own lock table. JetConnection and
LibRedConnection report whether #Dual exists; LibRed does so only in
compatible SQL mode, the only mode that reads from it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ChrisJollyAU
ChrisJollyAU requested a review from a team as a code owner November 11, 2026 14:05
ChrisJollyAU and others added 2 commits November 11, 2026 23:15
The .NET 11 migration pinned EFCoreVersion and MSLibVersion to an exact
build, so a preview or RC could not float underneath the baselines. The
GA update kept the exact pin at [11.0.0], so the packages required EF Core,
System.Data.Odbc and System.Data.OleDb 11.0.0 and nothing later. Both are
[11.0.0,11.0.999] again, as they were for EF 10.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
JetFactory still accepted 9.0.0 of either, though the packages have
required 11.0.0 since the move to .NET 11.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ChrisJollyAU
ChrisJollyAU merged commit 460a460 into CirrusRedOrg:master Nov 11, 2026
5 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant