From 67ec0250aeca808619f698d0ff85f1a991984ae3 Mon Sep 17 00:00:00 2001 From: Jakob Blomer Date: Tue, 25 Aug 2026 14:31:28 +0200 Subject: [PATCH 1/5] [NFC][ntuple] match naming style for Tuning.md --- tree/ntuple/doc/{tuning.md => Tuning.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename tree/ntuple/doc/{tuning.md => Tuning.md} (100%) diff --git a/tree/ntuple/doc/tuning.md b/tree/ntuple/doc/Tuning.md similarity index 100% rename from tree/ntuple/doc/tuning.md rename to tree/ntuple/doc/Tuning.md From fff624a4e34f4660a01e20c77a65b3630277d94f Mon Sep 17 00:00:00 2001 From: Jakob Blomer Date: Tue, 25 Aug 2026 14:33:12 +0200 Subject: [PATCH 2/5] [NFC][ntuple] minor updates to README.md --- tree/ntuple/doc/README.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/tree/ntuple/doc/README.md b/tree/ntuple/doc/README.md index b898957f59e7e..ddfdcb015aa39 100644 --- a/tree/ntuple/doc/README.md +++ b/tree/ntuple/doc/README.md @@ -67,11 +67,10 @@ consecutive entries. Clusters provide a unit of writing and provide the means f To get first information about an RNTuple, ROOT::RNTupleReader provides RNTupleReader::PrintInfo(). To show entries, it provides RNTupleReader::Show(). +The ROOT browsers show RNTuple contents. + To get more details such as the achieved compression, there is RNTupleInspector. For plotting and scanning through entries, use ROOT::RDataFrame. A table for translating TTree commands to RDataFrame can be found at [RDataFrame: Rosetta stone](https://root.cern/doc/master/classROOT_1_1RDataFrame.html#rosetta-stone). These commands work both with TTree as well as RNTuple. - -## Related classes - From cab56a69e226996d0521c2ccd6e86cf636cf9e14 Mon Sep 17 00:00:00 2001 From: Jakob Blomer Date: Thu, 3 Sep 2026 11:35:50 +0200 Subject: [PATCH 3/5] [NFC][ntuple] add RNTupleExporter to Architecture.md --- tree/ntuple/doc/Architecture.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/tree/ntuple/doc/Architecture.md b/tree/ntuple/doc/Architecture.md index 7ffd270328f55..fbcfb743a33d9 100644 --- a/tree/ntuple/doc/Architecture.md +++ b/tree/ntuple/doc/Architecture.md @@ -317,6 +317,11 @@ The RNTupler merger is used by the `TFileMerger` and thus provides RNTuple merge The RNTupleImporter creates RNTuple data sets from ROOT trees. It is part of the `ROOTNTupleUtil` library. +### RNTupleExporter +The RNTupleExporter writes the pages of an RNTuple as individual files. +This can be useful for compression studies. +It is part of the `ROOTNTupleUtil` library. + ### RNTupleInspector The RNTupleInspector provides insights of an RNTuple, e.g. the distribution of data volume wrt. column types. It is part of the `ROOTNTupleUtil` library. From 444cb7e4cde2aae5388ed9b373ab437e8cdbe03f Mon Sep 17 00:00:00 2001 From: Jakob Blomer Date: Thu, 3 Sep 2026 11:39:03 +0200 Subject: [PATCH 4/5] [NFC][ntuple] mention type emulation in Architecture.md --- tree/ntuple/doc/Architecture.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tree/ntuple/doc/Architecture.md b/tree/ntuple/doc/Architecture.md index fbcfb743a33d9..264f57d5ce845 100644 --- a/tree/ntuple/doc/Architecture.md +++ b/tree/ntuple/doc/Architecture.md @@ -503,6 +503,10 @@ For user-defined classes as well as sets and maps, RNTuple uses `TClass`. Simple types and other stdlib classes are natively supported and do not require dictionaries. See the format specification for an exhaustive list of types supported in RNTuple. The streamer field uses the standard ROOT streaming machinery. +Without dictionaries, RNTuple can still read data with an emulated schema derived from the RNTuple fields. +Type emulation reads classes as untyped records, collections as `std::vector`s, and custom enums as integers. +Type emulation works for all fields except streamer fields. +Emulated types cannot be used for writing, but they can be used for merging. Integration to RDataFrame is provided through an RNTuple data source. A universal RDataFrame constructor can create a data frame from either a TTree or an RNTuple with the same syntax. From 95d209d05ed0310f693e29b1c75f8c1ca68ec8ee Mon Sep 17 00:00:00 2001 From: Jakob Blomer Date: Mon, 31 Aug 2026 15:15:54 +0200 Subject: [PATCH 5/5] [NFC][ntuple] add SoA design document --- tree/ntuple/doc/SoA.md | 111 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 111 insertions(+) create mode 100644 tree/ntuple/doc/SoA.md diff --git a/tree/ntuple/doc/SoA.md b/tree/ntuple/doc/SoA.md new file mode 100644 index 0000000000000..da71574cadd5e --- /dev/null +++ b/tree/ntuple/doc/SoA.md @@ -0,0 +1,111 @@ +# SoA I/O in RNTuple + +RNTuple has a mechanism to represent a collection of a class, i.e. an "array of struct" (AoS), +as "struct of array" (SoA) in memory. +Note that because the data of an AoS in an RNTuple are stored in columnar layout, +the on-disk layout allows for the effecient transformation into a SoA in-memory layout. + +RNTuple provides SoA I/O through the `RSoAField`. +The `RSoAField` stores "RNTuple SoA types", classes with `RVec` data types that meet certain properties (see below). +There is some degree of freedom on the in-memory layout of a SoA type +but every SoA type has one, well-defined on-disk representation +based on the corresponding "underlying record type" (see below). + +Being a regular class, a SoA type could be stored just through the normal RNTuple I/O. +Storing them using "SoA I/O", however, has the following advantages: + + - The length of the vector members is not duplicated on disk. + - The same on-disk representation can be used to populate different, compatible SoA in-memory layouts, + picked at runtime + - As a runtime decision, the data can also be read into an AoS layout (e.g., a `vector` of the underlying record type) + +## RNTuple SoA Types + +An RNTuple SoA type can only exist in combination with its "underlying record type". +The underlying record type defines the AoS layout to which the SoA type corresponds. +During writing, the underlying record type may not be used directly by the user but its dictionary must exists. + +Concretely, an RNTuple SoA type is a user-defined class that has exactly one associated underlying record type, +with the following constraints: + + - The RNTuple SoA type must meet all the conditions for doing RNTuple I/O (see binary format specification). + - Likewise, its underlying record type must be a user-defined class that meets the conditions for RNTuple I/O. + - SoA type `A` is allowed to inherit from a SoA type `B` whose underlying record type is `X` + if and only if the underlying record type of `A` inherits from `X`. + SoA types must only inherit from other SoA types. + - For every persistent member of type `T` in the underlying record type, + there must be a member with the same name in the SoA type. + The data type of that data member in the SoA class must be either `RVec` or + a SoA type that has `T` as an underlying record type (nested SoA type). + The SoA type must have no additional persistent data members. + - The SoA type and its underlying record type must have the same class version number. + +These conditions are checked at runtime when an `RSoAField` is created. +Equal vector lengths are ensured by construction when reading from disk and checked when writing to disk. + +The underlying record type of a SoA type can be specified in the dictionary or, at runtime, as a class attribute. + +Emulated reading (see Architecture.md) reads SoA types as `std::vector`. + +### Example + +For the underlying record type(s) + +``` +struct Properties { + int fId; + int fColor; +}; + +struct Point { + float fX; + float fY; + Properties fProperties; +}; +``` + +a possible SoA layout is + +``` +struct PointSoA { + ROOT::RVec fX; + ROOT::RVec fY; + ROOT::RVec fProperties; +} +``` + +Another possible SoA layout is + +``` +struct PropertiesSoA { + ROOT::RVec fId; + ROOT::RVec fColor; +}; + +struct PointSoA { + PropertiesSoA fProperties; + ROOT::RVec fY; + ROOT::RVec fX; +} +``` + +### Choice of `ROOT::RVec` + +For the SoA vectors, the `ROOT::RVec` type is used because it can own or adopt memory. +As a result, optimized code can prepare a memory region and initialize an `RVec` with that region and the right length +in order to directly read into adopted memory. +SoA fields can also be read without any additional logic, in which case the `RVec`s own their memory. + +## Schema Evolution of SoA types + +Schema evolution of SoA types is identical to normal user-defined classes except for the following caveats. + +For added members, reading will set the corresponding vector(s) to the collection length +and default-initialize the vector elements. +This is different to added members of normal classes, for which reading is a no-op. + +For I/O customization rules, there is no check if the rules of the underlying record type are consistent +with the rules of the SoA types. +A rule mismatch means that data values are different depending on whether data is read into an SoA or the AoS layout. +When reading through the `RSoAField`, the rules of the SoA type apply. +When reading data as a collection of underlying record type, the rules of the underlying record type apply.