Proposal: support C/C++ Bit-fields

[xushiwei]

Summary

Go has no bit-fields, so llcppg needs a convention for turning a C/C++ struct that declares bit-fields into something an LLGo program can declare, embed, pass to C functions, and read or write field by field.

Throughout this document:

• S is a C/C++ struct, class or union that declares at least one bit-field.
• X is the Go type llcppg generates for S.
• foo is the C name of a named bit-field of S.
• T is the Go type llcppg generates for the bit-field's declared C type (unsigned int gives c.Uint), exactly as it would for a regular field of that type.
• o and w are the bit offset of the bit-field from the start of S and its width in bits, both as reported by libclang for the target.

The proposal has two parts:

1. X keeps the size, alignment and member offsets of S. Every run of consecutive bit-fields is stored as a byte array at the position where C puts those bits.

2. For every named bit-field foo, llcppg generates a getter and a setter:

   func (*X) XGo_bitget_foo() T
   func (*X) XGo_bitset_foo(v T)

   var p Packet
   p.XGo_bitset_mode(5)
   println(p.XGo_bitget_mode())  // 5

Motivation

A bit-field binding has to get three things right:

• Layout. Several bit-fields share bytes, and a C compiler decides how they are packed. If X does not have the same size, alignment and member offsets as S, arrays of it, structs containing it, and functions that take it by value or by pointer all break.
• Access. Users must be able to read and write each bit-field as an ordinary integer of its declared type, without shifting and masking by hand.
• ABI. Passing a struct with bit-fields by value must follow the C calling convention.

A Go integer field cannot stand in for a bit-field: it would occupy whole bytes at natural alignment, which changes every offset after it.

Design

1. Where a bit-field lives

Where the bits of a bit-field end up depends on the target ABI: the allocation unit, whether a field may straddle a unit boundary, how :0 behaves, and the effect of #pragma pack and attributes. llcppg does not re-implement any of these rules. It takes o and w from libclang (clang_Cursor_getOffsetOfField and clang_getFieldDeclBitWidth), so the result is correct for whatever ABI and packing the headers are compiled with.

Bit k of S means bit k mod 8 of byte k / 8 of the memory of S, where bit 0 of a byte is its least significant bit. The bit-field foo occupies exactly bits o to o + w - 1.

2. Go representation

The members of S are converted in declaration order. Regular fields keep their usual conversion. Bit-fields are grouped into runs.

Runs. A run is a maximal sequence of consecutive bit-field members, named or unnamed, including zero-width ones. Let lo be the smallest o and hi the largest o + w among the members of the run that have w > 0. If there is no such member, the run produces nothing. Otherwise the run is stored in one field

_xgo_bits_<N> [n]uint8

where n = ceil(hi / 8) - floor(lo / 8) and N is the 0-based index of the run within X. The field is placed at byte offset floor(lo / 8), and bit offsets in accessors stay relative to the start of X.

Placement. Every item, regular field or run, must end up at its offset in S. A run has alignment 1 and a regular field has its natural alignment, so Go's own layout matches C whenever nothing but padding separates two items. If Go's natural offset for an item is smaller than its offset in S, llcppg inserts an explicit padding field _ [k]uint8 before it. If Go's natural offset is larger, which happens only in packed structs, the layout cannot be represented (section 8).

Alignment and size. Let A be the alignment of S reported by libclang. Bit-fields can raise the alignment of a struct through their declared type, but a byte array cannot. After all items are placed:

• If the alignment Go computes for X is less than A, llcppg adds _xgo_align [0]uintW as the first field, where W is A bytes wide. If A is larger than 8, uint64 is used and llcppg prints a warning, since no Go scalar has a larger alignment.
• If the size Go computes for X is still less than the size of S, llcppg appends _ [k]uint8 as trailing padding.

Guarantee. X has the size and alignment of S, every regular member has the same offset in X as in S, and the bits of every bit-field occupy the same bit positions in X as in S.

Two examples on a little-endian target. First, a struct where a zero-width field moves the next bit-field to a new allocation unit:

typedef struct {
    unsigned int ready : 1;
    unsigned int mode  : 3;
    int          delta : 4;
    unsigned int       : 0;
    unsigned int level : 6;
    int          id;
} Packet;

Bit-field	o	w	Signed
ready	0	1	no
mode	1	3	no
delta	4	4	yes
level	32	6	no

The whole run covers bits 0 to 37, so it needs 5 bytes. id lives at byte 8, where Go's natural alignment already puts it.

type Packet struct {
	_xgo_bits_0 [5]uint8
	Id          c.Int
}

Second, a struct whose alignment comes only from the declared type of its bit-fields:

typedef struct {
    unsigned int r : 1;
    unsigned int w : 1;
    unsigned int x : 1;
} Perm;

type Perm struct {
	_xgo_align  [0]uint32 // alignment 4, as in C
	_xgo_bits_0 [1]uint8  // bits 0..2
}

The zero-length array adds no size, and Go rounds the size of Perm up to 4, which matches sizeof(Perm) in C.

3. Accessors

For every named bit-field foo with a convertible declared type, llcppg generates:

func (p *X) XGo_bitget_foo() T
func (p *X) XGo_bitset_foo(v T)

For Packet:

func (p *Packet) XGo_bitget_ready() c.Uint { return c.Uint(_xgo_bitget(unsafe.Pointer(p), 0, 1)) }
func (p *Packet) XGo_bitset_ready(v c.Uint) { _xgo_bitset(unsafe.Pointer(p), 0, 1, uint64(v)) }

func (p *Packet) XGo_bitget_mode() c.Uint { return c.Uint(_xgo_bitget(unsafe.Pointer(p), 1, 3)) }
func (p *Packet) XGo_bitset_mode(v c.Uint) { _xgo_bitset(unsafe.Pointer(p), 1, 3, uint64(v)) }

func (p *Packet) XGo_bitget_delta() c.Int { return c.Int(_xgo_bitget_signed(unsafe.Pointer(p), 4, 4)) }
func (p *Packet) XGo_bitset_delta(v c.Int) { _xgo_bitset(unsafe.Pointer(p), 4, 4, uint64(v)) }

func (p *Packet) XGo_bitget_level() c.Uint { return c.Uint(_xgo_bitget(unsafe.Pointer(p), 32, 6)) }
func (p *Packet) XGo_bitset_level(v c.Uint) { _xgo_bitset(unsafe.Pointer(p), 32, 6, uint64(v)) }

Semantics.

• The getter returns the w bits at offset o. The value is zero-extended when the declared type is unsigned and sign-extended when it is signed. Signedness comes from the declared type as libclang reports it; for an enum it is the signedness of the underlying type.
• The setter stores the low w bits of v at offset o and leaves every other bit of X unchanged. A value that does not fit is truncated modulo 2^w, as in a C assignment. Negative values of a signed type are stored in two's complement, so XGo_bitset_delta(-3) stores 0b1101, and the getter then returns -3.
• If T is Go bool (from _Bool or C++ bool), the getter reports whether the stored bits are non-zero, and the setter stores 1 for true and 0 for false.
• Both methods take a pointer receiver, so the value must be addressable. The getter uses a pointer receiver too, so that reading does not copy the struct and both methods belong to the same method set.
• Accessors are not atomic. Bit-fields that share a byte form one memory location, so concurrent writes to different bit-fields of the same run race, exactly as in C.
• Accessors are ordinary Go code with no llgo:link directive and no C call. Each doc comment carries the original C declaration, for example // unsigned int mode : 3.

Usage:

var p Packet
p.XGo_bitset_ready(1)
p.XGo_bitset_mode(5)
p.XGo_bitset_delta(-3)
p.XGo_bitset_level(40)
p.Id = 42

println(p.XGo_bitget_delta()) // -3

4. Bit access helpers

Accessors call three unexported helpers, emitted once into every generated package that contains at least one bit-field:

func _xgo_bitget(base unsafe.Pointer, off, width uintptr) uint64
func _xgo_bitget_signed(base unsafe.Pointer, off, width uintptr) int64
func _xgo_bitset(base unsafe.Pointer, off, width uintptr, v uint64)

For 1 <= width <= 64 and bits numbered as in section 1:

• _xgo_bitget returns bits off to off + width - 1 of the memory at base, zero-extended into the low bits of the result.
• _xgo_bitget_signed returns the same bits sign-extended from bit width - 1.
• _xgo_bitset replaces exactly those bits with the low width bits of v and leaves all other bits unchanged.
• All three access only the bytes that contain at least one bit of the range. This keeps every access inside the run and therefore inside X. A field of up to 64 bits can span 9 bytes when it is not byte-aligned, and the helpers must handle that case.

5. Naming rules

• The C name is kept verbatim. A bit-field named mode gives XGo_bitget_mode and XGo_bitset_mode. There is no case conversion and no keyword escaping.
• The two prefixes are disjoint. Every getter starts with XGo_bitget_ and every setter with XGo_bitset_. These differ at the letter right after XGo_bit (g versus s), so no getter can equal any setter, even for bit-fields named get_x or set_x.
• They are reserved namespaces. Names starting with XGo_bitget_ or XGo_bitset_ cannot collide with methods llcppg derives from C functions, with the XGo_union_ accessors of the union proposal, or with each other, because C forbids duplicate member names.
• Accessors are exported, helpers and storage are not. Accessors start with an uppercase X. The helpers _xgo_bitget, _xgo_bitget_signed and _xgo_bitset, the storage fields _xgo_bits_<N> and the marker _xgo_align start with an underscore.

6. Bit-fields in unions and in nested records

Unions. Bit-fields can be members of a union. They start at bit 0 of the union, and they get the same XGo_bitget_ and XGo_bitset_ accessors, on the Go type of the union. The storage of the union is unchanged, since it is already an array of the right size and alignment. This replaces the rule in the union proposal that bit-field members of a union get no accessor.

Unnamed struct or union members. In C11 the members of an unnamed struct or union member are members of the enclosing record. Bit-fields inside such a member are promoted to accessors on the enclosing named type, with o being the offset of the unnamed member, in bits, plus the offset of the bit-field within it.

Named-field nested structs. A tagless struct declared inline with a field name (struct { unsigned a : 3; } inner;) needs a named Go type so that methods can be attached to it. Accessors are generated on that type, and bit offsets are relative to its start. See open question 2 for its name.

7. Bit-fields without accessors

The following get no accessor. Each is reported by a comment in the generated file. Their bits still belong to the run, so layout is not affected:

• Unnamed bit-fields, which are padding, and zero-width bit-fields.
• Bit-fields whose declared type llcppg cannot convert, such as __int128.
• Bit-fields whose width exceeds the bit size of T, which C++ allows.

8. Layouts that cannot be represented

Two situations break the rule that every item can be placed at its offset in S:

• A regular member sits at an offset smaller than its natural Go alignment allows. This happens only in packed structs.
• The bytes of a run overlap the bytes of another member. This can happen in C++ for non-POD types, where a compiler may reuse the tail padding of a base class or member.

In both cases llcppg converts X as an opaque struct with the exact size and alignment, _xgo_align followed by _xgo_opaque [S]uint8, generates no accessors for it, and prints a warning that names the struct and the reason. This rule applies only to structs that contain bit-fields. Structs without bit-fields are not affected by this proposal.

9. C++ notes

• struct and class follow the same rules, and so does a bit-field of an enum or bool type.
• Bit-fields with default member initializers are treated like any other bit-field. Initializers do not affect layout or accessors.
• Constructors are not generated, so a bit-field of a Go value starts zeroed, and any C++ initializer is the user's responsibility.

Assumptions

The design depends on the following. Each should be checked before implementation.

1. libclang reports the offset, the width and the declared type, including its signedness, of every bit-field for the target being generated.
2. Bits are allocated from the least significant bit of each byte, as on little-endian x86-64 and AArch64. On a big-endian target llcppg still preserves the layout of the struct but generates no accessors, and prints a warning.
3. When LLGo passes a Go struct by value to a C function, it lowers it according to the platform C ABI, based on the field types. C classifies bit-field storage as integer, and so does a [n]uint8 array.
4. LLGo honors the alignment of a zero-length array field, so _xgo_align [0]uintW as the first field raises the alignment of the struct.

Complete example

C header:

typedef struct {
    unsigned int ready : 1;
    unsigned int mode  : 3;
    int          delta : 4;
    unsigned int       : 0;
    unsigned int level : 6;
    int          id;
} Packet;

void packet_dump(const Packet *p);

Generated Go (abridged):

type Packet struct {
	_xgo_bits_0 [5]uint8
	Id          c.Int
}

func (p *Packet) XGo_bitget_ready() c.Uint { return c.Uint(_xgo_bitget(unsafe.Pointer(p), 0, 1)) }
func (p *Packet) XGo_bitset_ready(v c.Uint) { _xgo_bitset(unsafe.Pointer(p), 0, 1, uint64(v)) }
func (p *Packet) XGo_bitget_mode() c.Uint { return c.Uint(_xgo_bitget(unsafe.Pointer(p), 1, 3)) }
func (p *Packet) XGo_bitset_mode(v c.Uint) { _xgo_bitset(unsafe.Pointer(p), 1, 3, uint64(v)) }
func (p *Packet) XGo_bitget_delta() c.Int { return c.Int(_xgo_bitget_signed(unsafe.Pointer(p), 4, 4)) }
func (p *Packet) XGo_bitset_delta(v c.Int) { _xgo_bitset(unsafe.Pointer(p), 4, 4, uint64(v)) }
func (p *Packet) XGo_bitget_level() c.Uint { return c.Uint(_xgo_bitget(unsafe.Pointer(p), 32, 6)) }
func (p *Packet) XGo_bitset_level(v c.Uint) { _xgo_bitset(unsafe.Pointer(p), 32, 6, uint64(v)) }

//go:linkname PacketDump C.packet_dump
func PacketDump(p *Packet)

The helpers _xgo_bitget, _xgo_bitget_signed and _xgo_bitset are emitted once in the same package.

User code:

var p Packet
p.XGo_bitset_ready(1)
p.XGo_bitset_mode(5)
p.XGo_bitset_delta(-3)
p.XGo_bitset_level(40)
p.Id = 42
PacketDump(&p)

Open questions

1. Where the helpers live. Emitting them into every generated package keeps bindings self-contained but duplicates the code. Placing them in a shared package such as github.com/goplus/lib/c removes the duplication but adds a dependency and requires a change to that package.
2. Naming of hoisted structs. A tagless struct with a field name that contains bit-fields needs a named Go type (section 6). To stay consistent with the union proposal, this document suggests _llcppg_struct_<order>, with its own package-wide counter starting at 0 and the same assignment rules as _llcppg_union_<order>.
3. Big-endian targets. Assumption 2 leaves them without accessors. If big-endian targets matter, the helpers need a second bit numbering, and libclang's offsets need to be interpreted from the most significant bit.