What about: using pointers to fields of an enum to change the active variant?

How do we feel about this code -- UB or not?

fn main() { unsafe {
    let mut x = Some(&0); 
    match x {
        Some(ref mut b) => {
            let u = b as *mut &i32 as *mut usize;
            // Just writing into a *mut u8
            *u = 0;
        }
        None => panic!(),
    }
    assert!(x.is_some());
} }

This "magically" changes the discriminant. OTOH, it is very similar to

fn main() { unsafe {
    let mut x = Some(&0); 
    (&mut x as *mut _ as *mut usize) = 0;
    assert!(x.is_some());
} }

which is definitely allowed (it makes assumptions about layout, but we do guarantee layout of Option<&T>.

Other example:

fn main() { unsafe {
    let mut x = Some(true); 
    match x {
        Some(ref mut b) => {
            let u = b as *mut bool as *mut u8;
            // Just writing into a *mut u8
            *u = 2;
        }
        None => panic!(),
    }
    assert!(x.is_some());
} }

#![feature(rustc_attrs)]

#[rustc_layout_scalar_valid_range_start(1)]
#[repr(transparent)]
pub(crate) struct NonZero(u32);

fn main() { unsafe {
    let mut x = Some(NonZero(1)); 
    match x {
        Some(NonZero(ref mut c)) => {
            // Just writing 0 into an &mut u32
            *c = 0;
        }
        None => panic!(),
    }
    assert!(x.is_some());
} }

Relevant questions:

Is changing the discriminant through a pointer derived from an enum variant okay if there's never again an access with the outer pointer? In other words, does this act like a protector where "invalidating" the discriminant seen by parent pointers is immediate UB? If no, there are still two variants: one where the discriminant can be "repaired" again (i.e. we just check the discriminant again when the parent pointer is used), and one where it is permanently invalidated (when overwriting the discriminant the parent pointer becomes unusable but the child pointers remain usable). This was brought up here.

To clarify, the last example doesn't need the unsafe {}, right? It's not "safe code" (you can modify it to be unquestionably UB by omitting the Option and reading x after the end) but that's due to the use of #[rustc_layout_scalar_valid_range_start], not due to any individual operation in main.

@rkruppe It needs the unsafe, we have fixed unsafety checks in rustc to require an unsafe block to create a reference into a rustc_layout_scalar_valid_range_start struct.

See #189 for me describing this problem again. ;)

An interesting observation: the following code is certainly fine; FFI code does something like this all the time when using Option<&T> on the interface:

fn main() {
    let mut x = Some(NonZeroU32::new(1).unwrap());
    let xref = &mut x as *mut _ as *mut u32;
    unsafe { *xref = 0 };
    println!("{:?}", x);
}

Operationally, this is basically the same as the problematic

use std::num::NonZeroU32;

fn main() {
    let mut x = Some(NonZeroU32::new(1).unwrap());
    let xref = match x {
        Some(ref mut c) => c as *mut NonZeroU32 as *mut u32,
        None => panic!(),
    };
    unsafe { *xref = 0 }; // writing 0 into a `*mut u32`
    println!("{:?}", x);
}

To distinguish the two, xref would need to have some kind of "provenance" remembering that it was created from a pointer to NonZeroU32, not a pointer to u32.

Just some musings I came up with:

When moving a !Copy type, invalidating the old name is obviously correct.

Example:

let a: Thing1 = default();
let aref = &a as *const _;
let b: Thing2 = transmute(a);
// `aref` is invalid _even if_ we guaranteed `b` were in the same physical location
// as a logical move has occurred

So maybe the last copy of a Copy value could be a move as well?

Example:

let mut x = Some(&0);
match x {
    Some(ref mut b) => {
        let u = b as *mut bool as *mut u8;
        // `b` is no longer required to be valid, as it has been "moved"
        // Just writing into a *mut u8
        *u = 2;
    }
    None => panic!(),
}
dbg!(x);

On the other hand, the origin provenance makes sense as well. Using this example for clarity:

let mut x = Some(NonZeroU32::new(1).unwrap());
let xref = match x {
    Some(ref mut c) => c as *mut NonZeroU32 as *mut u32,
    None => panic!(),
};
unsafe { *xref = 0 }; // writing 0 into a `*mut u32`
dbg!(x);

It "makes sense" that the c binding only "owns" the right to write to the NonZeroU32.

Naively, it feels like this should behave the same as

let mut x = (1u32, 0u32);
let x1ref = &mut x.1 as *mut u32;
let xref = xref.sub(1) as *mut (u32, u32);
*xref = (0, 0);
dbg!(x);

Specifically, it's taking a reference into the "interior" of an allocated object and "upgrading" it to a pointer to the full object to write the full object. It just so happens that the physical pointers are the same.

My intuition here is suggesting something like guaranteed niche optimizations shouldn't matter for what operations you are allowed do with a pointer, just how you're able to accomplish said operations, as well as what memory reinterpretation is allowed. I don't know how practical that is, but it seems somewhat logical. (e.g. <*const Option<NonZeroU32>>::cast::<u32> is valid because of the guaranteed niche optimization, but <*const NonZeroU32>::cast::<Option<NonZeroU32>> is not valid, even if said pointer lives in that type (unless it was derived from a wider pointer).)

Specifically, it's taking a reference into the "interior" of an allocated object and "upgrading" it to a pointer to the full object to write the full object. It just so happens that the physical pointers are the same.

Yes, that is a way to look at it. Your "upgrading" example would be rejected by Stacked Borrows because that pointer is not allowed to write to the "outer" memory.
On the other hand, physical overlap is the thing that matters for aliasing, which is what Stacked Borrows is all about.
And note that there is nothing in the validity machinery that has any problem with deriving "outer" pointers from "inner" pointers". Only Stacked Borrows cares.

(Moved to new issue)

I asked about this on zulip, and was told to write my use case for why I'm doing it here. Here's more words about it than you asked for:

I'm trying to write a highly tuned string type, using as many performance tricks as I can manage without hurting the average case. This includes (among other tricks) zero-cost conversion from &'static str, using atomic refcounts + clone-on-write for large strings, and of course, why I'm here, the classic small-string-optimization/inline-string-optimization well known from its use in C++ stdlibs.

While I have a bunch of representations, all of them except SSO are more-or-less representationally equivalent (keeps most branches out of the non-small path). When in the SSO case though, it is essentially [u8; size_of::<String>()] where the last byte is used for length and tags.

It's very important to me* that I not increase the size relative to std::string::String, and that includes for representing things like Option<String> or Result<String, E>.

This means I need the NonNull byte pointer, and can't use this as a union, even though conceptually it's used as one. This all means it just looks like a pretty normal string:

#[repr(C)]
pub struct String {
    data: NonNull<u8>,
    len: usize,
    cap_flags: CapFlags, // just a usize**
}

But it's a bit of a lie -- I just interpret &mut self as a &mut [u8; N] when writing as a small string, and &self as a &[u8; N] when reading as one. The only true parts about it when we're small are that NonNull stays zeroed, and that CapFlags

Anyway, The issue is this: naively, the push_byte code (e.g. the part of the code handling when an ascii character is the arg to push for the small string case ended up needing too many checks to avoid writing to NonNull<u8> -- essentially you want to ask a subset of (on 64 bit) 'are our first 7 bytes nul' && 'is the incoming c == '\0'' && 'will it get written to the 8th byte'. this is hot code, so more checks, even if they're correctly predicted, are bad.***

I tried a bit, and am probably going to try harder****, but ultimately ended up just performing the write unconditionally, and testing if afterwards our first field when read as a usize is zero (if it is quickly writing a non-zero to that byte and calling a #[cold] function).

This is a branch in a relatively hot path that I'm annoyed with (it's easily predictable but oh my god it seems like such a ridiculous edge case), but it ended up with better codegen than the alternatives I've tried so far. That said I haven't really finished tuning things so it's very possible that sort of write/read pattern is ultimately problematic in other ways...

Anyway that was extremely rambling, but probably at least explained enough to get a good idea of the picture.

* Mostly because I suspect it matters a lot in practice, even though llvm makes the cost of memcpys often hard to measure. I doubt increasing memcpy traffic would be favorable for very much rust code...

** CapFlags is a transparent wrapper for a usize that just holds both the capacity and two flag bits, while managing endian-specific stuff to keep them in the right place so that reading flags from it and from the final byte of the string both work.

*** The most promising is probably the c == '\0' test, which the compiler should be able to do away with when c is constant (which appears to be the common case for arguments to push much of the time), and should be rare enough. Unfortunately, the only place this is convenient to punt it to is the non-ascii push case (which is less hot, but very far from cold), and doing so isn't exactly ideal...

**** since this makes me uncomfortable -- and rightfully so, wasn't properly accounting for the case where a macro that can expand to contain profiling data could possibly panic if the right --cfg was passed...

(alright thats it i'm leaving before i think to give one of my footnotes a footnote)

@digama0

@digama0 raised concerns about this current "feature" of the semantics proposed so far. They also say

I won't elaborate on it more in this thread, but my proposal could enforce this property without requiring a typed memory.

I assume this is using the same notion of "typed memory" as I do, i.e., a memory that contains types in its state. I am curious how you want to do that -- the proposal so far says

The property we would like to establish is that when the lifetime of the &mut b ends, b is again valid at type bool, but in an operational model we can't just say that, so instead, when &mut b as *mut bool is executed, we push on the borrow stack of b a record that, when popped, reasserts the bool invariant at this byte.

So this sounds to me like bool is stored in the borrow stack, which is part of memory. So I would call this "typed memory".

The memory doesn't contain bool on the borrow stack, it contains one of a small set of predicates on bytes representing byte sized slices of a validity property. One of those predicates is "this byte is 0 or 1", so in that sense yes it's put bool on the stack, but that doesn't mean that you have some recursive type going into the memory or a chicken and egg problem defining what types and memory are.

@rfcbot

@rfcbot fcp cancel

@RalfJung

@RalfJung proposal cancelled.

Honestly this issue has also went quite far from what it was about originally, we should have a new issue just about that sub-discussion.

Actually looking at it again, most of the issue was about the enum discriminant case. Nobody seems motivated to have any kind of "validity outside of typed copies" except when it comes to enum discriminants. I'm not sure how to phrase this precisely enough to propose FCP on that, but it seems reasonable to go forward assuming consensus on this question.

In the all-hands discussion, Mario described their detailed proposal for a model that makes this kind of code UB. We also discussed various potential simplifications of the proposal. The proposal requires bringing together the aliasing model and the "data path" much closer than they are right now.

One of my simplifications, with the goal of removing the concept of "niche decoding sites", would also make code like this UB:

let data = 1u32;
let x = &raw mut data as *mut Option<NonZeroI32>;

let inner = match *x { Some(i) => i, None => panic!() };

ptr::from_mut(inner).cast::<i32>().write(0);

let _val = data; // UB?

Mario's original proposal spent a bunch of effort to avoid this UB, given that the read of data is at a type that doesn't have any enum discriminants. OTOH that requires defining what the enum discriminants involved in any load are and lining that up with any pointers that have niche claims attached to them -- a lot of extra complexity in an already complicated model.

More discussion will be needed.

I also think there isn't a conflict, but several people seem to disagree with me on this; for example, this comment thread on the RFC was claiming an unsoundness based on reasoning based on "you could use a pointer to an enum field to change the active variant". In such cases, it is worth cross-linking the relevant threads to ensure that different/incompatible decisions aren't being made in different places (and a general agreement that no conflict exists would be a great outcome).

This issue will only make decisions about reference types that exist in the language today, not about all possible future language extensions. That is implicitly true for every issue in this repo, so IMO it's not worth spelling it out everywhere, given the infinite number of possible extensions we'd have to hedge against.

As already mentioned elsewhere, I would personally be fine with promising that as long as you put a valid T back into the &mut by the time the lifetime ends, it's fine if there was some garbage in that memory temporarily. So, the discriminant has to be back where it was when the reference was created, but it doesn't have to be valid for the entire duration of the reference's lifetime.

I proposed this for FCP in #621.

Navigation Menu

Repository navigation

What about: using pointers to fields of an enum to change the active variant? #84

85 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

What about: using pointers to fields of an enum to change the active variant? #84

Description

Activity

hanna-kruppe commented on Feb 6, 2019

RalfJung commented on Feb 6, 2019

RalfJung commented on Aug 2, 2019

CAD97 commented on Aug 2, 2019

RalfJung commented on Aug 2, 2019

RalfJung commented on Sep 16, 2019

thomcc commented on Dec 12, 2019

RalfJung commented on Oct 8, 2020

digama0 commented on Oct 8, 2020

85 remaining items

RalfJung commented on Apr 24, 2026

RalfJung commented on Apr 24, 2026

rust-rfcbot commented on Apr 24, 2026

RalfJung commented on Apr 27, 2026

RalfJung commented on May 24, 2026

ais523 commented on Aug 22, 2026

RalfJung commented on Aug 23, 2026

ais523 commented on Aug 23, 2026

RalfJung commented on Aug 23, 2026

ais523 commented on Aug 23, 2026

RalfJung commented on Aug 23, 2026

RalfJung commented on Aug 23, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions