[Kobzol]

View all comments

Right now, it is not possible to test how the compiler behaves on the stable toolchain in UI tests. Even though #132993 added a mechanism to emulate stable rustc on the nightly channel, using the RUSTC_BOOTSTRAP=-1 environment variable, this mechanism is not currently usable within compiletest and UI tests directly, because compiletest unconditionally passes several unstable -Z flags that are load-bearing for the testing infra itself to work (notably -Zui-testing and friends).

That is quite annoying, because we do in fact want to test stable behavior in a couple of tests, but today we have to write those as run-make tests (latest example: https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler/topic/Testing.20stable.20vs.20nightly.20behavior.20in.20UI.20tests/with/628206707), even if the tested situation is very simple. Those are harder to read, write and maintain than UI tests and they do not support blessing outputs as well as UI tests. Recently, I at least added #163444 as a way of centralizing the logic of testing a stable rustc in a run-make test, but that does not really help with the fact that we still have to use run-make tests for it, even when a normal UI test would do.

For #162848, I would also like to have a way of passing -Zthreads=1 to all run-make tests, even if they are testing stable rustc, but that's not easily possible, because those tests are using RUSTC_BOOTSTRAP=-1.

This PR proposes to add a new mode to RUSTC_BOOTSTRAP, with the value of -allow-unstable-compiler-flags=<flags>. This can be used to emulate the behavior of stable rustc, while still allowing passing specific unstable flags to the compiler.

The stability check in the compiler is thus now separated into two questions:

• Are we an unstable/nightly build?
• Can we accept specific unstable compiler flags?

The latter is checked only at a few places, so the necessary change to the compiler is rather small (IMO).

To demonstrate how this would be used, I also implemented a new directive into compiletest in this PR, act-as-stable. That can be used in a UI test to emulate stable rustc. Some tests before used only-stable to "achieve" this, which is very misleading, because it doesn't work at all 😆, due to compiletest unconditionally passing RUSTC_BOOTSTRAP=1 to all executed tests. act-as-stable fixes that.

The last two commits demonstrate how we could replace run-make tests that tested different output on stable/nightly with UI tests.

I suppose that this mode might be interesting for other people outside of rust-lang/rust, but if you don't want that, I can gate it behind -Zui-testing.

r? jieyouxu

[rustbot]

compiletest directives have been modified. Please add or update docs for the
new or modified directive in src/doc/rustc-dev-guide/.

The rustc-dev-guide subtree was changed. If your future PRs only touch the subtree, consider submitting them directly to rust-lang/rustc-dev-guide, which is where the document is primarily maintained (and has faster CI).

cc @BoxyUwU, @tshepang

Some changes occurred in src/tools/compiletest

cc @jieyouxu

[bjorn3]

You can probably remove

rust/compiler/rustc_codegen_cranelift/scripts/test_rustc_tests.sh

Lines 162 to 167 in c165834

	rm -r tests/run-make/missing-unstable-trait-bound # This disables support for unstable features, but running cg_clif needs some unstable features
	rm -r tests/run-make/const-trait-stable-toolchain # same
	rm -r tests/run-make/print-request-help-stable-unstable # same
	rm -r tests/run-make/issue-149402-suggest-unresolve # same
	rm -r tests/run-make/const-destruct-stable-toolchain # same
	rm -r tests/run-make/unstable-target-feature # same

[rustbot]

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

[jieyouxu]

General idea makes sense to me. Some discussions

View changes since this review